Using pagination with the BigQuery API

This document describes how to read table data and query results in large datasets by using pagination with the BigQuery API.

With pagination , BigQuery breaks a large dataset into smaller chunks called pages . For most users, Cloud Client Libraries handle this process automatically, but you can also manually control pagination for specific use cases like web applications.

Use automatic pagination

Cloud Client Libraries handle the low-level details of API pagination and provide an iterator-like experience. When you iterate through results, the library automatically fetches the next page of data when it's needed.

The following samples demonstrate how to automatically iterate through BigQuery table data.

C#

Before trying this sample, follow the C# setup instructions in the BigQuery quickstart using client libraries . For more information, see the BigQuery C# API reference documentation .

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries .

  using 
  
  Google.Api.Gax 
 
 ; 
 using 
  
 Google.Apis.Bigquery.v2.Data 
 ; 
 using 
  
  Google.Cloud.BigQuery.V2 
 
 ; 
 using 
  
 System 
 ; 
 using 
  
 System.Linq 
 ; 
 public 
  
 class 
  
 BigQueryBrowseTable 
 { 
  
 public 
  
 void 
  
 BrowseTable 
 ( 
  
 string 
  
 projectId 
  
 = 
  
 "your-project-id" 
  
 ) 
  
 { 
  
  BigQueryClient 
 
  
 client 
  
 = 
  
  BigQueryClient 
 
 . 
  Create 
 
 ( 
 projectId 
 ); 
  
 TableReference 
  
 tableReference 
  
 = 
  
 new 
  
 TableReference 
 () 
  
 { 
  
 TableId 
  
 = 
  
 "shakespeare" 
 , 
  
 DatasetId 
  
 = 
  
 "samples" 
 , 
  
 ProjectId 
  
 = 
  
 "bigquery-public-data" 
  
 }; 
  
 // Load all rows from a table 
  
 PagedEnumerable<TableDataList 
 , 
  
 BigQueryRow 
>  
 result 
  
 = 
  
 client 
 . 
  ListRows 
 
 ( 
  
 tableReference 
 : 
  
 tableReference 
 , 
  
 schema 
 : 
  
 null 
  
 ); 
  
 // Print the first 10 rows 
  
 foreach 
  
 ( 
  BigQueryRow 
 
  
 row 
  
 in 
  
 result 
 . 
 Take 
 ( 
 10 
 )) 
  
 { 
  
 Console 
 . 
 WriteLine 
 ( 
 $"{row[" 
 corpus 
 "]}: {row[" 
 word_count 
 "]}" 
 ); 
  
 } 
  
 } 
 }

Go

Before trying this sample, follow the Go setup instructions in the BigQuery quickstart using client libraries . For more information, see the BigQuery Go API reference documentation .

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries .

  import 
  
 ( 
  
 "context" 
  
 "fmt" 
  
 "io" 
  
 "cloud.google.com/go/bigquery" 
  
 "google.golang.org/api/iterator" 
 ) 
 // browseTable demonstrates reading data from a BigQuery table directly without the use of a query. 
 // For large tables, we also recommend the BigQuery Storage API. 
 func 
  
 browseTable 
 ( 
 w 
  
 io 
 . 
 Writer 
 , 
  
 projectID 
 , 
  
 datasetID 
 , 
  
 tableID 
  
 string 
 ) 
  
 error 
  
 { 
  
 // projectID := "my-project-id" 
  
 // datasetID := "mydataset" 
  
 // tableID := "mytable" 
  
 ctx 
  
 := 
  
 context 
 . 
 Background 
 () 
  
 client 
 , 
  
 err 
  
 := 
  
 bigquery 
 . 
 NewClient 
 ( 
 ctx 
 , 
  
 projectID 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 fmt 
 . 
 Errorf 
 ( 
 "bigquery.NewClient: %v" 
 , 
  
 err 
 ) 
  
 } 
  
 defer 
  
 client 
 . 
 Close 
 () 
  
 table 
  
 := 
  
 client 
 . 
 Dataset 
 ( 
 datasetID 
 ). 
 Table 
 ( 
 tableID 
 ) 
  
 it 
  
 := 
  
 table 
 . 
 Read 
 ( 
 ctx 
 ) 
  
 for 
  
 { 
  
 var 
  
 row 
  
 [] 
 bigquery 
 . 
  Value 
 
  
 err 
  
 := 
  
 it 
 . 
 Next 
 ( 
& row 
 ) 
  
 if 
  
 err 
  
 == 
  
 iterator 
 . 
 Done 
  
 { 
  
 break 
  
 } 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 err 
  
 } 
  
 fmt 
 . 
 Fprintln 
 ( 
 w 
 , 
  
 row 
 ) 
  
 } 
  
 return 
  
 nil 
 }

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries . For more information, see the BigQuery Java API reference documentation .

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries .

  import 
  
 com.google.cloud.bigquery. BigQuery 
 
 ; 
 import 
  
 com.google.cloud.bigquery. BigQuery 
. TableDataListOption 
 
 ; 
 import 
  
 com.google.cloud.bigquery. BigQueryException 
 
 ; 
 import 
  
 com.google.cloud.bigquery. BigQueryOptions 
 
 ; 
 import 
  
 com.google.cloud.bigquery. TableId 
 
 ; 
 import 
  
 com.google.cloud.bigquery. TableResult 
 
 ; 
 // Sample to directly browse a table with optional paging 
 public 
  
 class 
 BrowseTable 
  
 { 
  
 public 
  
 static 
  
 void 
  
 runBrowseTable 
 () 
  
 { 
  
 // TODO(developer): Replace these variables before running the sample. 
  
 String 
  
 table 
  
 = 
  
 "MY_TABLE_NAME" 
 ; 
  
 String 
  
 dataset 
  
 = 
  
 "MY_DATASET_NAME" 
 ; 
  
 browseTable 
 ( 
 dataset 
 , 
  
 table 
 ); 
  
 } 
  
 public 
  
 static 
  
 void 
  
 browseTable 
 ( 
 String 
  
 dataset 
 , 
  
 String 
  
 table 
 ) 
  
 { 
  
 try 
  
 { 
  
 // Initialize client that will be used to send requests. This client only needs to be created 
  
 // once, and can be reused for multiple requests. 
  
 BigQuery 
  
 bigquery 
  
 = 
  
 BigQueryOptions 
 . 
 getDefaultInstance 
 (). 
 getService 
 (); 
  
 // Identify the table itself 
  
 TableId 
  
 tableId 
  
 = 
  
 TableId 
 . 
 of 
 ( 
 dataset 
 , 
  
 table 
 ); 
  
 // Page over 100 records. If you don't need pagination, remove the pageSize parameter. 
  
 TableResult 
  
 result 
  
 = 
  
 bigquery 
 . 
 listTableData 
 ( 
 tableId 
 , 
  
 TableDataListOption 
 . 
 pageSize 
 ( 
 100 
 )); 
  
 // Print the records 
  
 result 
  
 . 
 iterateAll 
 () 
  
 . 
 forEach 
 ( 
  
 row 
  
 - 
>  
 { 
  
 row 
 . 
 forEach 
 ( 
 fieldValue 
  
 - 
>  
 System 
 . 
 out 
 . 
 print 
 ( 
 fieldValue 
 . 
 toString 
 () 
  
 + 
  
 ", " 
 )); 
  
 System 
 . 
 out 
 . 
 println 
 (); 
  
 }); 
  
 System 
 . 
 out 
 . 
 println 
 ( 
 "Query ran successfully" 
 ); 
  
 } 
  
 catch 
  
 ( 
  BigQueryException 
 
  
 e 
 ) 
  
 { 
  
 System 
 . 
 out 
 . 
 println 
 ( 
 "Query failed to run \n" 
  
 + 
  
 e 
 . 
 toString 
 ()); 
  
 } 
  
 } 
 }

Node.js

Before trying this sample, follow the Node.js setup instructions in the BigQuery quickstart using client libraries . For more information, see the BigQuery Node.js API reference documentation .

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries .

  // Import the Google Cloud client library using default credentials 
 const 
  
 { 
 BigQuery 
 } 
  
 = 
  
 require 
 ( 
 ' @google-cloud/bigquery 
' 
 ); 
 const 
  
 bigquery 
  
 = 
  
 new 
  
  BigQuery 
 
 (); 
 async 
  
 function 
  
 browseTable 
 () 
  
 { 
  
 // Retrieve a table's rows using manual pagination. 
  
 /** 
 * TODO(developer): Uncomment the following lines before running the sample. 
 */ 
  
 // const datasetId = 'my_dataset'; // Existing dataset 
  
 // const tableId = 'my_table'; // Table to create 
  
 const 
  
 query 
  
 = 
  
 `SELECT name, SUM(number) as total_people 
 FROM \`bigquery-public-data.usa_names.usa_1910_2013\` 
 GROUP BY name 
 ORDER BY total_people 
 DESC LIMIT 100` 
 ; 
  
 // Create table reference. 
  
 const 
  
 dataset 
  
 = 
  
 bigquery 
 . 
 dataset 
 ( 
 datasetId 
 ); 
  
 const 
  
 destinationTable 
  
 = 
  
 dataset 
 . 
 table 
 ( 
 tableId 
 ); 
  
 // For all options, see https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#jobconfigurationquery 
  
 const 
  
 queryOptions 
  
 = 
  
 { 
  
 query 
 : 
  
 query 
 , 
  
 destination 
 : 
  
 destinationTable 
 , 
  
 }; 
  
 // Run the query as a job 
  
 const 
  
 [ 
 job 
 ] 
  
 = 
  
 await 
  
 bigquery 
 . 
 createQueryJob 
 ( 
 queryOptions 
 ); 
  
 // For all options, see https://cloud.google.com/bigquery/docs/reference/v2/jobs/getQueryResults 
  
 const 
  
 queryResultsOptions 
  
 = 
  
 { 
  
 // Retrieve zero resulting rows. 
  
 maxResults 
 : 
  
 0 
 , 
  
 }; 
  
 // Wait for the job to finish. 
  
 await 
  
  job 
 
 . 
  getQueryResults 
 
 ( 
 queryResultsOptions 
 ); 
  
 function 
  
 manualPaginationCallback 
 ( 
 err 
 , 
  
 rows 
 , 
  
 nextQuery 
 ) 
  
 { 
  
 rows 
 . 
 forEach 
 ( 
 row 
  
 = 
>  
 { 
  
 console 
 . 
 log 
 ( 
 `name: 
 ${ 
 row 
 . 
 name 
 } 
 , 
 ${ 
 row 
 . 
 total_people 
 } 
 total people` 
 ); 
  
 }); 
  
 if 
  
 ( 
 nextQuery 
 ) 
  
 { 
  
 // More results exist. 
  
 destinationTable 
 . 
 getRows 
 ( 
 nextQuery 
 , 
  
 manualPaginationCallback 
 ); 
  
 } 
  
 } 
  
 // For all options, see https://cloud.google.com/bigquery/docs/reference/v2/tabledata/list 
  
 const 
  
 getRowsOptions 
  
 = 
  
 { 
  
 autoPaginate 
 : 
  
 false 
 , 
  
 maxResults 
 : 
  
 20 
 , 
  
 }; 
  
 // Retrieve all rows. 
  
 destinationTable 
 . 
 getRows 
 ( 
 getRowsOptions 
 , 
  
 manualPaginationCallback 
 ); 
 } 
 browseTable 
 ();

PHP

Before trying this sample, follow the PHP setup instructions in the BigQuery quickstart using client libraries . For more information, see the BigQuery PHP API reference documentation .

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries .

  use Google\Cloud\BigQuery\BigQueryClient; 
 /** Uncomment and populate these variables in your code */ 
 // $projectId = 'The Google project ID'; 
 // $datasetId = 'The BigQuery dataset ID'; 
 // $tableId   = 'The BigQuery table ID'; 
 // $maxResults = 10; 
 $maxResults = 10; 
 $startIndex = 0; 
 $options = [ 
 'maxResults' => $maxResults, 
 'startIndex' => $startIndex 
 ]; 
 $bigQuery = new BigQueryClient([ 
 'projectId' => $projectId, 
 ]); 
 $dataset = $bigQuery->dataset($datasetId); 
 $table = $dataset->table($tableId); 
 $numRows = 0; 
 foreach ($table->rows($options) as $row) { 
 print('---'); 
 foreach ($row as $column => $value) { 
 printf('%s: %s' . PHP_EOL, $column, $value); 
 } 
 $numRows++; 
 }

Python

Before trying this sample, follow the Python setup instructions in the BigQuery quickstart using client libraries . For more information, see the BigQuery Python API reference documentation .

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries .

  from 
  
 google.cloud 
  
 import 
  bigquery 
 
 # Construct a BigQuery client object. 
 client 
 = 
  bigquery 
 
 . 
  Client 
 
 () 
 # TODO(developer): Set table_id to the ID of the table to browse data rows. 
 # table_id = "your-project.your_dataset.your_table_name" 
 # Download all rows from a table. 
 rows_iter 
 = 
 client 
 . 
  list_rows 
 
 ( 
 table_id 
 ) 
 # Make an API request. 
 # Iterate over rows to make the API requests to fetch row data. 
 rows 
 = 
 list 
 ( 
 rows_iter 
 ) 
 print 
 ( 
 "Downloaded 
 {} 
 rows from table 
 {} 
 " 
 . 
 format 
 ( 
 len 
 ( 
 rows 
 ), 
 table_id 
 )) 
 # Download at most 10 rows. 
 rows_iter 
 = 
 client 
 . 
  list_rows 
 
 ( 
 table_id 
 , 
 max_results 
 = 
 10 
 ) 
 rows 
 = 
 list 
 ( 
 rows_iter 
 ) 
 print 
 ( 
 "Downloaded 
 {} 
 rows from table 
 {} 
 " 
 . 
 format 
 ( 
 len 
 ( 
 rows 
 ), 
 table_id 
 )) 
 # Specify selected fields to limit the results to certain columns. 
 table 
 = 
 client 
 . 
  get_table 
 
 ( 
 table_id 
 ) 
 # Make an API request. 
 fields 
 = 
 table 
 . 
 schema 
 [: 
 2 
 ] 
 # First two columns. 
 rows_iter 
 = 
 client 
 . 
  list_rows 
 
 ( 
 table_id 
 , 
 selected_fields 
 = 
 fields 
 , 
 max_results 
 = 
 10 
 ) 
 rows 
 = 
 list 
 ( 
 rows_iter 
 ) 
 print 
 ( 
 "Selected 
 {} 
 columns from table 
 {} 
 ." 
 . 
 format 
 ( 
 len 
 ( 
 rows_iter 
 . 
 schema 
 ), 
 table_id 
 )) 
 print 
 ( 
 "Downloaded 
 {} 
 rows from table 
 {} 
 " 
 . 
 format 
 ( 
 len 
 ( 
 rows 
 ), 
 table_id 
 )) 
 # Print row data in tabular format. 
 rows 
 = 
 client 
 . 
  list_rows 
 
 ( 
 table 
 , 
 max_results 
 = 
 10 
 ) 
 format_string 
 = 
 " 
 {!s:<16} 
 " 
 * 
 len 
 ( 
  rows 
 
 . 
 schema 
 ) 
 field_names 
 = 
 [ 
 field 
 . 
 name 
 for 
 field 
 in 
  rows 
 
 . 
 schema 
 ] 
 print 
 ( 
 format_string 
 . 
 format 
 ( 
 * 
 field_names 
 )) 
 # Prints column headers. 
 for 
 row 
 in 
 rows 
 : 
 print 
 ( 
 format_string 
 . 
 format 
 ( 
 * 
 row 
 )) 
 # Prints row data.

Ruby

Before trying this sample, follow the Ruby setup instructions in the BigQuery quickstart using client libraries . For more information, see the BigQuery Ruby API reference documentation .

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries .

  require 
  
 "google/cloud/bigquery" 
 def 
  
 browse_table 
  
 bigquery 
  
 = 
  
 Google 
 :: 
 Cloud 
 :: 
  Bigquery 
 
 . 
  new 
 
  
 project_id 
 : 
  
 "bigquery-public-data" 
  
 dataset 
  
 = 
  
 bigquery 
 . 
 dataset 
  
 "samples" 
  
 table 
  
 = 
  
 dataset 
 . 
 table 
  
 "shakespeare" 
  
 # Load all rows from a table 
  
 rows 
  
 = 
  
 table 
 . 
 data 
  
 # Load the first 10 rows 
  
 rows 
  
 = 
  
 table 
 . 
 data 
  
 max 
 : 
  
 10 
  
 # Print row data 
  
 rows 
 . 
 each 
  
 { 
  
 | 
 row 
 | 
  
 puts 
  
 row 
  
 } 
 end

Control page size

You can specify the maximum number of rows returned in each network request by setting a page size. Setting the page size is useful for optimizing network usage or fitting data into memory.

In most client libraries, you can use a max_results or page_size parameter when you call methods like list_rows or query .

Use manual pagination with page tokens

Manual pagination is useful for stateless applications, such as a web service where a user clicks Nextto see the next set of results. In this scenario, the server does not maintain an active iterator between requests.

Instead, you use a page token as follows:

Request a page.Call the API and receive a pageToken parameter along with the rows.
Resume.In the next request, pass that same pageToken parameter back to BigQuery to retrieve the next chunk of data.

The following samples show how to retrieve a page token and use it to fetch the next page of query results.

API

Read the jobs.config.query.destinationTable field to determine the table that query results have been written to. Call the tabledata.list to read the query results.

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries . For more information, see the BigQuery Java API reference documentation .

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries .

  import 
  
 com.google.cloud.bigquery. BigQuery 
 
 ; 
 import 
  
 com.google.cloud.bigquery. BigQueryException 
 
 ; 
 import 
  
 com.google.cloud.bigquery. BigQueryOptions 
 
 ; 
 import 
  
 com.google.cloud.bigquery. QueryJobConfiguration 
 
 ; 
 import 
  
 com.google.cloud.bigquery. TableId 
 
 ; 
 import 
  
 com.google.cloud.bigquery. TableResult 
 
 ; 
 // Sample to run query with pagination. 
 public 
  
 class 
 QueryPagination 
  
 { 
  
 public 
  
 static 
  
 void 
  
 main 
 ( 
 String 
 [] 
  
 args 
 ) 
  
 { 
  
 String 
  
 datasetName 
  
 = 
  
 "MY_DATASET_NAME" 
 ; 
  
 String 
  
 tableName 
  
 = 
  
 "MY_TABLE_NAME" 
 ; 
  
 String 
  
 query 
  
 = 
  
 "SELECT name, SUM(number) as total_people" 
  
 + 
  
 " FROM `bigquery-public-data.usa_names.usa_1910_2013`" 
  
 + 
  
 " GROUP BY name" 
  
 + 
  
 " ORDER BY total_people DESC" 
  
 + 
  
 " LIMIT 100" 
 ; 
  
 queryPagination 
 ( 
 datasetName 
 , 
  
 tableName 
 , 
  
 query 
 ); 
  
 } 
  
 public 
  
 static 
  
 void 
  
 queryPagination 
 ( 
 String 
  
 datasetName 
 , 
  
 String 
  
 tableName 
 , 
  
 String 
  
 query 
 ) 
  
 { 
  
 try 
  
 { 
  
 // Initialize client that will be used to send requests. This client only needs to be created 
  
 // once, and can be reused for multiple requests. 
  
  BigQuery 
 
  
 bigquery 
  
 = 
  
  BigQueryOptions 
 
 . 
 getDefaultInstance 
 (). 
 getService 
 (); 
  
  TableId 
 
  
 tableId 
  
 = 
  
  TableId 
 
 . 
 of 
 ( 
 datasetName 
 , 
  
 tableName 
 ); 
  
  QueryJobConfiguration 
 
  
 queryConfig 
  
 = 
  
  QueryJobConfiguration 
 
 . 
 newBuilder 
 ( 
 query 
 ) 
  
 // save results into a table. 
  
 . 
 setDestinationTable 
 ( 
 tableId 
 ) 
  
 . 
 build 
 (); 
  
 bigquery 
 . 
  query 
 
 ( 
 queryConfig 
 ); 
  
  TableResult 
 
  
 results 
  
 = 
  
 bigquery 
 . 
  listTableData 
 
 ( 
 tableId 
 , 
  
 BigQuery 
 . 
 TableDataListOption 
 . 
 pageSize 
 ( 
 20 
 )); 
  
 // First Page 
  
 results 
  
 . 
 getValues 
 () 
  
 . 
 forEach 
 ( 
 row 
  
 - 
>  
 row 
 . 
 forEach 
 ( 
 val 
  
 - 
>  
 System 
 . 
 out 
 . 
 printf 
 ( 
 "%s,\n" 
 , 
  
 val 
 . 
 toString 
 ()))); 
  
 while 
  
 ( 
 results 
 . 
  hasNextPage 
 
 ()) 
  
 { 
  
 // Remaining Pages 
  
 results 
  
 = 
  
 results 
 . 
  getNextPage 
 
 (); 
  
 results 
  
 . 
 getValues 
 () 
  
 . 
 forEach 
 ( 
 row 
  
 - 
>  
 row 
 . 
 forEach 
 ( 
 val 
  
 - 
>  
 System 
 . 
 out 
 . 
 printf 
 ( 
 "%s,\n" 
 , 
  
 val 
 . 
 toString 
 ()))); 
  
 } 
  
 System 
 . 
 out 
 . 
 println 
 ( 
 "Query pagination performed successfully." 
 ); 
  
 } 
  
 catch 
  
 ( 
  BigQueryException 
 
  
 | 
  
 InterruptedException 
  
 e 
 ) 
  
 { 
  
 System 
 . 
 out 
 . 
 println 
 ( 
 "Query not performed \n" 
  
 + 
  
 e 
 . 
 toString 
 ()); 
  
 } 
  
 } 
 }

To set the number of rows returned on each page, use a GetQueryResults job and set the pageSize option of the QueryResultsOption object that you pass in, as shown in the following example:

  TableResult 
  
 result 
  
 = 
  
 job 
 . 
 getQueryResults 
 (); 
 QueryResultsOption 
  
 queryResultsOption 
  
 = 
  
 QueryResultsOption 
 . 
 pageSize 
 ( 
 20 
 ); 
 TableResult 
  
 result 
  
 = 
  
 job 
 . 
 getQueryResults 
 ( 
 queryResultsOption 
 );

Node.js

Before trying this sample, follow the Node.js setup instructions in the BigQuery quickstart using client libraries . For more information, see the BigQuery Node.js API reference documentation .

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries .

  // Import the Google Cloud client library using default credentials 
 const 
  
 { 
 BigQuery 
 } 
  
 = 
  
 require 
 ( 
 ' @google-cloud/bigquery 
' 
 ); 
 const 
  
 bigquery 
  
 = 
  
 new 
  
  BigQuery 
 
 (); 
 async 
  
 function 
  
 queryPagination 
 () 
  
 { 
  
 // Run a query and get rows using automatic pagination. 
  
 const 
  
 query 
  
 = 
  
 `SELECT name, SUM(number) as total_people 
 FROM \`bigquery-public-data.usa_names.usa_1910_2013\` 
 GROUP BY name 
 ORDER BY total_people DESC 
 LIMIT 100` 
 ; 
  
 // Run the query as a job. 
  
 const 
  
 [ 
 job 
 ] 
  
 = 
  
 await 
  
 bigquery 
 . 
 createQueryJob 
 ( 
 query 
 ); 
  
 // Wait for job to complete and get rows. 
  
 const 
  
 [ 
 rows 
 ] 
  
 = 
  
 await 
  
  job 
 
 . 
  getQueryResults 
 
 (); 
  
 console 
 . 
 log 
 ( 
 'Query results:' 
 ); 
  
 rows 
 . 
 forEach 
 ( 
 row 
  
 = 
>  
 { 
  
 console 
 . 
 log 
 ( 
 `name: 
 ${ 
 row 
 . 
 name 
 } 
 , 
 ${ 
 row 
 . 
 total_people 
 } 
 total people` 
 ); 
  
 }); 
 } 
 queryPagination 
 ();

Python

The QueryJob.result method returns an iterable of the query results. Alternatively,

Read the QueryJob.destination property. If this property is not configured, it is set by the API to a reference to a temporary anonymous table .
Get the table schema with the Client.get_table method.
Create an iterable over all rows in the destination table with the Client.list_rows method.

Before trying this sample, follow the Python setup instructions in the BigQuery quickstart using client libraries . For more information, see the BigQuery Python API reference documentation .

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries .

  from 
  
 google.cloud 
  
 import 
  bigquery 
 
 # Construct a BigQuery client object. 
 client 
 = 
  bigquery 
 
 . 
  Client 
 
 () 
 query 
 = 
 """ 
 SELECT name, SUM(number) as total_people 
 FROM `bigquery-public-data.usa_names.usa_1910_2013` 
 GROUP BY name 
 ORDER BY total_people DESC 
 """ 
 query_job 
 = 
 client 
 . 
  query 
 
 ( 
 query 
 ) 
 # Make an API request. 
  query_job 
 
 . 
 result 
 () 
 # Wait for the query to complete. 
 # Get the destination table for the query results. 
 # 
 # All queries write to a destination table. If a destination table is not 
 # specified, the BigQuery populates it with a reference to a temporary 
 # anonymous table after the query completes. 
 destination 
 = 
  query_job 
 
 . 
 destination 
 # Get the schema (and other properties) for the destination table. 
 # 
 # A schema is useful for converting from BigQuery types to Python types. 
 destination 
 = 
 client 
 . 
  get_table 
 
 ( 
 destination 
 ) 
 # Download rows. 
 # 
 # The client library automatically handles pagination. 
 print 
 ( 
 "The query data:" 
 ) 
 rows 
 = 
 client 
 . 
  list_rows 
 
 ( 
 destination 
 , 
 max_results 
 = 
 20 
 ) 
 for 
 row 
 in 
 rows 
 : 
 print 
 ( 
 "name= 
 {} 
 , count= 
 {} 
 " 
 . 
 format 
 ( 
 row 
 [ 
 "name" 
 ], 
 row 
 [ 
 "total_people" 
 ]))

Optimize with ETags

When you page backwards or jump to arbitrary pages using cached pageToken values, it is possible that the data in your pages might have changed since it was last viewed. To mitigate this outcome, you can use the etag property.

Every collection.list method (except for Tabledata ) returns an etag property in the result. This property is a hash of the page results that can be used to verify whether the page has changed since the last request. When you make a request to BigQuery with an ETag value, BigQuery compares the ETag value to the ETag value returned by the API and responds based on whether the ETag values match. You can use ETags to avoid redundant list calls as follows:

To return values only if they have changed, make a list call with a previously-stored ETag using the HTTP If-None-Match header . If the ETags match, BigQuery returns an HTTP 304 Not Modified status code and no data, saving bandwidth.
To return values only if they have notchanged, use the HTTP If-Match header . BigQuery returns a 412 Precondition Failed if the page has changed.

Note:Although ETags are a great way to avoid making redundant list calls, you can apply the same methods to identifying if any objects have changed. For example, you can perform a `GET` request for a specific table and use ETags to determine if the table has changed before returning the full response.

Reference: API limits and criteria

All *collection*.list methods return paginated results under certain circumstances. The maxResults property limits the number of results per page.

Method	Pagination criteria	Default `maxResults` limit	Maximum `maxResults` limit	Maximum `maxFieldValues` limit
`tabledata.list`	Returns paginated results if the response size is more than 10 MB ¹ of data or more than `maxResults` rows.	Unlimited	Unlimited	Unlimited
All other `collection.list` methods	Returns paginated results if the response is more than `maxResults` rows and also less than the maximum limits.	10,000	Unlimited	300,000

If the result is larger than the byte or field limit, the result is trimmed to fit the limit. If one row is greater than the byte or field limit, the tabledata.list method can return up to 100 MB of data ¹ , which is consistent with the maximum row size limit for query results. There is no minimum size per page, and some pages might return more rows than others.

The jobs.getQueryResults REST API method can return 20 MB of data unless you explicitly request more through support.

¹ The row size is approximate, as the size is based on the internal representation of row data. The maximum row size limit is enforced during certain stages of query job execution.