Create dashboards, charts, and alerts

This document describes how to create charts and alerts to monitor BigQuery resources using Cloud Monitoring.

Before you begin

Before you use Cloud Monitoring, ensure that you have the following:

A Cloud Billing account.
A BigQuery project with billing enabled.

One way to ensure that you have both is to complete the Quickstart using the Google Cloud console .

View and create dashboards, charts, and alerts

View the Cloud Monitoring dashboard

To use Cloud Monitoring to monitor your BigQuery project:

In the Google Cloud console, go to the Monitoringpage.

Go to Monitoring
Select the name of your project if it is not already selected at the top of the page.
To view BigQuery resources, select Dashboards > BigQuery. On this page you see a list of tables, events, and incident reporting that are user-configurable as well as charts of project metrics or dataset metrics.

Visualize slots available and slots allocated

To visualize the slots available and slots allocated to your project, go to the dashboard for BigQuery described in Viewing the Cloud Monitoring dashboard :

In the Google Cloud console, go to the Monitoringpage.

Go to Monitoring
Select Dashboards > BigQuery.
On the Cloud Monitoring dashboard for BigQuery, scroll to the chart named Slot Utilization.

The Slot Utilizationchart appears on both the main Cloud Monitoring default dashboard and the Cloud Monitoring dashboard for BigQuery.

Create a dashboard and chart

Display the metrics collected by Cloud Monitoring in your own charts and dashboards:

In the Google Cloud console, go to the Monitoringpage.

Go to Monitoring
Select Dashboards > Create Dashboard.
Click Add Chart. You see the Add Chart page:
In the Find resource type and metricpanel fields:
- For the Resource typedrop-down list, select Global. You might need to expand the list of Resource typesfor the Globaloption to be visible.
- For the Metricdrop-down list, select Query execution time.
The Aggregationpane fields control how the execution-time data are displayed. You can modify the default settings for these fields.
Click Save.

View quota usage and limits

In Cloud Monitoring, you can view metrics for quota usage and limits:

In the Google Cloud console, go to the Monitoringpage.

Go to Monitoring
In the navigation pane, select Metrics explorer .
In the toolbar, select Explorer > Configuration.
In the Resource & Metricsection, click Select a metric.
Select Consumer Quota > Quota > Quota limit, and then click Apply.
Click Add filter, and then in the Labelmenu, select limit_name.
In the Valuemenu, select the quota for which you want to view the metrics.

Create an alert

To create an alerting policy that triggers when the 99th percentile of the execution time of a BigQuery query exceeds a user-defined limit, use the following settings.

Steps to create an alerting policy .

To create an alerting policy, do the following:

In the Google Cloud console, go to the Alerting page:

Go to Alerting

If you use the search bar to find this page, then select the result whose subheading is Monitoring .
If you haven't created your notification channels and if you want to be notified, then click Edit Notification Channels and add your notification channels. Return to the Alerting page after you add your channels.
From the Alerting page, select Create policy .
To select the resource, metric, and filters, expand the Select a metric menu and then use the values in the New condition table:
1. Optional: To limit the menu to relevant entries, enter the resource or metric name in the filter bar.
2. Select a Resource type . For example, select VM instance .
3. Select a Metric category . For example, select instance .
4. Select a Metric . For example, select CPU Utilization .
5. Select Apply .
Click Next and then configure the alerting policy trigger. To complete these fields, use the values in the Configure alert trigger table.
Click Next .
Optional: To add notifications to your alerting policy, click Notification channels . In the dialog, select one or more notification channels from the menu, and then click OK .

To be notified when incidents are openend and closed, check Notify on incident closure . By default, notifications are sent only when incidents are openend.
Optional: Update the Incident autoclose duration . This field determines when Monitoring closes incidents in the absence of metric data.
Optional: Click Documentation , and then add any information that you want included in a notification message.
Click Alert name and enter a name for the alerting policy.
Click Create Policy .

New condition Field	Value
Resource and Metric	In the Resources menu, select BigQuery Project . In the Metric categories menu, select Query . In the Metrics menu, select Query execution times .
Filter
Across time series Time series group by	`priority`
Across time series Time series aggregation	`99th percentile`
Rolling window	`5 m`
Rolling window function	`sum`

Configure alert trigger Field	Value
Condition type	`Threshold`
Alert trigger	`Any time series violates`
Threshold position	`Above threshold`
Threshold value	You determine this value; however, a threshold of 60 seconds is recommended.
Retest window	`most recent value`

To create an alerting policy that triggers when the total scanned bytes billed for a BigQuery project exceeds a user-defined limit, use the following alerting policy configuration.

New condition Field	Value
Resource and Metric	In the Resources menu, select BigQuery Project . In the Metric categories menu, select Query . In the Metrics menu, select Statement scanned bytes billed .
Filter	(No filter needed for a project-wide alert)
Across time series Time series group by	(Leave blank to aggregate all series)
Across time series Time series aggregation	`sum`
Rolling window	`5 m`
Rolling window function	`sum`

Configure alert trigger Field	Value
Condition type	`Threshold`
Alert trigger	`Any time series violates`
Threshold position	`Above threshold`
Threshold value	You determine this value. For example, to trigger an alert when usage exceeds 1 TiB, enter `1000000000000` .

While the alerting policy monitors the total bytes scanned, you can set a threshold based on a specific budget. To accomplish this budget based alerting policy, you must first convert your desired cost threshold into the equivalent number of bytes. The formula relies on the BigQuery on-demand compute pricing. For more information, see On-demand compute pricing .

You can use the following formula to convert your cost threshold into bytes:

 Threshold in Bytes = (Target Amount / (price per TiB)) * 1,000,000,000,000

Example: Trigger an alert when usage exceeds $100

Let's say you want to be alerted when your project's query costs exceed $100.

Calculate the equivalent data volume in TiB:
$100 / (price per TiB) = Equivalent Data Volume in TiB
Convert data volume to bytes:
(Equivalent Data Volume in TiB) * 1,000,000,000,000 = Threshold Value in Bytes
Set the Threshold Value:
In the Configure alert triggersection of your policy, enter the "Threshold Value in Bytes" as the Threshold value.

Now, your alerting policy triggers when the total scanned bytes billed in the rolling window corresponds to approximately $100in on-demand query costs.

Metrics available for visualization

The following metrics are available, time delayed up to several hours.

Resource type

Name

Units

Description

BigQuery

Scanned bytes

Bytes per minute

Number of bytes scanned.

BigQuery

Scanned bytes billed

Bytes per minute

Number of bytes sent for billing when using the on-demand analysis model. Scanned bytes and scanned bytes billed can differ as charges are rounded up, with a minimum amount of data processed per query.

BigQuery

BI Engine Query Fallback Count ( Preview )

Queries

The amount of queries that did not use BI Engine as a rate. You can set the Group By option to reason to separate the count into different fallback reasons, including:

NO_RESERVATION
INSUFFICIENT_RESERVATION
UNSUPPORTED_SQL_TEXT
INPUT_TOO_LARGE
OTHER_REASON

BigQuery

Query count

Queries

Queries in flight.

BigQuery

Query execution count ( Preview )

Queries

The number of queries executed.

BigQuery

Query execution times
- 5th percentile
- 50th percentile
- 95th percentile
- 99th percentile

Seconds

Non-cached query execution times.

BigQuery

Slots used by project

Slots

Number of BigQuery slots allocated for query jobs in the project.

Slots are allocated per billing account and multiple projects can share the same reservation of slots.

BigQuery

Slots used by project and job type

Slots

Number of slots allocated to the project at any time separated by job type . This can also be thought of as the number of slots being utilized by that project. Load and export jobs are free operations, and they run in a public pool of resources.

Slots are allocated per billing account and multiple projects can share the same reservation of slots.

BigQuery

Slots used by project, reservation, and job type

Slots

Number of BigQuery slots allocated for project. Slot allocation can be broken down based on reservation and job type.

BigQuery

Total slots

Slots

Total number of slots available to the project.

If the project shares a reservation of slots with other projects the slots being used by the other projects is not depicted.

BigQuery

Slots used across projects in reservations

Slots

Number of BigQuery slots allocated across projects in the reservation. Note that the metric data is only reported while at least one project has been assigned to the reservation and is consuming slots. As an alternative, consider querying reservations information from INFORMATION_SCHEMA .

BigQuery

Slots used by project in reservation

Slots

Number of BigQuery slots allocated for project in the reservation.

BigQuery continuous job

Estimated backlog logical bytes

Bytes

The number of bytes in the backlog for each stage of the continuous job.

BigQuery continuous job

Estimated backlog records

Records

The estimated number of backlog records for each stage of the continuous job.

BigQuery continuous job

Estimated bytes processed

Bytes

The estimated number of bytes processed for each stage of the continuous job.

BigQuery continuous job

Output watermark

Timestamp

The most recent timestamp, in microseconds since the epoch, up to which all data has been processed by this stage of the continuous job.

BigQuery continuous job

Records read

Records

The number of input records read for each stage of the continuous job.

BigQuery continuous job

Records written

Records

The number of output records written for each stage of the continuous job.

BigQuery continuous job

Slots used

Slot milliseconds

The total slot milliseconds used by the continuous job.

BigQuery dataset

Stored bytes

Bytes

Bytes stored in the dataset - For the 100 largest tables in the dataset, bytes stored is displayed for each individual table (by name). Any additional tables in the dataset (beyond the 100 largest) are reported as single sum, and the table name for the summary is an empty string.

BigQuery dataset

Table count

Tables

Number of tables in the dataset.

BigQuery dataset

Uploaded bytes

Bytes per minute

Number of bytes uploaded to any table in the dataset.

BigQuery dataset

Uploaded rows

Rows per minute

Number of records uploaded to any table in the dataset.

For a complete list of available Google Cloud metrics, see Google Cloud metrics .

Known issues

If no queries are running, then no data is returned for slots allocated, slots available, or any query-related variables. Zoom out to see data.
If queries are running in both the US and the EU, then slots allocated and slots available might be incorrect.
Slots allocated is reported as the average value in a time window (the width of the time window depends on the zoom level of the chart). Zooming in and out can change the value of slots allocated. Zooming in to a time window of 1 hr or less shows the true values of slots allocated. At this range for any time visible on the chart, avg(slots allocated) = slots allocated .
The data in Cloud Monitoring charts pertains only to the selected project.
Metrics are instantaneous values, sampled at a point in time, and might miss data points between sample intervals. For example, the job count metric is sampled every minute. The value is the number of jobs at that particular time, not the maximum number of jobs throughout the entire minute.