Labels
The AlloyDB Omni Kubernetes operator exposes the following types of labels.
Resource labels
AlloyDB Omni Kubernetes operator exposes the following resource labels that uniquely identify the database container that the metrics belong to. These resource labels match the names of the Kubernetes resource that owns the database container:
| Label key | Label value |
|---|---|
| dbnamespace | Namespace of the dbcluster CR |
| dbcluster | Name of the dbcluster CR |
| dbinstance | Name of the dbinstance CR. Only the dbinstance of ReadPool type is supported. If the database container does not belong to a ReadPool dbinstance, this value is n/a
|
| dbnode | Name of the instance CR. Every instance CR has a one-to-one mapping to a database container. |
System metadata labels
System metadata labels will change dynamically when the roles
of
the DB container changes. For example, When your dbcluster is promoted from
secondary to primary, dbcluster_type will change from Secondary
to Primary
.
| Label key | Label value |
|---|---|
| dbcluster_type | Disaster recovery (DR) role of the dbcluster CR. Can be Primary
or Secondary
|
| dbinstance_type | Type of the dbinstance CR. If the container belongs to a ReadPool dbinstance, this value is ReadPool
, otherwise this value is n/a
|
| dbnode_type | HA role of the dbnode, can be Primary
or Standby
|
Metric labels
The specific labels of each metric are listed in the following tables. For
example, database
means the name of a Postgres database hosted inside the AlloyDB Omni database container.
Metrics
AlloyDB Omni Kubernetes operator exposes the following metrics.
The metrics list mentions only metrics labels. All metrics start with alloydb_omni
.
To learn more about metric types, see Metric types
.
Database container-level metrics
Database container-level metrics metrics are collected per AlloyDB Omni database container. Each database container-level metric has resource and system metadata labels.
Name
Description
Label
Unit
Type
alloydb_omni_database_postgresql_backends
The number of active and idle connections to the AlloyDB Omni instance.
gauge
alloydb_omni_database_postgresql_chill_cache_get_entry_calls_total
The total number of calls to retrieve data entries from the ultra fast disk cache.
counter
alloydb_omni_database_postgresql_chill_cache_num_hits_total
The total number of successful retrievals of data entries from the ultra fast disk cache.
prewarm: the hits that occur during the prewarm phase
counter
alloydb_omni_database_postgresql_max_connections
The current value of the Postgres max_connections runtime parameter.
gauge
alloydb_omni_database_postgresql_up
Whether the Postgres main process is running, 1 if running, 0 if down.
gauge
alloydb_omni_database_postgresql_uptime_second
Time elapsed since the Postgres main process started.
second
gauge
alloydb_omni_database_postgresql_vacuum_oldest_transaction_age
The current age of the oldest uncommitted transaction that is blocking vacuum operation. It's measured in the number of transactions that started after the oldest transaction.
type: one of [running, prepared, replication_slot, replica]
gauge
alloydb_omni_database_postgresql_vacuum_transaction_id_utilization_percentage
The ratio of transaction ID space consumed. 1 means 100%.
gauge
alloydb_omni_instance_postgresql_backends_by_state
Current number of connections.
state: state of the connections that is one of [idle, active, idle_in_transaction, idle_in_transaction_aborted, disabled, fastpath_function_call]
gauge
alloydb_omni_instance_postgresql_backends_for_top_applications
Current number of connections per application.
application_name: name of the application
gauge
alloydb_omni_instance_postgresql_blks_hit_count_total
Total number of times Postgres found the requested block in the buffer cache.
counter
alloydb_omni_instance_postgresql_blks_read_count_total
Total number of blocks read by Postgres that were not in the Postgres buffer cache.
counter
alloydb_omni_instance_postgresql_committed_transactions_count_total
Total number of transactions committed.
counter
alloydb_omni_instance_postgresql_deadlock_count_total
Number of deadlocks detected.
counter
alloydb_omni_instance_postgresql_new_connections_count_total
Total number of new connections.
counter
alloydb_omni_instance_postgresql_rolledback_transactions_count_total
Total number of transactions rolled back.
counter
alloydb_omni_instance_postgresql_temp_bytes_written_count_total
Total amount of data written to temporary files by queries.
byte
counter
alloydb_omni_instance_postgresql_temp_files_written_count_total
Total number of temporary files used for writing data while performing internal algorithms.
counter
alloydb_omni_instance_postgresql_tuples_deleted_count_total
Total number of rows deleted.
counter
alloydb_omni_instance_postgresql_tuples_fetched_count_total
Total number of rows fetched.
counter
alloydb_omni_instance_postgresql_tuples_inserted_count_total
Total number of rows inserted.
counter
alloydb_omni_instance_postgresql_tuples_returned_count_total
Total number of rows returned.
counter
alloydb_omni_instance_postgresql_tuples_updated_count_total
Total number of rows updated.
counter
alloydb_omni_instance_postgresql_version
The version of the PostgreSQL database. In
MAJOR.MINOR
format, for example, 16.3
gauge
alloydb_omni_instance_postgresql_wait_count_total
Total wait count for a wait event.
- wait_event_name: name of the wait event
- wait_event_type: type of the wait event
counter
alloydb_omni_instance_postgresql_wait_time_us_total
Total time elapsed for a wait event.
- wait_event_name: name of the wait event
- wait_event_type: type of the wait event
us
counter
alloydb_omni_instance_postgresql_replication_flush_lag_ms
Time elapsed between flushing recent WAL locally and receiving notification that the replica server has written and flushed it (but not yet applied it).
- application_name: application_name in the replica's connection string to the primary that matches the name of the replica instance CR.
- client_addr: IP address of the replica pod.
ms
gauge
alloydb_omni_instance_postgresql_replication_replay_lag_ms
Time elapsed between flushing recent WAL locally and receiving notification that the replica server has written, flushed and applied it.
- application_name: application_name in the replica's connection string to the primary. It matches the name of the replica instance CR.
- client_addr: IP address of the replica pod.
ms
gauge
alloydb_omni_instance_postgresql_replication_state
State of each replica connected to the primary. Its value is always
1
.- application_name: application_name in the replica's connection string to the primary. It matches the name of the replica instance CR.
- client_addr: IP address of the replica pod.
- state: one of [startup, catchup, streaming, backup, stopping]
gauge
alloydb_omni_instance_postgresql_replication_write_lag_ms
Time elapsed between flushing recent WAL locally and receiving notification that the replica server has written it (but not yet flushed it or applied it).
- application_name: application_name in the replica's connection string to the primary. It matches the name of the replica instance CR.
- client_addr: IP address of the replica pod.
ms
gauge
alloydb_omni_memory_available_byte
Estimate of the amount of memory available for allocation.
byte
gauge
alloydb_omni_memory_limit_byte
Memory limit of the database container that's specified in
the
primarySpec
section of spec
portion of the database cluster manifest file.byte
gauge
alloydb_omni_memory_used_byte
Estimate of the used database container memory that can't be freed.
byte
gauge
alloydb_omni_node_cpu_mcpu
Number of mCPUs allocated. 1000 mCPU = 1 CPU
mCPU
gauge
alloydb_omni_node_cpu_usage_second_total
Total CPU seconds used.
second
counter
alloydb_omni_node_network_received_bytes_count_total
Network received bytes count on the AlloyDB Omni pod.
byte
counter
alloydb_omni_node_network_sent_bytes_count_total
Network sent bytes count on the AlloyDB Omni pod.
byte
counter
alloydb_omni_node_storage_limit_per_disk_byte
Storage limit in byte.
disk: name of the disk
byte
gauge
alloydb_omni_node_storage_read_bytes_count_total
Number of bytes read from disk.
byte
counter
alloydb_omni_node_storage_read_ops_count_total
Number of disk read IO operations.
counter
alloydb_omni_node_storage_usage_per_disk_byte
Storage used in bytes per disk.
disk: name of the disk
byte
gauge
alloydb_omni_node_storage_write_bytes_count_total
Number of bytes written to disk.
byte
counter
alloydb_omni_node_storage_write_ops_count_total
Number of disk write IO operations.
counter
Database-level metrics
These metrics are collected on a per AlloyDB Omni database container per Postgres database level. You can create multiple Postgres databases in one database container. All these metrics have resource, system metadata, and "database" labels. The database label is the name of the Postgres database that the metric belongs to.
Name
Description
label
Unit
Type
alloydb_omni_database_postgresql_backends_for_top_databases
The current number of connections per database.
gauge
alloydb_omni_database_postgresql_blks_hit_for_top_databases_count_total
Total number of times Postgres found the requested block in the buffer cache per database.
counter
alloydb_omni_database_postgresql_blks_read_for_top_databases_count_total
Total number of blocks read by Postgres that were not in the Postgres buffer cache per database.
counter
alloydb_omni_database_postgresql_committed_transactions_for_top_databases_count_total
Total number of transactions committed per database.
counter
alloydb_omni_database_postgresql_deadlock_for_top_databases_count_total
The number of deadlocks per database.
counter
alloydb_omni_database_postgresql_insights_aggregate_execution_time_us_total
Total execution time across all queries.
- user: Postgres user that ran the queries.
- client_addr: IP address of the client if available, otherwise empty.
us
counter
alloydb_omni_database_postgresql_insights_aggregate_io_time_us_total
Total time spent doing IO across all queries.
- user: Postgres user that ran the queries
- io_type:
readorwrite
us
counter
alloydb_omni_database_postgresql_new_connections_for_top_databases_count_total
The number of new connections per database.
counter
alloydb_omni_database_postgresql_rolledback_transactions_for_top_databases_count_total
Total number of transactions rolled back per database.
counter
alloydb_omni_database_postgresql_size_byte
Size of the database.
byte
gauge
alloydb_omni_database_postgresql_statements_executed_count_total
Total count of statements executed per database.
operation_type: name of the operation, one of [SELECT, UPDATE, INSERT, DELETE, MERGE, UTILITY, NOTHING, UNKNOWN]
counter
alloydb_omni_database_postgresql_temp_bytes_written_for_top_databases_count_total
Total amount of data written to temporary files by queries per database.
byte
counter
alloydb_omni_database_postgresql_temp_files_written_for_top_databases_count_total
Total number of temporary files used for writing data while performing internal algorithms per database.
counter
alloydb_omni_database_postgresql_tuples
Number of rows in the database.
state: one of [live, dead]
gauge
alloydb_omni_database_postgresql_tuples_deleted_for_top_databases_count_total
The total number of rows deleted per database.
counter
alloydb_omni_database_postgresql_tuples_fetched_for_top_databases_count_total
The total number of rows fetched per database.
counter
alloydb_omni_database_postgresql_tuples_inserted_for_top_databases_count_total
The total number of rows inserted per database.
counter
alloydb_omni_database_postgresql_tuples_returned_for_top_databases_count_total
The total number of rows returned per database.
counter
alloydb_omni_database_postgresql_tuples_updated_for_top_databases_count_total
The total number of rows updated per database.
counter
Metrics collection metrics
These metrics indicate the status of each metric collection cycle. They have the resource labels mentioned in Labels .
| Name | Description | Unit | Type |
|---|---|---|---|
|
alloydb_omni_monitor_collect_ms
|
Number of milliseconds spent to collect metrics. | ms | gauge |
|
alloydb_omni_monitor_error_count
|
Number of errors encountered while trying to collect metrics this cycle. | gauge | |
|
alloydb_omni_monitor_metric_count
|
Number of metrics collected successfully this cycle. | gauge |
Prometheus metric handler metrics
These metrics are automatically generated by Prometheus for each collection cycle.
| Name | Description | Cause | Type |
|---|---|---|---|
|
promhttp_metric_handler_errors_total
|
Total number of internal errors encountered by the promhttp metric handler. | Cause of the error | counter |
Controller-runtime metrics
These metrics are exposed by the AlloyDB Omni Kubernetes operator for monitoring its health and performance. For more information, see the controller-runtime metrics reference .
| Name | Description | Label | Unit | Type |
|---|---|---|---|---|
workqueue_depth
|
Current depth of workqueue. | name | Gauge | |
workqueue_adds_total
|
Total number of adds handled by workqueue. | name | Counter | |
workqueue_queue_duration_seconds
|
How long in seconds an item stays in workqueue before being requested. | name | seconds | Histogram |
workqueue_work_duration_seconds
|
How long in seconds processing an item from workqueue takes. | name | seconds | Histogram |
workqueue_unfinished_work_seconds
|
How many seconds of work has been done that is in progress and hasn't been observed by work_duration
. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. |
name | seconds | Gauge |
workqueue_longest_running_processor_seconds
|
How many seconds has the longest running processor for workqueue been running. | name | seconds | Gauge |
workqueue_retries_total
|
Total number of retries handled by workqueue. | name | Counter | |
rest_client_requests_total
|
Number of HTTP requests, partitioned by status code, method, and host. | code, method, host | Counter | |
controller_runtime_reconcile_total
|
Total number of reconciliations per controller. | controller, result | Counter | |
controller_runtime_reconcile_errors_total
|
Total number of reconciliation errors per controller. | controller | Counter | |
controller_runtime_terminal_reconcile_errors_total
|
Total number of terminal errors from the reconciler. | controller | Counter | |
controller_runtime_reconcile_time_seconds
|
Length of time per reconciliation per controller. | controller | seconds | Histogram |
controller_runtime_max_concurrent_reconciles
|
Maximum number of concurrent reconciles per controller. | controller | Gauge | |
controller_runtime_active_workers
|
Number of used workers per controller. | controller | controller | Gauge |
controller_runtime_webhook_latency_seconds
|
Histogram of the latency of processing admission requests. | webhook | seconds | Histogram |
controller_runtime_webhook_requests_total
|
Total number of admission requests by HTTP status code. | webhook, code | Counter | |
controller_runtime_webhook_requests_in_flight
|
Current number of admission requests being served. | webhook | Gauge |
What's next
- To learn how to use metrics for monitoring, see Monitor AlloyDB Omni .
