Manage and monitor AlloyDB Omni

Single-server

AlloyDB Omni logs its activity in two locations:

• AlloyDB Omni logs database activity at data/log/postgres , relative to the directory that you defined in your dataplane.conf configuration file.

  You can customize the name and format of this log file through the various log_* directives defined in /var/alloydb/config/postgresql.conf . For more information, see Error Reporting and Logging .

• AlloyDB Omni logs its installation, startup, and shutdown activity at /var/alloydb/logs/alloydb.log .

To check the immediate running status of your server, see Check the status of AlloyDB Omni .

Kubernetes

Find your database cluster log files

You can find postgresql.audit and postgresql.log files on the file system of the database pod. To access these files, follow these steps:

1. Define an environment variable containing the name of the database pod.

     export 
     
    DB_POD 
    = 
    ` 
   kubectl  
   get  
   pod  
   -l  
   alloydbomni.internal.dbadmin.goog/dbcluster = 
    DB_CLUSTER_NAME 
   ,alloydbomni.internal.dbadmin.goog/task-type = 
   database  
   -o  
    jsonpath 
    = 
    '{.items[0].metadata.name}' 
    ` 
    
   

   Replace DB_CLUSTER_NAME with the name of your database cluster. It's the same database cluster name you declared when you created it .

2. Run a shell on the database pod as root.

    kubectl  
    exec 
     
    ${ 
    DB_POD 
    } 
     
   -it  
   --  
   /bin/bash 
   

3. Find the log files in the /obs/diagnostic/ directory:

   • /obs/diagnostic/postgresql.audit
   • /obs/diagnostic/postgresql.log

List monitoring services

 kubectl  
get  
svc  
-n  
 NAMESPACE 
  
 | 
  
grep  
monitoring 

 al-1060-dbc-monitoring-system  
ClusterIP  
 10 
.0.15.227  
<none>  
 9187 
/TCP  
7d20h
al-3de6-dbc-monitoring-system  
ClusterIP  
 10 
.0.5.205  
<none>  
 9187 
/TCP  
7d19h
al-4bc0-dbc-monitoring-system  
ClusterIP  
 10 
.0.15.92  
<none>  
 9187 
/TCP  
7d19h 

 kubectl  
get  
svc  
-n  
 NAMESPACE 
  
 | 
  
grep  
monitoring 

 al-2953-dbcluster-foo7-monitoring-db  
ClusterIP  
 10 
.36.3.243  
<none>  
 9187 
/TCP  
44m
al-2953-dbcluster-foo7-monitoring-system  
ClusterIP  
 10 
.36.7.72  
<none>  
 9187 
/TCP  
44m 

View Prometheus metrics from the command line

The port 9187 is named as metricsalloydbomni for all monitoring services.

1. Set up port forwarding from your local environment to the monitoring service.

    kubectl  
   port-forward  
   service/ MONITORING_SERVICE 
     
   -n  
    NAMESPACE 
     
    MONITORING_METRICS_PORT 
   :metricsalloydbomni 
   

   Replace the following:

   • MONITORING_SERVICE : The name of the monitoring service that you want to forward—for example, al-1060-dbc-monitoring-system .

   • NAMESPACE : The namespace where your cluster belongs.

   • MONITORING_METRICS_PORT : A local available TCP port.

   The following response shows that the services are being forwarded.

    Forwarding  
   from  
    127 
   .0.0.1:9187  
   ->  
    9187 
   Forwarding  
   from  
    [ 
   ::1 ] 
   :9187  
   ->  
    9187 
    
   

2. While the previous command runs, you can access monitoring metrics through HTTP on the port that you specified. For example, you can use curl to see all of the metrics as plain text:

    curl  
   http://localhost: MONITORING_METRICS_PORT 
   /metrics 
   

View metrics using the Prometheus API

The alloydbomni.internal.dbadmin.goog/task-type label key and the metricsalloydbomni port is available as a default for all monitoring services in AlloyDB Omni. You can use them together with a single ServiceMonitor custom resource to select all the services for all namespaces in your database cluster.

For more information about using the Prometheus API, see the Prometheus Operator documentation .

The following is an example spec field of the ServiceMonitor custom resource that includes the alloydbomni.internal.dbadmin.gdc.goog/task-type label key and metricsalloydbomni port. The ServiceMonitor custom resource monitors and collects all the kubernetes services in all namespaces

For more information about the complete ServiceMonitor definition, see the ServiceMonitor custom resource definition .

   
 spec 
 : 
  
 selector 
 : 
  
 matchLabels 
 : 
  
 alloydbomni.internal.dbadmin.goog/task-type 
 : 
  
 monitoring 
  
 namespaceSelector 
 : 
  
 any 
 : 
  
 true 
  
 endpoints 
 : 
  
 - 
  
 port 
 : 
  
 metricsalloydbomni 
 

   
 spec 
 : 
  
 selector 
 : 
  
 matchExpressions 
 : 
  
 - 
  
 key 
 : 
  
 alloydbomni.internal.dbadmin.gdc.goog/task-type 
  
 operator 
 : 
  
 Exists 
  
 values 
 : 
  
 [] 
  
 namespaceSelector 
 : 
  
 any 
 : 
  
 true 
  
 endpoints 
 : 
  
 - 
  
 port 
 : 
  
 metricsalloydbomni 
 

Single-server

These instructions apply only to AlloyDB Omni version 15.2.0 and later.

Before you begin

Your machine must have version 1.2 or later of the AlloyDB Omni CLI installed.

Perform the upgrade

To upgrade your AlloyDB Omni installation, run the following command:

 sudo  
alloydb  
database-server  
upgrade 

Kubernetes

The steps that you take to upgrade AlloyDB Omni in Kubernetes depend upon the version of AlloyDB Omni that you are running, and the version that you are upgrading to.

Determine your current version numbers

To check the version of AlloyDB Omni used by your database cluster, run this command:

 kubectl  
get  
dbclusters.alloydbomni.dbadmin.goog  
 DB_CLUSTER_NAME 
  
-n  
 NAMESPACE 
  
-o  
 jsonpath 
 = 
 '{.status.primary.currentDatabaseVersion}' 
 

Replace the following:

• DB_CLUSTER_NAME : the name of your database cluster. It's the same database cluster name you declared when you created it .

• NAMESPACE : the Kubernetes namespace of your database cluster.

If you are running version 1.0.0 or later of the AlloyDB Omni Operator, then this command prints the version of AlloyDB Omni in use by your database cluster.

To check the version of the AlloyDB Omni Operator installed on your Kubernetes cluster, run this command:

 kubectl  
get  
dbclusters.alloydbomni.dbadmin.goog  
 DB_CLUSTER_NAME 
  
-n  
 NAMESPACE 
  
-o  
 jsonpath 
 = 
 '{.status.primary.currentControlPlaneAgentsVersion}' 
 

If you are running version 1.0.0 or later of the AlloyDB Omni Operator, then this command prints the version number of the AlloyDB Omni Operator running on your Kubernetes cluster.

If you are running a version of AlloyDB Omni Operator earlier than 1.0.0, then you can't perform an in-place upgrade of your database cluster or of the AlloyDB Omni Operator. Instead, you must follow the instructions in Upgrade from a pre-1.0.0 AlloyDB Omni Operator .

Otherwise, continue to the next section.

Determine your target version numbers

If you are running a version of AlloyDB Omni Operator 1.0.0 or later, then your next steps depend upon the version of AlloyDB Omni that you want to upgrade to. This, in turn, requires understanding of the AlloyDB Omni version number.

The AlloyDB Omni version number has three parts:

• The major version number of its PostgreSQL compatibility
• The minor version number of its PostgreSQL compatibility
• The patch version number of this AlloyDB Omni release

For example, AlloyDB Omni version 15.5.2 is patch version 2 of the AlloyDB Omni that supports PostgreSQL version 15.5.

If you want to upgrade to a version of AlloyDB Omni that supports a newer version of PostgreSQL, then you must upgrade the AlloyDB Omni Operator itself, alongside your database cluster. Each set of AlloyDB Omni releases that support a particular PostgreSQL minor version has its own associated AlloyDB Omni Operator version number, which you can find in the release note for the AlloyDB Omni version.

If you want to upgrade only to a newer patch version of AlloyDB Omni, then you can upgrade only your database cluster, with no need to upgrade the AlloyDB Omni Operator itself. You can skip to the instructions in Upgrade your database cluster .

Otherwise, continue to the next section.

Upgrade the AlloyDB Omni Operator

To upgrade the AlloyDB Omni Operator, do the following:

1. Define necessary environment variables:

     export 
     
    GCS_BUCKET 
    = 
   alloydb-omni-operator 
     export 
     
    OPERATOR_VERSION 
    = 
    OPERATOR_VERSION 
    
     export 
     
    HELM_PATH 
    = 
    $OPERATOR_VERSION 
   /alloydbomni-operator- $OPERATOR_VERSION 
   .tgz 
   

   Replace OPERATOR_VERSION with the version of the AlloyDB Omni Operator that you are upgrading to—for example, 1.1.0 .

2. Download the newer AlloyDB Omni Operator:

    gcloud  
   storage  
   cp  
   gs:// $GCS_BUCKET 
   / $HELM_PATH 
     
   ./  
   --recursive 
    tar  
   -xvzf  
   alloydbomni-operator- ${ 
    OPERATOR_VERSION 
    } 
   .tgz 
   

3. Apply the newer AlloyDB Omni Operator custom resource definitions:

    kubectl  
   apply  
   -f  
   alloydbomni-operator/crds 
   

4. Upgrade the AlloyDB Omni Operator Helm chart:

    helm  
   upgrade  
   alloydbomni-operator  
   alloydbomni-operator- ${ 
    OPERATOR_VERSION 
    } 
   .tgz  
    \ 
   --namespace  
   alloydb-omni-system  
    \ 
   --atomic  
    \ 
   --timeout  
   5m 
   

To follow up on your AlloyDB Omni Operator upgrade by upgrading both your Kubernetes manifest and your database cluster, follow the instructions in the next section immediately after completing the previous steps.

Upgrade your database cluster

To upgrade your database cluster, update the following fields in the spec section of the manifest that defines it:

• Set databaseVersion to the full version number of AlloyDB Omni that you want to upgrade this database cluster to.

• If you have also upgraded AlloyDB Omni Operator, then set controlPlaneAgentsVersion to the full version number of the AlloyDB Omni Operator that you have upgraded.

If you are upgrading only the patch version of AlloyDB Omni—for example, updating databaseVersion from 15.5.1 to 15.5.2 —then this step is all that you need to accomplish.

If you are upgrading the minor version of PostgreSQL compatibility—for example, updating databaseVersion from 15.4.1 to 15.5.2 ‐then you must also update controlPlaneAgentsVersion . In that case, make sure that you have followed the extra steps listed in Upgrade the AlloyDB Omni Operator .

As an example, the following manifest excerpt defines a database cluster running AlloyDB Omni Operator version 15.5.2 , with AlloyDB Omni Operator version 1.0.0 :

 apiVersion: alloydbomni.dbadmin.goog/v1 
kind: DBCluster
metadata:
  name: dbcluster-sample
spec:
  databaseVersion: 15.5.2
  controlPlaneAgentsVersion: 1.0.0 

In this example, to upgrade the database cluster to run AlloyDB Omni version 15.5.3 , change databaseVersion: 15.5.2 to databaseVersion: 15.5.3 .

Upgrade from a pre-1.0.0 AlloyDB Omni Operator

If you are running a version of AlloyDB Omni Operator earlier than 1.0.0, then upgrading a Kubernetes-based AlloyDB Omni installation involves uninstalling and then re-installing the AlloyDB Omni Operator after backing up your data. Follow these steps:

1. List all of your database clusters:

    kubectl  
   get  
   dbclusters.alloydbomni.dbadmin.goog  
   --all-namespaces 
   

2. For each database cluster, use the pg_dumpall command to export all of its data.

3. Uninstall the AlloyDB Omni Operator. This includes deleting all of your database clusters.

4. Reinstall the AlloyDB Omni Operator. You can use the same commands that you used to install the previous version of the AlloyDB Omni Operator, with no need to specify a new version number.

5. Recreate your database clusters. You can adapt the same manifest files that you used when previously creating your database clusters. You might need to update the files to reflect API changes that version 1.0.0 of the AlloyDB Omni Operator introduced, such as the databaseVersion attribute replacing the older version attribute.

6. Use pg_restore or the \i command in psql to import your previously exported data into the recreated clusters.

Single-server

To uninstall AlloyDB Omni, run the following command:

 sudo  
alloydb  
database-server  
uninstall 

Your data directory remains on your file system after you uninstall AlloyDB Omni. You can move, archive, or delete this directory, depending on whether and how you want to preserve your data after uninstalling AlloyDB Omni.

Kubernetes

Delete your database cluster

To delete your database cluster, set isDeleted to true in its manifest. You can accomplish this with the following command.

 kubectl  
patch  
dbclusters.alloydbomni.dbadmin.goog  
 DB_CLUSTER_NAME 
  
-p  
 '{"spec":{"isDeleted":true}}' 
  
--type = 
merge 

Replace DB_CLUSTER_NAME with the name of your database cluster. It's the same database cluster name you declared when you created it .

Uninstall the AlloyDB Omni Operator

To uninstall the AlloyDB Omni Kubernetes Operator from your Kubernetes cluster, take the following steps:

1. Delete all of your database clusters:

     for 
     
   ns  
    in 
     
    $( 
   kubectl  
   get  
   dbclusters.alloydbomni.dbadmin.goog  
   --all-namespaces  
   -o = 
    jsonpath 
    = 
    '{range .items[*]}{.metadata.namespace}{"\n"}{end}' 
    ) 
    ; 
     
    do 
    for 
     
   cr  
    in 
     
    $( 
   kubectl  
   get  
   dbclusters.alloydbomni.dbadmin.goog  
   -n  
    $ns 
     
   -o = 
    jsonpath 
    = 
    '{range .items[*]}{.metadata.name}{"\n"}{end}' 
    ) 
    ; 
     
    do 
   kubectl  
   patch  
   dbclusters.alloydbomni.dbadmin.goog  
    $cr 
     
   -n  
    $ns 
     
   --type = 
   merge  
   -p  
    '{"spec":{"isDeleted":true}}' 
    done 
    done 
    
   

2. Wait for the AlloyDB Omni Kubernetes Operator to delete all of your database clusters. Use the following command to check whether any database resources remain:

    kubectl  
   get  
   dbclusters.alloydbomni.dbadmin.goog  
   --all-namespaces 
   

3. Delete other resources that the AlloyDB Omni Kubernetes Operator created:

    kubectl  
   delete  
   failovers.alloydbomni.dbadmin.goog  
   --all  
   --all-namespaces 
    kubectl  
   delete  
   restores.alloydbomni.dbadmin.goog  
   --all  
   --all-namespaces 
    kubectl  
   delete  
   switchovers.alloydbomni.dbadmin.goog  
   --all  
   --all-namespaces 
   

4. Uninstall the AlloyDB Omni Kubernetes Operator:

    helm  
   uninstall  
   alloydbomni-operator  
   --namespace  
   alloydb-omni-system 
   

5. Clean up secrets, custom resource descriptions, and namespaces related to the AlloyDB Omni Kubernetes Operator:

    kubectl  
   delete  
   certificate  
   -n  
   alloydb-omni-system  
   --all 
    kubectl  
   get  
   secrets  
   --all-namespaces  
   -o  
   custom-columns = 
   NAMESPACE:.metadata.namespace,NAME:.metadata.name,ANNOTATION:.metadata.annotations.cert-manager \. 
   io/issuer-name  
    | 
     
   grep  
   -E  
    'alloydbomni|dbs-al' 
     
    | 
     
   awk  
    '{print $1 " " $2}' 
     
    | 
     
   xargs  
   -n  
    2 
     
   kubectl  
   delete  
   secret  
   -n 
    kubectl  
   delete  
   crd  
   -l  
   alloydb-omni = 
    true 
    
    kubectl  
   delete  
   ns  
   alloydb-omni-system