Diagnose issues for homogeneous migrations to Cloud SQL for PostgreSQL

The migration job process might incur errors during runtime.

• Some errors, such as a bad password on the source database, are recoverable, meaning they can be fixed and the migration job resumes automatically.
• Some are unrecoverable, such as errors in data replication, meaning the migration job needs to be restarted from the beginning.

When an error occurs, the migration job status changes to Failed , and the substatus reflects the last status before failure.

To troubleshoot an error, navigate to the failed migration job to view the error and follow the steps outlined in the error message.

To view more details about the error, navigate to Cloud Monitoring using the link on the migration job. The logs are filtered to the specific migration job.

In the following table, you can find some examples of issues and how they can be solved:

Clear extra data from your existing destination instance

When you migrate to an existing destination instance , you receive the following error message: The destination instance contains existing data or user defined entities (for example databases, tables, or functions). You can only migrate to empty instances. Clear your destination instance and retry the migration job.

This issue can occur if your destination instance contains extra data. You can only migrate to existing instances that are empty. See Known limitations .

Things to try

Clear extra data from your destination instance and start the migration job again by performing the following steps:

1. Stop the migration job .
2. At this point, your destination Cloud SQL instance is in `read-only` mode. Promote the destination instance to gain write access.
3. Connect to your destination Cloud SQL instance .
4. Remove extra data from your destination instance databases. Your destination can only contain system configuration data. Destination databases can't contain user data (such as tables). There are different SQL statements you can run on your databases to find non-system data, for example:

   Example SQL statement to retrieve non-system databases (click to expand)

    SELECT 
     
    datname 
     
    FROM 
     
    pg_catalog 
    . 
    pg_database 
    WHERE 
     
    datname 
     
    NOT 
     
    IN 
     
    ( 
    'cloudsqladmin' 
    , 
     
    'template1' 
    , 
     
    'template0' 
    , 
     
    'postgres' 
    ); 
   

   Example SQL statement to retrieve non-system data in the postgres database (click to expand)

   The postgres database is a system database, but it can contain non-system data. Make sure you run these statements on the postgres database. If you use the psql client to connect to the destination instance, you can switch to another database without resetting your connection by using the \connect {database_name_here} command.

    SELECT 
     
    table_schema 
    , 
     
    table_name 
     
    FROM 
     
    information_schema 
    . 
    tables 
    WHERE 
     
    table_schema 
     
    != 
     
    'information_schema' 
     
    AND 
     
    table_schema 
     
    not 
     
    like 
     
    'pg\_%' 
    ; 
    SELECT 
     
    routine_schema 
    , 
     
    routine_name 
     
    FROM 
     
    information_schema 
    . 
    routines 
    WHERE 
     
    routine_schema 
     
    != 
     
    'information_schema' 
     
    AND 
     
    routine_schema 
     
    not 
     
    like 
     
    'pg\_%' 
    ; 
    SELECT 
     
    extname 
     
    FROM 
     
    pg_extension 
     
    WHERE 
     
    extname 
     
    != 
     
    'plpgsql' 
    ; 
     
   

5. Start the migration job .

Clean up replication slots

You see one of the following messages:

• Cleanup may have failed on source due to error: generic::unknown: failed to connect to on-premises database.
• Error promoting EM replica: finished drop replication with errors.

The issue might be

When promoting a Cloud SQL instance, if the source instance isn't reachable from the Cloud SQL instance (for example, the source instance isn't running, or you removed the Cloud SQL instance from the allow list of source instances), then the settings needed for the replication can't be cleaned up during the promotion of a migration job. You must clean up the replication slots manually.

Things to try

For each database, run the following commands as a user with the superuser privilege:

1. Get the replication slot names from the error message, and then run the following command to drop the slots, one by one:

   select pg_drop_replication_slot({ slot_name 
   });

2. If the replication slot names aren't available in the error message, then run the following command to query for the existing replication slots:

   select pg_drop_replication_slot( slot_name 
   ) from pg_replication_slots where slot_name 
   like '%cloudsql%' and active = 'f';

3. If there are no Cloud SQL replicas using the source instance, then run the following command to clean up pglogical settings:

   select pglogical.drop_node( node_name 
   ) from pglogical.node where node_name 
   like 'cloudsql'; 
   

4. If the pglogical extension isn't needed anymore, then run the following command to uninstall the extension:

   DROP EXTENSION IF EXISTS pglogical;

Error message: Cannot connect to invalid database

When migrating to PostgreSQL version 15, after multiple subsequent connection retry attempts one of the following symptoms occur:

• You receive a Cannot connect to invalid database error message.
• The Storage usage migration job metric shows no progress after a long time when the migration job is performing the full database dump.

The issue might be

This problem is often attributed to the deadlock issue in the pglogical extension. For more information, see the pglogical issue tracker in GitHub .

Things to try

Perform the migration job again with a new destination instance

Try deleting the destination database where you experienced the issue and re-create your migration job. Follow these steps:

1. Delete the destination instance where you experienced the problems. See Delete instances in Cloud SQL for PostgreSQL documentation.
2. Delete the failed migration job. See Review a migration job .
3. Re-create your migration job. See Create a migration job .

Migrate to an intermediate version

Consider migrating to an earlier PostgreSQL version, such as PostgreSQL 14. After a successful migration, you can try upgrade to the desired PostgreSQL 15 instance. See Upgrade the database major version by migrating data in Cloud SQL for PostgreSQL documentation.

Manage users and roles

Migrate existing users

Currently, Database Migration Service doesn't support migrating existing users from a source instance into a destination Cloud SQL instance. You can manage this migration by creating the users in Cloud SQL manually.

About the cloudsqlexternalsync user

During the migration, all objects on the Cloud SQL replica are owned by the cloudsqlexternalsync user. After the data is migrated, you can modify the ownership of the objects to other users by completing the following steps:

• Run the GRANT cloudsqlexternalsync to {USER} command.
• On each database, run the reassign owned by cloudsqlexternalsync to {USER} ; command.
• To remove the cloudsqlexternalsync user, run the drop role cloudsqlexternalsync command.

Import data into a new Cloud SQL instance

If you first export data from a Cloud SQL instance that Database Migration Service migrated into Cloud Storage, and then import the data from Cloud Storage into a stand-alone Cloud SQL instance, the import might fail because the cloudsqlexternalsync user doesn't exist on the destination instance.

To mitigate the issue, either create the cloudsqlexternalsync user on the destination instance or remove the user from the migrated instance.