We recommend using the following tools to assist you in various stages of your Spanner migration, depending on your source database and other factors. Some tools only support certain source databases. For some steps of the migration process, no tool is available, so you must complete those steps manually.
-
Spanner migration tool (SMT) is an open-source tool that performs assessments, schema conversion, and data migrations. For information on setting up the Spanner migration tool, see Set up Spanner migration tool .
-
Datastream is a Google Cloud service that lets you read change data capture (CDC) events and bulk data from a source database and write to a specified destination.
-
Bulk data migration is a Dataflow template that lets you migrate large MySQL data sets directly to Spanner.
-
Live data migration uses Datastream and Dataflow to migrate:
- Existing data in your source database.
- Stream of changes that are made to your source database during the migration.
-
Data Validation Tool (DVT) is a standardized data validation method built by Google and supported by the open source community. You can integrate DVT into existing Google Cloud products.
-
Database Migration Assessment (DMA) offers a basic assessment to migrate MySQL and PostgreSQL to Spanner.
Migration tools for MySQL source databases
If your source database is MySQL, then you can perform some of the initial migration stages using MySQL dump files. You must directly connect to your running source MySQL database to complete a production migration.
The following table recommends migration tools based on the migration stage and whether you're using a dump file or directly connecting your source database:
- If your sample dump file is less than 100 GB, use SMT in POC mode.
- If your sample dump file is more than 100 GB, export the file to Cloud SQL and perform a bulk migration .
- If your sample dump file is in a CSV, Avro, or Parquet file format, load the file to BigQuery and copy to Spanner using reverse ETL.
Migration tools for PostgreSQL source databases
If your source database uses PostgreSQL, then you can perform some of the migration stages using a PostgreSQL dump file. You must directly connect to your running source PostgreSQL database to complete the migration.
The following table recommends migration tools based on the migration stage and whether you're working with a dump file or directly connecting from your source database:
- If your sample dump file is less than 100GB, use SMT in POC mode.
- If your sample dump file is more than 100GB, export the file to Cloud SQL and perform a minimal downtime migration .
- If your sample dump file is in a CSV, Avro, or Parquet file format, load the file to BigQuery and copy to Spanner using reverse ETL.
Migration tools for a Cassandra-source databases
Spanner offers a Cassandra-compatible interface that supports near-zero application code changes when migrating from Cassandra to Spanner. For more information about compatibility details, see the Cassandra overview .
The following table recommends migration tools based on the migration stage:
- Bulk migrate: Use the Sourcedb to Spanner template
- Live data: Use the DataStax ZDM Proxy .
For large scale databases (>10 million rows): build your own tooling. For more information, see Tips to validate Cassandra using row matching .
For small scale databases (<10 million rows), see the sample validation scripts in GitHub.
