In this page, you learn how to use the Datastream API to:
- Create connection profiles for a source Oracle database, a source MySQL database, a source PostgreSQL database, a source Salesforce org, a BigQuery destination, and a Cloud Storage destination
- Retrieve information about, update, and delete connection profiles
- Discover the structure of source or destination connection profiles
There are two ways that you can use the Datastream API. You can make REST API calls or you can use the Google Cloud CLI (CLI).
For high-level information about using gcloud
to manage Datastream connection profiles, click Google Cloud SDK documentation
.
Create a connection profile for a source Oracle database
The following code shows a request to create a connection profile for an Oracle database using a secret and the IP allowlist connectivity method.
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : "[connection-profile-display-name]" , "oracleProfile" : { "hostname" : " HOST_ID_ADDRESS " , "port" : PORT , "username" : " USERNAME " , "databaseService" : " DATABASE " }, "secretManagerStoredPassword" : " SECRET_MANAGER_RESOURCE " , "staticServiceIpConnectivity" : {} }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=myOracleCP { "displayName" : "my Oracle database" , "oracleProfile" : { "hostname" : "1.2.3.4" , "port" : 1521 , "username" : "admin" , "databaseService" : "ORCL" , }, "secretManagerStoredPassword" : "projects/myProjectId/secrets/mySecret/versions/latest" , "staticServiceIpConnectivity" : {} }
The following code shows a request to create a connection profile for an Oracle database, and specifies the connectivity method as private connectivity (VPC peering).
This method establishes secure connectivity between Datastream and the source database (internally within Google Cloud, or with external sources connected over VPN or Interconnect). This communication happens through a VPC peering connection.
POST h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : "[connection-profile-display-name]" , "oracleProfile" : { "hostname" : " HOST_ID_ADDRESS " , "port" : PORT , "username" : " USERNAME " , "password" : " PASSWORD " , "databaseService" : " DATABASE " }, "privateConnectivity" : { "privateConnection" : "https://datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /privateConnections/ [private-connectivity-configuration-id]" } }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=myOracleVPCPeeri n gCP { "displayName" : "my Oracle database" , "oracleProfile" : { "hostname" : "1.2.3.4" , "port" : 1521 , "username" : "admin" , "password" : "12345" "databaseService" : "ORCL" , }, "privateConnectivity" : { "privateConnection" : "https://datastream.googleapis.com/v1/projects/ myProjectId/locations/us-central1/privateConnections/myPrivateConnection" } }
After the connection profile is created, you can view information about it by calling the connectionProfiles/get
method. Your output appears, similar to the following:
{ "name" : "projects/projectId/location/us-central1/connectionProfiles/myOracleVPCPeeringCP" , "createTime" : "2019-12-22T16:17:37.159786963Z" , "updateTime" : "2019-12-22T16:17:37.159786963Z" , "displayName" : "my Oracle database" , "oracleProfile" : { "hostname" : "1.2.3.4" , "port" : 1521 , "databaseService" : "ORCL" , "username" : "admin" }, "privateConnectivity" : { "privateConnection" : "https://datastream.googleapis.com/v1/projects/ myProjectId/locations/us-central1/privateConnections/myPrivateConnection" } }
gcloud
For more information about using gcloud
to create a connection profile for a source Oracle database, see Google Cloud SDK documentation
.
Create a connection profile for a source MySQL database
The following code shows a request to create a connection profile for a MySQL database, and specifies the connectivity method as using static IP addresses (IP allowlist).
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : "[connection-profile-display-name]" , "mysqlProfile" : { "hostname" : " HOST_ID_ADDRESS " , "port" : PORT , "username" : " USERNAME " , "password" : " PASSWORD " }, "staticServiceIpConnectivity" : {} }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=mysql - cp { "displayName" : "my MySQL database" , "mysqlProfile" : { "hostname" : "1.2.3.4" , "port" : 3306 , "username" : "root" , "password" : "12345" }, "staticServiceIpConnectivity" : {} }
gcloud
For more information about using gcloud
to create a connection profile for a source MySQL database, see Google Cloud SDK documentation
.
Create a connection profile for a source PostgreSQL database
The following code shows a request to create a connection profile for a PostgreSQL database.
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : "[connection-profile-display-name]" , "postgresqlProfile" : { "hostname" : " HOST_ID_ADDRESS " , "port" : PORT , "username" : " USERNAME " , "password" : " PASSWORD " , "database" : " DATABASE " }, "staticServiceIpConnectivity" : {} }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=pos t gres - cp { "displayName" : "my PostgreSQL database" , "postgresqlProfile" : { "hostname" : "1.2.3.4" , "database" : "postgres" , "port" : 5432 , "username" : "root" , "password" : "12345" }, "staticServiceIpConnectivity" : {} }
gcloud
For more information about using gcloud
to create connection profiles, see Google Cloud SDK documentation
.
Create a connection profile for a source SQL Server database
The following code shows a request to create a connection profile for a SQL Server database, and specifies the connectivity method as using static IP addresses (IP allowlist).
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : "[connection-profile-display-name]" , "sqlserverProfile" : { "hostname" : " HOST_ID_ADDRESS " , "port" : PORT , "username" : " USERNAME " , "password" : " PASSWORD " , "database" : " DATABASE " }, "staticServiceIpConnectivity" : {} }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=sqlserver - cp { "displayName" : "my SQL Server database" , "mysqlProfile" : { "hostname" : "1.2.3.4" , "port" : 1433 , "username" : "root" , "password" : "12345" "database" : "database1" }, "staticServiceIpConnectivity" : {} }
gcloud
For more information about using gcloud
to create connection profiles, see Google Cloud SDK documentation
.
Create a connection profile for a Salesforce source
The following code shows a request to create a connection profile for a source Salesforce instance using a username and a password as the authentication method.
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : " CONNECTION_PROFILE_NAME " , "salesforceProfile" : { "domain" : " DOMAIN_NAME " , "userCredentials" : { "username" : " USERNAME " , "password" : " PASSWORD " , "securitytoken" : " SECURITY_TOKEN " } } }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=sale f orce - cp { "displayName" : "My Salesforce connection profile" , "salesforceProfile" : { "domain" : "domain.my.salesforce.com" , "userCredentials" : { "username" : "mySalesforceUser" , "password" : "12345" , "securitytoken" : "C08120F510542FFB1C3640F57AF19E2D5D700556A25F7D665C3B428407709D8C" } } }
gcloud
For more information about using gcloud
to create connection profiles, see Google Cloud SDK documentation
.
The following code shows a request to create a connection profile for a source Salesforce instance using OAuth 2.0 client credentials as the authentication method.
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : " CONNECTION_PROFILE_NAME " , "salesforceProfile" : { "domain" : " DOMAIN_NAME " , "oauth2ClientCredentials" : { "clientId" : " CLIENT_ID " , "clientSecret" : " CLIENT_SECRET " } } }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=sale f orce - cp { "displayName" : "My Salesforce connection profile" , "salesforceProfile" : { "domain" : "domain.my.salesforce.com" , "oauth2ClientCredentials" : { "clientId" : "myClientId" , "clientSecret" : "projects/myProject/secrets/sf-client-secret" } } }
gcloud
For more information about using gcloud
to create connection profiles, see Google Cloud SDK documentation
.
Create a connection profile for a MongoDB source
The following code shows a request to create a connection profile for a source MongoDB instance using an SRV connection string format.
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : " CONNECTION_PROFILE_NAME " , "mongodbProfile" : { "hostAddresses" : [ { "hostname" : " HOST_URI " } ], "username" : " USERNAME " , "password" : " PASSWORD " , "srvConnectionFormat" : {} }, "staticServiceIpConnectivity" : {} }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=mo n godb - cp { "displayName" : "mongodb-cp" , "mongodbProfile" : { "hostAddresses" : [ { "hostname" : "cluster85828.ebeek.mongodb.net" } ], "username" : "Cluster85828" , "password" : "password" , "srvConnectionFormat" : {} }, "staticServiceIpConnectivity" : {} }
gcloud
For more information about using gcloud
to create connection profiles, see Google Cloud SDK documentation
.
The following code shows a request to create a connection profile for a source MongoDB instance using a standard connection string format.
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : " CONNECTION_PROFILE_NAME " , "mongodbProfile" : { "hostAddresses" : [ { "hostname" : " HOST_IP " , "port" : " PORT_NUMBER " } ], "replica_set" : " SET_ID " , "username" : " USERNAME " , "password" : " PASSWORD " , "standardConnectionFormat" : { "directConnection" : false } }, "staticServiceIpConnectivity" : {} }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=mo n godb - cp { "displayName" : "mongodb-cp" , "mongodbProfile" : { "hostAddresses" : [ { "hostname" : "34.19.10.194" , "port" : "27017" } ], "replicaset" : "rs0" "username" : "root" , "password" : "password" , "standardConnectionFormat" : { "directConnection" : false } }, "staticServiceIpConnectivity" : {} }
gcloud
For more information about using gcloud
to create connection profiles, see Google Cloud SDK documentation
.
Create a connection profile for a BigQuery destination
The following code shows a request to create a connection profile for a BigQuery destination.
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ [project-id /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : "connection-profile-display-name" , "bigqueryProfile" : {} }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=myBigqueryCP { "displayName" : "my BigQuery destination" , "bigqueryProfile" : {} }
gcloud
For more information about using gcloud
to create a connection profiles, see Google Cloud SDK documentation
.
Create a connection profile for a Cloud Storage destination
The following code shows a request to create a connection profile for a Cloud Storage bucket. Because Cloud Storage is in the same network as Datastream, no special connectivity is required. Therefore, no connectivity method is specified.
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ [project-id /locations/ LOCATION /co nne c t io n Pro f iles?co nne c t io n Pro f ileId= CONNECTION_PROFILE_ID { "displayName" : "connection-profile-display-name" , "gcsProfile" : { "bucketName" : "bucket-name" , "rootPath" : "prefix-inside-bucket" } }
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles?co nne c t io n Pro f ileId=myGcsCP { "displayName" : "my Cloud Storage bucket" , "gcsProfile" : { "bucketName" : "myBucket" , "rootPath" : "prefix-inside-bucket" } }
gcloud
For more information about using gcloud
to create a connection profile for Cloud Storage, see Cloud SDK documentation
.
Get information about a connection profile
The following code shows a request to retrieve information about a connection profile for a source Oracle database, a source MySQL database, a BigQuery destination, or a Cloud Storage destination that has been created.
REST
GET h tt ps : //datastream.googleapis.com/v1/projects/ project-id /locations/ LOCATION /co nne c t io n Pro f iles/ CONNECTION_PROFILE_ID
For example:
GET h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles/myOracleCP
gcloud
For more information about using gcloud
to retrieve information about your connection profile, see Google Cloud SDK documentation
.
List connection profiles
The following code shows a request to retrieve information about all of your connection profiles.
REST
GET h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles
gcloud
For more information about using gcloud
to retrieve information about all of your connection profiles, see Google Cloud SDK documentation
.
Update a connection profile
The following code shows a request to change the username and password of an existing connection profile for a source database.
By using the updateMask
parameter in the request, only the fields that you specify have to be included in the body of the request (for this example, the username
and password
fields, represented by the oracle_profile.username
and oracle_profile.password
flags, respectively).
REST
PATCH h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles/ CONNECTION_PROFILE_ID ? upda te Mask=oracle_pro f ile.user na me , oracle_pro f ile.password { "oracleProfile" : { "username" : " USERNAME " , "password" : " PASSWORD " } }
For example:
PATCH h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles/myOracleCP? upda te Mask=oracle_pro f ile.user na me , oracle_pro f ile.password { "oracleProfile" : { "username" : "username" , "password" : "password" } }
The following code shows a request to change the private connectivity configuration that's assigned to a connection profile. This connection profile uses the VPC peering network connectivity method.
For this example, you're assigning the new_private_connection
configuration to the connection profile.
PATCH - d { \ "private_connectivity\":{\"private_connection_name\":\ " h tt ps : //datastream.googleapis.com/v1/ projec ts / PROJECT_ID /loca t io ns / LOCATION /priva te Co nne c t io ns / ne w_priva te _co nne c t io n \ "}}" - H "Authorization: Bearer $TOKEN" - H "Content-Type: application/json" h tt ps : //datastream.googleapis.com/v1/ projec ts / PROJECT_ID /loca t io ns / LOCATION /co nne c t io n Pro f iles/ CONNECTION_PROFILE_ID ?upda te _mask=priva te _co nne c t ivi t y.priva te _co nne c t io n _ na me
For example:
PATCH - d { \ "private_connectivity\":{\"private_connection_name\":\ " h tt ps : //datastream.googleapis.com/v1/ projec ts /myProjec t Id/loca t io ns /us - ce ntral 1 /priva te Co nne c t io ns / ne w_priva te _co nne c t io n \ "}}" - H "Authorization: Bearer $TOKEN" - H "Content-Type: application/json" h tt ps : //datastream.googleapis.com/v1/projects/ myProjec t Id/loca t io ns /us - ce ntral 1 /co nne c t io n Pro f iles/ myOracleVPCPeeri n gCP?upda te _mask=priva te _co nne c t ivi t y.priva te _co nne c t io n _ na me
gcloud
For more information about using gcloud
to update your connection profile, see Google Cloud SDK documentation
.
Delete a connection profile
The following code shows a request to delete a connection profile. After the connection profile is deleted, any streams using it will fail.
REST
DELETE h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles/ CONNECTION_PROFILE_ID
For example:
DELETE h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles/myOracleCP
gcloud
For more information about using gcloud
to delete your connection profile, see Google Cloud SDK documentation
.
Discover the structure of a source database
Use the discoverConnectionProfile
API to retrieve the list of entities (for example, schemas and tables) from a source, and the metadata associated with the entities.
The API can receive as a parameter either an ID of an existing connection profile or a complete connection profile object definition. The function can return a single level (for example, all schemas in a database or all tables in a schema), or all entities recursively (for example, schemas, tables, and columns).
REST
POST h tt ps : //datastream.googleapis.com/v1/projects/ PROJECT_ID /locations/ LOCATION /co nne c t io n Pro f iles : discoverCo nne c t io n Pro f ile? CONNECTION_PROFILE_ID
For example:
POST h tt ps : //datastream.googleapis.com/v1/projects/myProjectId/locations/ us - ce ntral 1 /co nne c t io n Pro f iles : discoverCo nne c t io n Pro f ile?myOracleCP
gcloud
For more information about using gcloud
to discover the structure of your source database, see Google Cloud SDK documentation
.
What's next
- Learn how to use the Datastream API to manage streams .
- Learn how to use the Datastream API to manage private connectivity configurations .
- For more information about using the Datastream API, see the reference documentation .
