Console
    Contact Us Start free

    REST Resource: projects.locations.streams

    Resource: Stream

    A resource representing streaming data from a source to a destination.

    JSON representation 
     { 
 "name" 
 : 
 string 
 , 
 "createTime" 
 : 
 string 
 , 
 "updateTime" 
 : 
 string 
 , 
 "labels" 
 : 
 { 
 string 
 : 
 string 
 , 
 ... 
 } 
 , 
 "displayName" 
 : 
 string 
 , 
 "sourceConfig" 
 : 
 { 
 object (  SourceConfig 
 
) 
 } 
 , 
 "destinationConfig" 
 : 
 { 
 object (  DestinationConfig 
 
) 
 } 
 , 
 "state" 
 : 
 enum (  State 
 
) 
 , 
 "errors" 
 : 
 [ 
 { 
 object (  Error 
 
) 
 } 
 ] 
 , 
 "lastRecoveryTime" 
 : 
 string 
 , 
 // Union field backfill_strategy 
can be only one of the following: 
 "backfillAll" 
 : 
 { 
 object (  BackfillAllStrategy 
 
) 
 } 
 , 
 "backfillNone" 
 : 
 { 
 object (  BackfillNoneStrategy 
 
) 
 } 
 // End of list of possible types for union field backfill_strategy 
. 
 "customerManagedEncryptionKey" 
 : 
 string 
 , 
 "satisfiesPzs" 
 : 
 boolean 
 , 
 "satisfiesPzi" 
 : 
 boolean 
 }
    Fields
    name

    string

    Output only. Identifier. The stream's name.

    createTime

    string ( Timestamp format)

    Output only. The creation time of the stream.

    Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z" , "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30" .

    updateTime

    string ( Timestamp format)

    Output only. The last update time of the stream.

    Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z" , "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30" .

    labels

    map (key: string, value: string)

    Labels.

    An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" } .

    displayName

    string

    Required. Display name.

    sourceConfig

    object ( SourceConfig )

    Required. Source connection profile configuration.

    destinationConfig

    object ( DestinationConfig )

    Required. Destination connection profile configuration.

    state

    enum ( State )

    The state of the stream.

    errors[]

    object ( Error )

    Output only. Errors on the Stream.

    lastRecoveryTime

    string ( Timestamp format)

    Output only. If the stream was recovered, the time of the last recovery. Note: This field is currently experimental.

    Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z" , "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30" .

    Union field backfill_strategy . Stream backfill strategy. backfill_strategy can be only one of the following:
    backfillAll

    object ( BackfillAllStrategy )

    Automatically backfill objects included in the stream source configuration. Specific objects can be excluded.

    backfillNone

    object ( BackfillNoneStrategy )

    Do not automatically backfill any objects.

    customerManagedEncryptionKey

    string

    Immutable. A reference to a KMS encryption key. If provided, it will be used to encrypt the data. If left blank, data will be encrypted using an internal Stream-specific encryption key provisioned through KMS.

    satisfiesPzs

    boolean

    Output only. Reserved for future use.

    satisfiesPzi

    boolean

    Output only. Reserved for future use.

    SourceConfig

    The configuration of the stream source.

    JSON representation 
     { 
 "sourceConnectionProfile" 
 : 
 string 
 , 
 // Union field source_stream_config 
can be only one of the following: 
 "oracleSourceConfig" 
 : 
 { 
 object (  OracleSourceConfig 
 
) 
 } 
 , 
 "mysqlSourceConfig" 
 : 
 { 
 object (  MysqlSourceConfig 
 
) 
 } 
 , 
 "postgresqlSourceConfig" 
 : 
 { 
 object (  PostgresqlSourceConfig 
 
) 
 } 
 , 
 "sqlServerSourceConfig" 
 : 
 { 
 object (  SqlServerSourceConfig 
 
) 
 } 
 , 
 "salesforceSourceConfig" 
 : 
 { 
 object (  SalesforceSourceConfig 
 
) 
 } 
 , 
 "mongodbSourceConfig" 
 : 
 { 
 object (  MongodbSourceConfig 
 
) 
 } 
 // End of list of possible types for union field source_stream_config 
. 
 }
    Fields
    sourceConnectionProfile

    string

    Required. Source connection profile resource. Format: projects/{project}/locations/{location}/connectionProfiles/{name}

    Union field source_stream_config . Stream configuration that is specific to the data source type. source_stream_config can be only one of the following:
    oracleSourceConfig

    object ( OracleSourceConfig )

    Oracle data source configuration.

    mysqlSourceConfig

    object ( MysqlSourceConfig )

    MySQL data source configuration.

    postgresqlSourceConfig

    object ( PostgresqlSourceConfig )

    PostgreSQL data source configuration.

    sqlServerSourceConfig

    object ( SqlServerSourceConfig )

    SQLServer data source configuration.

    salesforceSourceConfig

    object ( SalesforceSourceConfig )

    Salesforce data source configuration.

    mongodbSourceConfig

    object ( MongodbSourceConfig )

    MongoDB data source configuration.

    OracleSourceConfig

    Oracle data source configuration

    JSON representation 
     { 
 "includeObjects" 
 : 
 { 
 object (  OracleRdbms 
 
) 
 } 
 , 
 "excludeObjects" 
 : 
 { 
 object (  OracleRdbms 
 
) 
 } 
 , 
 "maxConcurrentCdcTasks" 
 : 
 integer 
 , 
 "maxConcurrentBackfillTasks" 
 : 
 integer 
 , 
 // Union field large_objects_handling 
can be only one of the following: 
 "dropLargeObjects" 
 : 
 { 
 object (  DropLargeObjects 
 
) 
 } 
 , 
 "streamLargeObjects" 
 : 
 { 
 object (  StreamLargeObjects 
 
) 
 } 
 // End of list of possible types for union field large_objects_handling 
. 
 // Union field cdc_method 
can be only one of the following: 
 "logMiner" 
 : 
 { 
 object (  LogMiner 
 
) 
 } 
 , 
 "binaryLogParser" 
 : 
 { 
 object (  BinaryLogParser 
 
) 
 } 
 // End of list of possible types for union field cdc_method 
. 
 }
    Fields
    includeObjects

    object ( OracleRdbms )

    Oracle objects to include in the stream.

    excludeObjects

    object ( OracleRdbms )

    Oracle objects to exclude from the stream.

    maxConcurrentCdcTasks

    integer

    Maximum number of concurrent CDC tasks. The number should be non-negative. If not set (or set to 0), the system's default value is used.

    maxConcurrentBackfillTasks

    integer

    Maximum number of concurrent backfill tasks. The number should be non-negative. If not set (or set to 0), the system's default value is used.

    Union field large_objects_handling . The configuration for handle Oracle large objects. large_objects_handling can be only one of the following:
    dropLargeObjects

    object ( DropLargeObjects )

    Drop large object values.

    streamLargeObjects

    object ( StreamLargeObjects )

    Stream large object values.

    Union field cdc_method . Configuration to select the CDC method. cdc_method can be only one of the following:
    logMiner

    object ( LogMiner )

    Use LogMiner.

    binaryLogParser

    object ( BinaryLogParser )

    Use Binary Log Parser.

    OracleRdbms

    Oracle database structure.

    JSON representation 
     { 
 "oracleSchemas" 
 : 
 [ 
 { 
 object (  OracleSchema 
 
) 
 } 
 ] 
 }
    Fields
    oracleSchemas[]

    object ( OracleSchema )

    Oracle schemas/databases in the database server.

    OracleSchema

    Oracle schema.

    JSON representation 
     { 
 "schema" 
 : 
 string 
 , 
 "oracleTables" 
 : 
 [ 
 { 
 object (  OracleTable 
 
) 
 } 
 ] 
 }
    Fields
    schema

    string

    Schema name.
    oracleTables[]

    object ( OracleTable )

    Tables in the schema.

    OracleTable

    Oracle table.

    JSON representation 
     { 
 "table" 
 : 
 string 
 , 
 "oracleColumns" 
 : 
 [ 
 { 
 object (  OracleColumn 
 
) 
 } 
 ] 
 }
    Fields
    table

    string

    Table name.
    oracleColumns[]

    object ( OracleColumn )

    Oracle columns in the schema. When unspecified as part of include/exclude objects, includes/excludes everything.

    OracleColumn

    Oracle Column.

    JSON representation 
     { 
 "column" 
 : 
 string 
 , 
 "dataType" 
 : 
 string 
 , 
 "length" 
 : 
 integer 
 , 
 "precision" 
 : 
 integer 
 , 
 "scale" 
 : 
 integer 
 , 
 "encoding" 
 : 
 string 
 , 
 "primaryKey" 
 : 
 boolean 
 , 
 "nullable" 
 : 
 boolean 
 , 
 "ordinalPosition" 
 : 
 integer 
 }
    Fields
    column

    string

    Column name.
    dataType

    string

    The Oracle data type.
    length

    integer

    Column length.
    precision

    integer

    Column precision.
    scale

    integer

    Column scale.
    encoding

    string

    Column encoding.
    primaryKey

    boolean

    Whether or not the column represents a primary key.
    nullable

    boolean

    Whether or not the column can accept a null value.
    ordinalPosition

    integer

    The ordinal position of the column in the table.

    DropLargeObjects

    This type has no fields.

    Configuration to drop large object values.

    StreamLargeObjects

    This type has no fields.

    Configuration to stream large object values.

    LogMiner

    This type has no fields.

    Configuration to use LogMiner CDC method.

    BinaryLogParser

    Configuration to use Binary Log Parser CDC technique.

    JSON representation 
     { 
 // Union field log_file_access 
can be only one of the following: 
 "oracleAsmLogFileAccess" 
 : 
 { 
 object (  OracleAsmLogFileAccess 
 
) 
 } 
 , 
 "logFileDirectories" 
 : 
 { 
 object (  LogFileDirectories 
 
) 
 } 
 // End of list of possible types for union field log_file_access 
. 
 }
    Fields
    Union field log_file_access . Configuration to specify how the log file should be accessed. log_file_access can be only one of the following:
    oracleAsmLogFileAccess

    object ( OracleAsmLogFileAccess )

    Use Oracle ASM.

    logFileDirectories

    object ( LogFileDirectories )

    Use Oracle directories.

    OracleAsmLogFileAccess

    This type has no fields.

    Configuration to use Oracle ASM to access the log files.

    LogFileDirectories

    Configuration to specify the Oracle directories to access the log files.

    JSON representation 
     { 
 "onlineLogDirectory" 
 : 
 string 
 , 
 "archivedLogDirectory" 
 : 
 string 
 }
    Fields
    onlineLogDirectory

    string

    Required. Oracle directory for online logs.
    archivedLogDirectory

    string

    Required. Oracle directory for archived logs.

    MysqlSourceConfig

    MySQL source configuration

    JSON representation 
     { 
 "includeObjects" 
 : 
 { 
 object (  MysqlRdbms 
 
) 
 } 
 , 
 "excludeObjects" 
 : 
 { 
 object (  MysqlRdbms 
 
) 
 } 
 , 
 "maxConcurrentCdcTasks" 
 : 
 integer 
 , 
 "maxConcurrentBackfillTasks" 
 : 
 integer 
 , 
 // Union field cdc_method 
can be only one of the following: 
 "binaryLogPosition" 
 : 
 { 
 object (  BinaryLogPosition 
 
) 
 } 
 , 
 "gtid" 
 : 
 { 
 object (  Gtid 
 
) 
 } 
 // End of list of possible types for union field cdc_method 
. 
 }
    Fields
    includeObjects

    object ( MysqlRdbms )

    MySQL objects to retrieve from the source.

    excludeObjects

    object ( MysqlRdbms )

    MySQL objects to exclude from the stream.

    maxConcurrentCdcTasks

    integer

    Maximum number of concurrent CDC tasks. The number should be non negative. If not set (or set to 0), the system's default value will be used.

    maxConcurrentBackfillTasks

    integer

    Maximum number of concurrent backfill tasks. The number should be non negative. If not set (or set to 0), the system's default value will be used.

    Union field cdc_method . The CDC method to use for the stream. cdc_method can be only one of the following:
    binaryLogPosition

    object ( BinaryLogPosition )

    Use Binary log position based replication.

    gtid

    object ( Gtid )

    Use GTID based replication.

    MysqlRdbms

    MySQL database structure

    JSON representation 
     { 
 "mysqlDatabases" 
 : 
 [ 
 { 
 object (  MysqlDatabase 
 
) 
 } 
 ] 
 }
    Fields
    mysqlDatabases[]

    object ( MysqlDatabase )

    Mysql databases on the server

    MysqlDatabase

    MySQL database.

    JSON representation 
     { 
 "database" 
 : 
 string 
 , 
 "mysqlTables" 
 : 
 [ 
 { 
 object (  MysqlTable 
 
) 
 } 
 ] 
 }
    Fields
    database

    string

    Database name.
    mysqlTables[]

    object ( MysqlTable )

    Tables in the database.

    MysqlTable

    MySQL table.

    JSON representation 
     { 
 "table" 
 : 
 string 
 , 
 "mysqlColumns" 
 : 
 [ 
 { 
 object (  MysqlColumn 
 
) 
 } 
 ] 
 }
    Fields
    table

    string

    Table name.
    mysqlColumns[]

    object ( MysqlColumn )

    MySQL columns in the database. When unspecified as part of include/exclude objects, includes/excludes everything.

    MysqlColumn

    MySQL Column.

    JSON representation 
     { 
 "column" 
 : 
 string 
 , 
 "dataType" 
 : 
 string 
 , 
 "length" 
 : 
 integer 
 , 
 "collation" 
 : 
 string 
 , 
 "primaryKey" 
 : 
 boolean 
 , 
 "nullable" 
 : 
 boolean 
 , 
 "ordinalPosition" 
 : 
 integer 
 , 
 "precision" 
 : 
 integer 
 , 
 "scale" 
 : 
 integer 
 }
    Fields
    column

    string

    Column name.
    dataType

    string

    The MySQL data type. Full data types list can be found here: https://dev.mysql.com/doc/refman/8.0/en/data-types.html

    length

    integer

    Column length.
    collation

    string

    Column collation.
    primaryKey

    boolean

    Whether or not the column represents a primary key.
    nullable

    boolean

    Whether or not the column can accept a null value.
    ordinalPosition

    integer

    The ordinal position of the column in the table.
    precision

    integer

    Column precision.
    scale

    integer

    Column scale.

    BinaryLogPosition

    This type has no fields.

    Use Binary log position based replication.

    Gtid

    This type has no fields.

    Use GTID based replication.

    PostgresqlSourceConfig

    PostgreSQL data source configuration

    JSON representation 
     { 
 "includeObjects" 
 : 
 { 
 object (  PostgresqlRdbms 
 
) 
 } 
 , 
 "excludeObjects" 
 : 
 { 
 object (  PostgresqlRdbms 
 
) 
 } 
 , 
 "replicationSlot" 
 : 
 string 
 , 
 "publication" 
 : 
 string 
 , 
 "maxConcurrentBackfillTasks" 
 : 
 integer 
 }
    Fields
    includeObjects

    object ( PostgresqlRdbms )

    PostgreSQL objects to include in the stream.
    excludeObjects

    object ( PostgresqlRdbms )

    PostgreSQL objects to exclude from the stream.
    replicationSlot

    string

    Required. Immutable. The name of the logical replication slot that's configured with the pgoutput plugin.
    publication

    string

    Required. The name of the publication that includes the set of all tables that are defined in the stream's includeObjects.
    maxConcurrentBackfillTasks

    integer

    Maximum number of concurrent backfill tasks. The number should be non negative. If not set (or set to 0), the system's default value will be used.

    PostgresqlRdbms

    PostgreSQL database structure.

    JSON representation 
     { 
 "postgresqlSchemas" 
 : 
 [ 
 { 
 object (  PostgresqlSchema 
 
) 
 } 
 ] 
 }
    Fields
    postgresqlSchemas[]

    object ( PostgresqlSchema )

    PostgreSQL schemas in the database server.

    PostgresqlSchema

    PostgreSQL schema.

    JSON representation 
     { 
 "schema" 
 : 
 string 
 , 
 "postgresqlTables" 
 : 
 [ 
 { 
 object (  PostgresqlTable 
 
) 
 } 
 ] 
 }
    Fields
    schema

    string

    Schema name.
    postgresqlTables[]

    object ( PostgresqlTable )

    Tables in the schema.

    PostgresqlTable

    PostgreSQL table.

    JSON representation 
     { 
 "table" 
 : 
 string 
 , 
 "postgresqlColumns" 
 : 
 [ 
 { 
 object (  PostgresqlColumn 
 
) 
 } 
 ] 
 }
    Fields
    table

    string

    Table name.
    postgresqlColumns[]

    object ( PostgresqlColumn )

    PostgreSQL columns in the schema. When unspecified as part of include/exclude objects, includes/excludes everything.

    PostgresqlColumn

    PostgreSQL Column.

    JSON representation 
     { 
 "column" 
 : 
 string 
 , 
 "dataType" 
 : 
 string 
 , 
 "length" 
 : 
 integer 
 , 
 "precision" 
 : 
 integer 
 , 
 "scale" 
 : 
 integer 
 , 
 "primaryKey" 
 : 
 boolean 
 , 
 "nullable" 
 : 
 boolean 
 , 
 "ordinalPosition" 
 : 
 integer 
 }
    Fields
    column

    string

    Column name.
    dataType

    string

    The PostgreSQL data type.
    length

    integer

    Column length.
    precision

    integer

    Column precision.
    scale

    integer

    Column scale.
    primaryKey

    boolean

    Whether or not the column represents a primary key.
    nullable

    boolean

    Whether or not the column can accept a null value.
    ordinalPosition

    integer

    The ordinal position of the column in the table.

    SqlServerSourceConfig

    SQLServer data source configuration

    JSON representation 
     { 
 "includeObjects" 
 : 
 { 
 object (  SqlServerRdbms 
 
) 
 } 
 , 
 "excludeObjects" 
 : 
 { 
 object (  SqlServerRdbms 
 
) 
 } 
 , 
 "maxConcurrentCdcTasks" 
 : 
 integer 
 , 
 "maxConcurrentBackfillTasks" 
 : 
 integer 
 , 
 // Union field cdc_method 
can be only one of the following: 
 "transactionLogs" 
 : 
 { 
 object (  SqlServerTransactionLogs 
 
) 
 } 
 , 
 "changeTables" 
 : 
 { 
 object (  SqlServerChangeTables 
 
) 
 } 
 // End of list of possible types for union field cdc_method 
. 
 }
    Fields
    includeObjects

    object ( SqlServerRdbms )

    SQLServer objects to include in the stream.

    excludeObjects

    object ( SqlServerRdbms )

    SQLServer objects to exclude from the stream.

    maxConcurrentCdcTasks

    integer

    Max concurrent CDC tasks.

    maxConcurrentBackfillTasks

    integer

    Max concurrent backfill tasks.

    Union field cdc_method . Configuration to select the CDC read method for the stream. cdc_method can be only one of the following:
    transactionLogs

    object ( SqlServerTransactionLogs )

    CDC reader reads from transaction logs.

    changeTables

    object ( SqlServerChangeTables )

    CDC reader reads from change tables.

    SqlServerRdbms

    SQLServer database structure.

    JSON representation 
     { 
 "schemas" 
 : 
 [ 
 { 
 object (  SqlServerSchema 
 
) 
 } 
 ] 
 }
    Fields
    schemas[]

    object ( SqlServerSchema )

    SQLServer schemas in the database server.

    SqlServerSchema

    SQLServer schema.

    JSON representation 
     { 
 "schema" 
 : 
 string 
 , 
 "tables" 
 : 
 [ 
 { 
 object (  SqlServerTable 
 
) 
 } 
 ] 
 }
    Fields
    schema

    string

    Schema name.
    tables[]

    object ( SqlServerTable )

    Tables in the schema.

    SqlServerTable

    SQLServer table.

    JSON representation 
     { 
 "table" 
 : 
 string 
 , 
 "columns" 
 : 
 [ 
 { 
 object (  SqlServerColumn 
 
) 
 } 
 ] 
 }
    Fields
    table

    string

    Table name.
    columns[]

    object ( SqlServerColumn )

    SQLServer columns in the schema. When unspecified as part of include/exclude objects, includes/excludes everything.

    SqlServerColumn

    SQLServer Column.

    JSON representation 
     { 
 "column" 
 : 
 string 
 , 
 "dataType" 
 : 
 string 
 , 
 "length" 
 : 
 integer 
 , 
 "precision" 
 : 
 integer 
 , 
 "scale" 
 : 
 integer 
 , 
 "primaryKey" 
 : 
 boolean 
 , 
 "nullable" 
 : 
 boolean 
 , 
 "ordinalPosition" 
 : 
 integer 
 }
    Fields
    column

    string

    Column name.
    dataType

    string

    The SQLServer data type.
    length

    integer

    Column length.
    precision

    integer

    Column precision.
    scale

    integer

    Column scale.
    primaryKey

    boolean

    Whether or not the column represents a primary key.
    nullable

    boolean

    Whether or not the column can accept a null value.
    ordinalPosition

    integer

    The ordinal position of the column in the table.

    SqlServerTransactionLogs

    This type has no fields.

    Configuration to use Transaction Logs CDC read method.

    SqlServerChangeTables

    This type has no fields.

    Configuration to use Change Tables CDC read method.

    SalesforceSourceConfig

    Salesforce source configuration

    JSON representation 
     { 
 "includeObjects" 
 : 
 { 
 object (  SalesforceOrg 
 
) 
 } 
 , 
 "excludeObjects" 
 : 
 { 
 object (  SalesforceOrg 
 
) 
 } 
 , 
 "pollingInterval" 
 : 
 string 
 }
    Fields
    includeObjects

    object ( SalesforceOrg )

    Salesforce objects to retrieve from the source.
    excludeObjects

    object ( SalesforceOrg )

    Salesforce objects to exclude from the stream.
    pollingInterval

    string ( Duration format)

    Required. Salesforce objects polling interval. The interval at which new changes will be polled for each object. The duration must be between 5 minutes and 24 hours.

    A duration in seconds with up to nine fractional digits, ending with ' s '. Example: "3.5s" .

    SalesforceOrg

    Salesforce organization structure.

    JSON representation 
     { 
 "objects" 
 : 
 [ 
 { 
 object (  SalesforceObject 
 
) 
 } 
 ] 
 }
    Fields
    objects[]

    object ( SalesforceObject )

    Salesforce objects in the database server.

    SalesforceObject

    Salesforce object.

    JSON representation 
     { 
 "objectName" 
 : 
 string 
 , 
 "fields" 
 : 
 [ 
 { 
 object (  SalesforceField 
 
) 
 } 
 ] 
 }
    Fields
    objectName

    string

    Object name.
    fields[]

    object ( SalesforceField )

    Salesforce fields. When unspecified as part of include objects, includes everything, when unspecified as part of exclude objects, excludes nothing.

    SalesforceField

    Salesforce field.

    JSON representation 
     { 
 "name" 
 : 
 string 
 , 
 "dataType" 
 : 
 string 
 , 
 "nillable" 
 : 
 boolean 
 }
    Fields
    name

    string

    Field name.
    dataType

    string

    The data type.
    nillable

    boolean

    Indicates whether the field can accept nil values.

    MongodbSourceConfig

    MongoDB source configuration.

    JSON representation 
     { 
 "includeObjects" 
 : 
 { 
 object (  MongodbCluster 
 
) 
 } 
 , 
 "excludeObjects" 
 : 
 { 
 object (  MongodbCluster 
 
) 
 } 
 , 
 "maxConcurrentBackfillTasks" 
 : 
 integer 
 }
    Fields
    includeObjects

    object ( MongodbCluster )

    MongoDB collections to include in the stream.
    excludeObjects

    object ( MongodbCluster )

    MongoDB collections to exclude from the stream.
    maxConcurrentBackfillTasks

    integer

    Optional. Maximum number of concurrent backfill tasks. The number should be non-negative and less than or equal to 50. If not set (or set to 0), the system's default value is used

    MongodbCluster

    MongoDB Cluster structure.

    JSON representation 
     { 
 "databases" 
 : 
 [ 
 { 
 object (  MongodbDatabase 
 
) 
 } 
 ] 
 }
    Fields
    databases[]

    object ( MongodbDatabase )

    MongoDB databases in the cluster.

    MongodbDatabase

    MongoDB Database.

    JSON representation 
     { 
 "database" 
 : 
 string 
 , 
 "collections" 
 : 
 [ 
 { 
 object (  MongodbCollection 
 
) 
 } 
 ] 
 }
    Fields
    database

    string

    Database name.
    collections[]

    object ( MongodbCollection )

    Collections in the database.

    MongodbCollection

    MongoDB Collection.

    JSON representation 
     { 
 "collection" 
 : 
 string 
 , 
 "fields" 
 : 
 [ 
 { 
 object (  MongodbField 
 
) 
 } 
 ] 
 }
    Fields
    collection

    string

    Collection name.
    fields[]

    object ( MongodbField )

    Fields in the collection.

    MongodbField

    MongoDB Field.

    JSON representation 
     { 
 "field" 
 : 
 string 
 }
    Fields
    field

    string

    Field name.

    DestinationConfig

    The configuration of the stream destination.

    JSON representation 
     { 
 "destinationConnectionProfile" 
 : 
 string 
 , 
 // Union field destination_stream_config 
can be only one of the following: 
 "gcsDestinationConfig" 
 : 
 { 
 object (  GcsDestinationConfig 
 
) 
 } 
 , 
 "bigqueryDestinationConfig" 
 : 
 { 
 object (  BigQueryDestinationConfig 
 
) 
 } 
 // End of list of possible types for union field destination_stream_config 
. 
 }
    Fields
    destinationConnectionProfile

    string

    Required. Destination connection profile resource. Format: projects/{project}/locations/{location}/connectionProfiles/{name}

    Union field destination_stream_config . Stream configuration that is specific to the data destination type. destination_stream_config can be only one of the following:
    gcsDestinationConfig

    object ( GcsDestinationConfig )

    A configuration for how data should be loaded to Cloud Storage.

    bigqueryDestinationConfig

    object ( BigQueryDestinationConfig )

    BigQuery destination configuration.

    GcsDestinationConfig

    Google Cloud Storage destination configuration

    JSON representation 
     { 
 "path" 
 : 
 string 
 , 
 "fileRotationMb" 
 : 
 integer 
 , 
 "fileRotationInterval" 
 : 
 string 
 , 
 // Union field file_format 
can be only one of the following: 
 "avroFileFormat" 
 : 
 { 
 object (  AvroFileFormat 
 
) 
 } 
 , 
 "jsonFileFormat" 
 : 
 { 
 object (  JsonFileFormat 
 
) 
 } 
 // End of list of possible types for union field file_format 
. 
 }
    Fields
    path

    string

    Path inside the Cloud Storage bucket to write data to.

    fileRotationMb

    integer

    The maximum file size to be saved in the bucket.

    fileRotationInterval

    string ( Duration format)

    The maximum duration for which new events are added before a file is closed and a new file is created. Values within the range of 15-60 seconds are allowed.

    A duration in seconds with up to nine fractional digits, ending with ' s '. Example: "3.5s" .

    Union field file_format . File Format that the data should be written in. file_format can be only one of the following:
    avroFileFormat

    object ( AvroFileFormat )

    AVRO file format configuration.

    jsonFileFormat

    object ( JsonFileFormat )

    JSON file format configuration.

    AvroFileFormat

    This type has no fields.

    AVRO file format configuration.

    JsonFileFormat

    JSON file format configuration.

    JSON representation 
     { 
 "schemaFileFormat" 
 : 
 enum (  SchemaFileFormat 
 
) 
 , 
 "compression" 
 : 
 enum (  JsonCompression 
 
) 
 }
    Fields
    schemaFileFormat

    enum ( SchemaFileFormat )

    The schema file format along JSON data files.
    compression

    enum ( JsonCompression )

    Compression of the loaded JSON file.

    SchemaFileFormat

    Schema file format.

    Enums
    SCHEMA_FILE_FORMAT_UNSPECIFIED Unspecified schema file format.
    NO_SCHEMA_FILE Do not attach schema file.
    AVRO_SCHEMA_FILE Avro schema format.

    JsonCompression

    Json file compression.

    Enums
    JSON_COMPRESSION_UNSPECIFIED Unspecified json file compression.
    NO_COMPRESSION Do not compress JSON file.
    GZIP Gzip compression.

    BigQueryDestinationConfig

    BigQuery destination configuration

    JSON representation 
     { 
 "dataFreshness" 
 : 
 string 
 , 
 "blmtConfig" 
 : 
 { 
 object (  BlmtConfig 
 
) 
 } 
 , 
 // Union field dataset_config 
can be only one of the following: 
 "singleTargetDataset" 
 : 
 { 
 object (  SingleTargetDataset 
 
) 
 } 
 , 
 "sourceHierarchyDatasets" 
 : 
 { 
 object (  SourceHierarchyDatasets 
 
) 
 } 
 // End of list of possible types for union field dataset_config 
. 
 // Union field write_mode 
can be only one of the following: 
 "merge" 
 : 
 { 
 object (  Merge 
 
) 
 } 
 , 
 "appendOnly" 
 : 
 { 
 object (  AppendOnly 
 
) 
 } 
 // End of list of possible types for union field write_mode 
. 
 }
    Fields
    dataFreshness

    string ( Duration format)

    The guaranteed data freshness (in seconds) when querying tables created by the stream. Editing this field will only affect new tables created in the future, but existing tables will not be impacted. Lower values mean that queries will return fresher data, but may result in higher cost.

    A duration in seconds with up to nine fractional digits, ending with ' s '. Example: "3.5s" .

    blmtConfig

    object ( BlmtConfig )

    Optional. Big Lake Managed Tables (BLMT) configuration.

    Union field dataset_config . Target dataset(s) configuration. dataset_config can be only one of the following:
    singleTargetDataset

    object ( SingleTargetDataset )

    Single destination dataset.

    sourceHierarchyDatasets

    object ( SourceHierarchyDatasets )

    Source hierarchy datasets.

    Union field write_mode .

    write_mode can be only one of the following:

    merge

    object ( Merge )

    The standard mode

    appendOnly

    object ( AppendOnly )

    Append only mode

    SingleTargetDataset

    A single target dataset to which all data will be streamed.

    JSON representation 
     { 
 "datasetId" 
 : 
 string 
 }
    Fields
    datasetId

    string

    The dataset ID of the target dataset. DatasetIds allowed characters: https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#datasetreference .

    SourceHierarchyDatasets

    Destination datasets are created so that hierarchy of the destination data objects matches the source hierarchy.

    JSON representation 
     { 
 "datasetTemplate" 
 : 
 { 
 object (  DatasetTemplate 
 
) 
 } 
 , 
 "projectId" 
 : 
 string 
 }
    Fields
    datasetTemplate

    object ( DatasetTemplate )

    The dataset template to use for dynamic dataset creation.
    projectId

    string

    Optional. The project id of the BigQuery dataset. If not specified, the project will be inferred from the stream resource.

    DatasetTemplate

    Dataset template used for dynamic dataset creation.

    JSON representation 
     { 
 "location" 
 : 
 string 
 , 
 "datasetIdPrefix" 
 : 
 string 
 , 
 "kmsKeyName" 
 : 
 string 
 }
    Fields
    location

    string

    Required. The geographic location where the dataset should reside. See https://cloud.google.com/bigquery/docs/locations for supported locations.
    datasetIdPrefix

    string

    If supplied, every created dataset will have its name prefixed by the provided value. The prefix and name will be separated by an underscore. i.e.

    _ .
    kmsKeyName

    string

    Describes the Cloud KMS encryption key that will be used to protect destination BigQuery table. The BigQuery Service Account associated with your project requires access to this encryption key. i.e. projects/{project}/locations/{location}/keyRings/{key_ring}/cryptoKeys/{cryptoKey}. See https://cloud.google.com/bigquery/docs/customer-managed-encryption for more information.

    BlmtConfig

    The configuration for BLMT.

    JSON representation 
     { 
 "bucket" 
 : 
 string 
 , 
 "rootPath" 
 : 
 string 
 , 
 "connectionName" 
 : 
 string 
 , 
 "fileFormat" 
 : 
 enum (  FileFormat 
 
) 
 , 
 "tableFormat" 
 : 
 enum (  TableFormat 
 
) 
 }
    Fields
    bucket

    string

    Required. The Cloud Storage bucket name.
    rootPath

    string

    The root path inside the Cloud Storage bucket.
    connectionName

    string

    Required. The bigquery connection. Format: {project}.{location}.{name}

    fileFormat

    enum ( FileFormat )

    Required. The file format.
    tableFormat

    enum ( TableFormat )

    Required. The table format.

    FileFormat

    Supported file formats for BigLake managed tables.

    Enums
    FILE_FORMAT_UNSPECIFIED Default value.
    PARQUET Parquet file format.

    TableFormat

    Supported table formats for BigLake managed tables.

    Enums
    TABLE_FORMAT_UNSPECIFIED Default value.
    ICEBERG Iceberg table format.

    Merge

    This type has no fields.

    Merge mode defines that all changes to a table will be merged at the destination table.

    AppendOnly

    This type has no fields.

    AppendOnly mode defines that all changes to a table will be written to the destination table.

    State

    Stream state.

    Enums
    STATE_UNSPECIFIED Unspecified stream state.
    NOT_STARTED The stream has been created but has not yet started streaming data.
    RUNNING The stream is running.
    PAUSED The stream is paused.
    MAINTENANCE

    The stream is in maintenance mode.

    Updates are rejected on the resource in this state.
    FAILED The stream is experiencing an error that is preventing data from being streamed.
    FAILED_PERMANENTLY The stream has experienced a terminal failure.
    STARTING The stream is starting, but not yet running.
    DRAINING The Stream is no longer reading new events, but still writing events in the buffer.

    BackfillAllStrategy

    Backfill strategy to automatically backfill the Stream's objects. Specific objects can be excluded.

    JSON representation 
     { 
 // Union field excluded_objects 
can be only one of the following: 
 "oracleExcludedObjects" 
 : 
 { 
 object (  OracleRdbms 
 
) 
 } 
 , 
 "mysqlExcludedObjects" 
 : 
 { 
 object (  MysqlRdbms 
 
) 
 } 
 , 
 "postgresqlExcludedObjects" 
 : 
 { 
 object (  PostgresqlRdbms 
 
) 
 } 
 , 
 "sqlServerExcludedObjects" 
 : 
 { 
 object (  SqlServerRdbms 
 
) 
 } 
 , 
 "salesforceExcludedObjects" 
 : 
 { 
 object (  SalesforceOrg 
 
) 
 } 
 , 
 "mongodbExcludedObjects" 
 : 
 { 
 object (  MongodbCluster 
 
) 
 } 
 // End of list of possible types for union field excluded_objects 
. 
 }
    Fields
    Union field excluded_objects . List of objects to exclude. excluded_objects can be only one of the following:
    oracleExcludedObjects

    object ( OracleRdbms )

    Oracle data source objects to avoid backfilling.

    mysqlExcludedObjects

    object ( MysqlRdbms )

    MySQL data source objects to avoid backfilling.

    postgresqlExcludedObjects

    object ( PostgresqlRdbms )

    PostgreSQL data source objects to avoid backfilling.

    sqlServerExcludedObjects

    object ( SqlServerRdbms )

    SQLServer data source objects to avoid backfilling

    salesforceExcludedObjects

    object ( SalesforceOrg )

    Salesforce data source objects to avoid backfilling

    mongodbExcludedObjects

    object ( MongodbCluster )

    MongoDB data source objects to avoid backfilling

    BackfillNoneStrategy

    This type has no fields.

    Backfill strategy to disable automatic backfill for the Stream's objects.

    Methods

    create

    		Use this method to create a stream.

    delete

    		Use this method to delete a stream.

    get

    		Use this method to get details about a stream.

    list

    		Use this method to list streams in a project and location.

    patch

    		Use this method to update the configuration of a stream.

    run

    		Use this method to start, resume or recover a stream with a non default CDC strategy.
    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: