Generate metadata for translation and assessment

Turns on assessment mode when generating database logs or extracting metadata. The dwh-migration-dumper tool generates required metadata statistics for BigQuery migration assessment when used for metadata extraction. When used for query logs it extracts additional columns for BigQuery migration assessment.

A list of the databases to extract, separated by commas. The database names might be case-sensitive, depending on the Teradata server configuration.

If this flag is used in combination with the teradata connector, then the dwh-migration-dumper tool filters the metadata tables and views by the provided list of databases. The exceptions are the DatabasesV and RoleMembersV views - the dwh-migration-dumper tool extracts the databases and users from these views without filtering by the database name.

This flag cannot be used in combination with the teradata-logs connector. Query logs are always extracted for all the databases.

The username to use for the database connection.

For the teradata-logs connector only.

To extract the query logs from an alternative location, we recommend that you use the -Dteradata-logs.query-logs-table and -Dteradata-logs.sql-logs-table flags instead.

By default, the query logs are extracted from the tables dbc.DBQLogTbl and dbc.DBQLSQLTbl . If you use the --assessment flag, then the query logs are extracted from the view dbc.QryLogV and from the table dbc.DBQLSQLTbl . If you need to extract the query logs from an alternative location, you can specify the fully-qualified names of the tables or views by using the --query-log-alternates flag. The first parameter references the alternative to the dbc.DBQLogTbl table, and the second parameter references the alternative to the dbc.DBQLSQLTbl table. Both parameters are required.
The -Dteradata-logs.log-date-column flag can be used to improve extraction performance when both tables have an indexed column of type DATE .

Example: --query-log-alternates historicdb.ArchivedQryLogV,historicdb.ArchivedDBQLSqlTbl

The transaction mode for the connection. The following values are supported:

• ANSI : ANSI mode. This is the default mode (if the flag is not specified)
• TERA : Teradata transaction mode (BTET)
• DEFAULT : use the default transaction mode configured on the database server
• NONE : no mode is set for the connection

Example (Bash): -Dteradata.tmode=TERA

Example (Windows PowerShell): "-Dteradata.tmode=TERA"

For the teradata-logs connector only.

To improve performance of joining tables that are specified by the -Dteradata-logs.query-logs-table and -Dteradata-logs.sql-logs-table flags, you can include an additional column of type DATE in the JOIN condition. This column must be defined in both tables and it must be part of the Partitioned Primary Index.

Example (Bash): -Dteradata-logs.log-date-column=ArchiveLogDate

Example (Windows PowerShell): "-Dteradata-logs.log-date-column=ArchiveLogDate"

For the teradata-logs connector only.

By default, the query logs are extracted from the dbc.DBQLogTbl table. If you use the --assessment flag, then the query logs are extracted from the view dbc.QryLogV . If you need to extract the query logs from an alternative location, you can specify the fully-qualified name of the table or view by using this flag.
See -Dteradata-logs.log-date-column flag to improve extraction performance.

Example (Bash): -Dteradata-logs.query-logs-table=historicdb.ArchivedQryLogV

Example (Windows PowerShell): "-Dteradata-logs.query-logs-table=historicdb.ArchivedQryLogV"

For the teradata-logs connector only.

By default, the query logs containing SQL text are extracted from the dbc.DBQLSqlTbl table. If you need to extract them from an alternative location, you can specify the fully-qualified name of the table or view by using this flag.See -Dteradata-logs.log-date-column flag to improve extraction performance.

Example (Bash): -Dteradata-logs.sql-logs-table=historicdb.ArchivedDBQLSqlTbl

Example (Windows PowerShell): "-Dteradata-logs.sql-logs-table=historicdb.ArchivedDBQLSqlTbl"

For the teradata-logs connector only.

By default, the utility logs are extracted from the table dbc.DBQLUtilityTbl . If you need to extract the utility logs from an alternative location, you can specify the fully-qualified name of the table by using the -Dteradata-logs.utility-logs-table flag.

Example (Bash): -Dteradata-logs.utility-logs-table=historicdb.ArchivedUtilityLogs

Example (Windows PowerShell): "-Dteradata-logs.utility-logs-table=historicdb.ArchivedUtilityLogs"

For the teradata-logs connector only.

By default, the SCPU resource usage logs are extracted from the table dbc.ResUsageScpu . If you need to extract these from an alternative location, you can specify the fully-qualified name of the table by using the -Dteradata-logs.res-usage-scpu-table flag.

Example (Bash): -Dteradata-logs.res-usage-scpu-table=historicdb.ArchivedResUsageScpu

Example (Windows PowerShell): "-Dteradata-logs.res-usage-scpu-table=historicdb.ArchivedResUsageScpu"

For the teradata-logs connector only.

By default, the SPMA resource usage logs are extracted from the table dbc.ResUsageSpma . If you need to extract these logs from an alternative location, you can specify the fully-qualified name of the table by using the -Dteradata-logs.res-usage-spma-table flag.

Example (Bash): -Dteradata-logs.res-usage-spma-table=historicdb.ArchivedResUsageSpma

Example (Windows PowerShell): "-Dteradata-logs.res-usage-spma-table=historicdb.ArchivedResUsageSpma"

The start time (inclusive) for query logs to extract. The value is truncated to the hour. This flag is only available for the teradata-logs connector.

Example: --query-log-start "2023-01-01 14:00:00"

The end time (exclusive) for query logs to extract. The value is truncated to the hour. This flag is only available for the teradata-logs connector.

Example: --query-log-end "2023-01-15 22:00:00"

For the teradata connector only.

Limit the number of rows extracted from the view TableSizeV . The rows are grouped by the columns DatabaseName , AccountName , and TableName , and then sorted in descending order by the size of the permanent space (the expression SUM(CurrentPerm) ). Then, the specified number of rows are extracted.

Example (Bash): -Dteradata.metadata.tablesizev.max-rows=100000

Example (Windows PowerShell): "-Dteradata.metadata.tablesizev.max-rows=100000"

For the teradata connector only.

Limit the number of rows extracted from the view DiskSpaceV . The rows are sorted in descending order by the size of the permanent space (column CurrentPerm ), and then the specified number of rows are extracted.

Example (Bash): -Dteradata.metadata.diskspacev.max-rows=100000

Example (Windows PowerShell): "-Dteradata.metadata.diskspacev.max-rows=100000"

For the teradata connector only.

Limit the number of rows that represent users ( DBKind='U' ) that are extracted from the view DatabasesV . The rows are sorted in descending order by the column PermSpace , and then the specified number of rows are extracted.

Example (Bash): -Dteradata.metadata.databasesv.users.max-rows=100000

Example (Windows PowerShell): "-Dteradata.metadata.databasesv.users.max-rows=100000"

For the teradata connector only.

Limit the number of rows that represent databases ( DBKind='D' ) that are extracted from the view DatabasesV . The rows are sorted in descending order by the column PermSpace , and then the specified number of rows are extracted.

Example (Bash): -Dteradata.metadata.databasesv.dbs.max-rows=100000

Example (Windows PowerShell): "-Dteradata.metadata.databasesv.dbs.max-rows=100000"

For the teradata connector only.

Maximum length of the text column when extracting the data from the TableTextV view. Text longer than the defined limit will be split into multiple rows. Allowed range: between 5000 and 32000 (inclusive).

Example (Bash): -Dteradata.metadata.max-text-length=10000

Example (Windows PowerShell): "-Dteradata.metadata.max-text-length=10000"

For the teradata-logs connector only.

Maximum length of the DBQLSqlTbl.SqlTextInfo column. Query text longer than the defined limit will be split into multiple rows. Allowed range: between 5000 and 31000 (inclusive).

Example (Bash): -Dteradata-logs.max-sql-length=10000

Example (Windows PowerShell): "-Dteradata-logs.max-sql-length=10000"

Turning on assessment mode when generating database logs or extracting metadata. It generates required metadata statistics for BigQuery migration assessment when used for metadata extraction. When used for query logs extraction it generates query metrics statistics for BigQuery migration assessment.

The name of the database to connect to.

The AWS IAM access key ID to use for authentication. The access key is a string of characters, something like AKIAIOSFODNN7EXAMPLE .

Use in conjunction with the --iam-secretaccesskey flag. Do not use this flag when specifying the --iam-profile or --password flags.

Not explicitly, but you must provide authentication information through one of the following methods:

• Using this flag in conjunction with the --iam-secretaccesskey flag.
• Using the --iam-profile flag.
• Using the --password flag in conjunction with the --user flag.

The AWS IAM profile to use for authentication. You can retrieve a profile value to use by examining the $HOME/.aws/credentials file or by running aws configure list-profiles .

Do not use this flag with the --iam-accesskeyid , --iam-secretaccesskey or --password flags.

Not explicitly, but you must provide authentication information through one of the following methods:

• Using this flag.
• Using the --iam-accesskeyid flag in conjunction with the --iam-secretaccesskey flag.
• Using the --password flag in conjunction with the --user flag.

The AWS IAM secret access key to use for authentication. The secret access key is a string of characters, something like wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY .

Use in conjunction with the --iam-accesskeyid flag. Do not use this flag with the --iam-profile or --password flags.

Not explicitly, but you must provide authentication information through one of the following methods:

• Using this flag in conjunction with the --iam-accesskeyid flag.
• Using the --iam-profile flag.
• Using the --password flag in conjunction with the --user flag.

Do not use this flag with the --iam-accesskeyid , --iam-secretaccesskey or --iam-profile flags.

Not explicitly, but you must provide authentication information through one of the following methods:

• Using this flag in conjunction with the --user flag.
• Using the --iam-accesskeyid flag in conjunction with the --iam-secretaccesskey flag.
• Using the --password flag.

The start time (inclusive) for query logs to extract. The value is truncated to the hour. This flag is only available for the redshift-raw-logs connector.

Example: --query-log-start "2023-01-01 14:00:00"

The end time (exclusive) for query logs to extract. The value is truncated to the hour. This flag is only available for the redshift-raw-logs connector.

Example: --query-log-end "2023-01-15 22:00:00"

Turns on assessment mode when extracting metadata. The dwh-migration-dumper tool generates required metadata statistics for BigQuery migration assessment when used for metadata extraction.

Causes the dwh-migration-dumper tool to extract partition metadata. You might want to set this flag to false for production metastore with a significant number of partitions, due to Thrift client performance implications. This improves the extraction tool performance, but causes some loss of partition optimization on the BigQuery side.

Don't use this flag with the --assessment flag, as it will have no effect.

When you run the dwh-migration-dumper tool, it selects the appropriate Thrift specification to use for communicating with your Apache Hive server, based on the value of this flag. If the extraction tool doesn't have an appropriate Thrift specification, it uses the 2.3.6 client and emits a warning to stdout . If this occurs, please contact Support and provide the Apache Hive version number you requested.

The RPC protection configuration level. This determines the Quality of Protection (QOP) of the Simple Authentication and Security Layer (SASL) connection between cluster and the dwh-migration-dumper tool.

Must be equal to the value of the hadoop.rpc.protection parameter inside the /etc/hadoop/conf/core-site.xml file on the cluster, with one of the following values:

• authentication
• integrity
• privacy

Example (Bash): -Dhiveql.rpc.protection=privacy

Example (Windows PowerShell): "-Dhiveql.rpc.protection=privacy"