File system transfer logs

This document describes the current file system transfer log format.

View transfer logs

File system transfers produce detailed transfer logs that you can use to verify the results of your transfer job. Each job produces a collection of transfer logs that are stored in the transer's Cloud Storage bucket: the source bucket, destination bucket, or intermediate bucket, depending on the transfer direction.

Logs are produced while the transfer job is running. The complete logs are typically available within 15 minutes of job completion.

View logs in the Cloud Storage bucket

Transfer logs are stored in the transfer's Cloud Storage bucket at the following path:

bucket-name /storage-transfer/logs/transferJobs/ job-name /transferOperations/ operation-name

where:

• bucket-name is the name of the Cloud Storage bucket involved in this transfer.
• job-name is the job name, as displayed in the job list .
• operation-name is the name of the individual transfer operation, comprised as the IS08601 timestamp and generated ID.

Logs are aggregated and stored as objects. Each batch of logs is named by its creation time. For example:

my bucket/storage-transfer/logs/transferOperations/job1/2019-10-19T10_52_56.519081644-07_00.log

Run BigQuery queries on transfer logs

To run BigQuery queries on your transfer logs:

1. Load the CSV log data into BigQuery .

2. Run your BigQuery query .

Example queries

Display the number of files that attempted transfer and whether they failed or succeeded

select ActionStatus, count(*) as num_files
from big-query-table 
where Action="TRANSFER"
group by 1;

Where big-query-table is the name of the BigQuery table that contains the transfer log.

Display all files that failed to transfer

select Src_File_Path
from big-query-table 
where Action="TRANSFER" and ActionStatus="FAILED";

Where big-query-table is the name of the BigQuery table that contains the transfer log.

Display checksum and timestamp for each file that successfully transferred

select Timestamp, Action, ActionStatus, Src_File_Path, Src_File_Size,
Src_File_Crc32C, Dst_Gcs_BucketName, Dst_Gcs_ObjectName, Dst_Gcs_Size,
Dst_Gcs_Crc32C, Dst_Gcs_Md5
from big-query-table 
where Action="TRANSFER" and ActionStatus="SUCCEEDED";

Where big-query-table is the name of the BigQuery table that contains the transfer log.

Display all error information for directories that failed to transfer

select FailureDetails_ErrorType, FailureDetails_GrpcCode, FailureDetails_Message
from big-query-table 
where Action="FIND" and ActionStatus="FAILED";

Where big-query-table is the name of the BigQuery table that contains the transfer log.

Format description

File system transfer logs are saved in tab-separated values (TSV) format, and contain a header row with field names. The order that the fields are displayed are generally stable, but aren't guaranteed, and must be inferred from the header row.

Each line represents a single log record, that is a status update on part of the overall transfer operation.

The following table describes the fields in the log file:

Example log entry

The following is an example of two lines of log output, the header and a data line:

Timestamp       OperationName   Action  ActionStatus    FailureDetails.ErrorType        FailureDetails.GrpcCode FailureDetails.Message     Src.Type        Src.File.Path   Src.File.LastModified   Src.File.Size   Src.File.Crc32C Dst.Type   Dst.Gcs.BucketName      Dst.Gcs.ObjectName      Dst.Gcs.LastModified    Dst.Gcs.Size    Dst.Gcs.Crc32C  Dst.Gcs.Md5
2019-10-18T21:06:17Z    transferOperations/agent-manual_transferJobs/OPI1494275376193533620_0000000001571432448     TRANSFER        SUCCEEDED                               ON_PREM_FILE    /mnt/disks/disk-1/directory1/file.txt      1561409931      406     3089075447      GCS_OBJECT      destination-gcs-bucket    file.txt  1571432776      406     3089075447      FDjya7dWbd0OrgnZ7g9ZJw==