Troubleshoot file system transfers

This document describes how to troubleshoot and resolve transfer and agent issues, and where to find agent logs to help you troubleshoot issues.

Errors

The following table describes transfer error messages, and how to resolve them:

Viewing agent logs

Agent logs contain information relevant to agent processes, and can help you troubleshoot agent connection problems. If your agents are listed as connected in Google Cloud console and you are experiencing transfer failures, see Viewing errors to view a sample of transfer errors. To view logs that contain a record of every file Storage Transfer Service considered during a transfer, see Viewing transfer logs .

By default, agent logs are stored in /tmp . You can change the location with the --log-dir= logs-directory command-line option .

The logs are named:

agent. hostname . username .log. log-level . timestamp

Where:

• hostname - hostname the agent is running on.
• username - username running the agent.
• log-level is one of:
  • INFO - informational messages
  • ERROR - errors encountered during transfer, but that don't prevent the transfer job from continuing.
  • FATAL - errors encountered that prevent the transfer job from continuing.
• timestamp - timestamp in YYYYMMDD-hhmmss.thread-id format.

The logs directory contains symlinks to the most recent logs for each of the priority levels:

• agent.ERROR
• agent.FATAL
• agent.INFO

Slow transfer speed

If your data is taking a long time to transfer, check the following:

1. The read throughput of your file system should be approximately 1.5 times your desired upload speed. You can use FIO to test the read throughput of your file system.

   Install fio:

    sudo apt install -y fio 
   

   Create a new directory fiotest :

    TEST_DIR=/mnt/mnt_dir/fiotest 
    sudo mkdir -p $TEST_DIR 
   

   Test read throughput:

    sudo fio --directory=$TEST_DIR --direct=1
       --rw=randread --randrepeat=0 --ioengine=libaio --bs=1M --iodepth=8
       --time_based=1 --runtime=180 --name=read_test --size=1G 
   

   After you run the commands above, Fio generates a report. The line labeled "bw" represents the total aggregate bandwidth of all threads, and it can be used as a proxy for read throughput.

2. Use iPerf3 to check your available internet bandwidth to Storage Transfer Service.

3. Make sure each of your transfer agents have at least 4 vCPU and 8GB of RAM.

If you've checked the conditions above and are still experiencing long transfer times, you can add additional agents to increase the number of concurrent connections to your data's file system.

For more information on how to maximize the performance of your transfer agents, see Agent best practices .

Troubleshooting agent errors

The following sections describe how to troubleshoot and resolve transfer agent errors:

Agents are not connected

If transfer agents are not displayed as connected within Google Cloud console:

1. Verify that agents can connect to Cloud Storage APIs:

   1. Run the following command from the same machine as the transfer agent to test the agent's connection to Cloud Storage APIs:

      gcloud storage cp test.txt gs:// my-bucket

      Replace:

      my-bucket with the name of your Cloud Storage bucket.

2. If your project uses VPC Service Controls, view the agent logs for errors. If VPC Service Controls is misconfigured, the INFO agent logs will contain the following error:

   Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: id

   In this output:

   • id is the VPC Service Controls error's unique id

Agents are connected but jobs fail

If agents are displayed as connected but transfer jobs fail, check the error details of the failed jobs .

Proxy rejects IP addresses

If you run behind a proxy like Squid and you use an allowlist, you may see requests being denied because of IP addresses being used instead of hostnames.

To resolve this issue, use the docker run command to run the agents and add the following flag:

 --transfer-service-endpoint=storagetransfer.googleapis.com:443 

If you use an alternative endpoint to reach googleapis.com (e.g. for Private Service Connect), replace googleapis.com with the alternative endpoint.

Agent is unable to connect to an HTTPS endpoint using a self-signed certificate

If your agent is unable to connect to an HTTPS endpoint using a self-signed certificate, use an extra -v option to mount your custom certificate bundle into the container's default location ( /etc/ssl/certs ).

To do this, modify the docker run command by adding the following flag:

 -v PATH_TO_LOCAL_CERT 
:/etc/ssl/certs/ca-certificates.crt 

Replace:

• PATH_TO_LOCAL_CERT with the path to your custom certificate file.