Transfer from S3 via CloudFront

Transfers from Amazon S3 can use an Amazon CloudFront distribution as an egress path.

Data transfers through CloudFront may benefit from lower AWS egress costs compared to transferring directly from S3. See CloudFront pricing and S3 egress charges for details.

Using CloudFront as the egress path does not expose your S3 objects to the public. See Does using CloudFront expose my objects to the public?

Overview

In order to transfer S3 data through CloudFront, you must follow these steps:

• Configure IAM permissions
• Configure access to your S3 bucket
• Create a CloudFront distribution for your S3 bucket
• Create a transfer job

Configure IAM permissions

Follow the instructions in Agentless transfer permissions to grant the required Google Cloud permissions.

Configure access to your S3 bucket

Follow the instructions in Configure access to a source: Amazon S3 to configure access to your data in Amazon S3.

Create a CloudFront distribution for your S3 bucket

1. In your AWS account, go to CloudFront.
2. Click Create a CloudFront distribution.
3. Under Origin domainselect your S3 bucket.
4. Origin pathmust be left empty.
5. Accept the auto-filled Origin nameor specify your own value.
6. In the Origin accesssection, select Public . This does not make your bucket public; instead it indicates to CloudFront that no access mechanism should be configured.
7. In the Cache key and origin requestssection:
   1. For Cache policyselect CachingDisabled . This prevents CloudFront from caching requests and serving them to unauthenticated viewers.
   2. For Origin request policyselect AllViewerExceptHostHeader . This allows CloudFront to forward authentication headers to S3, so that Storage Transfer Service can access your bucket with your secure credentials.
8. In the Web Application Firewall (WAF)section, select Do not enable .
9. Optionally, choose a Price class. Storage Transfer Service selects worker pools based on the source bucket's region, so CloudFront pricing is incurred in that region. To obtain the lowest pricing, either ensure that your source bucket is in the US or Europe, or select Use only North America and Europe as the Price classin CloudFront.

10. Click Create CloudFront distribution.

    Once successfully created, the CloudFront distribution details page is displayed.

11. Note the Distribution domain name. For example: https://dy1h2n3l4ob56.cloudfront.net . If the details page doesn't include the https:// protocol in front of the distribution domain name, you'll need to add it yourself when creating the transfer job.

Create a transfer job

Transferring via a CloudFront distribution is supported in the Google Cloud console and the REST API.

  "transferSpec" 
 : 
  
 { 
  
 "awsS3DataSource" 
 : 
  
 { 
  
 "bucketName" 
 : 
  
 "AWS_SOURCE_NAME" 
 , 
  
 "cloudfrontDomain" 
 : 
  
 "https://dy1h2n3l4ob56.cloudfront.net" 
 , 
  
 "awsAccessKey" 
 : 
  
 { 
  
 "accessKeyId" 
 : 
  
 "AWS_ACCESS_KEY_ID" 
 , 
  
 "secretAccessKey" 
 : 
  
 "AWS_SECRET_ACCESS_KEY" 
  
 } 
  
 }, 
  
 ... 
 } 
 

Frequently asked questions

Does using CloudFront expose my objects to the public?

No. If you've followed the configuration steps on this page, your objects are not exposed to the public.

• Cloudfront does not have direct access to your S3 objects.
• Users receive a permission denied error if they try to access your objects either directly or through CloudFront (if your bucket is private).
• Storage Transfer Service signs requests to CloudFront using the credentials you provided in the transfer job, which allows us to download your objects securely as if we were downloading directly from S3. This works due to the header forwarding setting AllViewerExceptHostHeader .