Collect Fastly CDN logs

This document explains how to ingest Fastly CDN logs to Google Security Operations using Amazon S3.

Fastly is a content delivery network (CDN) and edge cloud platform that provides real-time content delivery, security, and edge computing services. Fastly's Real-Time Log Streaming feature can send CDN access logs, WAF events, and other telemetry data to various destinations including Amazon S3 for security monitoring and analysis.

Before you begin

Make sure that you have the following prerequisites:

• A Google SecOps instance
• Privileged access to Fastlycontrol panel or API
• Privileged access to AWS(S3, IAM)
• A Fastly API token with global or global:read scope

Create Fastly API token

1. Sign in to the Fastly control panelat https://manage.fastly.com.
2. Click your user iconin the upper-right corner.
3. Go to Account > Personal API tokens.
4. Click Create Token.
5. In the Create API Tokendialog, provide the following:
   • Name: Enter a descriptive name (for example, Google SecOps S3 Logging Configuration ).
   • Type: Select User token.
   • Scope: Select Global access(required to configure logging endpoints).
   • Expiration: Select Never expireor set an appropriate expiration date.
6. Click Create Token.
7. Copy and save the API token securely.

Configure AWS S3 bucket and IAM for Google SecOps

1. Create Amazon S3 bucketfollowing this user guide: Creating a bucket .
2. Save bucket Nameand Regionfor future reference (for example, fastly-cdn-logs ).
3. Create a Userfollowing this user guide: Creating an IAM user .
4. Select the created User.
5. Select Security credentialstab.
6. Click Create Access Keyin section Access Keys.
7. Select Third-party serviceas Use case.
8. Click Next.
9. Optional: Add description tag.
10. Click Create access key.
11. Click Download .csv fileto save the Access Keyand Secret Access Keyfor future reference.
12. Click Done.
13. Select the Permissionstab.
14. Click Add permissionsin section Permissions policies.
15. Select Add permissions.
16. Select Attach policies directly.
17. Search for AmazonS3FullAccesspolicy.
18. Select the policy.
19. Click Next.
20. Click Add permissions.

Configure Fastly S3 logging endpoint

Option A: Configure via Fastly control panel

1. Sign in to the Fastly control panelat https://manage.fastly.com
2. Select the serviceyou want to configure logging for.
3. Click Edit configurationand select Clone active versionto create a new draft version.
4. Click Loggingin the left navigation menu.
5. In the Amazon Web Services S3section, click Create endpoint.

6. In the Create an Amazon S3 endpointform, provide the following configuration details:

   • Basic settings:

     • Name: Enter a descriptive name (for example, secops-s3-logs ).
     • Placement: Select Format Version Default(recommended).

     • Log format: Enter the following JSON format string:

        {"timestamp":"%{begin:%Y-%m-%dT%H:%M:%S%z}t","client_ip":"%{req.http.Fastly-Client-IP}V","geo_country":"%{client.geo.country_name}V","geo_city":"%{client.geo.city}V","geo_region":"%{client.geo.region}V","datacenter":"%{server.datacenter}V","host":"%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V","url":"%{json.escape(req.url)}V","request_method":"%{json.escape(req.method)}V","request_protocol":"%{json.escape(req.proto)}V","request_referer":"%{json.escape(req.http.referer)}V","request_user_agent":"%{json.escape(req.http.User-Agent)}V","response_state":"%{json.escape(fastly_info.state)}V","response_status":%{resp.status}V,"response_reason":%{if(resp.response, "\"%22\"+json.escape(resp.response)+\"%22\", \"null\")}V,"response_body_size":%{resp.body_bytes_written}V,"response_header_size":%{resp.header_bytes_written}V,"request_body_size":%{req.body_bytes_read}V,"request_header_size":%{req.header_bytes_read}V,"cache_status":"%{fastly_info.state}V","is_tls":"%{if(req.is_ssl, \"true\", \"false\")}V","tls_protocol":"%{cstr_escape(tls.client.protocol)}V","tls_cipher":"%{cstr_escape(tls.client.cipher)}V","server_identity":"%{json.escape(server.identity)}V","is_edge":%{if(fastly.ff.visits_this_service == 0, "true", "false")}V,"time_elapsed_usec":%{time.elapsed.usec}V,"time_start_sec":%{time.start.sec}V} 
       

     • Timestamp format: Leave as default (strftime compatible string).

   S3 bucket configuration:

   • Bucket name: Enter the S3 bucket name (for example, fastly-cdn-logs ).
   • Domain: Leave empty if your bucket is in US Standard region. For other regions, enter the appropriate S3 endpoint (for example, s3-us-west-2.amazonaws.com for us-west-2 region).
   • Access method: Select User Credentials
   • Access key: Enter the AWS access key from step 11.
   • Secret key: Enter the AWS secret access key from step 11.
   • Period: Enter 3600 (logs are finalized every hour).

7. Click Advanced optionsto expand additional settings:

   Advanced settings:

   • Path: Enter fastly-cdn/ (optional prefix for organizing logs within the bucket).
   • PGP public key: Leave empty (encryption not required for this integration).
   • Select a log line format: Select Blank(we're using custom JSON format).
   • Compression: Select gzip(recommended to reduce storage costs).
   • Redundancy level: Select Standard.
   • ACL: Select private(recommended).
   • Server side encryption: Select None(or configure as per your security requirements).
   • Maximum bytes: Enter 0 (no maximum file size limit).

8. Click Createto create the logging endpoint.

9. Click Activatein the top-right corner.

10. Select Activate on Productionto deploy your configuration changes.

11. In the confirmation dialog, click Activateto confirm.

Option B: Configure via Fastly API

If you prefer to configure logging via API, use the following approach:

1. Get your Fastly service ID from the control panel or by listing services:

    curl  
   -H  
    "Fastly-Key: YOUR_FASTLY_TOKEN" 
     
   https://api.fastly.com/service 
   

2. Clone the active version to create a new draft version:

    curl  
   -X  
   PUT  
   -H  
    "Fastly-Key: YOUR_FASTLY_TOKEN" 
     
    \ 
     
    "https://api.fastly.com/service/SERVICE_ID/version/ACTIVE_VERSION/clone" 
    
   

3. Create the S3 logging endpoint on the new draft version:

    curl  
   -X  
   POST  
   -H  
    "Fastly-Key: YOUR_FASTLY_TOKEN" 
     
    \ 
     
   -H  
    "Content-Type: application/x-www-form-urlencoded" 
     
    \ 
     
    "https://api.fastly.com/service/SERVICE_ID/version/NEW_VERSION/logging/s3" 
     
    \ 
     
   --data-urlencode  
    'name=secops-s3-logs' 
     
    \ 
     
   --data-urlencode  
    'bucket_name=fastly-cdn-logs' 
     
    \ 
     
   --data-urlencode  
    'access_key=YOUR_AWS_ACCESS_KEY' 
     
    \ 
     
   --data-urlencode  
    'secret_key=YOUR_AWS_SECRET_KEY' 
     
    \ 
     
   --data-urlencode  
    'path=fastly-cdn/' 
     
    \ 
     
   --data-urlencode  
    'period=3600' 
     
    \ 
     
   --data-urlencode  
    'gzip_level=9' 
     
    \ 
     
   --data-urlencode  
    'format_version=2' 
     
    \ 
     
   --data-urlencode  
    'format={"timestamp":"%{begin:%Y-%m-%dT%H:%M:%S%z}t","client_ip":"%{req.http.Fastly-Client-IP}V","geo_country":"%{client.geo.country_name}V","geo_city":"%{client.geo.city}V","host":"%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V","url":"%{json.escape(req.url)}V","request_method":"%{json.escape(req.method)}V","response_status":%{resp.status}V,"cache_status":"%{fastly_info.state}V"}' 
    
   

4. Activate the new version:

    curl  
   -X  
   PUT  
   -H  
    "Fastly-Key: YOUR_FASTLY_TOKEN" 
     
    \ 
     
    "https://api.fastly.com/service/SERVICE_ID/version/NEW_VERSION/activate" 
    
   

Verify log delivery to S3

1. Wait approximately 1 hour (based on the configured period) for the first log file to be finalized.
2. In the AWS Console, go to S3 > Buckets.
3. Click your bucket name (for example, fastly-cdn-logs ).
4. Navigate to the fastly-cdn/ prefix folder.

5. Verify that log files are being created with naming pattern similar to:

    fastly-cdn/YYYY-MM-DD-HH-MM-SS-RANDOM_STRING.log.gz 
   

6. Download a sample log file and verify the JSON format matches your configuration.

Configure a feed in Google SecOps to ingest Fastly CDN logs

1. Go to SIEM Settings > Feeds.
2. Click Add New Feed.
3. On the next page, click Configure a single feed.
4. Enter a unique name for the Feed name.
5. Select Amazon S3 V2as the Source type.
6. Select Fastly CDNas the Log type.
7. Click Nextand then click Submit.

8. Specify values for the following fields:

   • S3 URI: s3://fastly-cdn-logs/fastly-cdn/ .
   • Source deletion option: Select the deletion option according to your preference.
   • Maximum File Age: Include files modified in the last number of days (default is 180 days).
   • Access Key ID: User access key with access to the S3 bucket.
   • Secret Access Key: User secret key with access to the S3 bucket.
   • Asset namespace: The asset namespace .
   • Ingestion labels: The label to be applied to the events from this feed.

9. Click Nextand then click Submit.

UDM mapping table

Need more help? Get answers from Community members and Google SecOps professionals.