Console
    Contact Us Start free

    Collect Akamai Cloud Monitor logs

    Supported in:

    This document explains how to ingest Akamai Cloud Monitor (Load Balancer, Traffic Shaper, ADC) logs to Google Security Operations using AWS S3. Akamai pushes JSON events to your HTTPS endpoint; an API Gateway + Lambdareceiver writes the events to S3 (JSONL, gz). The parser transforms the JSON logs into UDM. It extracts fields from the JSON payload, performs data type conversions, renames fields to match the UDM schema, and handles specific logic for custom fields and URL construction. It also incorporates error handling and conditional logic based on field presence.

    Before you begin

    Make sure you have the following prerequisites:

    • Google SecOps instance
    • Privileged access to Akamai Control Center and Property Manager
    • Privileged access to AWS*(S3, IAM, Lambda, API Gateway)

    Configure AWS S3 bucket and IAM for Google SecOps

    1. Create Amazon S3 bucketfollowing this user guide: Creating a bucket
    2. Save bucket Nameand Regionfor future reference (for example, akamai-cloud-monitor ).
    3. Create a user following this user guide: Creating an IAM user .
    4. Select the created User.
    5. Select the Security credentialstab.
    6. Click Create Access Keyin the Access Keyssection.
    7. Select Third-party serviceas the Use case.
    8. Click Next.
    9. Optional: add a description tag.
    10. Click Create access key.
    11. Click Download CSV fileto save the Access Keyand Secret Access Keyfor later use.
    12. Click Done.
    13. Select the Permissionstab.
    14. Click Add permissionsin the Permissions policiessection.
    15. Select Add permissions.
    16. Select Attach policies directly
    17. Search for and select the AmazonS3FullAccesspolicy.
    18. Click Next.
    19. Click Add permissions.

    Configure the IAM policy and role for S3 uploads (Lambda)

    1. In the AWS Console, go to IAM > Policies > Create policy > JSONand paste the policy below.

    2. JSON Policy (replace akamai-cloud-monitor with your S3 bucket name):

        { 
 "Version" 
 : 
  
 "2012-10-17" 
 , 
 "Statement" 
 : 
  
 [ 
  
 { 
  
 "Sid" 
 : 
  
 "AllowPutAkamaiObjects" 
 , 
  
 "Effect" 
 : 
  
 "Allow" 
 , 
  
 "Action" 
 : 
  
 [ 
 "s3:PutObject" 
 ], 
  
 "Resource" 
 : 
  
 "arn:aws:s3:::akamai-cloud-monitor/*" 
  
 } 
 ] 
 }

    3. Click Next > Create policy.

    4. Go to IAM > Roles > Create role > AWS service > Lambda.

    5. Attach the JSON policy.

    6. Name the role WriteAkamaiCMToS3Role and click Create role.

    Create the Lambda function

    Setting Value
    Name akamai_cloud_monitor_to_s3
    Runtime Python 3.13
    Architecture x86_64
    Execution role WriteAkamaiCMToS3Role

    1. After the function is created, open the Codetab, delete the stub and enter the following code ( akamai_cloud_monitor_to_s3.py ):

        #!/usr/bin/env python3 
 # Lambda: Receive Akamai Cloud Monitor POST, write JSONL (gz) to S3 
 import 
  
 os 
 , 
  
 json 
 , 
  
 gzip 
 , 
  
 io 
 , 
  
 uuid 
 , 
  
 base64 
 , 
  
 datetime 
  
 as 
  
 dt 
 import 
  
 boto3 
 S3_BUCKET 
 = 
 os 
 . 
 environ 
 [ 
 "S3_BUCKET_NAME" 
 ] 
 S3_PREFIX 
 = 
 os 
 . 
 environ 
 . 
 get 
 ( 
 "S3_PREFIX" 
 , 
 "akamai/cloud-monitor/json/" 
 ) 
 . 
 strip 
 ( 
 "/" 
 ) 
 + 
 "/" 
 INGEST_TOKEN 
 = 
 os 
 . 
 environ 
 . 
 get 
 ( 
 "INGEST_TOKEN" 
 ) 
 # optional shared secret in URL query (?token=...) 
 s3 
 = 
 boto3 
 . 
 client 
 ( 
 "s3" 
 ) 
 def 
  
 _write_jsonl_gz 
 ( 
 objs 
 : 
 list 
 ) 
 - 
> str 
 : 
 key 
 = 
 f 
 " 
 { 
 dt 
 . 
 datetime 
 . 
 utcnow 
 () 
 : 
 %Y/%m/%d 
 } 
 /akamai-cloud-monitor- 
 { 
 uuid 
 . 
 uuid4 
 () 
 } 
 .json.gz" 
 buf 
 = 
 io 
 . 
 BytesIO 
 () 
 with 
 gzip 
 . 
 GzipFile 
 ( 
 fileobj 
 = 
 buf 
 , 
 mode 
 = 
 "w" 
 ) 
 as 
 gz 
 : 
 for 
 o 
 in 
 objs 
 : 
 gz 
 . 
 write 
 (( 
 json 
 . 
 dumps 
 ( 
 o 
 , 
 separators 
 = 
 ( 
 "," 
 , 
 ":" 
 )) 
 + 
 "n" 
 ) 
 . 
 encode 
 ()) 
 buf 
 . 
 seek 
 ( 
 0 
 ) 
 s3 
 . 
 upload_fileobj 
 ( 
 buf 
 , 
 S3_BUCKET 
 , 
 f 
 " 
 { 
 S3_PREFIX 
 }{ 
 key 
 } 
 " 
 , 
 ExtraArgs 
 = 
 { 
 "ContentType" 
 : 
 "application/json" 
 , 
 "ContentEncoding" 
 : 
 "gzip" 
 , 
 }, 
 ) 
 return 
 f 
 "s3:// 
 { 
 S3_BUCKET 
 } 
 / 
 { 
 S3_PREFIX 
 }{ 
 key 
 } 
 " 
 def 
  
 _parse_records_from_event 
 ( 
 event 
 ) 
 - 
> list 
 : 
 # HTTP API (Lambda proxy) event: body is a JSON string 
 body 
 = 
 event 
 . 
 get 
 ( 
 "body" 
 ) 
 or 
 "" 
 if 
 event 
 . 
 get 
 ( 
 "isBase64Encoded" 
 ): 
 body 
 = 
 base64 
 . 
 b64decode 
 ( 
 body 
 ) 
 . 
 decode 
 ( 
 "utf-8" 
 , 
 "replace" 
 ) 
 try 
 : 
 data 
 = 
 json 
 . 
 loads 
 ( 
 body 
 ) 
 except 
 Exception 
 : 
 # accept line-delimited JSON as pass-through 
 try 
 : 
 return 
 [ 
 json 
 . 
 loads 
 ( 
 line 
 ) 
 for 
 line 
 in 
 body 
 . 
 splitlines 
 () 
 if 
 line 
 . 
 strip 
 ()] 
 except 
 Exception 
 : 
 return 
 [] 
 if 
 isinstance 
 ( 
 data 
 , 
 list 
 ): 
 return 
 data 
 if 
 isinstance 
 ( 
 data 
 , 
 dict 
 ): 
 return 
 [ 
 data 
 ] 
 return 
 [] 
 def 
  
 lambda_handler 
 ( 
 event 
 , 
 context 
 = 
 None 
 ): 
 # Optional shared-secret verification via query parameter (?token=...) 
 if 
 INGEST_TOKEN 
 : 
 qs 
 = 
 event 
 . 
 get 
 ( 
 "queryStringParameters" 
 ) 
 or 
 {} 
 token 
 = 
 qs 
 . 
 get 
 ( 
 "token" 
 ) 
 if 
 token 
 != 
 INGEST_TOKEN 
 : 
 return 
 { 
 "statusCode" 
 : 
 403 
 , 
 "body" 
 : 
 "forbidden" 
 } 
 records 
 = 
 _parse_records_from_event 
 ( 
 event 
 ) 
 if 
 not 
 records 
 : 
 return 
 { 
 "statusCode" 
 : 
 204 
 , 
 "body" 
 : 
 "no content" 
 } 
 key 
 = 
 _write_jsonl_gz 
 ( 
 records 
 ) 
 return 
 { 
 "statusCode" 
 : 
 200 
 , 
 "headers" 
 : 
 { 
 "Content-Type" 
 : 
 "application/json" 
 }, 
 "body" 
 : 
 json 
 . 
 dumps 
 ({ 
 "ok" 
 : 
 True 
 , 
 "s3_key" 
 : 
 key 
 , 
 "count" 
 : 
 len 
 ( 
 records 
 )}), 
 }

    2. Go to Configuration > Environment variables > Edit.

    3. Click Add new environment variableand set the following values:

      Environment variables

      Key Example
      S3_BUCKET_NAME akamai-cloud-monitor
      S3_PREFIX akamai/cloud-monitor/json/
      INGEST_TOKEN random-shared-secret

    4. Go to Configuration > General configuration.

    5. Click Editand set Timeoutto 5 minutes (300 seconds).

    6. Click Save.

    Create Amazon API Gateway (HTTPS endpoint for Akamai)

    1. In the AWS Console, go to API Gateway > Create API.
    2. Select HTTP API > Build.
    3. Provide the following configuration details:
      • Integrations: Choose Lambdaand select akamai_cloud_monitor_to_s3 .
      • Routes: Add ANY /{proxy+} or create a specific route (for example, POST /akamai/cloud-monitor ).
      • Stages: Create or use $default.
    4. Deploy the API and copy the Invoke URL(for example, https://abc123.execute-api.<region>.amazonaws.com ).

    Configure Akamai Cloud Monitor to push logs

    1. In Akamai Control Center, open your Propertyin Property Manager.
    2. Click Add Rule > choose Cloud Management.
    3. Add Cloud Monitor Instrumentationand select required Datasets.
    4. Add Cloud Monitor Data Delivery.
      • Delivery Hostname: enter your API Gateway Invoke URL(for example, abc123.execute-api.<region>.amazonaws.com ).
      • Delivery URL Path: your route plusan optional query token, for example: /akamai/cloud-monitor?token=<INGEST_TOKEN> .
    5. Saveand Activatethe property version.

    Configure a feed in Google SecOps to ingest Akamai Cloud Monitor (S3 JSON)

    1. Go to SIEM Settings > Feeds.
    2. Click Add New Feed.
    3. In the Feed namefield, enter a name for the feed (for example, Akamai Cloud Monitor — S3 ).
    4. Select Amazon S3 V2as the Source type.
    5. Select Akamai Cloud Monitoras the Log type.
    6. Click Next.
    7. Specify values for the following input parameters:
      • S3 URI: s3://akamai-cloud-monitor/akamai/cloud-monitor/json/
      • Source deletion options: Whether to delete files and/or directories after transferring.
      • Maximum File Age: Includes files modified in the last number of days. Default is 180 days.
      • Access Key ID: A 20-character alphanumeric account access key (e.g., AKIAIOSFODNN7EXAMPLE).
      • Secret Access Key: A 40-character alphanumeric account secret access key (e.g., wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY).
      • Asset namespace: akamai.cloud_monitor
      • Ingestion labels: Labels are added to all the events from this feed (for example, source=akamai_cloud_monitor , format=json ).
    8. Click Next.
    9. Review your new feed configuration in the Finalizescreen, and then click Submit.

    UDM Mapping Table

    Log Field UDM Mapping Logic
    accLang
    		network.http.user_agent Directly mapped if not "-" or empty string.
    city
    		principal.location.city Directly mapped if not "-" or empty string.
    cliIP
    		principal.ip Directly mapped if not empty string.
    country
    		principal.location.country_or_region Directly mapped if not "-" or empty string.
    cp
    		additional.fields Mapped as a key-value pair with key "cp".
    customField
    		about.ip , about.labels , src.ip Parsed as key-value pairs. Special handling for "eIp" and "pIp" to map to src.ip and about.ip respectively. Other keys are mapped as labels within about .
    errorCode
    		security_result.summary , security_result.severity If present, sets security_result.severity to "ERROR" and maps the value to security_result.summary .
    geo.city
    		principal.location.city Directly mapped if city is "-" or empty string.
    geo.country
    		principal.location.country_or_region Directly mapped if country is "-" or empty string.
    geo.lat
    		principal.location.region_latitude Directly mapped, converted to float.
    geo.long
    		principal.location.region_longitude Directly mapped, converted to float.
    geo.region
    		principal.location.state Directly mapped.
    id
    		metadata.product_log_id Directly mapped if not empty string.
    message.cliIP
    		principal.ip Directly mapped if cliIP is empty string.
    message.fwdHost
    		principal.hostname Directly mapped.
    message.reqHost
    		target.hostname , target.url Used to construct target.url and extract target.hostname .
    message.reqLen
    		network.sent_bytes Directly mapped, converted to unsigned integer if totalBytes is empty or "-".
    message.reqMethod
    		network.http.method Directly mapped if reqMethod is empty string.
    message.reqPath
    		target.url Appended to target.url .
    message.reqPort
    		target.port Directly mapped, converted to integer if reqPort is empty string.
    message.respLen
    		network.received_bytes Directly mapped, converted to unsigned integer.
    message.sslVer
    		network.tls.version Directly mapped.
    message.status
    		network.http.response_code Directly mapped, converted to integer if statusCode is empty or "-".
    message.UA
    		network.http.user_agent Directly mapped if UA is "-" or empty string.
    network.asnum
    		additional.fields Mapped as a key-value pair with key "asnum".
    network.edgeIP
    		intermediary.ip Directly mapped.
    network.network
    		additional.fields Mapped as a key-value pair with key "network".
    network.networkType
    		additional.fields Mapped as a key-value pair with key "networkType".
    proto
    		network.application_protocol Used to determine network.application_protocol .
    queryStr
    		target.url Appended to target.url if not "-" or empty string.
    referer
    		network.http.referral_url , about.hostname Directly mapped if not "-". Extracted hostname is mapped to about.hostname .
    reqHost
    		target.hostname , target.url Used to construct target.url and extract target.hostname .
    reqId
    		metadata.product_log_id , network.session_id Directly mapped if id is empty string. Also mapped to network.session_id .
    reqMethod
    		network.http.method Directly mapped if not empty string.
    reqPath
    		target.url Appended to target.url if not "-".
    reqPort
    		target.port Directly mapped, converted to integer.
    reqTimeSec
    		metadata.event_timestamp , timestamp Used to set event timestamp.
    start
    		metadata.event_timestamp , timestamp Used to set event timestamp if reqTimeSec is empty string.
    statusCode
    		network.http.response_code Directly mapped, converted to integer if not "-" or empty string.
    tlsVersion
    		network.tls.version Directly mapped.
    totalBytes
    		network.sent_bytes Directly mapped, converted to unsigned integer if not empty or "-".
    type
    		metadata.product_event_type Directly mapped.
    UA
    		network.http.user_agent Directly mapped if not "-" or empty string.
    version
    		metadata.product_version Directly mapped.
    xForwardedFor
    		principal.ip Directly mapped if not "-" or empty string.
    (Parser Logic)
    		 metadata.vendor_name Set to "Akamai".
    (Parser Logic)
    		 metadata.product_name Set to "DataStream".
    (Parser Logic)
    		 metadata.event_type Set to "NETWORK_HTTP".
    (Parser Logic)
    		 metadata.product_version Set to "2" if version is empty string.
    (Parser Logic)
    		 metadata.log_type Set to "AKAMAI_CLOUD_MONITOR".
    (Parser Logic)
    		 network.application_protocol Determined from proto or message.proto . Set to "HTTPS" if either contains "HTTPS" (case-insensitive), "HTTP" otherwise.
    (Parser Logic)
    		 security_result.severity Set to "INFORMATIONAL" if errorCode is "-" or empty string.
    (Parser Logic)
    		 target.url Constructed from protocol , reqHost (or message.reqHost ), reqPath (or message.reqPath ), and queryStr .

    Need more help? Get answers from Community members and Google SecOps professionals.

    Create a Mobile Website
    View Site in Mobile | Classic
    Share by: