Collect Team Cymru Scout Threat Intelligence data
Supported in:
This document explains how to ingest Team Cymru Scout Threat Intelligence data to Google Security Operations using Amazon S3.
Before you begin
Make sure you have the following prerequisites:
- A Google SecOps instance
- Privileged access to Team Cymru Scouttenant
- Privileged access to AWS(S3, IAM, Lambda, EventBridge)
Get Team Cymru Scout prerequisites
- Sign in to the Team Cymru Scout Platform.
- Go to the API Keys web .
- Click the Createbutton.
- Provide the description for the key, if needed.
- Click the Create Keybutton to generate the API key.
- Copy and save in a secure location the following details:
- SCOUT_API_KEY- API access key
- SCOUT_BASE_URL- Scout API base URL
Configure AWS S3 bucket and IAM for Google SecOps
- Create Amazon S3 bucketfollowing this user guide: Creating a bucket .
- Save bucket Nameand Regionfor future reference (for example,
team-cymru-scout-ti). - Create a Userfollowing this user guide: Creating an IAM user .
- Select the created User.
- Select Security credentialstab.
- Click Create Access Keyin section Access Keys.
- Select Third-party serviceas Use case.
- Click Next.
- Optional: Add a description tag.
- Click Create access key.
- Click Download CSV fileto save the Access Keyand Secret Access Keyfor future reference.
- Click Done.
- Select Permissionstab.
- Click Add permissionsin section Permissions policies.
- Select Add permissions.
- Select Attach policies directly.
- Search for AmazonS3FullAccesspolicy.
- Select the policy.
- Click Next.
- Click Add permissions.
Configure the IAM policy and role for S3 uploads
- In the AWS console, go to IAM > Policies.
- Click Create policy > JSON tab.
-
Enter the following policy:
{ "Version" : "2012-10-17" , "Statement" : [ { "Sid" : "AllowPutObjects" , "Effect" : "Allow" , "Action" : "s3:PutObject" , "Resource" : "arn:aws:s3:::team-cymru-scout-ti/*" }, { "Sid" : "AllowGetStateObject" , "Effect" : "Allow" , "Action" : "s3:GetObject" , "Resource" : "arn:aws:s3:::team-cymru-scout-ti/team-cymru/scout-ti/state.json" } ] }- Replace
team-cymru-scout-tiif you entered a different bucket name.
- Replace
-
Click Next > Create policy.
-
Go to IAM > Roles > Create role > AWS service > Lambda.
-
Attach the newly created policy.
-
Name the role
TeamCymruScoutToS3Roleand click Create role.
Create the Lambda function
- In the AWS Console, go to Lambda > Functions > Create function.
- Click Author from scratch.
-
Provide the following configuration details:
Setting Value Name team_cymru_scout_ti_to_s3Runtime Python 3.13 Architecture x86_64 Execution role TeamCymruScoutToS3Role -
After the function is created, open the Codetab, delete the stub and enter the following code (
team_cymru_scout_ti_to_s3.py):``` python #!/usr/bin/env python3 # Lambda: Pull Team Cymru Scout Threat Intelligence exports to S3 (no transform) import os , json , time from urllib.request import Request , urlopen from urllib.error import HTTPError , URLError import boto3 S3_BUCKET = os . environ [ "S3_BUCKET" ] S3_PREFIX = os . environ . get ( "S3_PREFIX" , "team-cymru/scout-ti/" ) STATE_KEY = os . environ . get ( "STATE_KEY" , "team-cymru/scout-ti/state.json" ) WINDOW_SEC = int ( os . environ . get ( "WINDOW_SECONDS" , "3600" )) HTTP_TIMEOUT = int ( os . environ . get ( "HTTP_TIMEOUT" , "60" )) HTTP_RETRIES = int ( os . environ . get ( "HTTP_RETRIES" , "3" )) MODE = os . environ . get ( "MODE" , "GET" ) . upper () API_HEADERS = json . loads ( os . environ . get ( "API_HEADERS" , " {} " )) MAX_PAGES = int ( os . environ . get ( "MAX_PAGES" , "10" )) # GET mode DOWNLOAD_URL_TEMPLATE = os . environ . get ( "DOWNLOAD_URL_TEMPLATE" , "" ) # POST_JSON mode API_URL = os . environ . get ( "API_URL" , "" ) JSON_BODY_TEMPLATE = os . environ . get ( "JSON_BODY_TEMPLATE" , "" ) # Team Cymru Scout specific SCOUT_BASE_URL = os . environ . get ( "SCOUT_BASE_URL" , "https://api.scout.cymru.com" ) SCOUT_API_KEY = os . environ . get ( "SCOUT_API_KEY" , "" ) s3 = boto3 . client ( "s3" ) def _iso ( ts : float ) - > str : return time . strftime ( "%Y-%m- %d T%H:%M:%SZ" , time . gmtime ( ts )) def _get_state () - > dict : try : obj = s3 . get_object ( Bucket = S3_BUCKET , Key = STATE_KEY ) b = obj [ "Body" ] . read () return json . loads ( b ) if b else {} except Exception : return {} def _put_state ( st : dict ): s3 . put_object ( Bucket = S3_BUCKET , Key = STATE_KEY , Body = json . dumps ( st , separators = ( "," , ":" )) . encode ( "utf-8" ), ContentType = "application/json" , ) def _http ( url : str , method : str = "GET" , body : bytes | None = None ) - > tuple [ bytes , str ]: attempt = 0 while True : try : req = Request ( url , method = method ) # Add headers headers = API_HEADERS . copy () if SCOUT_API_KEY and "Authorization" not in headers : headers [ "Authorization" ] = f "Bearer { SCOUT_API_KEY } " headers . setdefault ( "Accept" , "application/json" ) for k , v in headers . items (): req . add_header ( k , v ) if body is not None : req . add_header ( "Content-Type" , "application/json" ) with urlopen ( req , data = body , timeout = HTTP_TIMEOUT ) as r : return r . read (), r . headers . get ( "Content-Type" , "application/json" ) except HTTPError as e : if e . code in ( 429 , 500 , 502 , 503 , 504 ) and attempt < HTTP_RETRIES : delay = 1 + attempt try : delay = int ( e . headers . get ( "Retry-After" , delay )) except Exception : pass time . sleep ( max ( 1 , delay )) attempt += 1 continue raise except URLError : if attempt < HTTP_RETRIES : time . sleep ( 1 + attempt ) attempt += 1 continue raise def _write ( blob : bytes , ctype : str , from_ts : float , to_ts : float , page : int ) - > str : date_path = time . strftime ( "%Y/%m/ %d " , time . gmtime ( to_ts )) key = f " { S3_PREFIX } / { date_path } /scout_ti_ { int ( from_ts ) } _ { int ( to_ts ) } _p { page : 03d } .json" s3 . put_object ( Bucket = S3_BUCKET , Key = key , Body = blob , ContentType = ctype or "application/json" ) return key def _next_cursor ( obj : dict ) - > str | None : if not isinstance ( obj , dict ): return None for container in ( obj , obj . get ( "meta" , {}) or {}, obj . get ( "metadata" , {}) or {}): for k in ( "next" , "next_cursor" , "nextCursor" , "nextPageToken" , "continuation" , "cursor" , "pagedResultsCookie" ): v = container . get ( k ) if v : return str ( v ) return None def _loop ( from_ts : float , to_ts : float ) - > dict : cursor , page , written = None , 0 , 0 while page < MAX_PAGES : if MODE == "GET" : if DOWNLOAD_URL_TEMPLATE : url = ( DOWNLOAD_URL_TEMPLATE . replace ( " {FROM} " , _iso ( from_ts )) . replace ( " {TO} " , _iso ( to_ts )) . replace ( " {CURSOR} " , cursor or "" )) else : # Default Scout API endpoint (adjust based on actual API) url = f " { SCOUT_BASE_URL } /v1/threat-intelligence?start= { _iso ( from_ts ) } & end= { _iso ( to_ts ) } " if cursor : url += f "&cursor= { cursor } " blob , ctype = _http ( url , method = "GET" ) else : assert API_URL and JSON_BODY_TEMPLATE , "API_URL and JSON_BODY_TEMPLATE required for MODE=POST_JSON" body = ( JSON_BODY_TEMPLATE . replace ( " {FROM} " , _iso ( from_ts )) . replace ( " {TO} " , _iso ( to_ts )) . replace ( " {CURSOR} " , cursor or "" )) . encode ( "utf-8" ) blob , ctype = _http ( API_URL , method = "POST" , body = body ) # Normalize to JSON bytes for storage try : parsed = json . loads ( blob . decode ( "utf-8" )) normalized = json . dumps ( parsed , separators = ( "," , ":" )) . encode ( "utf-8" ) ctype_out = "application/json" except Exception : normalized = blob ctype_out = ctype or "application/octet-stream" _ = _write ( normalized , ctype_out , from_ts , to_ts , page ) written += 1 page += 1 # Follow cursor if JSON and cursor exists try : if parsed and isinstance ( parsed , dict ): cursor = _next_cursor ( parsed ) if not cursor : break except Exception : break return { "pages" : page , "objects" : written } def lambda_handler ( event = None , context = None ): st = _get_state () now = time . time () from_ts = st . get ( "last_to_ts" ) or ( now - WINDOW_SEC ) to_ts = now res = _loop ( from_ts , to_ts ) st [ "last_to_ts" ] = to_ts _put_state ( st ) return { "ok" : True , "window" : { "from" : _iso ( from_ts ), "to" : _iso ( to_ts )}, ** res } if __name__ == "__main__" : print ( lambda_handler ()) ``` -
Go to Configuration > Environment variables.
-
Click Edit > Add new environment variable.
-
Enter the following environment variables provided, replacing with your values.
Key Example value S3_BUCKETteam-cymru-scout-tiS3_PREFIXteam-cymru/scout-ti/STATE_KEYteam-cymru/scout-ti/state.jsonSCOUT_BASE_URLhttps://api.scout.cymru.comSCOUT_API_KEYyour-scout-api-keyWINDOW_SECONDS3600HTTP_TIMEOUT60HTTP_RETRIES3MODEGETorPOST_JSONAPI_HEADERS{"Authorization":"Bearer <token>","Accept":"application/json"}DOWNLOAD_URL_TEMPLATE(GET mode) Custom URL template with {FROM},{TO},{CURSOR}API_URL(POST_JSON mode) API endpoint URL JSON_BODY_TEMPLATE(POST_JSON mode) JSON body with {FROM},{TO},{CURSOR}MAX_PAGES10 -
After the function is created, stay on its page (or open Lambda > Functions > your-function).
-
Select the Configurationtab.
-
In the General configurationpanel click Edit.
-
Change Timeoutto 5 minutes (300 seconds)and click Save.
Create an EventBridge schedule
- Go to Amazon EventBridge > Scheduler > Create schedule.
- Provide the following configuration details:
- Recurring schedule: Rate(
1 hour). - Target: Your Lambda function
team_cymru_scout_ti_to_s3. - Name:
team-cymru-scout-ti-1h.
- Recurring schedule: Rate(
- Click Create schedule.
Optional: Create read-only IAM user & keys for Google SecOps
- Go to AWS Console > IAM > Users > Add users.
- Click Add users.
- Provide the following configuration details:
- User: Enter
secops-reader. - Access type: Select Access key – Programmatic access.
- User: Enter
- Click Create user.
- Attach minimal read policy (custom): Users > secops-reader > Permissions > Add permissions > Attach policies directly > Create policy.
-
In the JSON editor, enter the following policy:
{ "Version" : "2012-10-17" , "Statement" : [ { "Effect" : "Allow" , "Action" : [ "s3:GetObject" ], "Resource" : "arn:aws:s3:::team-cymru-scout-ti/*" }, { "Effect" : "Allow" , "Action" : [ "s3:ListBucket" ], "Resource" : "arn:aws:s3:::team-cymru-scout-ti" } ] } -
Set the name to
secops-reader-policy. -
Go to Create policy > search/select > Next > Add permissions.
-
Go to Security credentials > Access keys > Create access key.
-
Download the CSV(these values are entered into the feed).
Configure a feed in Google SecOps to ingest Team Cymru Scout Threat Intelligence
- Go to SIEM Settings > Feeds.
- Click Add New Feed.
- In the Feed namefield, enter a name for the feed (for example,
Team Cymru Scout Threat Intelligence). - Select Amazon S3 V2as the Source type.
- Select Team Cymru Scout Threat Intelligenceas the Log type.
- Click Next.
- Specify values for the following input parameters:
- S3 URI:
s3://team-cymru-scout-ti/team-cymru/scout-ti/ - Source deletion options: Select deletion option according to your preference.
- Maximum File Age: Include files modified in the last number of days. Default is 180 days.
- Access Key ID: User access key with access to the S3 bucket.
- Secret Access Key: User secret key with access to the S3 bucket.
- Asset namespace: The asset namespace .
- Ingestion labels: The label applied to the events from this feed.
- S3 URI:
- Click Next.
- Review your new feed configuration in the Finalizescreen, and then click Submit.
Need more help? Get answers from Community members and Google SecOps professionals.
