Collect CircleCI audit logs

Supported in:

Google secops SIEM

This parser extracts fields from CircleCI audit logs in CSV and JSON formats, transforming them into the Unified Data Model (UDM). It handles both formats, performs data transformations and enrichments, and maps the extracted fields to their corresponding UDM fields within the eventobject. It focuses on user actions, resource access, and update events, categorizing them and populating relevant UDM fields like principal, target, network, and metadata.

Before you begin

Ensure that you have the following prerequisites:

Google SecOps instance.
Privileged access to CircleCI.

Set up feeds

To configure a feed, follow these steps:

Go to SIEM Settings > Feeds.
Click Add New Feed.
On the next page, click Configure a single feed.
In the Feed namefield, enter a name for the feed (for example, CircleCI Logs).
Select Webhookas the Source type.
Select CircleCIas the Log type.
Click Next.
Optional: Specify values for the following input parameters:
- Split delimiter: the delimiter that is used to separate log lines, such as \n .
- Asset namespace: the asset namespace .
- Ingestion labels: the label applied to the events from this feed.
Click Next.
Review the feed configuration in the Finalizescreen, and then click Submit.
Click Generate Secret Keyto generate a secret key to authenticate this feed.
Copy and store the secret key. You cannot view this secret key again. If needed, you can regenerate a new secret key, but this action makes the previous secret key obsolete.
From the Detailstab, copy the feed endpoint URL from the Endpoint Informationfield. You need to specify this endpoint URL in your client application.
Click Done.

Create an API key for the webhook feed

Go to Google Cloud console > Credentials.

Go to Credentials
Click Create credentials, and then select API key.
Restrict the API key access to the Google Security Operations API.

Specify the endpoint URL

In your client application, specify the HTTPS endpoint URL provided in the webhook feed.
Enable authentication by specifying the API key and secret key as part of the custom header in the following format:
```
 X-goog-api-key = API_KEY 
X-Webhook-Access-Key = SECRET 
 
```
Recommendation: Specify the API key as a header instead of specifying it in the URL.
If your webhook client doesn't support custom headers, you can specify the API key and secret key using query parameters in the following format:
```
  ENDPOINT_URL 
?key= API_KEY 
&secret= SECRET 
 
```
Replace the following:
- ENDPOINT_URL : the feed endpoint URL.
- API_KEY : the API key to authenticate to Google SecOps.
- SECRET : the secret key that you generated to authenticate the feed.

Configuring a webhook in CircleCI

Sign in to the CircleCI web interface.
Select the project you want to ingest the logs from.
Click Project Settings.
Select Webhooks.
Click Add Webhook.
Specify values for the following input parameters:
- Webhook Name: provide a descriptive name (for example, Google SecOps).
- Endpoint URL: enter the <ENDPOINT_URL> of the Google SecOps API endpoint.
Important: Append the <API_KEY> and <SECRET> to the endpoint URL <ENDPOINT_URL>?key=<API_KEY>&secret=<SECRET> .
- Events:Select the CircleCI events that should trigger the webhook (for example, select workflow-completedto send data after a workflow finishes).
Click Saveto create the webhook.

UDM Mapping Table

Need more help? Get answers from Community members and Google SecOps professionals.

Log Field	UDM Mapping	Logic
account.id	read_only_udm.about.resource.attribute.labels.value	The value of account.idfrom the raw log is assigned to the UDM field read_only_udm.about.resource.attribute.labels.valuewhere the corresponding keyis account_id.
action	read_only_udm.metadata.product_event_type	The value of actionfrom the raw log is assigned to the UDM field read_only_udm.metadata.product_event_type.
actor.id	read_only_udm.principal.user.product_object_id	The value of actor.idfrom the raw log is assigned to the UDM field read_only_udm.principal.user.product_object_id.
actor.name	read_only_udm.principal.user.userid	The "github: " prefix is removed from the actor.namefield in the raw log. The remaining value is assigned to the UDM field read_only_udm.principal.user.userid. If actor.nameexists in the raw log, the value USER_RESOURCE_UPDATE_CONTENTis assigned to read_only_udm.metadata.event_type. Otherwise, USER_RESOURCE_ACCESSis assigned.
id	read_only_udm.metadata.product_log_id	The value of idfrom the raw log is assigned to the UDM field read_only_udm.metadata.product_log_id. The parser sets the read_only_udm.metadata.log_typeto CIRCLECI. The parser sets the read_only_udm.metadata.product_nameto CIRCLECI. The parser sets the read_only_udm.metadata.vendor_nameto CIRCLECI.
occurred_at	read_only_udm.metadata.event_timestamp	The value of occurred_atfrom the raw log is parsed as a timestamp and assigned to the UDM field read_only_udm.metadata.event_timestamp.
organization.name	read_only_udm.target.administrative_domain	The "github: " prefix is removed from the organization.namefield in the raw log. The remaining value is assigned to the UDM field read_only_udm.target.administrative_domain.
payload.job.id	read_only_udm.about.resource.attribute.labels.value	The value of payload.job.idfrom the raw log is assigned to the UDM field read_only_udm.about.resource.attribute.labels.valuewhere the corresponding keyis job_id.
payload.job.job_name	read_only_udm.about.resource.attribute.labels.value	The value of payload.job.job_namefrom the raw log is assigned to the UDM field read_only_udm.about.resource.attribute.labels.valuewhere the corresponding keyis job_name.
payload.job.job_status	read_only_udm.about.resource.attribute.labels.value	The value of payload.job.job_statusfrom the raw log is assigned to the UDM field read_only_udm.about.resource.attribute.labels.valuewhere the corresponding keyis job_status.
payload.workflow.id	read_only_udm.about.resource.attribute.labels.value	The value of payload.workflow.idfrom the raw log is assigned to the UDM field read_only_udm.about.resource.attribute.labels.valuewhere the corresponding keyis workflow_id.
request.id	read_only_udm.network.session_id	The value of request.idfrom the raw log is assigned to the UDM field read_only_udm.network.session_id.
scope.id	read_only_udm.about.resource.attribute.labels.value	The value of scope.idfrom the raw log is assigned to the UDM field read_only_udm.about.resource.attribute.labels.valuewhere the corresponding keyis scope_id. The parser initially sets sec_actionto BLOCK. If the successfield in the raw log is true, sec_actionis changed to ALLOW. The value of sec_actionis then assigned to the UDM field read_only_udm.security_result.action.
target.id	read_only_udm.target.resource.product_object_id	The value of target.idfrom the raw log is assigned to the UDM field read_only_udm.target.resource.product_object_id.
target.name	read_only_udm.target.resource.name	The "github: " prefix is removed from the target.namefield in the raw log. The remaining value is assigned to the UDM field read_only_udm.target.resource.name. The parser sets the read_only_udm.target.resource.resource_typeto STORAGE_OBJECT.
version	read_only_udm.target.resource.attribute.labels.value	The value of versionfrom the raw log is converted to a string and assigned to the UDM field read_only_udm.target.resource.attribute.labels.valuewhere the corresponding keyis version.