Console
    Contact Us Start free

    Collect Datadog logs

    Supported in:

    Overview

    This parser extracts fields from Datadog logs, performs several mutations and Grok matching to structure the data, and maps the extracted fields to the UDM. It handles different log formats within the messagefield, including key-value pairs and JSON objects, and converts specific fields into UDM-compliant labels and additional fields.

    Before you begin

    Ensure that you have the following prerequisites:

    • Google SecOps instance.
    • Privileged access to Google Cloud IAM.
    • Privileged access to Cloud Storage.
    • logs_write_archive user access to Datadog.

    Option 1: Datadog log sharing through Cloud Storage configuration

    Configure Datadog integration with Google Cloud Platform

    Create a Google Cloud Storage Bucket

    1. Sign in to the Google Cloud console.

    2. Go to the Cloud Storage Bucketspage.

      Go to Buckets

    3. Click Create.

    4. On the Create a bucketpage, enter your bucket information. After each of the following steps, click Continueto proceed to the next step:

      1. In the Get startedsection, do the following:

        1. Enter a unique name that meets the bucket name requirements (for example, datadog-data).

        2. To enable hierarchical namespace, click the expander arrow to expand the Optimize for file oriented and data-intensive workloadssection, and then select Enable Hierarchical namespace on this bucket.

        3. To add a bucket label, click the expander arrow to expand the Labelssection.

        4. Click Add label, and specify a key and a value for your label.

      2. In the Choose where to store your datasection, do the following:

        1. Select a Location type.
        2. Use the location type drop-down to select a Locationwhere object data within your bucket will be permanently stored.
          • If you select the dual-regionlocation type, you can also choose to enable turbo replicationby using the relevant checkbox.
        3. To set up cross-bucket replication, expand the Set up cross-bucket replicationsection.

      3. In the Choose a storage class for your datasection, either select a default storage classfor the bucket, or select Autoclassfor automatic storage class management of your bucket's data.

      4. In the Choose how to control access to objectssection, select notto enforce public access prevention, and select an access control modelfor your bucket's objects.

      5. In the Choose how to protect object datasection, do the following:

        1. Select any of the options under Data protectionthat you want to set for your bucket.
        2. To choose how your object data will be encrypted, click the expander arrow labeled Data encryption, and select a Data encryption method.

    5. Click Create.

    1. Go to IAM & Admin > Service Accounts.
    2. Create a new service account.
    3. Give it a descriptive name (For example, datadog-user).
    4. Grant the service account with Storage Object Adminrole on the Cloud Storage bucket you created in the previous step.
    5. Create an SSH key for the service account.
    6. Download a JSON key file for the service account. Keep this file secure.

    Configure Datadog to send logs to Cloud Storage

    1. Sign in to Datadog using a privileged account.
    2. Go to Logs > Log Forwarding.
    3. Click + Create New Archive.
    4. Select Google Cloud Storage.
    5. Input the required parameters and click Save.

    Option 2: Datadog log sharing through Webhook configuration

    Set up feeds

    To configure a feed, follow these steps:

    1. Go to SIEM Settings > Feeds.
    2. Click Add New Feed.
    3. On the next page, click Configure a single feed.
    4. In the Feed namefield, enter a name for the feed (for example, Datadog Logs).
    5. Select Webhookas the Source type.
    6. Select Datadogas the Log type.
    7. Click Next.
    8. Optional: Specify values for the following input parameters:
      • Split delimiter: the delimiter that is used to separate log lines, such as \n .
    9. Click Next.
    10. Review the feed configuration in the Finalizescreen, and then click Submit.
    11. Click Generate Secret Keyto generate a secret key to authenticate this feed.
    12. Copy and store the secret key. You cannot view this secret key again. If needed, you can regenerate a new secret key, but this action makes the previous secret key obsolete.
    13. From the Detailstab, copy the feed endpoint URL from the Endpoint Informationfield. You need to specify this endpoint URL in your client application.
    14. Click Done.

    Create an API key for the webhook feed

    1. Go to Google Cloud console > Credentials.

      Go to Credentials

    2. Click Create credentials, and then select API key.

    3. Restrict the API key access to the Chronicle API.

    Specify the endpoint URL

    1. In your client application, specify the HTTPS endpoint URL provided in the webhook feed.

    2. Enable authentication by specifying the API key and secret key as part of the custom header in the following format:

       X-goog-api-key = API_KEY 
X-Webhook-Access-Key = SECRET

      Recommendation: Specify the API key as a header instead of specifying it in the URL.

    3. If your webhook client doesn't support custom headers, you can specify the API key and secret key using query parameters in the following format:

        ENDPOINT_URL 
?key= API_KEY 
&secret= SECRET

      Replace the following:

      • ENDPOINT_URL : the feed endpoint URL.
      • API_KEY : the API key to authenticate to Google SecOps.
      • SECRET : the secret key that you generated to authenticate the feed.

    Configure Datadog to send logs to webhook

    1. Sign in to Datadog using a privileged account.
    2. Go to Logs > Log Forwarding.
    3. Select Custom Destinations.
    4. Click + Create a New Destination.
    5. Specify values for the following input parameters:
      1. Choose a destination type: Select HTTP.
      2. Name the destination: Provide a descriptive name for the webhook (for example, Google SecOps Webhook).
      3. Configure the destination: Enter the ENDPOINT_URL, followed by the API_KEYand SECRET.
      4. Configure authentication settings: Add a general header like the following, this won't malform the HTTP request and allow Datadog to complete webhook creation.
        • Header name: Accept .
        • Header value: application/json .
      5. Click Save.

    UDM Mapping Table

    Log Field UDM Mapping Logic
    _id
    		read_only_udm.metadata.product_log_id Directly mapped from the _id field.
    alert
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the alert field and added as a label within the security_result object.
    attributes.@timestamp
    		read_only_udm.metadata.event_timestamp The event timestamp is extracted from the attributes.@timestamp field and converted to seconds and nanoseconds.
    attributes.@version
    		read_only_udm.metadata.product_version Directly mapped from the attributes.@version field.
    attributes.level_value
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the attributes.level_value field and added as a label within the security_result object.
    attributes.logger_name
    		read_only_udm.principal.application Directly mapped from the attributes.logger_name field.
    attributes._trace.baggage._sli_service
    		read_only_udm.additional.fields Directly mapped from the attributes._trace.baggage._sli_service field and added as an additional field.
    attributes._trace.baggage.device_id
    		read_only_udm.principal.asset.asset_id Directly mapped from the attributes._trace.baggage.device_id field, prefixed with "Device Id:".
    attributes._trace.origin.operation
    		read_only_udm.metadata.product_event_type Directly mapped from the attributes._trace.origin.operation field.
    caller
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the caller field and added as a label within the security_result object.
    component
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the component field and added as a label within the security_result object.
    context.AlertName
    		read_only_udm.security_result.threat_name Directly mapped from the context.AlertName field.
    context.BusArch
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the context.BusArch field and added as a label within the security_result object.
    context.CANDBVersion
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the context.CANDBVersion field and added as a label within the security_result object.
    context.esn
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the context.esn field and added as a label within the security_result object.
    context.ftcpVersion
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the context.ftcpVersion field and added as a label within the security_result object.
    context.ingestMessageId
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the context.ingestMessageId field and added as a label within the security_result object.
    context.redactedVin
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the context.redactedVin field and added as a label within the security_result object.
    context.vehicleId
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the context.vehicleId field and added as a label within the security_result object.
    date
    		read_only_udm.metadata.collected_timestamp The collected timestamp is extracted from the date field (renamed to date1 in the parser) and converted to seconds and nanoseconds.
    host
    		read_only_udm.principal.hostname Directly mapped from the host field.
    message
    		read_only_udm.security_result.about.resource.attribute.labels The messagefield is parsed, and parts of it are used to populate the summaryand json_datafields. The remaining part is treated as key-value pairs and added as labels within the security_resultobject.
    msg
    		read_only_udm.security_result.about.resource.attribute.labels Extracted from the msg field and added as a label within the security_result object.
    service
    		read_only_udm.metadata.product_name Directly mapped from the service field.
    status
    		read_only_udm.security_result.severity The severity is determined based on the status field. "INFO", "DEBUG", "debug", and "info" map to "LOW", "WARN" maps to "MEDIUM", and other values are not explicitly mapped in the provided code snippet.
    tags
    		read_only_udm.additional.fields Each tag in the tags array is parsed into key-value pairs and added as additional fields.
    N/A
    		 read_only_udm.metadata.event_type Set to "STATUS_UPDATE" if the host field is present, and "GENERIC_EVENT" otherwise.

    Need more help? Get answers from Community members and Google SecOps professionals.

    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: