Create and manage feeds using the feed management UI

This document provides information about how to create, manage, and troubleshoot feeds using the feed management UI. Managing the feeds includes modifying, enabling, and deleting the feeds.

Before you begin

Each data feed has its own set of prerequisites that must be completed prior to setting up the feed in Google Security Operations. For information about prerequisites specific to a feed type, see Configuration by source type . Search for the data feed type you need to set up and follow the instructions provided.

Supported compression formats

Supported compression formats for feed ingestion include: .gz, .tar.gz, .tar, .targz, and solr.gz. The following table outlines the different file sizes that Google SecOps feeds transformation supports:

Ways to set up feeds

There are two ways for Google SecOps customers to set up this feed in the platform. Use the method that works best for your environment:

• SIEM Settings > Feeds
• Content Hub > Content Packs

View configured feeds

The Feedspage shows all the feeds you've configured.

1. Go to SIEM Settings > Feeds. The main page displays all your configured feeds.
2. Hold the pointer over each row to display the more_vert Moremenu.
3. In the menu, you can view feed details, edit, disable, or delete the feed.

Configure multiple feeds for a product family (Google SecOps customers only)

You can configure multiple feeds per product family, based on log type. Log types that Google has identified as a baseline are marked as required. The platform provides setup instructions, required procedures, and descriptions of all configuration parameters. Some parameters are predefined to simplify the configuration setup. For example, you can create multiple feeds under both requiredand optionallog types for CrowdStrike Falcon. For more information, see Configure multiple feeds .

Add a feed

To add a feed to your Google SecOps account, complete the following steps:

Use the following procedure to add a feed through the SIEM Settings* {and_then} Feedspage.

1. In the Google SecOps menu, select Settings, and then click Feeds. The data feeds listed on the Feedspage include all the feeds that Google has configured for your account in addition to the feeds that you have configured.

2. Click Add New Feed.

3. On the next page, click Configure a single feed. This step is not relevant for customers using the Google SecOps SIEM standalone platform.

4. Add a feed name.

5. In the Source typelist, select the source type for importing data into Google SecOps. You can select from the following feed source types:

   • Amazon Data Firehose
   • Amazon S3
   • Amazon S3 (Preview)
   • Amazon SQS
   • Amazon SQS (Preview)
   • Azure Blob Storage
   • Azure Blob Storage (Preview)
   • Google Cloud Pub/Sub
   • Cloud Storage
   • Cloud Storage (Preview)
   • Cloud Storage Event Driven (Preview)
   • HTTP(S) Files (non-API)
   • Third party API
   • Webhook

6. In the Log typelist, select the log type that corresponds to the logs you want to ingest. The available logs vary depending on the source type you selected earlier.

   If you select Cloud Storageas the source type, use the Get service accountoption to get a unique service account. See Google Cloud Storage feed setup example .

7. Click Next.

8. Specify the parameters needed from the Input Parameterstab. The options presented here vary depending on the source and log type selected on the Set Propertiestab. Hold the pointer over the question icon for each field to get additional information on what you need to provide.

9. Optional: You can specify a namespace in the Set Propertiestab. For more information about namespaces, see Work with asset namespaces .

10. Click Next.

11. Review your new feed configuration on the Finalizetab.

12. Click Submit. Google SecOps completes a validation check of the new feed. If the feed passes the check, a name is generated for the feed, it's submitted to Google SecOps, and Google SecOps begins to attempt to fetch data.

    

Google Cloud Storage feed setup example

1. From the Google SecOps menu, select Settings, and then click Feeds.
2. Click Add New Feed.
3. On the next page, click Configure a single feed. This step doesn't apply if you are using the Google SecOps SIEM standalone platform.
4. Select Cloud Storagefor Source Type.
5. Select the Log type. For example, to create a feed for Google Kubernetes Engine audit logs, select Google Kubernetes Engine audit logsas the Log Type.
6. Click Get service account. Google SecOps provides a unique service account that Google SecOps uses to ingest data.
7. Optional: Configure the service account. For more information, see Grant access to the Google SecOps service account .
8. Click Next.

9. Based on the Cloud Storage configuration that you created, specify values for the following fields:

   • Storage bucket URI

   • Source deletion option

   To learn more about how to set up Cloud Storage buckets, see Create Buckets .

10. Click Nextand then click Submit. Caution: You must enable allowlisting and add the IP ranges for all log types that ingest data from third-party APIs.

IP allowlisting

This section describes allowlisting requirements for third-party feeds and Storage Transfer Service (STS).

Allowlist third-party API feeds

Enable allowlisting and add the Google IP ranges for all log types that ingest data from third-party APIs.

Enable STS access for Amazon S3 and Azure Storage

The STS is used by the following Google Cloud Storage feeds to transfer data from Amazon S3 and Azure Storage blobstores to Google SecOps:

• Amazon S3 (Preview)
• Amazon SQS (Preview)
• Azure Blob Storage (Preview)

STS sends data transfer requests to the Amazon S3 and Azure storage services from a set of defined STS IP address ranges. These STS IP address ranges are published in the following JSON file: https://www.gstatic.com/storage-transfer-service/ipranges.json

To use these STS feed source types, you may need to enable access to your Amazon S3 and Azure Storage blobstores, for queries originating from an IP address defined in the STS IP address ranges.

You may need to adjust IP access restrictions to enable STS to access your Amazon S3 and Azure storage services:

1. Pull the latest IP ranges from the JSON file.

   We recommend reading data from this JSON file at least weekly to keep your security configuration up to date. When a new range is added to the file, the system waits at least 7 days before using that range for requests from STS.

   For a sample Python script that fetches IP ranges from a JSON file, see IP addresses for default domains .

2. Compare the current IP range creationTime to the IP range creationTime read from the previous JSON file. If these differ, update the IP access restrictions in Amazon S3 and Azure Storage blobstores.

   • For Amazon S3

     To update the IP access restrictions in your Amazon S3 blobstore:

     If your AWS project uses IP restrictions for access to storage, you must add the IP ranges used by STS workers to your list of allowed IPs.

     To add these ranges as allowed IPs, use the Condition field in a bucket policy , as described in the AWS S3 documentation: Managing access based on specific IP addresses .

   • For Azure Storage

     To update the IP access restrictions in your Azure Storage blobstore:

     If you restrict access to your Azure resources using an Azure Storage firewall, you must add the IP ranges used by STS workers to your list of allowed IPs.

     To add these ranges as allowed IPs, follow these instructions: Configure Azure Storage firewalls and virtual networks .

Delete source files

The source deletion optionlets you delete feed source objects (files and folders) from storage, after a successful transfer. This option is only available for selected feed source types, including Cloud Storage. These feed source types include the SOURCE DELETION OPTIONfield in their Add newand Edit feedworkflows.

Source deletion options

• For supported feed source types, including Cloud Storage, the SOURCE DELETION OPTIONfield offers these options:

  • Never delete files
  • Delete transferred files and empty directories
  • Delete transferred files

• Microsoft Azure Blob Storage (AZURE_BLOBSTORE) doesn't support deletion of source files. For the SOURCE DELETION OPTIONfield, only select the Never delete filesoption.

• For the following feed sources ( "feedSourceType" ): GOOGLE_CLOUD_STORAGE_V2 , GOOGLE_CLOUD_STORAGE_EVENT_DRIVEN , AMAZON_S3_V2 , AMAZON_SQS_V2 , and AZURE_BLOBSTORE_V2 , the SOURCE DELETION OPTIONfield offers two options:

  • NEVER: Never deletes any files after transfers.
  • ON_SUCCESS: Deletes all files and empty directories after transfer.

Set up a Pub/Sub push feed

To set up a Pub/Sub push feed, do the following:

1. Create a Pub/Sub push feed.
2. Specify the endpoint URL in a Pub/Sub subscription.

Create a Pub/Sub push feed

1. In the Google SecOps menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed namefield, enter a name for the feed.
4. In the Source typelist, select Google Cloud Pub/Sub Push.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF)as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n .
   • Asset namespace: the asset namespace .
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalizescreen, and then click Submit.
10. From the Detailstab, copy the feed endpoint URL from the Endpoint Informationfield. You need this endpoint URL to create a push subscription in Pub/Sub.
11. Optional: Click the Feed Enabledtoggle to disable the feed. The feed is enabled by default.
12. Click Done.

Specify the endpoint URL

After you create a Pub/Sub push feed, specify the endpoint URL as follows: In Pub/Sub, create a push subscription, specify the HTTPS endpoint. Select Enable authenticationand a service account.

1. Create a push subscription in Pub/Sub. For more information about how to create a push subscription, see Create push subscriptions .
2. Specify the endpoint URL, which is available in the Google Cloud Pub/Sub push feed.
3. Select Enable authentication, and select a service account.

Set up an Amazon Data Firehose feed

To set up an Amazon Data Firehose feed, do the following:

1. Create an Amazon Data Firehose feed and copy the endpoint URL and secret key.
2. Create an API key to authenticate to Google SecOps. You can also reuse your existing API key to authenticate to Google SecOps.
3. Specify the endpoint URL in Amazon Data Firehose.

Create an Amazon Data Firehose feed

1. In the Google SecOps menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed namefield, enter a name for the feed.
4. In the Source typelist, select Amazon Data Firehose.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF)as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n .
   • Asset namespace: the asset namespace .
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalizescreen, and then click Submit.
10. Click Generate Secret Keyto generate a secret key to authenticate this feed.
11. Copy and store the secret key as you cannot view this secret again. You can generate a new secret key again, but regeneration of the secret key makes the previous secret key obsolete.
12. On the Detailstab, copy the feed endpoint URL from the Endpoint Informationfield. You need this endpoint URL when you specify the destination settings for your delivery stream in Amazon Data Firehose.
13. Optional: Click the Feed Enabledtoggle to disable the feed. The feed is enabled by default.
14. Click Done.

Create an API key for the Amazon Data Firehose feed

To create an API key for the Amazon Data Firehose feed, do the following: 1. Go to the Google Cloud console Credentials page. 1. Click Create credentials, and then select API key. 1. Restrict the API key access to the Chronicle API.

Specify the endpoint URL

In Amazon Data Firehose, specify the HTTPS endpoint and access key, as follows:

1. Append the API key to the feed endpoint URL and specify this URL as the HTTP endpoint URL in the following format:

     ENDPOINT_URL 
   ?key= API_KEY 
    
   

   Replace the following:

   • ENDPOINT_URL : the feed endpoint URL.
   • API_KEY : the API key to authenticate to Google SecOps.

2. For the access key, specify the secret key that you obtained when you created the Amazon Data Firehose feed.

Set up an HTTPS webhook feed

To set up an HTTPS webhook feed, do the following:

1. Create an HTTPS webhook feed and copy the endpoint URL and secret key.
2. Create an API key that is specified with the endpoint URL. You can also reuse your existing API key to authenticate to Google SecOps.
3. Specify the endpoint URL in your application.

Prerequisites

• Ensure that a Google Cloud project for Google SecOps is configured and the Chronicle API is enabled for the project.

• Link a Google SecOps instance to Google Cloud services .

Create an HTTPS webhook feed

1. In the Google SecOps menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed namefield, enter a name for the feed.
4. In the Source typelist, select Webhook.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF)as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n .
   • Asset namespace: the asset namespace .
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalizescreen, and then click Submit.
10. Click Generate Secret Keyto generate a secret key to authenticate this feed.
11. Copy and store the secret key as you cannot view this secret again. You can generate a new secret key again, but regeneration of the secret key makes the previous secret key obsolete.
12. From the Detailstab, copy the feed endpoint URL from the Endpoint Informationfield. You need to specify this endpoint URL in your client application.
13. Optional: Click the Feed Enabledtoggle to disable the feed. The feed is enabled by default.
14. Click Done.

Create an API key for the webhook feed

1. Go to the Google Cloud console Credentials page.
2. Click Create credentials, and then select API key.
3. Restrict the API key access to the Chronicle API.

Specify the endpoint URL

1. In your client application, specify the HTTPS endpoint, which is available in the webhook feed.

2. Enable authentication by specifying the API key and secret key as part of the custom header in the following format:

   X-goog-api-key = API_KEY

   X-Webhook-Access-Key = SECRET

   We recommend that you specify the API key as a header instead of specifying it in the URL. If your webhook client doesn't support custom headers, you can specify the API key and secret key by using query parameters in the following format:

     ENDPOINT_URL 
   ?key= API_KEY 
   &secret= SECRET 
    
   

   Replace the following:

   • ENDPOINT_URL : the feed endpoint URL.
   • API_KEY : the API key to authenticate to Google SecOps.
   • SECRET : the secret key that you generated to authenticate the feed.

Grant access to the Google SecOps service account

1. In the Google Cloud console, go to the Cloud Storage Buckets page.
   

   Go to Buckets

2. Grant access to the service account to the relevant Cloud Storage objects.

   • To grant read permission to a specific file , complete the following steps:

     1. Select the file and click Edit access.
     2. Click Add principal.
     3. In the New principalsfield, enter the name of the Google SecOps service account.
     4. Assign a role that contains the read permission to the Google SecOps service account. For example, Storage Object Viewer ( roles/storage.objectViewer ). This can only be done if you've not enabled uniform bucket-level access .
     5. Click Save.

   • To grant read permission to multiple files , grant access at the bucket level as follows:

     • For "feedSourceType": "GOOGLE_CLOUD_STORAGE" :

       1. Add the Google SecOps service account as a principal to your storage bucket and grant it the IAM Storage Object Viewer ( roles/storage.objectViewer ) role.
       2. If you configure the feed to delete source files, you must add the Google SecOps service account as a principal on your bucket and grant it the IAM Storage Object Admin ( roles/storage.objectAdmin ) role.

     • For "feedSourceType": "GOOGLE_CLOUD_STORAGE_V2" :

       1. Grant either of these roles:

          • Storage Object Viewer( roles/storage.objectViewer ) if the transfer is to another Cloud Storage bucket.
          • Storage Object Creator( roles/storage.objectCreator ) if the transfer is to a file system.

       2. Grant either of these roles:

          • Storage Legacy Bucket Writer( roles/storage.legacyBucketWriter ) if object delete permission is required.
          • Storage Legacy Bucket Reader( roles/storage.legacyBucketReader ) if object delete permission is not required.

     • For "feedSourceType": "GOOGLE_CLOUD_STORAGE_EVENT_DRIVEN" :

       1. Grant either of these roles:

          • Storage Object Viewer( roles/storage.objectViewer ) if the transfer is to another Cloud Storage bucket.
          • Storage Object Creator( roles/storage.objectCreator ) if the transfer is to a file system.

       2. Grant either of these roles:

          • Storage Legacy Bucket Writer( roles/storage.legacyBucketWriter ) if object delete permission is required.
          • Storage Legacy Bucket Reader( roles/storage.legacyBucketReader ) if object delete permission is not required.

Configure VPC Service Controls

If VPC Service Controls is enabled, an ingress rule is required to provide access to the Cloud Storage bucket.

The following Cloud Storage methods must be allowed in the ingress rule:

• google.storage.objects.list . Required for a single file feed.
• google.storage.objects.get . Required for feeds that require directory or subdirectory access.
• google.storage.objects.delete . Required for feeds that require deletion of the source file.

Sample ingress rule

 - ingressFrom:
  identities:
    - serviceAccount:8911409095528497-0-account@partnercontent.gserviceaccount.com
  sources:
  - accessLevel: "*"
  ingressTo:
  operations:
  - serviceName: storage.googleapis.com
    methodSelectors:
    - method: google.storage.objects.list
    - method: google.storage.objects.get
    - method: google.storage.objects.delete
  resources:
  - projects/PROJECT_ID 

Feed status

You can monitor the status of the feed on the initial Feedspage, where feeds can have the following statuses:

• Active: Feed is configured and ready to ingest data into your Google SecOps account.
• InProgress: Google SecOps attempts to pull data from the configured third party.
• Completed: Data successfully retrieved by this feed.
• Archived: Disabled feed.

• Failed: Feed is failing to successfully fetch data. This is likely due to a configuration issue. Click the question to display the configuration error. Once you've corrected the error and resubmitted the feed, return to the Feedspage to determine whether or not the feed is now working.

Edit feeds

On the Feedspage, you can edit an existing feed, as follows:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click Edit Feed. You can now modify the input parameters for the feed and resubmit it to Google SecOps, which will attempt to use the updated feed.

Enable and disable feeds

In the Statuscolumn, enabled feeds are labeled as Active, InProgress, Completed, or Failed. Disabled fields are labeled as Archived. For a description, see the feed status .

From the Feedspage, you can enable or disable any of the existing feeds:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Optional: Click the Feed Enabledtoggle to disable the feed.

3. Optional: Click the Disable Feedtoggle to disable the feed. The feed is now labeled as Archived.

Delete feeds

On the Feedspage, you can also delete an existing feed:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click Delete Feed. The DELETE FEEDwindow opens. To permanently delete the feed, click Yes, delete it.

Control the rate of ingestion

When the data ingestion rate for a tenant reaches a certain threshold, Google Security Operations restricts the rate of ingestion for new data feeds to prevent a source with a high ingestion rate from affecting the ingestion rate of another data source. In this case, there is a delay but no data is lost. The ingestion volume and tenant's usage history determine the threshold.

You can request a rate limit increase by contacting Cloud Customer Care .

Troubleshooting

On the Feedspage, you can view details such as source type, log type, feed ID, and status of the existing feeds, as follows:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click View Feed. A dialog appears showing the feed details. For a failed feed, you can find error details under Details> Status.

For a failed feed, the details include the cause of the error and steps to fix it. The following table describes the error messages that you might encounter when working with data feeds:

Need more help? Get answers from Community members and Google SecOps professionals.