Set up and manage Data Processing Pipelines

The Data Processing Pipelinefeature provides robust control over Google Security Operations data ingestion. Data Processing Pipelines let you manipulate incoming data before it's parsed by Google Security Operations. For example, filter and transform events, or redact sensitive values. This process can help optimize data for Google SecOps, reduce costs, protect sensitive information, and improve compatibility.

This document shows how to use the Bindplane console to configure a connection to a Google SecOps destination instance, create a new Google SecOps Pipeline, set up the Data Processing Pipeline (streams and processors), roll it out to initiate data processing, and view pipeline streams and processors in the Google SecOps console. Example use cases include:

• Remove empty key-value pairs from raw logs.
• Redact sensitive data.
• Add ingestion labels from raw log content.
• In multi-instance environments, apply ingestion labels to direct-ingestion log data to identify the source stream instance (such as Google Cloud Workspace).
• Filter Palo Alto Cortex data by field values.
• Reduce SentinelOne data by category.
• Extract host information from feeds and direct-ingestion logs and map it to the ingestion_stream field for Cloud Monitoring.

You can configure Data Processing Pipelines for both on-premises and cloud data streams, using either the Bindplane management console or directly using the public Google SecOps Data Pipeline APIs .

A Data Processing Pipeline consists of the following elements:

• Streams: One or more streams feed data into the data processing pipeline. Each stream is configured for a specific stream type.
• Processor node: A Data Processing Pipeline has one Processor nodethat contains one or more processors. Each processor specifies an action to perform on the data (for example, filter, transform, and redact) as it flows through the pipeline.
• Destination: The Google SecOps destination instance is where the processed data is sent.

Prerequisites

If you intend to use the Bindplane console to manage your Google SecOps Data Processing Pipeline, perform the following steps:

1. In the Google Security Operations console, grant the installer the required predefined administrator roles. For details, see Assign the Project IAM Admin role in a dedicated project . Under Assign Roles, select the following predefined Identity and Access Management roles:

   • Chronicle API Admin ( roles/chronicle.admin )

2. Install the Bindplane Server console. For SaaS or on-premises, see Install the Bindplane Server console .

3. In the Bindplane console, connect a Google SecOps destination instance to your Bindplane project . For details, see Connect to a Google SecOps instance .

Manage low-volume SecOps data acknowledgment delays

Ingestion API users who configure their own agent, may experience a potential increase in acknowledgment time for low volume SecOps Pipelines in the Data processing pipeline. Latency averages may rise from 700 ms up to 2 seconds. In such a case. Increase timeout periods and memory as needed. Acknowledgment time drops when data throughput exceeds 4 MBps.

Connect to a Google SecOps instance

Connect to a Google SecOps instance, which will serve as the destination for the output from your Data Processing Pipelines.

To connect to a Google SecOps instance using the Bindplane console:

1. In the Bindplane console, go to the Manage your projectpage.
2. Go to the Integrationscard and click Connect to Google SecOps.

3. In the Edit Integrationwindow that opens, enter the details of the Google SecOps destination instance, that will ingest the output from your Data Processing Pipelines, as follows:

   Field	Description
   Region	The region of your Google SecOps instance. To find the instance, go to the Google Cloud console, navigate to the Google Security Operationspage, and click Instance details.
   Customer ID	The customer ID of your Google SecOps instance. In the Google SecOps console, go to Settings > Profile > Organization Details.
   Google Cloud project number	The Google Cloud Project Number of your Google SecOps instance. To find the project number in the Google SecOps console, go to Settings > Profile > Organization Details.
   Credentials	The Service Account credentials required credentials are the JSON value needed to authenticate and access the Google SecOps Data Pipeline APIs. The Service Account must be located in the same Google Cloud project as your Google SecOps instance and requires the Chronicle API Admin role ( roles/chronicle.admin ) privileges. You can get this JSON value from the Google Service Account credential file. For information about how to create a Service Account and download the JSON file, see Create and delete Service Account keys .

4. Click Connect. If your connection details are correct and you successfully connect to Google SecOps, you can expect the following:

   • A connection to the Google SecOps instance is opened.
   • The first time you connect, you can see the SecOps Pipelinesappears in the Bindplane console.
   • The Bindplane console now displays any Data Processing Pipelines you previously set up for this instance using the API. The system converts some processors you configured using the API into Bindplane processors, and displays others in their raw OpenTelemetry Transformation Language (OTTL) format. You can use the Bindplane console to edit pipelines and processors previously set up using the API.

5. After you successfully create a connection to a Google SecOps instance, you can create a SecOps Pipeline and set up the Data Processing Pipeline. For details, see Set up a Data Processing Pipeline using the Bindplane console .

Set up a Data Processing Pipeline using the Bindplane console

Using the Bindplane console, you can manage your Google SecOps Data Processing Pipelines, including pipelines set up using the API.

Follow these steps to provision and deploy a new log processing pipeline in Google SecOps, typically using the Bindplane console.

1. Create a new SecOps Pipeline
2. Configure a Data Processing Pipeline
   1. Configure streams
   2. Configure processors
3. Stage the deployment of a Data Processing Pipeline

Create a new Google SecOps pipeline

A Google SecOps pipeline is a container for you to configure one Data Processing Pipeline. To create a new Google SecOps pipeline container, do the following:

1. In the Bindplane console, click the SecOps Pipelinestab to open the SecOps Pipelinespage.
2. Click Create SecOps Pipeline.
3. In the Create new SecOps Pipelinewindow, set the SecOps Pipeline typeto Google SecOps(default).
4. Enter a SecOps Pipeline nameand Description.
5. Click Create. The new pipeline container is now displayed on the SecOps Pipelinespage. Proceed to configure the data processing pipeline streams and processors within this container.

Configure a Data Processing Pipeline

A Data Processing Pipeline specifies data Streams to ingest and Processors (for example, filter, transform, or redact) to manipulate the data as it flows to the Google SecOps Destination instance.

A Pipelineconfiguration card is a visualization of the data processing pipeline where you can configure the data Streamsand the Processornode. The Processornode consists of processors that manipulate the data as it flows to the Google SecOps Destinationinstance.

To configure a Data Processing Pipeline, first Create a new SecOps Pipeline , and then do the following:

1. In the Bindplane console, click the SecOps Pipelinestab to open the SecOps Pipelinespage.
2. Select the SecOps Pipeline where you want to configure the new Data Processing Pipeline. The Pipelineconfiguration card opens.

3. Configure the following:

   1. A Stream. See Configure streams for details.

   2. The Processor node:

      • To add a processor using the Bindplane console, see Configure processors for details.
      • Some custom processors let you edit their raw OTTL code directly.

4. Once these configurations are complete, see Roll out a Data Processing Pipeline to begin processing the data.

Configure streams

A Stream ingests data according to its configured specifications, and feeds it into the pipeline. A Data Processing Pipeline can have one or more Streams, each configured for a different stream.

To add a Stream, do the following:

1. In the Pipelineconfiguration card, click add Add Streamto open the Create Streamwindow.

2. In the Create SecOps Streamwindow, enter details for these fields:

   Field

   Description

   Log type

   Log type of the data to ingest.
   Select the log typeto ingest. For example, "CrowdStrike Falcon (CS_EDR)".
   
   Note: You can't select a log type with a warning warningicon.
   A warning icon indicates that the log type is already configured in another stream (in this pipeline or another pipeline in your Google SecOps instance).
   If you want to use such a log type, you must first delete it from the other stream configuration.
   To find the other stream configuration where the log type is configured, see Filter SecOps Pipeline configurations .

   Ingestion method

   Ingestion method to use to ingest the data for the selected Log type.
   These ingestion methods where previously defined for your Google SecOps instance.
   Select one of the following:
   

   • All Ingestion Methods
     < Note: Selecting All Ingestion Methodsfor a Log typeprevents you from adding subsequent Streams that use specific Ingestion Methodsfor that same Log type.
   • Select a specific ingestion method.
     For example, one of the following: " Cloud Native Ingestion", " Feed", " Ingestion API", or " Workspace".
     • Note: Selecting All Ingestion Methodsfor a Log typeprevents you from adding subsequent Streams that use specific Ingestion Methodsfor that same Log type.
       You will still be able to select other unconfigured specific Ingestion methodsfor this Log type.
     • If you select Feedas the Ingestion Method, the subsequent field displays a list of available feeds. Select the relevant Feedto define it as the ingestion stream for this configuration.

   Feed

   The specific feed configuration to use as the ingestion stream for data.
   If you select Feedin the Ingestion methodfield, the Feedfield displays a list of feed names (previously defined for your Google SecOps instance) for the selected Log type.
   Select a specific Feed from the list.
   
   Note: To see a list of your feeds in your Google SecOps console, go to Settings > Feeds table.

3. Click Add Streamto save the new stream.

   • The new data Streamis immediately displayed on the Pipelineconfiguration card.
   • It is automatically connected to the Processornode and the Google SecOps Destination.

Filter SecOps Pipeline configurations

The search bar on the SecOps Pipelinespage lets you filter your SecOps Pipelines (Data Processing Pipelines) based on multiple configuration elements. You can locate pipelines by searching for specific criteria, such as log type, ingestion method, or feed name. You can use the following syntax to filter: logtype:value , ingestionmethod:value and feed:value .

For example, to identify stream configurations containing a specific log type, enter the filter logtype: in the search bar and select the log type from the resulting list.

Configure processors

A Data Processing Pipeline has one Processor node, containing one or more processors. Each processor manipulates the stream data sequentially:

1. The first processor processes the raw stream data.
2. The resulting output from the first processor is then processed by the next processor.
3. This sequence continues for all subsequent processors in the order they appear in the Processorspane, with the output of one becoming the input of the next.

Configure the Processor nodeby adding, removing, or changing the sequence of one or more processors.

To add a processor, follow these steps:

1. In the Pipelineconfiguration card, click the Processornode to open the Edit Processorswindow.
   The Edit Processorswindow consists of the following panes, arranged by data flow:

   • Input pane (or source pane): Recent incoming stream log data (before processing)
   • Configuration pane (or processor list): Processors and their configurations
   • Output pane (or result pane): Recent outgoing result log data (after processing)

   If the pipeline has been rolled out before, then the system shows the recent incoming log data (before processing) and the recent outgoing log data (after processing) in the panes.

2. To add a processor, click Add Processorto display the processor list. For your convenience, the processor list is grouped by processor type.(To organize the processor list, you can add your own bundles by selecting one or more processors and clicking Add new Processor bundles.)

3. Select a Processorto add from the list.

4. Configure the processor as necessary.

5. Click Saveto save the processor configuration in the Processornode.

The system tests the new processor configuration by processing a fresh sample of the incoming stream log data (from the Input pane) and displays the outgoing result data (in the Output pane).

Deploy a Data Processing Pipeline

Once the stream and processor configurations are complete, deploy the pipeline to begin processing data.

To deploy a Data Processing Pipeline, click Start rollout. This activates the Data Processing Pipeline and allows Google's secure infrastructure to begin processing data according to the Data Processing Pipeline configuration.

If the deployment is successful, the Data Processing Pipeline configuration version number is incremented and displayed next to the Data Processing Pipeline name.

View Data Processing Pipeline information from the Google SecOps console

The following sections describe how to view Data Processing Pipeline information from the Google SecOps console:

View configured feeds

The Feedspage shows all the feeds that you configured.

1. In the Google SecOps console, go to Settings > Feeds. The main page displays all your configured feeds.
2. Hold the pointer over each row to display the ⋮ Moremenu. From the menu, you can view feed details, edit, disable, or delete the feed.
3. Click View Detailsto view the details window.
4. Click Open in Bindplaneto open the stream configuration for that feed in the Bindplane console.

View Data Processing Pipeline information from the Logtypes page

The Logtypespage shows all available log types. To view Data Processing Pipeline details:

1. In the Google SecOps console, go to Settings > Logtypes. The main page displays all your log types.
2. Hold the pointer over each row to display the ⋮ Moremenu. From the menu, you can view logtype details.
3. Click View Data Processingto view the details window.
4. Click Open in Bindplaneto open the processor configuration for that processor in the Bindplane console.

Use Google SecOps Data Pipeline APIs

The Google SecOps Data Pipeline APIs allow you to manage your Data Processing Pipelines. The APIs cover all the Data Pipeline functionality, such as creating, updating, deleting, and listing pipelines and associated feeds and log types within them.

Need more help? Get answers from Community members and Google SecOps professionals.