Integrate OpenSearch with Google SecOps

Integration version: 1.0

This document explains how to integrate OpenSearch with Google Security Operations.

Use cases

The OpenSearchintegration addresses the following security orchestration and investigation use cases:

Automated threat hunting: Periodically search across all OpenSearch indexes for specific Indicators of Compromise (IoCs) or suspicious patterns using Lucene and DSL queries.
Centralized alert ingestion: Ingest documents from OpenSearch as actionable alerts into Google SecOps, ensuring that security events from your data lake are prioritized and investigated.
Contextual data retrieval: Enrich existing security cases by automatically pulling related logs or historical data from OpenSearch to provide analysts with deeper context during an investigation.
Flexible data mapping: Dynamically map severity levels and environment tags from OpenSearch documents to Google SecOps standards to maintain consistency across different log sources.

Before you begin

Before you configure the integration in the Google SecOps platform, verify that you have the following:

CA Certificate file: A valid Certificate Authority file to secure the connection.
Latest integration version: Ensure you are using the most recent version of the OpenSearch integration.
Base64-encoded certificate: The content of your CA certificate file must be converted into a Base64 string before configuration.

To configure the integration with a CA certificate, complete the following steps:

Open the integration configuration parameters page.
Paste the Base64-encoded string into the CA Certificate Filefield.
Select the Verify SSLcheckbox.
Click Testto verify that the integration is successfully configured.

Integration parameters

The OpenSearchintegration requires the following parameters:

Parameter	Description
`Server Address`	Required. The endpoint URL of the OpenSearch instance, including the protocol and port (for example, `https://10.0.0.1:9200` ).
`Username`	Optional. The username used to authenticate with the OpenSearch instance.
`Password`	Optional. The password associated with the specified username.
`API Token`	Optional. The API token used for authentication as an alternative to a username and password.
`Authenticate`	Optional. If selected, the integration authenticates the connection to the OpenSearch server using the provided credentials.
`Verify SSL`	Optional. If selected, the integration validates the SSL certificate when connecting to the OpenSearch server.
`CA Certificate File`	Optional. The content of the Certificate Authority (CA) certificate file in PEM format, used to verify the server's SSL certificate.

For instructions about how to configure an integration in Google SecOps, see Configure integrations .

You can make changes at a later stage, if needed. After you configure an integration instance, you can use it in playbooks. For more information about how to configure and support multiple instances, see Supporting multiple instances .

Actions

For more information about actions, see Respond to pending actions from Your Workdesk and Perform a manual action .

Advanced ES Search

Use the Advanced ES Searchaction to execute a pre-configured search query on the OpenSearch server and retrieve the results in a dictionary format.

This action doesn't run on Google SecOps entities.

Action inputs

The Advanced ES Searchaction requires the following parameters:

Parameter	Description
`Index`	Optional. The search pattern for the OpenSearch index (for example, `smp_playbooks-2019.06.13` ). You can use wildcards to search multiple indexes (for example, `smp_playbooks-` ). The default value is `` .
`Query`	Optional. The search query to perform using Lucene syntax (for example, `level:error OR level:warning` ). The default value is `*` .
`Limit`	Optional. The maximum number of documents to return. Set this value to `0` for no limit. The default value is `100` .
`Display Field`	Optional. A comma-separated list of specific fields to return. The default value is `*` and returns all fields.
`Search Field`	Optional. The specific field to search when `Query` doesn't include a target field name. This acts as the default search target for free-text queries. The default value is `_all` , which searches across all indexed fields.
`Timestamp Field`	Optional. The name of the field used for time-based filtering. If both `Earliest Date` and `Oldest Date` are empty, the action doesn't perform time-based filtering. The default value is `@timestamp` .
`Oldest Date`	Optional. The start date for the search. The action returns records that occurred on or after this timestamp. You can provide an exact UTC timestamp ( `YYYY-MM-DDTHH:MM:SSZ` ) or relative date-math (for example, `now-1d` ). The default value is `now-1d` .
`Earliest Date`	Optional. The end date for the search. The action returns records that occurred on or before this timestamp. You can provide an exact UTC timestamp ( `YYYY-MM-DDTHH:MM:SSZ` ) or relative date-math (for example, `now` ). The default value is `now` .

Action outputs

The Advanced ES Searchaction provides the following outputs:

Action output type	Availability
Case wall attachment	Not available
Case wall link	Not available
Case wall table	Not available
Enrichment table	Not available
JSON result	Not available
Output messages	Available
Script result	Available

Script result

The following table lists the value for the script result output when using the Advanced ES Searchaction:

Script result name	Value
`results`	`RESULTS`

DSL Search

Use the DSL Searchaction to perform a comprehensive search across all OpenSearch indexes and retrieve results in a dictionary format.

This action doesn't run on Google SecOps entities.

Action inputs

The DSL Searchaction requires the following parameters:

Parameter Description

Index

Optional.

The search pattern for the OpenSearch index (for example, smp_playbooks-2019.06.13 ). You can use wildcards to search multiple indexes (for example, smp_playbooks-* ).

The default value is * .

Query

Optional.

The search query to perform using Lucene syntax (for example, level:error OR level:warning ).

The default value is * .

Limit

Optional.

The maximum number of documents to return. Set this value to 0 for no limit.

The default value is 100 .

Action outputs

The DSL Searchaction provides the following outputs:

Action output type	Availability
Case wall attachment	Not available
Case wall link	Not available
Case wall table	Not available
Enrichment table	Not available
JSON result	Available
Output messages	Available
Script result	Available

JSON result

The following example shows the JSON result output received when using the DSL Searchaction:

  [ 
  
 { 
  
 "_score" 
 : 
  
 0.2876821 
 , 
  
 "_type" 
 : 
  
 "person" 
 , 
  
 "_id" 
 : 
  
 "2" 
 , 
  
 "_source" 
 : 
  
 { 
  
 "lastname" 
 : 
  
 "Smith" 
 , 
  
 "name" 
 : 
  
 "John" 
 , 
  
 "job_description" 
 : 
  
 "Systems administrator" 
  
 }, 
  
 "_index" 
 : 
  
 "accounts" 
  
 }, 
  
 { 
  
 "_score" 
 : 
  
 0.28582606 
 , 
  
 "_type" 
 : 
  
 "person" 
 , 
  
 "_id" 
 : 
  
 "1" 
 , 
  
 "_source" 
 : 
  
 { 
  
 "lastname" 
 : 
  
 "Doe" 
 , 
  
 "name" 
 : 
  
 "John" 
 , 
  
 "job_description" 
 : 
  
 "Systems administrator and Linux specialist" 
  
 }, 
  
 "_index" 
 : 
  
 "accounts" 
  
 } 
 ]

Script result

The following table lists the value for the script result output when using the DSL Searchaction:

Script result name	Value
`results`	`RESULTS`

Ping

Use the Pingaction to test the connectivity to OpenSearch.

This action doesn't run on Google SecOps entities.

Action inputs

None.

Action outputs

The Pingaction provides the following outputs:

Action output type	Availability
Case wall attachment	Not available
Case wall link	Not available
Case wall table	Not available
Enrichment table	Not available
JSON result	Not available
Output messages	Available
Script result	Available

Script result

The following table lists the value for the script result output when using the Pingaction:

Script result name	Value
is_success	`true` or `false`

Simple ES Search

Use the Simple ES Searchaction to perform a basic search across all indexed fields in OpenSearch and retrieve the results in a dictionary format.

This action doesn't run on Google SecOps entities.

Action inputs

The Simple ES Searchaction requires the following parameters:

Parameter Description

Index

Optional.

The search pattern for the OpenSearch index (for example, smp_playbooks-2019.06.13 ). You can use wildcards to search multiple indexes (for example, smp_playbooks-* ).

The default value is * .

Query

Optional.

The search query to perform using Lucene syntax (for example, level:error OR level:warning ).

The default value is * .

Limit

Optional.

The maximum number of documents to return. Set this value to 0 for no limit.

The default value is 100 .

Action outputs

The Simple ES Searchaction provides the following outputs:

Action output type	Availability
Case wall attachment	Not available
Case wall link	Not available
Case wall table	Not available
Enrichment table	Not available
JSON result	Available
Output messages	Available
Script result	Available

JSON result

The following example shows the JSON result output received when using the Simple ES Searchaction:

  [{ 
  
 "_score" 
 : 
  
 0.2876821 
 , 
  
 "_type" 
 : 
  
 "person" 
 , 
  
 "_id" 
 : 
  
 "2" 
 , 
  
 "_source" 
 : 
  
 { 
  
 "lastname" 
 : 
  
 "Smith" 
 , 
  
 "name" 
 : 
  
 "John" 
 , 
  
 "job_description" 
 : 
  
 "Systems administrator" 
  
 }, 
  
 "_index" 
 : 
  
 "accounts" 
  
 }, 
  
 { 
  
 "_score" 
 : 
  
 0.28582606 
 , 
  
 "_type" 
 : 
  
 "person" 
 , 
  
 "_id" 
 : 
  
 "1" 
 , 
  
 "_source" 
 : 
  
 { 
  
 "lastname" 
 : 
  
 "Doe" 
 , 
  
 "name" 
 : 
  
 "John" 
 , 
  
 "job_description" 
 : 
  
 "Systems administrator and Linux specialist" 
  
 }, 
  
 "_index" 
 : 
  
 "accounts" 
  
 } 
  
 ]

Script result

The following table lists the value for the script result output when using the Simple ES Searchaction:

Script result name	Value
`results`	`RESULTS`

Connectors

For more information about how to configure connectors in Google SecOps, see Ingest your data (connectors) .

OpenSearch Connector

Use the OpenSearch Connectorto search specified OpenSearch indexes with a provided Lucene query and ingest documents as alerts into Google SecOps.

Connector inputs

The OpenSearch Connectorrequires the following parameters:

Parameter	Description
`Product Field Name`	Required. The name of the field where the product name is stored. The product name primarily impacts mapping. To streamline and improve the mapping process for the connector, the default value resolves to a fallback value that is referenced from the code. Any invalid input for this parameter resolves to a fallback value by default. The default value is `Product Name` .
`Event Field Name`	Required. The name of the field that determines the event name (subtype). The default value is `name` .
`Script Timeout (Seconds)`	Required. The timeout limit, in seconds, for the Python process that runs the current script. The default value is `180` .
`Server Address`	Required. The endpoint URL of the OpenSearch instance, including the protocol and port (for example, `https://10.0.0.1:9200` ).
`Username`	Optional. The username used to authenticate with the OpenSearch instance.
`Password`	Optional. The password associated with the specified username.
`Authenticate`	Optional. If selected, the integration authenticates the connection to the OpenSearch server using the provided credentials.
`API Token`	Optional. The API token used for authentication as an alternative to a username and password.
`Verify SSL`	Optional. If selected, the integration validates the SSL certificate when connecting to the OpenSearch server.
`Alert Name Field`	Required. The name of the field that contains the alert name (for example, `_source_alert_info_alert` ).
`Timestamp Field`	Required. The name of the field that contains the timestamp (for example, `_source_@timestamp` ).
`Environment Field Name`	Optional. The name of the field where the environment name is stored. If the environment field is missing, the connector uses the default value. The default value is `""` .
`Indexes`	Optional. The index pattern to search. Use `*` to search across all indexes.
`Query`	Optional. The search query to perform using Lucene syntax.
`Alerts Count Limit`	Required. The maximum number of alerts to pull in a single cycle. The default value is `20` .
`Max Days Backwards`	Required. The number of days prior to now to retrieve alerts. This parameter can apply to the initial connector iteration after you enable the connector for the first time, or the fallback value for an expired connector timestamp. The default value is `1` .
`Severity Field Name`	Optional. The field name used to extract the severity value for the alert. If the field contains numeric values (integers or floats), the connector automatically maps them to Google SecOps severity levels (for example, `100` is `Critical` ). If the field contains string values, you must define the mapping rules in the `severity_map_config.json` file.
`Proxy Server Address`	Optional. The address of the proxy server to use.
`Proxy Username`	Optional. The username used to authenticate with the proxy server.
`Proxy Password`	Optional. The password used to authenticate with the proxy server.
`Environment Regex Pattern`	Optional. A regular expression pattern to run on the value found in the `Environment Field Name` field. This parameter lets you manipulate the environment field using the regular expression logic. Use the default value `.*` to retrieve the required raw `Environment Field Name` value. If the regular expression pattern is null or empty, or the environment value is null, the final environment result is the default environment.

OpenSearch DSL Connector

Use the OpenSearch DSL Connectorto ingest documents as alerts into Google SecOps by making REST API calls using Domain Specific Language (DSL) queries.

Connector inputs

The OpenSearch DSL Connectorrequires the following parameters:

Parameter	Description
`Product Field Name`	Required. The name of the field where the product name is stored. The product name primarily impacts mapping. To streamline and improve the mapping process for the connector, the default value resolves to a fallback value that is referenced from the code. Any invalid input for this parameter resolves to a fallback value by default. The default value is `Product Name` . The default value is `device_product` .
`Event Field Name`	Required. The name of the field that determines the event name (subtype). The default value is `name` .
`Script Timeout (Seconds)`	Required. The timeout limit, in seconds, for the Python process that runs the current script. The default value is `60` .
`Server Address`	Required. The endpoint URL of the OpenSearch instance, including the protocol (for example, `https://10.0.0.1` ).
`Port`	Required. The port used to connect to the OpenSearch API server.
`Query`	Required. The DSL query used for the search in JSON format. To improve connector stability, it is recommended to include a sorting timestamp key in ascending order within your query.
`Index`	Required. The index used for the search. Use `_all` to search across all indexes.
`Timestamp Field`	Required. The name of the field that contains the timestamp (for example, `_source_@timestamp` ).
`Alert Field Name`	Required. The name of the field that contains the alert name (for example, `_source_info_alertname` ).
`Description Field`	Optional. The name of the field that contains the alert description (for example, `_source_alert_info_description` ).
`Severity`	Required. The static severity level to assign to all ingested alerts. This parameter is used if `Alert Severity` and `Severity Field Name` are empty. The default value is `Medium` .
`Alert Severity`	Optional. The severity to assign to alerts. This parameter has priority over `Severity Field Name` . If you want to use dynamic mapping using the `Severity Field Name` , leave this field empty.
`Severity Field Name`	Optional. The field name used to extract the severity value for the alert. If the field contains numeric values (integers or floats), the connector automatically maps them to Google SecOps severity levels (for example, `100` is `Critical` ). If the field contains string values, you must define the mapping rules in the `severity_map_config.json` file.
`Alerts Count Limit`	Optional. The maximum number of alerts to pull in a single cycle. The default value is `100` .
`Authenticate`	Optional. If selected, the connector authenticates the connection using the provided credentials.
`Username`	Optional. The username used to authenticate with the OpenSearch instance.
`Password`	Optional. The password used to authenticate with the OpenSearch instance.
`Use SSL`	Optional. If selected, the connector enables an SSL/TLS connection to the server.
`Proxy Server Address`	Optional. The address of the proxy server to use for the connection.
`Proxy Username`	Optional. The username used to authenticate with the proxy server.
`Proxy Password`	Optional. The password used to authenticate with the proxy server.
`Environment Field Name`	Optional. The name of the field where the environment name is stored. If the environment field is missing, the connector uses the default value. The default value is `""` .
`Environment Regex Pattern`	Optional. A regular expression pattern to run on the value found in the `Environment Field Name` field. This parameter lets you manipulate the environment field using the regular expression logic. Use the default value `.` to retrieve the required raw `Environment Field Name` value. If the regular expression pattern is null or empty, or the environment value is null, the final environment result is the default environment. The default value is `.` .

Notation support

The connector supports three different notations for field mapping. For example, to map event.type , you can provide _source_event_type , event_type , or event.type .

This applies to the following parameters:

Product Field Name
Event Field Name
Severity Field Name
Environment Field Name
Timestamp Field
Alert Field Name
Description Field (DSL connector only)

How to map severity in the connector

If the Severity Field Name contains numeric values (integers or floats), the connector maps them as follows:

Critical: 100
High: 80 ≤ x < 100
Medium: 60 ≤ x < 80
Low: 40 ≤ x < 60
Informational: x < 40

For string values, you must update the severity_map_config.json file in the connector scripts folder.

Example configuration

To map an event.severity field containing "Malicious", "Unknown", or "Benign":

Enter event.severity in the Severity Field Nameparameter.
Update severity_map_config.json :

  { 
  
 "event.severity" 
 : 
  
 { 
  
 "Malicious" 
 : 
  
 100 
 , 
  
 "Unknown" 
 : 
  
 60 
 , 
  
 "Benign" 
 : 
  
 -1 
  
 }, 
  
 "Default" 
 : 
  
 50 
 }

Need more help? Get answers from Community members and Google SecOps professionals.