Console
    Contact Us Start free

    ElasticsearchV7

    Integration version: 17.0

    Configure ElasticsearchV7 to work with Google Security Operations

    How to create API token

    To create a new API token, perform the following request:

      curl 
  
 -- 
 location 
  
 -- 
 request 
  
 POST 
  
 ' 
 http 
 : 
 //<server address>:<port>/_security/api_key' \ 
 -- 
 header 
  
 ' 
 Authorization 
 : 
  
 Basic 
  
 Base64 
 ( 
 username 
 , 
  
 password 
 ) 
 ' 
  
\ -- 
 header 
  
 ' 
 Content 
 - 
 Type 
 : 
  
 application 
 / 
 json 
 ' 
  
\ -- 
 data 
 - 
 raw 
  
 ' 
 { 
  
 "name" 
 : 
  
 "siemplify-integration" 
 , 
  
 "role_descriptors" 
 : 
  
 {} 
 } 
 ' 
 :

    Example of the response:

     {
  "id": "G1NIWnI",
  "name": "siemplify-integration",
  "api_key": "dSwyjWJ_Ql"
}
    1. We take the "id" and "api_key" parameter from the response.
    2. Use base64 encoding of "id" and "api_key" joined by a colon, like "id:api_key".
    3. The result is used as an API token in integration.

    Accessing Elasticsearch

    Google SecOps accesses Elasticsearch through RESTful API on TCP port 9200 by default. The Google SecOps server will need access to the relevant Elasticsearch nodes on TCP 9200 (default) or an alternate port if the default port was not used during Elasticsearch deployment.

    Configure ElasticsearchV7 integration in Google SecOps

    For detailed instructions on how to configure an integration in Google SecOps, see Configure integrations .

    Configure Elasticsearch integration with a CA certificate

    You can verify your connection with a CA certificate file if needed.

    Before you start, ensure you have the following:

    • The CA certificate file
    • The latest Elasticsearch integration version

    To configure the integration with a CA certificate, complete the following steps:

    1. Parse your CA certificate file into a Base64 String.
    2. Open the integration configuration parameters page.
    3. Insert the string in the CA Certificate Filefield.
    4. To test that the integration is successfully configured, select the Verify SSLcheckbox and click Test.

    Integration parameters

    Use the following parameters to configure the integration:

    Parameter Display Name Type Default Value Is Mandatory Description
    Instance Name
    		 String N/A No Name of the Instance you intend to configure integration for.
    Description
    		 String N/A No Description of the Instance.
    Server Address
    		 String x.x.x.x Yes IP address of the Elasticsearch 7.0.0 server.
    Username
    		 String N/A Yes The email address of the user which should be used to connect to Elasticsearch 7.0.0.
    Password
    		 Password N/A Yes The password of the according user.
    API Token
    		 Password N/A No Elasticsearch XPack API Token.
    Authenticate
    		 Checkbox Unchecked No N/A
    Verify SSL
    		 Checkbox Unchecked No Use this checkbox, if your Elasticsearch 7.0.0 connection requires an SSL verification (unchecked by default).
    CA Certificate File
    		 String N/A No CA Certificate File.
    Run Remotely
    		 Checkbox Unchecked No Check the field in order to run the configured integration remotely. Once checked, the option appears to select the remote user (agent).

    Actions

    Description

    An Elasticsearch test that is pre-made, and returns a word dictionary.

    Parameters

    Parameters Type Default Value Description
    Index
    		 String *

    Search pattern for a Elasticsearch index.

    In Elastic, index is like a DatabaseName, and data is stored across various indexes.This parameter defines in what index(es) to search. It can be an exact name ie: "smp_playbooks-2019.06.13" or you can use a wildcard to search by a pattern. e: "smp_playbooks-2019.06 "or "smp ".

    To learn more about Elasticsearch indexes visit https://www.elastic.co/blog/what-is-an-elasticsearch-index

    Query
    		 String *

    The search query to perform. It is in Lucene syntax.

    IE1: "*" (this is a wildcard that will return all record)

    IE2: "level:error"

    IE3: "level:information"

    IE4: "level:error OR level:warning"

    To learn more about Lucene syntax, visit https://www.elastic.co/guide/en/kibana/current/lucene-query.html#lucene-query\r\nhttps://www.elastic.co/guide/en/elasticsearch/reference/7.1/query-dsl-query-string-query.html#query-string-syntax

    Limit
    		 String 100

    Limits the document return count, ie: 10.

    0 = No limit.
    Display Field
    		 String *

    Limits the returned fields. Default "*" = Return all fields.

    You can state a single field. ie: "level"
    Search Field
    		 String _all

    Search field for free text queries (When query doesn't specify a field name).

    Default is "_all", which means all fields are searched. It is best to use proper Lucene syntax on "_all" fields, or textual search on a specific field.

    Ie1: Search Field = "_all". Query = "level:error" Query will return all records where "level" field, equals "error".

    Ie2: Search Field = "Message", query = " Login Alarm ". Query will return all records, which their "Message" field, contains the text "Login Alarm"
    Timestamp Field
    		 String @timestamp The name of the field to run time-based filtering against. Default is @timestamp. If both Earliest Date and Oldest Date are empty, no time-based filtering will occur.
    Oldest Date
    		 String now-1d

    Start date of the search. Search will return only records equal or after this point in time.

    Input may be in exact UTC:

    Format: YYYY-MM-DDTHH:MM:SSZ

    ie: 2019-06-04T10:00:00Z

    Input may also be in relative form (using date-math): tie: "now", "now-1d", "now-1d/d", "now-2h/h"

    To learn more about date-math visit https://www.elastic.co/guide/en/elasticsearch/reference/7.1/common-options.html#date-math

    Earliest Date
    		 String now

    End date of the search. Search will return only records equal or before this point in time.

    Input may be in exact UTC:

    Format: YYYY-MM-DDTHH:MM:SSZ

    ie: 2019-06-04T10:00:00Z

    Input may also be in relative form (using date-math):

    ie: "now", "now-1d", "now-1d/d", "now-2h/h"

    To learn more about date-math visit https://www.elastic.co/guide/en/elasticsearch/reference/7.1/common-options.html#date-math

    Run On

    This action runs on all entities.

    Action Results

    Script Result
    Script Result Name Value Options Example
    results
    		 N/A N/A

    Description

    Searches through everything in Elasticsearch and returns back results in a dictionary format. This action supports only queries without time range, if you want to use time range in your query use Advanced ES Search action.

    Parameters

    Parameters Type Default Value Description
    Index
    		 String *

    Search pattern for a Elasticsearch index.

    In Elasticsearch, index is like a DatabaseName, and data is stored across various indexes.

    This param defines in what index(es) to search. It can be an exact name ie: \"smp_playbooks-2019.06.13\"\r\nor you can use a ( ) wildcard to search by a pattern. e: \"smp_playbooks-2019.06 \" or \"smp*\".

    To learn more about Elasticsearch indexes visit https://www.elastic.co/blog/what-is-an-elasticsearch-index

    Query
    		 String *

    The search query to perform. It is in Lucene syntax.

    IE1: \"*\" (this is a wildcard that will return all record)

    IE2: \"level:error\"

    IE3: \"level:information\"

    IE4: \"level:error OR level:warning\"

    To learn more about Lucene syntax, visit https://www.elastic.co/guide/en/kibana/current/lucene-query.html#lucene-query\r\nhttps://www.elastic.co/guide/en/elasticsearch/reference/7.1/query-dsl-query-string-query.html#query-string-syntax

    Limit
    		 String 100

    Limits the document return count, ie: 10.

    0 = No limit

    Run On

    This action runs on all entities.

    Action Results

    Script Result
    Script Result Name Value Options Example
    results
    		 N/A N/A
    JSON Result
      [ 
  
 { 
  
 "_score" 
 : 
  
 0.2876821 
 , 
  
 "_type" 
 : 
  
 "person" 
 , 
  
 "_id" 
 : 
  
 "2" 
 , 
  
 "_source" 
 : 
  
 { 
  
 "lastname" 
 : 
  
 "Smith" 
 , 
  
 "name" 
 : 
  
 "John" 
 , 
  
 "job_description" 
 : 
  
 "Systems administrator" 
  
 }, 
  
 "_index" 
 : 
  
 "accounts" 
  
 }, 
  
 { 
  
 "_score" 
 : 
  
 0.28582606 
 , 
  
 "_type" 
 : 
  
 "person" 
 , 
  
 "_id" 
 : 
  
 "1" 
 , 
  
 "_source" 
 : 
  
 { 
  
 "lastname" 
 : 
  
 "Doe" 
 , 
  
 "name" 
 : 
  
 "John" 
 , 
  
 "job_description" 
 : 
  
 "Systems administrator and Linux specialist" 
  
 }, 
  
 "_index" 
 : 
  
 "accounts" 
  
 } 
 ]

    Ping

    Description

    Test Verifies connectivity to the Elasticsearch server.

    Parameters

    N/A

    Run On

    This action runs on all entities.

    Action Results

    Script Result
    Script Result Name Value Options Example
    is_success
    		 True/False is_success:False

    Description

    Action searches through everything in Elasticsearch and returns back results in a dictionary format.

    Parameters

    Parameters Type Default Value Description
    Index
    		 String *

    Search pattern for a Elasticsearch index.

    In Elasticsearch, index is like a DatabaseName, and data is stored across various indexes.

    This param defines in what index(es) to search. It can be an exact name ie: \"smp_playbooks-2019.06.13\" or you can use a ( ) wildcard to search by a pattern. e: \"smp_playbooks-2019.06 \" or \"smp*\".

    To learn more about Elasticsearch indexes visit https://www.elastic.co/blog/what-is-an-elasticsearch-index

    Query
    		 String *

    The search query to perform. It is in Lucene syntax.

    IE1: \"*\" (this is a wildcard that will return all record)

    IE2: \"level:error\"

    IE3: \"level:information\"

    IE4: \"level:error OR level:warning\"

    To learn more about Lucene syntax, visit https://www.elastic.co/guide/en/kibana/current/lucene-query.html#lucene-query\r\nhttps://www.elastic.co/guide/en/elasticsearch/reference/7.1/query-dsl-query-string-query.html#query-string-syntax

    Limit
    		 String 100

    Limits the document return count, ie: 10.

    0 = No limit.

    Run On

    This action runs on all entities.

    Action Results

    Script Result
    Script Result Name Value Options Example
    results
    		 N/A N/A
    JSON Result
      [{ 
  
 "_score" 
 : 
  
 0.2876821 
 , 
  
 "_type" 
 : 
  
 "person" 
 , 
  
 "_id" 
 : 
  
 "2" 
 , 
  
 "_source" 
 : 
  
 { 
  
 "lastname" 
 : 
  
 "Smith" 
 , 
  
 "name" 
 : 
  
 "John" 
 , 
  
 "job_description" 
 : 
  
 "Systems administrator" 
  
 }, 
  
 "_index" 
 : 
  
 "accounts" 
  
 }, 
  
 { 
  
 "_score" 
 : 
  
 0.28582606 
 , 
  
 "_type" 
 : 
  
 "person" 
 , 
  
 "_id" 
 : 
  
 "1" 
 , 
  
 "_source" 
 : 
  
 { 
  
 "lastname" 
 : 
  
 "Doe" 
 , 
  
 "name" 
 : 
  
 "John" 
 , 
  
 "job_description" 
 : 
  
 "Systems administrator and Linux specialist" 
  
 }, 
  
 "_index" 
 : 
  
 "accounts" 
  
 } 
  
 ]

    Connectors

    Configure Elasticsearch v7 connectors in Google SecOps

    For detailed instructions on how to configure a connector in Google SecOps, see Configuring the connector .

    To configure the selected connector use the connector-specific parameters listed in the following tables:

    Elasticsearch Connector

    Description

    This topic shows how Google SecOps integrates Elasticsearch with the mechanism and configuration for ingesting and processing.

    Elasticsearch alert forwarding to Google SecOps

    Google SecOps will search specified Elasticsearch indexes with a provided query (using Lucene query syntax) and return Elasticsearch documents that will be translated and contextualized as "alerts" for cases.

    Connector parameters

    Use the following parameters to configure the connector:

    Parameter Display Name Type Default Value Is Mandatory Description
    Default Environment
    		 String N/A No Select the required environment. For example, "Customer One".
    Run Every
    		 Integer 0:0:0:10 No Select the amount of time to run the connection. For example, "every day".
    Product Field Name
    		 String device_product Yes The field name used to determine the device product. Example: _type.
    Event Field Name
    		 String name Yes The field name used to determine the event name (sub-type). Example: _source_match_event_id.
    Script Timeout (Seconds)
    		 String 60 Yes The timeout limit (in seconds) for the python process running current script.
    Server Address
    		 String N/A Yes The Elasticsearch server address, i.e.: http://{ip_address}:{port}
    Username
    		 String N/A Yes Elasticsearch username.
    Password
    		 Password N/A Yes Elasticsearch password.
    Authenticate
    		 Checkbox Unchecked No Whether to authenticate on connection or not.
    API Token
    		 Password N/A No Elasticsearch XPack API Token.
    Verify SSL
    		 Checkbox Unchecked No Whether to use ssl on connection or not.
    Alert Name Field
    		 String N/A Yes The name of the field where the alert name is located (flat field path). Example: _source_alert_info_alert
    Timestamp Field
    		 String N/A Yes The name of the field where the timestamp is located (flat field path). Example: source @timestamp
    Environment Field
    		 String N/A No The name of the field where the environment is located (flat field path). Example: _source_environment
    Indexes
    		 String N/A No Index pattern to search by. Example: '*'
    Query
    		 String N/A No Search pattern query (Lucene query syntax). Example: '*'
    Alerts Count Limit
    		 Integer 20 Yes Max count of alerts to pull in one cycle. Example: 20.
    Max Days Backwards
    		 Integer 1 Yes Max number of days to fetch alerts since. Example: 3.
    Severity Field Mapping
    		 String N/A No Name of the field where severity value is stored.
    Proxy Server Address
    		 String N/A No The address of the proxy server to use.
    Proxy Username
    		 String N/A No The proxy username to authenticate with.
    Proxy Password
    		 Password N/A No The proxy password to authenticate with.
    Severity Field Name
    		 String N/A No If you want to map severity based on the string value then you would need to create a mapping file. Please refer to documentation portal for more details.
    Environment Regex Pattern
    		 String .* No

    A regex pattern to run on the value found in the "Environment Field Name" field.

    Default is .* to catch all and return value unchanged.

    Used to allow the user to manipulate the environment field via regex logic

    If the regex pattern is null or empty, or the environment value is null, the final environment result is "".

    How to map severity in the connector

    In order to map severity you need to specify what field should be used to get value for severity in the "Severity Field Name" parameter. In the response you can get 3 types of values: integers, floats and strings. For integers and floats, you don't need to do any additional configuration. The connector will read those values and map them according to the Google SecOps standards. A quick reminder of how integer values are mapped:

    • 100 - Critical
    • 100 > x >= 80 High
    • 80 > x >=60 Medium
    • 60 > x >=40 Low
    • 40 > x Informational

    If in the response, we are working with strings then additional configuration is required. In the folder, where connector scripts are located you will have a config file name severity_map_config.json . This file defines mapping rules for the severity.

    Initially, the file will look like this:

      { 
  
 "Default" 
 : 
  
 50 
 }

    Imagine a situation, where the needed values are located in the event.severity . event.severity can contain the following values: "Malicious", "Benign", "Unknown".

    First, we have to specify in the "Severity Field Name" parameter that we will use event.severity .

    Secondly, we have to update the config file.

    After changes, this is how severity_map_config.json file should look like:

      { 
  
 "event.severity" 
 : 
  
 { 
  
 "Malicious" 
 : 
  
 100 
 , 
  
 "Unknown" 
 : 
  
 60 
 , 
  
 "Benign" 
 : 
  
 -1 
  
 }, 
  
 "Default" 
 : 
  
 50 
 }

    Now, when the connector will get an event with event.severity = "Malicious" it will give it Critical severity.

    Connector rules

    Whitelist/Blacklist

    The connector doesn't support Whitelist/Blacklist.

    Proxy support

    The connector supports proxy.

    Elasticsearch DSL Connector

    Description

    The connector works by making a REST API call with a DSL query.

    Use Cases and Examples

    Ability to use DSL queries as a search parameter in Elasticsearch.

    Connector parameters

    Use the following parameters to configure the connector:

    Parameter Display Name Type Default Value Is Mandatory Description
    Default Environment
    		 String N/A No Select the required environment. For example, "Customer One".
    Run Every
    		 Integer 0:0:0:10 No Select the amount of time to run the connection. For example, "every day".
    Product Field Name
    		 String device_product Yes Describes the name of the field where the product name is stored.
    Environment Field Name
    		 String "" No

    Describes the name of the field where the environment name is stored.

    If the environment field isn't found, the environment is "".
    Environment Regex Pattern
    		 String .* No

    A regex pattern to run on the value found in the "Environment Field Name" field.

    Default is .* to catch all and return value unchanged.

    Used to allow the user to manipulate the environment field via regex logic

    If the regex pattern is null or empty, or the environment value is null, the final environment result is "".
    Script Timeout (Seconds)
    		 Integer 60 Yes Timeout limit for the python process running the current script.
    Server Address
    		 String N/A Yes IP address of the Elasticsearch API server.
    Port
    		 String N/A Yes Port of Elasticsearch API server.
    Query
    		 String N/A Yes

    DSL Query that is used for the search.

    Valid JSON format needed.

    To make the connector more stable it is recommended to add a sorting timestamp key in the ascending order.
    Index
    		 String N/A Yes

    Index that is used for a search.

    For example: _all
    Timestamp Field
    		 String N/A Yes

    The name of the field where the timestamp is located.Example:

    source @timestamp
    Alert Field Name
    		 String N/A Yes

    The name of the field where the alert name is located.Example:

    _source_info_alertname
    Description Field
    		 String N/A No

    The name of the field where the description is located.Example:

    _source_alert_info_description
    Severity
    		 String Medium Yes

    Severity of the alerts.Possible value:

    Info

    Low

    Medium

    High

    Critical
    Alerts Count Limit
    		 Integer 100 No Limit the number of alerts returned by the connector per 1 iteration.
    Authenticate
    		 Checkbox Unchecked No Whether to authenticate on a connection or not.
    Username
    		 String N/A No Elasticsearch account username.
    Password
    		 Password N/A No Elasticsearch account password.
    Use SSL
    		 Checkbox Unchecked No Option to enable SSL/TLS connection.
    Severity Field Name
    		 String N/A No If you want to map severity based on the string value then you would need to create a mapping file. Please refer to documentation portal for more details.
    Alert Severity
    		 String N/A No

    The severity of the alerts.

    Possible value: Info, Low, Medium, High, Critical.

    Note:this parameter has priority over "Severity Field Name". If you want to work with "Severity Field Name", this field should be left empty.
    Proxy Server Address
    		 String N/A No The address of the proxy server to use.
    Proxy Username
    		 String N/A No The proxy username to authenticate with.
    Proxy Password
    		 Password N/A No The proxy password to authenticate with.

    Supported Notations

    Connector supports three notations. For example, if you want use event.type in the "Event Field Name" parameter. In that case, you can either provide _source_event_type, event_type or event.type. All of these values will behave the same way.

    For parameters:

    • Product Field Name
    • Event Field Name
    • Severity Field Name
    • Environment Field
    • Timestamp Field
    • Alert Name Field
    • Alert Description Field - this one is only for DSL connectors

    How to map severity in the connector

    In order to map severity you need to specify what field should be used to get value for severity in the "Severity Field Name" parameter. In the response you can get 3 types of values: integers, floats and strings. For integers and floats, you don't need to do any additional configuration. The connector will read those values and map them according to the Google SecOps standards. A quick reminder of how integer values are mapped:

    • 100 - Critical
    • 100 > x >= 80 High
    • 80 > x >=60 Medium
    • 60 > x >=40 Low
    • 40 > x Informational

    If in the response, we are working with strings then additional configuration is required. In the folder, where connector scripts are located you will have a config file name severity_map_config.json . This file defines mapping rules for the severity.

    Initially, the file will look like this:

      { 
  
 "Default" 
 : 
  
 50 
 }

    Imagine a situation, where the needed values are located in the event.severity . event.severity can contain the following values: "Malicious", "Benign", "Unknown".

    First, we have to specify in the "Severity Field Name" parameter that we will use event.severity .

    Secondly, we have to update the config file.

    After changes, this is how severity_map_config.json file should look like:

      { 
  
 "event.severity" 
 : 
  
 { 
  
 "Malicious" 
 : 
  
 100 
 , 
  
 "Unknown" 
 : 
  
 60 
 , 
  
 "Benign" 
 : 
  
 -1 
  
 }, 
  
 "Default" 
 : 
  
 50 
 }

    Now, when the connector will get an event with event.severity = "Malicious" it will give it Critical severity.

    Connector rules

    Whitelist/Blacklist

    The connector doesn't support Whitelist/Blacklist.

    Proxy support

    The connector supports proxy.

    Need more help? Get answers from Community members and Google SecOps professionals.

    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: