Console
    Contact Us Start free

    Collect Elastic Packet Beats logs

    Supported in:

    This document explains how to ingest Elastic Packet Beats logs to Google Security Operations using Bindplane. The parser first initializes default values for various fields found in Elastic Packet Beats logs. Then, it extracts data from the log messages using a combination of grok patterns and json filters, performs data type conversions, and maps the extracted fields to the corresponding fields in the Unified Data Model (UDM) based on the event dataset type (for example, flow, dns, http, tls, dhcpv4).

    Before you begin

    Make sure you have the following prerequisites:

    • A Google SecOps instance.
    • A Windows 2016 or later, or Linux host with systemd .
    • If running behind a proxy, make sure firewall ports are open per the Bindplane agent requirements.
    • Privileged access to the Elastic Packet Beats management console or appliance.
    • Logstash installed and configured.

    Get Google SecOps ingestion authentication file

    1. Sign in to the Google SecOps console.
    2. Go to SIEM Settings > Collection Agents.
    3. Download the Ingestion Authentication File.
      • Save the file securely on the system where Bindplane will be installed.

    Get Google SecOps customer ID

    1. Sign in to the Google SecOps console.
    2. Go to SIEM Settings > Profile.
    3. Copy and save the Customer IDfrom the Organization Detailssection.

    Install the Bindplane agent

    Install the Bindplane agent on your Windows or Linux operating system according to the following instructions.

    Windows installation

    1. Open the Command Promptor PowerShellas an administrator.

    2. Run the following command:

        msiexec 
  
 / 
 i 
  
 "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" 
  
 / 
 quiet

    Linux installation

    1. Open a terminal with root or sudo privileges.

    2. Run the following command:

       sudo  
sh  
-c  
 " 
 $( 
curl  
-fsSlL  
https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh ) 
 " 
  
install_unix.sh

    Additional installation resources

    For additional installation options, consult the installation guide .

    Configure the Bindplane agent to ingest Syslog and send to Google SecOps

    1. Access the configuration file:

      • Locate the config.yaml file. Typically, it's in the /etc/bindplane-agent/ directory on Linux or in the installation directory on Windows.
      • Open the file using a text editor (for example, nano , vi , or Notepad).

    2. Edit the config.yaml file as follows:

        receivers 
 : 
  
 udplog 
 : 
  
 # Replace the port and IP address as required 
  
 listen_address 
 : 
  
 "0.0.0.0:514" 
 exporters 
 : 
  
 chronicle/chronicle_w_labels 
 : 
  
 compression 
 : 
  
 gzip 
  
 # Adjust the path to the credentials file you downloaded in Step 1 
  
 creds_file_path 
 : 
  
 '/path/to/ingestion-authentication-file.json' 
  
 # Replace with your actual customer ID from Step 2 
  
 customer_id 
 : 
  
 YOUR_CUSTOMER_ID 
  
 endpoint 
 : 
  
 malachiteingestion-pa.googleapis.com 
  
 # Add optional ingestion labels for better organization 
  
 log_type 
 : 
  
 'ELASTIC_PACKETBEATS' 
  
 raw_log_field 
 : 
  
 body 
  
 ingestion_labels 
 : 
 service 
 : 
  
 pipelines 
 : 
  
 logs/source0__chronicle_w_labels-0 
 : 
  
 receivers 
 : 
  
 - 
  
 udplog 
  
 exporters 
 : 
  
 - 
  
 chronicle/chronicle_w_labels
      • Replace the port and IP address as required in your infrastructure.
      • Replace YOUR_CUSTOMER_ID with the actual customer ID.
      • Update /path/to/ingestion-authentication-file.json to the path where the authentication file was saved in the Get Google SecOps ingestion authentication file section.

    Restart the Bindplane agent to apply the changes

    • To restart the Bindplane agent in Linux, run the following command:

       sudo  
systemctl  
restart  
observiq-otel-collector

    • To restart the Bindplane agent in Windows, you can either use the Servicesconsole, or enter the following command:

       net stop observiq-otel-collector && net start observiq-otel-collector

    Configure Syslog forwarding on Elastic Packet Beats

    Since Packetbeat does not support direct syslog output, you must use Logstash as an intermediary.

    Configure Packetbeat to send logs to Logstash

    1. Sign in to the Elastic Packet Beats Management Console.
    2. Go to Settings > Log Forwarding.
    3. Click the + Addor Enablebutton.
    4. Provide the following configuration details:
      • Name: Enter a descriptive name (for example, Logstash Output ).
      • Host: Enter the Logstash server IP address.
      • Port: Enter the Logstash beats input port (typically 5044).
      • Protocol: Select Beats protocol.
      • Format: Select JSON.
      • Timezone: Select UTC time zone for universal consistency across systems.
      • Go to the Eventssection and select the relevant log types or all.

    5. Save the configuration.

      Alternative: Edit packetbeat.yml directly:

        # /etc/packetbeat/packetbeat.yml 
 packetbeat.protocols 
 : 
  
 - 
  
 type 
 : 
  
 dns 
  
 ports 
 : 
  
 [ 
 53 
 ] 
  
 - 
  
 type 
 : 
  
 http 
  
 ports 
 : 
  
 [ 
 80 
 , 
  
 8080 
 , 
  
 8000 
 , 
  
 5000 
 , 
  
 8002 
 ] 
  
 send_headers 
 : 
  
 true 
  
 send_all_headers 
 : 
  
 true 
  
 - 
  
 type 
 : 
  
 tls 
  
 ports 
 : 
  
 [ 
 443 
 , 
  
 993 
 , 
  
 995 
 , 
  
 5223 
 , 
  
 8443 
 , 
  
 8883 
 , 
  
 9243 
 ] 
  
 - 
  
 type 
 : 
  
 dhcpv4 
  
 ports 
 : 
  
 [ 
 67 
 , 
  
 68 
 ] 
 # Enable processors for additional fields 
 processors 
 : 
  
 - 
  
 add_network_direction 
 : 
  
 source 
 : 
  
 private 
  
 destination 
 : 
  
 private 
  
 internal_networks 
 : 
  
 - 
  
 private 
  
 - 
  
 community_id 
 : 
 # Send to Logstash using beats protocol 
 output.logstash 
 : 
  
 hosts 
 : 
  
 [ 
 "LOGSTASH_IP:5044" 
 ]

      Replace LOGSTASH_IP with your Logstash server's IP address.

    Configure Logstash to forward to BindPlane using Syslog

    1. Create a Logstash pipeline configuration file:

       sudo  
nano  
/etc/logstash/conf.d/packetbeat-to-bindplane.conf

    2. Add the following configuration:

        # Receive from Packetbeat 
 input 
  
 { 
  
 beats 
  
 { 
  
 port 
  
 = 
>  
 5044 
  
 } 
 } 
 # Optional: Add filters for data enrichment 
 filter 
  
 { 
  
 # Preserve original message structure 
  
 mutate 
  
 { 
  
 copy 
  
 = 
>  
 { 
  
 "@metadata" 
  
 = 
>  
 "[@metadata_backup]" 
  
 } 
  
 } 
 } 
 # Send to BindPlane via syslog 
 output 
  
 { 
  
 syslog 
  
 { 
  
 host 
  
 = 
>  
 "BINDPLANE_IP" 
  
 port 
  
 = 
>  
 514 
  
 protocol 
  
 = 
>  
 "udp" 
  
 rfc 
  
 = 
>  
 "rfc5424" 
  
 facility 
  
 = 
>  
 "local0" 
  
 severity 
  
 = 
>  
 "informational" 
  
 sourcehost 
  
 = 
>  
 "%{[agent][hostname]}" 
  
 appname 
  
 = 
>  
 "packetbeat" 
  
 procid 
  
 = 
>  
 "%{[agent][id]}" 
  
 msgid 
  
 = 
>  
 "ELASTIC_PACKETBEATS" 
  
 structured_data 
  
 = 
>  
 "packetbeat@32473" 
  
 message 
  
 = 
>  
 "%{message}" 
  
 } 
 }

      Replace BINDPLANE_IP with your BindPlane agent's IP address.

    • Restart Logstash to apply the configuration:

       sudo  
systemctl  
restart  
logstash

    UDM mapping table

    Log field UDM mapping Logic
    @timestamp
    		 metadata.event_timestamp Directly mapped from the raw log field @timestamp .
    agent.hostname
    		 observer.hostname Directly mapped from the raw log field agent.hostname .
    agent.id
    		 observer.asset_id Concatenated with agent.type to form the observer.asset_id field.
    agent.type
    		 observer.application Directly mapped from the raw log field agent.type .
    agent.version
    		 observer.platform_version Directly mapped from the raw log field agent.version .
    audit_category
    		 security_result.category_details Directly mapped from the raw log field audit_category .
    audit_cluster_name
    		 additional.fields.audit_cluster_name.value.string_value Directly mapped from the raw log field audit_cluster_name .
    audit_node_host_address
    		 observer.ip Directly mapped from the raw log field audit_node_host_address .
    audit_node_id
    		 additional.fields.audit_node_id.value.string_value Directly mapped from the raw log field audit_node_id .
    audit_node_name
    		 additional.fields.audit_node_name.value.string_value Directly mapped from the raw log field audit_node_name .
    audit_request_effective_user
    		 observer.user.userid Directly mapped from the raw log field audit_request_effective_user .
    audit_request_initiating_user
    		 additional.fields.audit_request_initiating_user.value.string_value Directly mapped from the raw log field audit_request_initiating_user .
    audit_request_remote_address
    		 observer.ip Directly mapped from the raw log field audit_request_remote_address if it's different from audit_node_host_address .
    client.bytes
    		 network.received_bytes (INBOUND) / network.sent_bytes (OUTBOUND) Mapped based on the network.direction field. If INBOUND, it's mapped to network.received_bytes . If OUTBOUND, it's mapped to network.sent_bytes .
    client.ip
    		 target.ip (INBOUND) / principal.ip (OUTBOUND) Mapped based on the network.direction field. If INBOUND, it's mapped to target.ip . If OUTBOUND, it's mapped to principal.ip .
    client.port
    		 target.port (INBOUND) / principal.port (OUTBOUND) Mapped based on the network.direction field. If INBOUND, it's mapped to target.port . If OUTBOUND, it's mapped to principal.port .
    cluster.uuid
    		 additional.fields.uuid.value.string_value Directly mapped from the raw log field cluster.uuid .
    component
    		 additional.fields.component.value.string_value Directly mapped from the raw log field component .
    destination.bytes
    		 network.sent_bytes Directly mapped from the raw log field destination.bytes for FLOW events.
    destination.ip
    		 target.ip Directly mapped from the raw log field destination.ip if network.direction is not INBOUND or OUTBOUND.
    destination.mac
    		 target.mac Directly mapped from the raw log field destination.mac for FLOW events.
    destination.port
    		 target.port Directly mapped from the raw log field destination.port for FLOW events.
    dhcpv4.assigned_ip
    		 network.dhcp.requested_address Directly mapped from the raw log field dhcpv4.assigned_ip .
    dhcpv4.client_ip
    		 network.dhcp.yiaddr (ACK) / network.dhcp.ciaddr (REQUEST) / source.ip (REQUEST, if dhcpv4.client_ip is empty) Mapped based on the network.dhcp.type field. If ACK, it's mapped to network.dhcp.yiaddr . If REQUEST, it's mapped to network.dhcp.ciaddr . If REQUEST and dhcpv4.client_ip is empty, it's mapped to source.ip .
    dhcpv4.client_mac
    		 network.dhcp.client_identifier Directly mapped from the raw log field dhcpv4.client_mac after converting it to bytes.
    dhcpv4.op_code
    		 network.dhcp.opcode Mapped to network.dhcp.opcode based on the value of dhcpv4.op_code . If dhcpv4.op_code is BOOTREPLY or BOOTREQUEST , the value is directly mapped. Otherwise, it's mapped to UNKNOWN_OPCODE .
    dhcpv4.option.hostname
    		 network.dhcp.client_hostname Directly mapped from the raw log field dhcpv4.option.hostname .
    dhcpv4.option.ip_address_lease_time_sec
    		 network.dhcp.lease_time_seconds Directly mapped from the raw log field dhcpv4.option.ip_address_lease_time_sec after converting it to an unsigned integer.
    dhcpv4.option.message_type
    		 network.dhcp.type Mapped to network.dhcp.type based on the value of dhcpv4.option.message_type . The mapping is as follows: ack -> ACK , nack -> NAK , discover -> DISCOVER , offer -> OFFER , request -> REQUEST , decline -> DECLINE , release -> RELEASE , info -> INFORM . If the value is not one of these, it's mapped to UNKNOWN_MESSAGE_TYPE .
    dhcpv4.option.server_identifier
    		 network.dhcp.sname Directly mapped from the raw log field dhcpv4.option.server_identifier .
    dns.answers.data
    		 network.dns.answers.data Directly mapped from the raw log field dns.answers.data .
    dns.answers.class
    		 network.dns.answers.class Mapped to network.dns.answers.class based on the value of dns.answers.class . The mapping is as follows: IN -> 1, NONE -> 254, ANY -> 255.
    dns.answers.name
    		 network.dns.answers.name Directly mapped from the raw log field dns.answers.name .
    dns.answers.ttl
    		 network.dns.answers.ttl Directly mapped from the raw log field dns.answers.ttl after converting it to an unsigned integer.
    dns.answers.type
    		 network.dns.answers.type Mapped to network.dns.answers.type based on the value of dns.answers.type . The mapping is as follows: A -> 1, NS -> 2, CNAME -> 5, SOA -> 6, PTR -> 12, MX -> 15, TXT -> 16, AAAA -> 28, SRV -> 33, NAPTR -> 35, DS -> 43, DNSKEY -> 48, IXFR -> 251, AXFR -> 252, TYPE99 -> 99, TKEY -> 249, ANY -> 255, ALL -> 255, URI -> 256, NULL -> 0.
    dns.flags.authoritative
    		 network.dns.authoritative Directly mapped from the raw log field dns.flags.authoritative if it's true.
    dns.flags.recursion_available
    		 network.dns.recursion_available Directly mapped from the raw log field dns.flags.recursion_available if it's true.
    dns.flags.recursion_desired
    		 network.dns.recursion_desired Directly mapped from the raw log field dns.flags.recursion_desired if it's true.
    dns.flags.truncated_response
    		 network.dns.truncated Directly mapped from the raw log field dns.flags.truncated_response if it's true.
    dns.id
    		 network.dns.id Directly mapped from the raw log field dns.id after converting it to an unsigned integer.
    dns.question.class
    		 network.dns.questions.class Mapped to network.dns.questions.class based on the value of dns.question.class . The mapping is as follows: IN -> 1, NONE -> 254, ANY -> 255.
    dns.question.name
    		 network.dns.questions.name Directly mapped from the raw log field dns.question.name .
    dns.question.type
    		 network.dns.questions.type Mapped to network.dns.questions.type based on the value of dns.question.type . The mapping is as follows: A -> 1, NS -> 2, CNAME -> 5, SOA -> 6, PTR -> 12, MX -> 15, TXT -> 16, AAAA -> 28, SRV -> 33, NAPTR -> 35, DS -> 43, DNSKEY -> 48, IXFR -> 251, AXFR -> 252, TYPE99 -> 99, TKEY -> 249, ANY -> 255, ALL -> 255, URI -> 256, NULL -> 0.
    dns.resolved_ip
    		 network.dns.additional.data Each element in the dns.resolved_ip array is processed and mapped to the network.dns.additional.data field.
    dns.response_code
    		 network.dns.response_code Mapped to network.dns.response_code based on the value of dns.response_code . The mapping is as follows: NOERROR -> 0, FORMERR -> 1, SERVFAIL -> 2, NXDOMAIN -> 3, NOTIMP -> 4, REFUSED -> 5, YXDOMAIN -> 6, YXRRSET -> 7, NXRRSET -> 8, NOTAUTH -> 9, NOTZONE -> 10.
    error.message
    		 security_result.summary Concatenated with status to form the security_result.summary field for HTTP events.
    event.dataset
    		 metadata.product_event_type Directly mapped from the raw log field event.dataset .
    flow.final
    		 Used to determine if the flow is final. If not, the event is dropped.
    flow.id
    		 network.session_id Directly mapped from the raw log field flow.id for FLOW events.
    headers.accept_encoding
    		 security_result.about.labels.Accept-Encoding Directly mapped from the raw log field headers.accept_encoding .
    headers.content_length
    		 additional.fields.content_length.value.string_value Directly mapped from the raw log field headers.content_length .
    headers.content_type
    		 additional.fields.content_type.value.string_value Directly mapped from the raw log field headers.content_type .
    headers.http_accept
    		 additional.fields.http_accept.value.string_value Directly mapped from the raw log field headers.http_accept .
    headers.http_host
    		 principal.hostname, principal.asset.hostname Directly mapped from the raw log field headers.http_host .
    headers.http_user_agent
    		 network.http.user_agent Directly mapped from the raw log field headers.http_user_agent .
    headers.request_method
    		 network.http.method Directly mapped from the raw log field headers.request_method .
    headers.x_b3_parentspanid
    		 additional.fields.x_b3_parentspanid.value.string_value Directly mapped from the raw log field headers.x_b3_parentspanid .
    headers.x_b3_sampled
    		 additional.fields.x_b3_sampled.value.string_value Directly mapped from the raw log field headers.x_b3_sampled .
    headers.x_envoy_attempt_count
    		 security_result.about.labels.x_envoy_attempt_count Directly mapped from the raw log field headers.x_envoy_attempt_count .
    headers.x_envoy_original_path
    		 additional.fields.x_envoy_original_path.value.string_value Directly mapped from the raw log field headers.x_envoy_original_path .
    headers.x_forwarded_client_cert
    		 additional.fields.client_cert.value.string_value Directly mapped from the raw log field headers.x_forwarded_client_cert .
    headers.x_forwarded_for
    		 principal.ip, principal.asset.ip Directly mapped from the raw log field headers.x_forwarded_for after extracting the IP address using grok.
    headers.x_forwarded_proto
    		 additional.fields.x_forwarded_proto.value.string_value Directly mapped from the raw log field headers.x_forwarded_proto .
    headers.x_request_id
    		 additional.fields.x_request_id.value.string_value Directly mapped from the raw log field headers.x_request_id .
    host
    		 principal.ip, principal.asset.ip Directly mapped from the raw log field host after extracting the IP address using grok.
    http.request.method
    		 network.http.method Directly mapped from the raw log field http.request.method .
    level
    		 security_result.severity Mapped to security_result.severity based on the value of level . The mapping is as follows: INFO -> INFORMATIONAL , ERROR -> ERROR , WARNING -> LOW .
    logger
    		 additional.fields.logger.value.string_value Directly mapped from the raw log field logger .
    method
    		 Used to determine if the event is a DNS event.
    msg
    		 security_result.description Directly mapped from the raw log field msg after removing double quotes.
    network.community_id
    		 network.community_id Directly mapped from the raw log field network.community_id .
    network.direction
    		 network.direction Directly mapped from the raw log field network.direction after converting it to uppercase. If the value is INGRESS or INBOUND , it's mapped to INBOUND . If the value is EGRESS or OUTBOUND , it's mapped to OUTBOUND .
    network.protocol
    		 network.application_protocol Directly mapped from the raw log field network.protocol .
    network.transport
    		 network.ip_protocol Directly mapped from the raw log field network.transport for TLS events.
    server.bytes
    		 network.sent_bytes (INBOUND) / network.received_bytes (OUTBOUND) Mapped based on the network.direction field. If INBOUND, it's mapped to network.sent_bytes . If OUTBOUND, it's mapped to network.received_bytes .
    server.domain
    		 principal.hostname, principal.asset.hostname (INBOUND) / target.hostname, target.asset.hostname (OUTBOUND) Mapped based on the network.direction field. If INBOUND, it's mapped to principal.hostname . If OUTBOUND, it's mapped to target.hostname .
    server.ip
    		 principal.ip, principal.asset.ip (INBOUND) / target.ip, target.asset.ip (OUTBOUND) Mapped based on the network.direction field. If INBOUND, it's mapped to principal.ip . If OUTBOUND, it's mapped to target.ip .
    server.port
    		 principal.port (INBOUND) / target.port (OUTBOUND) Mapped based on the network.direction field. If INBOUND, it's mapped to principal.port . If OUTBOUND, it's mapped to target.port .
    source.bytes
    		 network.received_bytes Directly mapped from the raw log field source.bytes for FLOW events.
    source.ip
    		 principal.ip, principal.asset.ip Directly mapped from the raw log field source.ip for FLOW events.
    source.mac
    		 principal.mac Directly mapped from the raw log field source.mac for FLOW events.
    source.port
    		 principal.port Directly mapped from the raw log field source.port for FLOW events.
    status
    		 metadata.description, security_result.summary Mapped to metadata.description if level is empty. Concatenated with error.message to form the security_result.summary field for HTTP and TLS events.
    tls.client.ja3
    		 network.tls.client.ja3 Directly mapped from the raw log field tls.client.ja3 .
    tls.client.server_name
    		 network.tls.client.server_name Directly mapped from the raw log field tls.client.server_name .
    tls.client.supported_ciphers
    		 network.tls.client.supported_ciphers Each element in the tls.client.supported_ciphers array is processed and mapped to the network.tls.client.supported_ciphers array.
    tls.cipher
    		 network.tls.cipher Directly mapped from the raw log field tls.cipher .
    tls.detailed.server_certificate.not_after
    		 network.tls.server.certificate.not_after Directly mapped from the raw log field tls.detailed.server_certificate.not_after after converting it to a timestamp.
    tls.detailed.server_certificate.not_before
    		 network.tls.server.certificate.not_before Directly mapped from the raw log field tls.detailed.server_certificate.not_before after converting it to a timestamp.
    tls.detailed.server_certificate.serial_number
    		 network.tls.server.certificate.serial Directly mapped from the raw log field tls.detailed.server_certificate.serial_number .
    tls.detailed.server_certificate.version
    		 network.tls.server.certificate.version Directly mapped from the raw log field tls.detailed.server_certificate.version after converting it to a string.
    tls.established
    		 network.tls.established Directly mapped from the raw log field tls.established .
    tls.next_protocol
    		 network.tls.next_protocol Directly mapped from the raw log field tls.next_protocol .
    tls.resumed
    		 network.tls.resumed Directly mapped from the raw log field tls.resumed .
    tls.server.hash.sha1
    		 network.tls.server.certificate.sha1 Directly mapped from the raw log field tls.server.hash.sha1 after converting it to lowercase.
    tls.server.issuer
    		 network.tls.server.certificate.issuer Directly mapped from the raw log field tls.server.issuer .
    tls.server.subject
    		 network.tls.server.certificate.subject Directly mapped from the raw log field tls.server.subject .
    tls.version
    		 network.tls.version Directly mapped from the raw log field tls.version .
    tls.version_protocol
    		 network.tls.version_protocol Directly mapped from the raw log field tls.version_protocol .
    type
    		 Used to determine if the event is a DNS event.
    url.full
    		 principal.url (INBOUND) / target.url (OUTBOUND) Mapped based on the network.direction field. If INBOUND, it's mapped to principal.url . If OUTBOUND, it's mapped to target.url .
    user_id
    		 target.user.userid Directly mapped from the raw log field user_id .
    user_name
    		 target.user.user_display_name Directly mapped from the raw log field user_name .
    		metadata.event_type Set to GENERIC_EVENT by default. Changed to specific event types based on the log source and event data.
    		metadata.vendor_name Set to Elastic by default.
    		metadata.product_name Set to PacketBeat by default.
    		security_result.action Set to ALLOW by default.
    		metadata.log_type Set to ELASTIC_PACKETBEATS by default.

    Need more help? Get answers from Community members and Google SecOps professionals.

    Create a Mobile Website
    View Site in Mobile | Classic
    Share by: