Collect Zeek (Bro) logs

This document describes how you can deploy Zeek (formerly Bro) and NXLog with Google Security Operations to collect Zeek logs in JSON format. This document also explains how Zeek log fields map to Google Security Operations Unified Data Model (UDM) fields.

For an overview about Google Security Operations data ingestion, see Data ingestion to Google Security Operations .

An ingestion label identifies the parser which normalizes raw log data to structured UDM format. The information in this document applies to the parser with the BRO_JSON ingestion label.

Before you begin

• To understand the components deployed to collect Zeek logs, review the deployment architecture. Each customer deployment might differ from this representation and might be more complex. The following diagram shows how you can configure a NXLog agent and a Google Security Operations forwarder on a Linux server and forward log data to Google Security Operations.

  

• Verify the Zeek versions that the Google Security Operations parser supports. The Google Security Operations parser supports the following Zeek versions:

  • Zeek 4.1.0
  • Zeek 4.0.1
  • Zeek 5.2.0
  • Zeek 6.0.0

• Before you use the Zeek parser, review the changes in field mappings between the previous parser and the current Zeek parser . As part of the migration, ensure that the rules, searches, dashboards, or other processes that depend on the original fields use the updated fields.

  For example, in the previous parser version, the server_name field is mapped to the target.hostname UDM field. In the current Zeek parser, the server_name field is mapped to the network.tls.client.server_name UDM field. If you migrate to the current Zeek parser and use the server_name field in your rules, you need to modify the rules to use the network.tls.client.server_name UDM field of the current parser.

• Verify the Zeek log types that the Google Security Operations parser supports. The following table lists the Zeek log types that the Google Security Operations parser supports:

• If you have not done so already, install and configure Zeek. For more information, see Zeek installation .

• Collect Zeek logs in JSON format. For more information, see Output Zeek logs to JSON .

• Ensure that all systems in the deployment architecture are configured with the UTC time zone.

Configure NXLog and Google Security Operations forwarder

1. Download and install NXLog Community Edition on the Linux machine on which Google Security Operations forwarder runs.
   • For more information on downloading NXLog Community Edition, see NXLog documentation .
   • For more information on installing the required NXLog packages and dependencies, see installing NXLog on a Linux system .
2. Create a configuration file for each NXLog instance.

3. Use the NXLog im_file module to read from the file and parse the lines into fields. Here is an example NXLog configuration:

    LogFile /var/log/nxlog/nxlog.log
   LogLevel INFO
   
   define ZEEK_OUTPUT_DESTINATION_ADDRESS <hostname>
   define ZEEK_OUTPUT_DESTINATION_PORT <port>
   
   <Input conn>
      Module      im_file
      File        '/opt/zeek/logs/current/conn.log'
      Exec $raw_event= "conn" + ' - ' + $raw_event;;
   </Input>
   
   <Input dce_rpc>
     Module      im_file
     File        '/opt/zeek/logs/current/dce_rpc.log'
     Exec $raw_event= "dce_rpc" + ' - ' + $raw_event;;
   </Input>
   
   <Output out_chronicle>
     Module  om_tcp
     Host    %ZEEK_OUTPUT_DESTINATION_ADDRESS%
     Port    %ZEEK_OUTPUT_DESTINATION_PORT%
   </Output>
   
   <Route zeek_to_chronicle>
     Path conn, dce_rpc => out_chronicle
   </Route> 
   

   To use the preceding example configuration, do the following:

   • Replace <hostname> and <port> values with information about the destination Linux server.
   • Add input, output, and route elements for each Zeek log type that you want to collect.

4. Configure Google Security Operations forwarder to send logs to Google Security Operations. For more information, see Installing and configuring the forwarder on Linux . Here is an example forwarder configuration.

    output:
     url: URL 
   identity:
     identity:
     collector_id: COLLECTOR_ID 
   customer_id: CUSTOMER_ID 
   secret_key: |
   
     {
     "type": "service_account",
     "project_id": "malachite-projectname",
     "private_key_id": ` PRIVATE_KEY_ID 
   `,
     "private_key": ` PRIVATE_KEY 
   `,
     "client_email":"` SERVICE_ACCOUNT_NAME 
   `@malachite-` PROJECT_ID 
   `.` SERVICE_ACCOUNT_DOMAIN 
   `",
     "client_id": ` CLIENT_ID 
   `,
     "auth_uri": "https://accounts.google.com/o/oauth2/auth",
     "token_uri": "https://oauth2.googleapis.com/token",
     "auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs",
     "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/` SERVICSERVICE_ACCOUNT_NAME 
   `%40malachite-` PROJECT_ID 
   `.` SERVICE_ACCOUNT_DOMAIN 
   `",
     }
   
     collectors:
     - syslog:
         common:
           enabled: true
           data_type: BRO_JSON
           batch_n_seconds: 10
           batch_n_bytes: 1048576
         tcp_address: 0.0.0.0:10518
         connection_timeout_sec: 60 
   

5. Replace the PRIVATE_KEY_ID , PRIVATE_KEY SERVICSERVICE_ACCOUNT_NAME , PROJECT_ID , CLIENT_ID , SERVICE_ACCOUNT_DOMAIN CUSTOMER_ID , URL , COLLECTOR_ID and CUSTOMER_ID with the respective values from the service account JSON file which you can download from Google Cloud Platform.

6. Start the NXLog service.

Forward Logs to Google SecOps using Bindplane agent

1. Install and set up a Linux Virtual Machine.
2. Install and configure the Bindplane agent on Linux to forward logs to Google SecOps. For more information about how to install and configure the Bindplane agent, see the Bindplane agent installation and configuration instructions .

If you encounter issues when you create feeds, contact Google SecOps support .

Supported Zeek (Bro) log formats

The Zeek (Bro) parser supports logs in JSON and SYSLOG+JSON format.

Supported Zeek (Bro) sample logs

• JOSN

   {
    "insertId": "1pvsdy2f8v21o8",
    "jsonPayload": {
      "message": "Jun 14 07:46:10 dummyhostname systemd[1]: Stopping System Logging Service..."
    },
    "resource": {
      "type": "gce_instance",
      "labels": {
        "project_id": "cl-tpt-dis-awkc-con17-p-922a",
        "zone": "us-central1-a",
        "instance_id": "4136884722753789246"
      }
    },
    "timestamp": "2024-09-03T19:31:32.353129233Z",
    "labels": {
      "compute.googleapis.com/resource_name": "dummyostname"
    },
    "logName": "projects/cl-tpt-dis-awkc-con17-p-922a/logs/syslog",
    "receiveTimestamp": "2024-09-03T19:31:33.388651657Z"
  } 
  

• SYSLOG + JSON

   <13>1 2021-12-21T23: 51: 25-08: 00 ia-cs-vubro-089 bro_http - - - {
    "ts": 1640159484.694295,
    "uid": "CTgT3z1adxn1EMPbmj",
    "id.orig_h": "198.51.100.27",
    "id.orig_p": 58729,
    "id.resp_h": "198.51.100.28",
    "id.resp_p": 8088,
    "trans_depth": 2284,
    "method": "POST",
    "host": "198.51.100.8",
    "uri": "/system/gateway",
    "version": "1.1",
    "user_agent": "Java/11.0.11",
    "request_body_len": 304,
    "response_body_len": 203,
    "status_code": 200,
    "status_msg": "OK",
    "tags": [],
    "orig_fuids": [
      "FefIdu4i8dzFTUONb5"
    ],
    "orig_mime_types": [
      "application/xml"
    ],
    "resp_fuids": [
      "Flqz7L3yyQR1eSN4Kf"
    ],
    "resp_mime_types": [
      "application/xml"
    ]
  } 
  

Field mapping reference: Zeek logs fields to UDM fields

To understand how the Google Security Operations parser maps Zeek log fields to Google Security Operations UDM event fields for each Zeek log type, refer to the following sections:

• Network protocols
• Files
• Netcontrol
• Detection
• Network observations

Network protocols

The following table lists the log fields of the network protocols log type and their corresponding UDM fields.

Files

The following table lists the log fields of the files log type and their corresponding UDM fields.

Netcontrol

The following table lists the log fields of the netcontrol log type and their corresponding UDM fields.

Detection

The following table lists the log fields of the detection log type and their corresponding UDM fields.

Network Observations

The following table lists the log fields of the network observations log type and their corresponding UDM fields.

Field mapping reference: Event ID to UDM event type

To understand how the parser maps log names to UDM event types, refer to the following sections:

• Network protocols
• Files
• Netcontrol
• Detection
• Network observations

Network protocols

The following table lists the log names of the network protocols log type and their corresponding UDM event types.

Files

The following table lists the log names of the files log type and their corresponding UDM event types.

Netcontrol

The following table lists the log names of the netcontrol log type and their corresponding UDM event types.

Detection

The following table lists the log names of the detection log type and their corresponding UDM event types.

Network observations

The following table lists the log names of the network observations log type and their corresponding UDM event types.

What's next

• Data ingestion to Google Security Operations

Need more help? Get answers from Community members and Google SecOps professionals.