Collect VPC Flow Logs

Supported in:

Google secops SIEM

This document explains how to export VPC Flow Logs to Google Security Operations using Google Cloud. The parser transforms the logs from their built-in JSON format into the Google Security Operations UDM. It extracts relevant fields like source and destination IP, port, protocol, and bytes sent, then maps them to corresponding UDM fields, taking into account network direction and special cases for accurate representation in Google SecOps.

Before You Begin

Ensure that you have the following prerequisites:

Google SecOps instance.
VPC Flow is set up and active in your Google Cloud environment.
Privileged access to Google Cloud.

Create a Google Cloud Storage Bucket

Sign in to the Google Cloud console .
Go to the Cloud Storage Bucketspage.

Go to Buckets
Click Create.
On the Create a bucketpage, enter your bucket information. After each of the following steps, click Continueto proceed to the next step:
1. In the Get startedsection, do the following:
  1. Enter a unique name that meets the bucket name requirements; for example, vpcflow-logs.
  2. To enable hierarchical namespace, click the expander arrow to expand the Optimize for file oriented and data-intensive workloadssection, and then select Enable Hierarchical namespace on this bucket.
    
    Note: You cannot enable hierarchical namespace in an existing bucket.
  3. To add a bucket label, click the expander arrow to expand the Labelssection.
  4. Click Add label, and specify a key and a value for your label.
2. In the Choose where to store your datasection, do the following:
  1. Select a Location type.
  2. Use the location type menu to select a Locationwhere object data within your bucket will be permanently stored.
    
    Note: If you select the dual-regionlocation type, you can also choose to enable turbo replicationby using the relevant checkbox.
  3. To set up cross-bucket replication, expand the Set up cross-bucket replicationsection.
3. In the Choose a storage class for your datasection, either select a default storage classfor the bucket, or select Autoclassfor automatic storage class management of your bucket's data.
4. In the Choose how to control access to objectssection, select notto enforce public access prevention, and select an access control modelfor your bucket's objects.
  
  Note: If public access prevention is already enforced by your project's organization policy, the Prevent public accesscheckbox is locked.
5. In the Choose how to protect object data section, do the following:
  1. Select any of the options under Data protection that you want to set for your bucket.
  2. To choose how your object data will be encrypted, click the expander arrow labeled Data encryption, and select a Data encryption method.
Click Create.

Configure Log Export in Google Cloud VPC Flow

Sign in to Google Cloudaccount using your privileged account.
On Welcomepage, click VPC Networks.
Click Defaultand a subnet page should appear.
Select all logs.
Click Flow Logs > Configure.
Select Aggregation Interval; for example, 30 SEC.
Provide Sample Rate; for example, 50%.
Click Save
Search Loggingin the search bar and click Enter.
In Log Explorer, filter the logs by choosing VPC_flowsin the Log Nameand click Apply.
Click More Actions.
Click Create Sink.
Provide the following configurations:
1. Sink Details: enter a name and description.
2. Click Next.
3. Sink Destination: select Cloud Storage Bucket.
4. Cloud Storage Bucket: select the bucket created earlier or create a new bucket.
5. Click Next.
6. Choose Logs to include in Sink: a default log is populated when you select an option in Cloud Storage Bucket.
7. Click Next.
8. Optional: Choose Logs to filter out of Sink: select the logs that you would like not to sink.
Click Create Sink.

Set up feeds

There are two different entry points to set up feeds in the Google SecOps platform:

SIEM Settings > Feeds > Add New
Content Hub > Content Packs > Get Started

How to set up the Google Cloud VPC Flow feed

Click the Google Cloud Compute platformpack.
Locate the GCP VPC Flow Feedlogtype.
Specify the values in the following fields.
- Source Type: Amazon SQS V2
- Queue Name: The SQS queue name to read from
- S3 URI: The bucket URI.
  - s3://your-log-bucket-name/
    - Replace your-log-bucket-name with the actual name of your S3 bucket.
- Source deletion options: Select the deletion option according to your ingestion preferences.
  
  Note: If you select the Delete transferred files or Delete transferred files and empty directories option, make sure that you granted appropriate permissions to the service account.
- Maximum File Age: Include files modified in the last number of days. Default is 180 days.
- SQS Queue Access Key ID: An account access key that is a 20-character alphanumeric string.
- SQS Queue Secret Access Key: An account access key that is a 40-character alphanumeric string.
Advanced options
- Feed Name: A prepopulated value that identifies the feed.
- Asset Namespace: Namespace associated with the feed.
- Ingestion Labels: Labels applied to all events from this feed.
Click Create feed.

For more information about configuring multiple feeds for different log types within this product family, see Configure feeds by product .

Supported VPC Flow Logs log formats

The VPC Flow Logs parser supports logs in JSON format.

Supported VPC Flow Logs Sample Logs

JSON

 {
  "insertId": "1wjp1y9f8vc6y6",
  "jsonPayload": {
    "bytes_sent": "0",
    "connection": {
      "dest_ip": "198.51.100.0",
      "dest_port": 32846,
      "protocol": 6,
      "src_ip": "198.51.100.1",
      "src_port": 443
    },
    "dest_instance": {
      "project_id": "logging-259109",
      "region": "us-west2",
      "vm_name": "demisto-01",
      "zone": "us-west2-a"
    },
    "dest_vpc": {
      "project_id": "logging-259109",
      "subnetwork_name": "default",
      "vpc_name": "default"
    },
    "end_time": "2020-03-28T10:44:41.896734136Z",
    "packets_sent": "2",
    "reporter": "DEST",
    "start_time": "2020-03-28T10:44:41.896734136Z"
  },
  "logName": "projects/logging-259109/logs/compute.googleapis.com%2Fvpc_flows",
  "receiveTimestamp": "2020-03-28T10:44:50.112903743Z",
  "resource": {
    "labels": {
      "location": "us-west2-a",
      "project_id": "dummy_project_id",
      "subnetwork_id": "subnetwork_id",
      "subnetwork_name": "default"
    },
    "type": "gce_subnetwork"
  },
  "timestamp": "2020-03-28T10:44:50.112903743Z"
}

UDM Mapping Table

Log field	UDM mapping	Logic
connection.dest_ip	target.asset.ip target.ip	Direct mapping when network.direction is OUTBOUND. Mapped from principal.ip when network.direction is INBOUND.
connection.dest_port	target.port	Converted to integer if greater than -1.
connection.protocol	network.ip_protocol	Converted to string, then mapped to integer. Based on the integer value, mapped to IP protocol name (e.g., TCP, UDP, ICMP).
connection.src_ip	principal.ip	Direct mapping.
connection.src_port	principal.port	Converted to integer.
dest_instance.region	target.location.name	Direct mapping.
dest_instance.vm_name	target.asset.hostname	Direct mapping.
dest_location.city	target.location.city	Direct mapping.
dest_location.country	target.location.country_or_region	Direct mapping.
dest_location.region	target.location.state	Direct mapping.
dest_vpc.project_id	target.namespace	Used with dest_vpc.vpc_name to form the target.namespace.
dest_vpc.vpc_name	target.namespace	Used with dest_vpc.project_id to form the target.namespace.
insertId	metadata.product_log_id	Direct mapping.
jsonPayload.bytes_sent	network.sent_bytes	Renamed to network.sent_bytes and converted to uinteger.
jsonPayload.packets_sent	network.sent_packets	Converted to integer.
labels.tunnel_id	additional.fields	Merged into additional.fields with the key `Tunnel Id` and type `string_value` .
logName	security_result.category_details	Direct mapping.
resource.labels.project_id	target.resource.name	Used to construct the target.resource.name with the format `//cloudresourcemanager.googleapis.com/projects/{resource.labels.project_id}` .
resource.labels.region	target.location.country_or_region	Direct mapping.
resource.labels.subnetwork_id	target.user.attribute.labels	Merged into target.user.attribute.labels with the key `subnetwork_id` .
resource.type	metadata.product_event_type	Direct mapping.
severity	security_result.severity	Mapped to `LOW` if the value is `DEBUG` .
src_gke_details.cluster.cluster_location	principal.resource.attribute.labels	Merged into principal.resource.attribute.labels with the key `cluster_location` .
src_gke_details.cluster.cluster_name	principal.resource.attribute.labels	Merged into principal.resource.attribute.labels with the key `cluster_name` .
src_gke_details.pod.pod_name	principal.resource.attribute.labels	Merged into principal.resource.attribute.labels with the key `pod_name` .
src_gke_details.pod.pod_namespace	principal.resource.attribute.labels	Merged into principal.resource.attribute.labels with the key `pod_namespace` .
src_instance.region	principal.location.name	Direct mapping.
src_instance.vm_name	principal.asset.hostname	Direct mapping.
src_location.city	principal.location.city	Direct mapping.
src_location.country	principal.location.country_or_region	Direct mapping.
src_location.region	principal.location.state	Direct mapping.
src_vpc.project_id	principal.namespace	Used with src_vpc.vpc_name to form the principal.namespace.
src_vpc.vpc_name	principal.namespace	Used with src_vpc.project_id to form the principal.namespace.
textPayload	additional.fields	Merged into additional.fields with the key `Textpayload` and type `string_value` .
timestamp	metadata.event_timestamp	Used to populate event_timestamp if jsonPayload.end_time is empty.
	metadata.description	A description of the network flow, including the reporter (SRC or DEST) and the direction (INBOUND or OUTBOUND), is generated based on the 'reporter' field.
	metadata.event_type	Set to `NETWORK_CONNECTION` for VPC flow logs and `USER_RESOURCE_ACCESS` for other log types.
	metadata.log_type	Set to `GCP_VPC_FLOW` .
	metadata.product_name	Set to `GCP VPC Flow Logs` .
	metadata.product_version	Set to `1.0` .
	metadata.vendor_name	Set to `Google Cloud` .
	network.direction	Determined based on the target.port. If the port is a well-known or reserved port, it's considered INBOUND; otherwise, it's considered OUTBOUND.
	security_result.severity	Set to `LOW` by default.
	target.resource.attribute.cloud.environment	Set to `GOOGLE_CLOUD_PLATFORM` .
	target.resource.resource_type	Set to `CLOUD_PROJECT` .

Need more help? Get answers from Community members and Google SecOps professionals.