About VPC Flow Logs records

This page describes the VPC Flow Logs record format, including which base and metadata fields are available. It also explains how you can use log filtering so that only logs that match certain criteria are generated.

Record format

Log records contain base fields, which are the core fields of every log record, and metadata fields that add additional information. Metadata fields may be omitted to save storage costs.

Some log fields are in a multi-field format, with more than one piece of data in a given field. For example, the connection field is of the IpConnection format, which contains the source and destination IP address and port, plus the protocol, in a single field. These multi-field fields are described below the record format table.

The values for metadata fields aren't based on the data plane path; they are approximations and might be missing or incorrect. Unlike metadata fields, the values for base fields are taken directly from packet headers.

IpConnection field format

GatewayDetails field format

GkeDetails field format

ClusterDetails field format

PodDetails field format

WorkloadDetails field format

ServiceDetails field format

Example:

If there are two services, the Service field looks like this:

service: [
 0: {
  service_name: "my-lb-service"
  service_namespace: "default"
 }
 1: {
  service_name: "my-lb-service2"
  service_namespace: "default"
 }
]

If there are more than two services, the Service field looks like this:

service: [
 0: {
  service_name: "MANY_SERVICES"
 }
]

GoogleServiceDetails field format

InstanceDetails field format

InstanceGroupDetails field format

GeographicDetails field format

VpcDetails field format

InternetRoutingDetails field format

AsPath field format

AsDetails field format

LoadBalancingDetails field format

NetworkServiceDetails field format

PrivateServiceConnectDetails field format

PrivateServiceConnectEndpointDetails field format

PrivateServiceConnectAttachmentDetails field format

Metadata annotations

Log records contain base fields and metadata fields. The Record format section lists which fields are type metadata and which are type base. All base fields are always included. You can customize which metadata fields you keep.

• If you select all metadata, all metadata fields in the VPC Flow Logs record format are included in the flow logs. When new metadata fields are added to the record format, the flow logs automatically include the new fields.

• If you select no metadata, this omits all metadata fields.

• If you select custom metadata, you can specify the metadata fields that you want to include by the parent field, such as src_vpc , or by their full names, such as src_vpc.project_id

  When new metadata fields are added to the record format, they're excluded from the flow logs unless they're within a parent field that you have specified to include.

  • If you specify custom metadata using parent fields, when new metadata fields are added to the record format within that parent field, the flow logs will automatically include the new fields.

  • If you specify custom metadata using the full name of the field, new metadata fields that are added to the parent field are excluded from the flow logs.

For information about how to customize metadata fields, see Enable VPC Flow Logs or Update VPC Flow Logs configuration .

GKE metadata annotations

Flows that have an endpoint in a GKE Cluster can be annotated with GKE metadata annotations , which can include details of the Cluster, Pod, and Service of the endpoint.

GKE Service annotations

Traffic sent to a ClusterIP, NodePort, or LoadBalancer can receive Service annotations. If sent to a NodePort or LoadBalancer, the flow receives the Service annotation on both hops of the connection.

Traffic sent directly to a Pod's Service port is annotated with a Service annotation on the destination endpoint.

Traffic sent to a Pod's Service port where the Pod is backing more than one Service on the same Service port is annotated with multiple Services on the destination endpoint. This is limited to two Services. If there are more than that, the endpoint will be annotated with a special MANY_SERVICES marker.

Pod annotations on internet traffic

Traffic between a Pod and the internet doesn't receive Pod annotations by default. VPC Flow Logs can't add Pod annotations because, for packets to the internet, the masquerade agent translates the Pod IP address to the node IP address before VPC Flow Logs sees the packet.

Because of the masquerade, Pod annotations are only visible if the destinations are within either the default non-masquerade destinations or in a custom nonMasqueradeCIDRs list . If you include internet destinations in a custom nonMasqueradeCIDRs list, you need to provide a way for the internal Pod IP addresses to be translated before they are delivered to the internet. For both private and non-private clusters, you can use Cloud NAT. See GKE interaction for more details.

Cross-project annotations

If VPC Flow Logs is enabled at the organization level , flows through Shared VPC, VPC Network Peering, and Network Connectivity Center are annotated with cross-project annotations. Cross-project annotations are enabled by default. You can optionally disable these annotations.

• If cross-project annotations are enabled, log records for flows between resources in different projects include information about both sides of the flow .
• If cross-project annotations are disabled, log records include information only about the reporter of the flow.

If VPC Flow Logs is enabled at the project level, flows between resources in different projects aren't annotated with cross-project annotations.

Log filtering

When you enable VPC Flow Logs, you can set a filter based on both base and metadata fields that only preserves logs that match the filter. All other logs are discarded before being written to Logging.

You can filter on any subset of fields listed in Record format , except for the following fields:

• rtt_msec
• bytes_sent
• packets_sent
• start_time
• end_time

VPC Flow Logs filtering uses CEL, an embedded expression language for attribute-based logic expressions. Filter expressions for VPC Flow Logs have a limit of 2,048 characters. For more information, see Supported CEL logic operators .

For more information about CEL, see the CEL introduction and the language definition . The generation filter feature supports a limited subset of CEL syntax.

To create a VPC Flow Logs configuration that uses log filtering, see Enable VPC Flow Logs . To configure log filtering for an existing VPC Flow Logs configuration, see Update VPC Flow Logs configuration .

Supported CEL logic operators

Examples of log filters

If you enabled VPC Flow Logs for a subnet by using the Compute Engine API , use the gcloud compute networks subnets update command to configure filtering (Example 1–3).

For all other VPC Flow Logs configurations, use the gcloud network-management vpc-flow-logs-configs update command (Example 4–6). The filter expressions in Example 1–3 can be used with the gcloud network-management vpc-flow-logs-configs update command.

Example 1. Limit logs collection to a specific VM named my-vm . In this case, only logs where the src_instance field as reported by the source of the traffic is my-vm or the dst_instance field as reported by the destination of the traffic is my-vm are recorded.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr="(src_instance.vm_name == 'my-vm' && reporter=='SRC') || (dest_instance.vm_name == 'my-vm' && reporter=='DEST')"

Example 2. Limit logs collection to packets whose source IP addresses are in the 10.0.0.0/8 subnet.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr="inIpRange(connection.src_ip, '10.0.0.0/8')"

Example 3. Limit logs collection to traffic that is external to a VPC network.

gcloud compute networks subnets update my-subnet \
    --logging-filter-expr '!(has(src_vpc.vpc_name) && has(dest_vpc.vpc_name))'

Example 4. Limit logs collection to a specific destination VLAN attachment or Cloud VPN tunnel, my-gateway .

gcloud network-management vpc-flow-logs-configs update my-config \
    --location=global \
    --filter-expr="dest_gateway.name == 'my-gateway'"

Example 5. Limit logs collection to VLAN attachments.

gcloud network-management vpc-flow-logs-configs update my-config \
    --location=global \
    --filter-expr="dest_gateway.type == 'INTERCONNECT_ATTACHMENT'"

Example 6. Limit logs collection to a specific source VPC network, my-network .

gcloud beta network-management vpc-flow-logs-configs update my-config \
    --location=global \
    --filter-expr="src_vpc.vpc_name == 'my-network'"

What's next

• Configure VPC Flow Logs
• Access flow logs