Collect Gmail logs

This document explains how to ingest Gmail logs to Google Security Operations using Google Cloud Storage V2.

Gmail is Google Workspace's email service that provides secure, intelligent email with built-in spam and phishing protection. Gmail logs capture detailed information about email delivery, security events, and message flow through the Gmail infrastructure.

Before you begin

Ensure that you have the following prerequisites:

• A Google SecOps instance
• GCP project with Cloud Storage API and BigQuery API enabled
• Google Workspace account with appropriate edition (Enterprise Standard, Enterprise Plus, Education Standard, or Education Plus)
• Super administrator access to Google Workspace Admin console
• Permissions to create and manage GCS buckets
• Permissions to manage IAM policies on GCS buckets
• Permissions to create BigQuery scheduled queries
• The service account gapps-reports@system.gserviceaccount.com must have editor role on the BigQuery project

Create Google Cloud Storage bucket

Using Google Cloud Console

1. Go to the Google Cloud Console .
2. Select your project or create a new one.
3. In the navigation menu, go to Cloud Storage > Buckets.
4. Click Create bucket.

5. Provide the following configuration details:

   Setting	Value
   Name your bucket	Enter a globally unique name (for example, gmail-logs-export )
   Location type	Choose based on your needs (Region, Dual-region, Multi-region)
   Location	Select the location (for example, us-central1 )
   Storage class	Standard (recommended for frequently accessed logs)
   Access control	Uniform (recommended)
   Protection tools	Optional: Enable object versioning or retention policy

6. Click Create.

Using gcloud command-line tool

• Alternatively, create a bucket using the gcloud command:

   gcloud  
  storage  
  buckets  
  create  
  gs://gmail-logs-export  
   \ 
    
  --location = 
  us-central1  
   \ 
    
  --default-storage-class = 
  STANDARD 
  

  • Replace:
    • gmail-logs-export : Your desired bucket name (globally unique).
    • us-central1 : Your preferred region (for example, us-central1 , europe-west1 ).

Configure Google Workspace to export logs to BigQuery

Google Workspace logs, including Gmail logs, are exported to BigQuery through the unified Workspace logs and reports feature.

Enable BigQuery export for Workspace logs

1. Sign in with a super administrator account to the Google Admin console .
2. Go to Menu > Reporting > Data integrations.
3. Point to the BigQuery Exportcard and click Edit.
4. Check the Enable Google Workspace data export to Google BigQuerybox.
5. Under BigQuery project ID, select the project where you want to store the logs.
6. Under New dataset within project, enter the name of the dataset to use for storing the logs (for example, workspace_logs ).
7. Optional: Check the Restrict the dataset to a specific geographic locationbox and select the location from the menu.

8. Click Save.

After enabling the export, activity log events are typically available within 10 minutes. The data is exported to tables named activity_YYYYMMDD in the specified dataset.

Create scheduled query to export from BigQuery to GCS

To automatically export Gmail logs from BigQuery to Cloud Storage on a recurring schedule, create a scheduled query using the EXPORT DATA statement.

Using BigQuery Console

1. In the Google Cloud Console , go to BigQuery.
2. In the left navigation, click Scheduled queries.
3. Click Create scheduled query.

4. In the Query editor, enter the following SQL:

     EXPORT 
     
    DATA 
     
    OPTIONS 
    ( 
     
    uri 
    = 
    'gs://gmail-logs-export/gmail-logs/*.json' 
    , 
     
    format 
    = 
    'JSON' 
    , 
     
    overwrite 
    = 
    false 
    ) 
     
    AS 
    SELECT 
     
     
    * 
    FROM 
     
     
    ` 
    PROJECT_ID 
    . 
    workspace_logs 
    . 
    activity_ 
    *` 
    WHERE 
     
     
    record_type 
     
    = 
     
    'gmail' 
     
    AND 
     
    _TABLE_SUFFIX 
     
    = 
     
    FORMAT_DATE 
    ( 
    '%Y%m%d' 
    , 
     
    DATE_SUB 
    ( 
    CURRENT_DATE 
    (), 
     
    INTERVAL 
     
    1 
     
    DAY 
    )) 
    
   

   • Replace:
     • gmail-logs-export : Your GCS bucket name
     • PROJECT_ID : Your GCP project ID
     • workspace_logs : Your BigQuery dataset name

5. In the Schedule optionssection:

   • Repeat: Select Hours.
   • Every: Enter 1 .
   • Start date and run time: Select current date and time.

6. In the Destination for query resultssection:

   • Dataset: Select a dataset for query metadata (not the exported data).

7. Click Save.

Using bq command-line tool

• Alternatively, create a scheduled query using the bq command:

   bq  
  mk  
   \ 
    
  --transfer_config  
   \ 
    
  --project_id = 
  PROJECT_ID  
   \ 
    
  --data_source = 
  scheduled_query  
   \ 
    
  --display_name = 
   'Gmail Logs Export to GCS' 
    
   \ 
    
  --schedule = 
   'every 1 hours' 
    
   \ 
    
  --params = 
   '{ 
   "query":"EXPORT DATA OPTIONS(uri=\"gs://gmail-logs-export/gmail-logs/*.json\", format=\"JSON\", overwrite=false) AS SELECT * FROM `PROJECT_ID.workspace_logs.activity_*` WHERE record_type = \"gmail\" AND _TABLE_SUFFIX = FORMAT_DATE(\"%Y%m%d\", DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))", 
   "destination_table_name_template":"gmail_export_metadata", 
   "write_disposition":"WRITE_TRUNCATE" 
   }' 
   
  

  • Replace:
    • PROJECT_ID : Your GCP project ID
    • gmail-logs-export : Your GCS bucket name
    • workspace_logs : Your BigQuery dataset name

Retrieve the Google SecOps service account

Google SecOps uses a unique service account to read data from your GCS bucket. You must grant this service account access to your bucket.

Configure a feed in Google SecOps to ingest Gmail logs

1. Go to SIEM Settings > Feeds.
2. Click Add New Feed.
3. Click Configure a single feed.
4. In the Feed namefield, enter a name for the feed (for example, Gmail Logs ).
5. Select Google Cloud Storage V2as the Source type.

6. Select GMAIL Logsas the Log type.

7. Click Get Service Account. A unique service account email is displayed, for example:

    chronicle-12345678@chronicle-gcp-prod.iam.gserviceaccount.com 
   

8. Copy this email address for use in the next step.

9. Click Next.

10. Specify values for the following input parameters:

    • Storage bucket URL: Enter the GCS bucket URI with the prefix path:

       gs://gmail-logs-export/gmail-logs/ 
      

      • Replace:
        • gmail-logs-export : Your GCS bucket name
        • gmail-logs : The prefix/folder path where logs are stored

    • Source deletion option: Select the deletion option according to your preference:

      • Never: Never deletes any files after transfers (recommended for testing).
      • Delete transferred files: Deletes files after successful transfer.

      • Delete transferred files and empty directories: Deletes files and empty directories after successful transfer.

    • Maximum File Age: Include files modified in the last number of days. Default is 180 days.

    • Asset namespace: The asset namespace .

    • Ingestion labels: The label to be applied to the events from this feed.

11. Click Next.

12. Review your new feed configuration in the Finalizescreen, and then click Submit.

Grant IAM permissions to the Google SecOps service account

The Google SecOps service account needs Storage Object Viewerrole on your GCS bucket.

Using Google Cloud Console

1. Go to Cloud Storage > Buckets.
2. Click your bucket name.
3. Go to the Permissionstab.
4. Click Grant access.
5. Provide the following configuration details:
   • Add principals: Paste the Google SecOps service account email.
   • Assign roles: Select Storage Object Viewer.
6. Click Save.

Using gcloud command-line tool

• Alternatively, grant permissions using the gcloud command:

   gcloud  
  storage  
  buckets  
  add-iam-policy-binding  
  gs://gmail-logs-export  
   \ 
    
  --member = 
   "serviceAccount:SECOPS_SERVICE_ACCOUNT_EMAIL" 
    
   \ 
    
  --role = 
   "roles/storage.objectViewer" 
   
  

  • Replace:
    • gmail-logs-export : Your bucket name.
    • SECOPS_SERVICE_ACCOUNT_EMAIL : The Google SecOps service account email.

Using gsutil command-line tool (legacy)

• Grant the SecOps service account Object Viewer access to the Gmail logs export bucket.

   gsutil  
  iam  
  ch  
  serviceAccount:SECOPS_SERVICE_ACCOUNT_EMAIL:objectViewer  
   \ 
    
  gs://gmail-logs-export 
  

Verify permissions

• To verify the permissions were granted correctly:

   gcloud  
  storage  
  buckets  
  get-iam-policy  
  gs://gmail-logs-export  
   \ 
    
  --flatten = 
   "bindings[].members" 
    
   \ 
    
  --filter = 
   "bindings.role:roles/storage.objectViewer" 
   
  

You should see the Google SecOps service account email in the output.

UDM mapping table

Need more help? Get answers from Community members and Google SecOps professionals.