Collect Fivetran logs

This document explains how to configure Fivetran to push logs to Google Security Operations using webhooks.

Fivetran is a data integration platform that automates data pipelines from various sources to data warehouses. Fivetran generates operational events including connector sync events, transformation events, and connection status changes. These events can be sent to external endpoints via outbound webhooks for monitoring, alerting, and security analysis.

Before you begin

Make sure you have the following prerequisites:

• A Google SecOps instance.
• A Fivetran account with admin or account-level permissions.
• Access to the Google Cloud Console (for API key creation).
• Fivetran account with REST API access enabled.

Create webhook feed in Google SecOps

Create the feed

1. Go to SIEM Settings > Feeds.
2. Click + Add New.
3. Select Configure a single feed.
4. In the Feed namefield, enter a name (for example, Fivetran Events ).
5. Select Webhookas the Source type.
6. Select Fivetranas the Log type.
7. Click Next.
8. Specify values for the following input parameters:
   • Split delimiter(optional): Leave empty.
   • Asset namespace: The asset namespace .
   • Ingestion labels: The label to be applied to the events from this feed.
9. Click Next.
10. Review your new feed configuration and click Submit.

Generate and save secret key

1. On the feed details page, click Generate Secret Key.
2. A dialog displays the secret key.
3. Copy and savethe secret key securely.

Get the feed endpoint URL

1. Go to the Detailstab of the feed.
2. In the Endpoint Informationsection, copy the Feed endpoint URL.
3. Save this URL for the next steps.
4. Click Done.

Create Google Cloud API key

Google SecOps requires an API key for authentication.

Create the API key

1. Go to the Google Cloud Console Credentials page .
2. Select your project.
3. Click Create credentials > API key.
4. Click Edit API keyto restrict the key.

Restrict the API key

1. In the API keysettings:
   • Name: Enter a descriptive name (for example, SecOps Webhook API Key ).
2. Under API restrictions:
   1. Select Restrict key.
   2. In the dropdown, search for and select Google SecOps API(or Chronicle API).
3. Click Save.
4. Copythe API key value and save it securely.

Configure Fivetran webhook

Construct the webhook URL

• Combine the endpoint URL and API key:

   <ENDPOINT_URL>?key=<API_KEY> 
  

Create webhook using Fivetran REST API

Get Fivetran API credentials

1. Sign in to your Fivetran account .
2. Go to Account Settings > API Config.
3. Click Generate API Keyif you don't have one.
4. Copy the API Keyand API Secret.

Create account-level webhook

Use this method to receive events from all connectors in your account.

1. Open a terminal and run the following command:

    curl  
   -X  
   POST  
    [ 
   https://api.fivetran.com/v1/webhooks/account ]( 
   https://api.fivetran.com/v1/webhooks/account ) 
     
    \ 
     
   -u  
    "API_KEY:API_SECRET" 
     
    \ 
     
   -H  
    "Content-Type: application/json" 
     
    \ 
     
   -H  
    "Accept: application/json" 
     
    \ 
     
   -d  
    '{ 
    "url": "[https://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate?key=YOUR_CHRONICLE_API_KEY](https://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate?key=YOUR_CHRONICLE_API_KEY)", 
    "events": [ 
    "sync_start", 
    "sync_end", 
    "transformation_start", 
    "transformation_succeeded", 
    "transformation_failed", 
    "connection_successful", 
    "connection_failure", 
    "create_connector", 
    "pause_connector", 
    "resume_connector", 
    "edit_connector", 
    "delete_connector", 
    "force_update_connector", 
    "resync_connector", 
    "resync_table" 
    ], 
    "active": true, 
    "secret": "YOUR_CHRONICLE_SECRET_KEY" 
    }' 
    
   

Webhook details

Available webhook events

Event	Description
sync_start	Connector sync started
sync_end	Connector sync completed
transformation_start	Transformation started
transformation_succeeded	Transformation completed successfully
transformation_failed	Transformation failed
connection_successful	Connection test succeeded
connection_failure	Connection test failed
create_connector	New connector created

Webhook retry behavior

Fivetran automatically retries failed webhooks for up to 24 hourswith the following schedule:

Retry	Time After Initial Attempt
1st retry	6 minutes
2nd retry	27 minutes
3rd retry	1 hour 45 minutes
4th retry	6 hours 25 minutes
5th retry	23 hours 13 minutes

UDM mapping table

Log Field	UDM Mapping	Logic
jsonPayload.connector_id	additional.connector_id	Value copied directly.
jsonPayload.connector_type	additional.connector_type	Value copied directly.
jsonPayload.data.query	additional.query	Value copied directly.
N/A	metadata.event_type	Set based on presence of principal and target.
jsonPayload.event	metadata.product_event_type	Value copied directly.
jsonPayload.sync_id	metadata.product_log_id	Value copied directly.
jsonPayload.connector_name	principal.hostname	Value copied directly.
resource.labels.email_id	principal.user.email_addresses	Mapped if format is valid email.
resource.labels.unique_id	principal.user.userid	Value copied directly.
severity	security_result.severity	"INFO" maps to INFORMATIONAL.
logName	target.resource.name	Value copied directly.
N/A	target.resource.type	Set to DATABASE.
N/A	metadata.product_name	Set to FIVETRAN.
N/A	metadata.vendor_name	Set to FIVETRAN.

Change Log

View the Change Log for this parser

Need more help? Get answers from Community members and Google SecOps professionals.