Collect AWS ECS Metrics logs

Supported in:

Google secops SIEM

This document explains how to ingest AWS ECS Metrics logs to Google Security Operations using Amazon S3.

Amazon Elastic Container Service (Amazon ECS) is a fully managed container orchestration service that helps you deploy, manage, and scale containerized applications. CloudWatch Container Insights collects, aggregates, and summarizes metrics and logs from ECS clusters, services, tasks, and containers, including CPU utilization, memory usage, network traffic, and storage I/O. This integration uses Amazon Data Firehose to stream Container Insights performance log events from CloudWatch Logs to an S3 bucket, which Google SecOps then ingests using an Amazon S3 V2 feed.

Before you begin

Make sure you have the following prerequisites:

A Google SecOps instance
Privileged access to the AWS Management Consolewith permissions to manage:
- Amazon ECS(clusters, account settings)
- Amazon CloudWatch Logs(log groups, subscription filters)
- Amazon Data Firehose(delivery streams)
- Amazon S3(buckets)
- AWS IAM(roles, policies, users)

Enable Container Insights on Amazon ECS

Container Insights must be enabled to generate performance log events in CloudWatch Logs for your ECS clusters.

Enable Container Insights at the account level

Sign in to the Amazon ECS consoleat https://console.aws.amazon.com/ecs/v2 .
In the navigation pane, select Account Settings.
Click Update.
Select Container Insights with enhanced observability.
Click Save changes.
On the confirmation screen, click Confirm.

Note: After enabling the account setting, all new ECS clusters automatically use Container Insights with enhanced observability. Existing clusters require a separate update.

Enable Container Insights on an existing cluster

To update an existing ECS cluster, run the following AWS CLI command (replace your-cluster-name with your cluster name):
```
 aws  
ecs  
update-cluster-settings  
--cluster  
your-cluster-name  
--settings  
 name 
 = 
containerInsights,value = 
enhanced 
```
Note: Amazon ECS agent version 1.29 or later is required for EC2 launch type instances. Fargate launch type clusters do not require any agent updates.

Verify the CloudWatch log group

After enabling Container Insights, Amazon ECS automatically creates a CloudWatch Logs log group with the following naming convention:
```
 /aws/ecs/containerinsights/your-cluster-name/performance 
```

To verify:

In the AWS Console, go to CloudWatch > Logs > Log groups.
Search for /aws/ecs/containerinsights/ .
Confirm that a log group exists for each ECS cluster with Container Insights enabled.

Configure an AWS S3 bucket

Create an Amazon S3 bucketfollowing this user guide: Creating a bucket .
Save the bucket Nameand Regionfor future reference (for example, ecs-metrics-to-secops ).

Configure the IAM role for Amazon Data Firehose

Amazon Data Firehose requires an IAM role to write logs to your S3 bucket.

Create the IAM policy

In the AWS Console, go to IAM > Policies > Create policy.
Select the JSONtab.

Paste the following policy (replace ecs-metrics-to-secops with your actual bucket name):

  { 
  
 "Version" 
 : 
  
 "2012-10-17" 
 , 
  
 "Statement" 
 : 
  
 [ 
  
 { 
  
 "Sid" 
 : 
  
 "S3Delivery" 
 , 
  
 "Effect" 
 : 
  
 "Allow" 
 , 
  
 "Action" 
 : 
  
 [ 
  
 "s3:AbortMultipartUpload" 
 , 
  
 "s3:GetBucketLocation" 
 , 
  
 "s3:GetObject" 
 , 
  
 "s3:ListBucket" 
 , 
  
 "s3:ListBucketMultipartUploads" 
 , 
  
 "s3:PutObject" 
  
 ], 
  
 "Resource" 
 : 
  
 [ 
  
 "arn:aws:s3:::ecs-metrics-to-secops" 
 , 
  
 "arn:aws:s3:::ecs-metrics-to-secops/*" 
  
 ] 
  
 }, 
  
 { 
  
 "Sid" 
 : 
  
 "CloudWatchLogging" 
 , 
  
 "Effect" 
 : 
  
 "Allow" 
 , 
  
 "Action" 
 : 
  
 [ 
  
 "logs:PutLogEvents" 
  
 ], 
  
 "Resource" 
 : 
  
 "arn:aws:logs:*:*:log-group:/aws/kinesisfirehose/ecs-metrics-to-secops:log-stream:*" 
  
 } 
  
 ] 
 }

Click Next.
In the Policy namefield, enter ECSMetricsFirehoseS3Policy .
Click Create policy.

Create the IAM role

Go to IAM > Roles > Create role.
Select Custom trust policy.

Paste the following trust policy:

  { 
  
 "Version" 
 : 
  
 "2012-10-17" 
 , 
  
 "Statement" 
 : 
  
 [ 
  
 { 
  
 "Effect" 
 : 
  
 "Allow" 
 , 
  
 "Principal" 
 : 
  
 { 
  
 "Service" 
 : 
  
 "firehose.amazonaws.com" 
  
 }, 
  
 "Action" 
 : 
  
 "sts:AssumeRole" 
  
 } 
  
 ] 
 }

Click Next.
Search for and select ECSMetricsFirehoseS3Policy.
Click Next.
In the Role namefield, enter ECSMetricsFirehoseToS3Role .
Click Create role.

Create the Amazon Data Firehose stream

Open the Kinesis consoleat https://console.aws.amazon.com/kinesis .
In the navigation pane, select Amazon Data Firehose.
Click Create Firehose stream.
Under Choose source and destination, provide the following configuration:
- Source: Select Direct PUT.
- Destination: Select Amazon S3.
In the Firehose stream namefield, enter ecs-metrics-to-secops .
Under Transform records, in the Decompress source records from Amazon CloudWatch Logssection:
1. Select Turn on decompression.
2. Do notselect Turn on message extraction.
Note: Keeping message extraction disabled preserves CloudWatch metadata (logGroup, logStream, subscriptionFilters) in the delivered logs. The AWS_ECS_METRICS parser supports the full CloudWatch Logs envelope format including the logEvents structure.
Under Destination settings:
- S3 bucket: Select the S3 bucket ecs-metrics-to-secops .
- S3 bucket prefix(optional): Enter ecs-metrics/ .
- S3 bucket error output prefix(optional): Enter firehose-errors/ .
Under Buffer hints:
- Buffer size: 5 MiB (default).
- Buffer interval: 300 seconds (default).
Under Advanced settings:
- Server-side encryption: Optional. Enable if encryption is required.
- Error logging: Select Enabled(recommended).
- Permissions: Select Choose existing IAM role, then select ECSMetricsFirehoseToS3Role .
Click Create Firehose stream.
Wait for the stream Statusto show Active.

Configure the IAM role for CloudWatch Logs

CloudWatch Logs requires an IAM role to send log data to the Firehose stream.

Create the IAM policy

Go to IAM > Policies > Create policy.
Select the JSONtab.

Paste the following policy (replace <region> and <account-id> with your AWS region and account ID):

  { 
  
 "Version" 
 : 
  
 "2012-10-17" 
 , 
  
 "Statement" 
 : 
  
 [ 
  
 { 
  
 "Effect" 
 : 
  
 "Allow" 
 , 
  
 "Action" 
 : 
  
 [ 
  
 "firehose:PutRecord" 
 , 
  
 "firehose:PutRecordBatch" 
  
 ], 
  
 "Resource" 
 : 
  
 "arn:aws:firehose:<region>:<account-id>:deliverystream/ecs-metrics-to-secops" 
  
 } 
  
 ] 
 }

Click Next.
In the Policy namefield, enter ECSMetricsCWLtoFirehosePolicy .
Click Create policy.

Create the IAM role

Go to IAM > Roles > Create role.
Select Custom trust policy.

Paste the following trust policy (replace <region> with your AWS region):

  { 
  
 "Version" 
 : 
  
 "2012-10-17" 
 , 
  
 "Statement" 
 : 
  
 [ 
  
 { 
  
 "Effect" 
 : 
  
 "Allow" 
 , 
  
 "Principal" 
 : 
  
 { 
  
 "Service" 
 : 
  
 "logs.<region>.amazonaws.com" 
  
 }, 
  
 "Action" 
 : 
  
 "sts:AssumeRole" 
  
 } 
  
 ] 
 }

Click Next.
Search for and select ECSMetricsCWLtoFirehosePolicy.
Click Next.
In the Role namefield, enter ECSMetricsCWLtoFirehoseRole .
Click Create role.

Create a CloudWatch Logs subscription filter

In the AWS Console, go to CloudWatch > Logs > Log groups.
Select the log group /aws/ecs/containerinsights/your-cluster-name/performance .
Select the Subscription filterstab.
Click Create > Create Amazon Data Firehose subscription filter.
Provide the following configuration details:
- Destination: Select the Firehose stream ecs-metrics-to-secops .
- Grant permission: Select the role ECSMetricsCWLtoFirehoseRole .
- Subscription filter name: Enter a descriptive name (for example, ecs-metrics-to-secops-filter ).
- Log format: Select Other.
- Subscription filter pattern: Leave empty to send all Container Insights performance events.
Click Start streaming.

Note: Repeat steps 1-6 for each additional ECS cluster log group you want to stream to Google SecOps. Each cluster has a separate log group at /aws/ecs/containerinsights/<cluster-name>/performance . All subscription filters can use the same Firehose stream and IAM roles.

Configure an IAM user for Google SecOps

Google SecOps needs an IAM user with access to the S3 bucket to ingest the delivered logs.

Create a Userfollowing this user guide: Creating an IAM user .
Select the created User.
Select Security credentialstab.
Click Create Access Keyin section Access Keys.
Select Third-party serviceas Use case.
Click Next.
- Optional: Add a description tag.
Click Create access key.
Click Download .csv fileto save the Access Keyand Secret Access Keyfor future reference.
Click Done.
Select Permissionstab.
Click Add permissionsin the Permissions policiessection.
Select Add permissions.
Select Attach policies directly.
Search for AmazonS3FullAccesspolicy.
Select the policy.
Click Next.
Click Add permissions.

Configure a feed in Google SecOps to ingest AWS ECS Metrics logs

Go to SIEM Settings > Feeds.
Click Add New Feed.
On the next page, click Configure a single feed.
Enter a unique name for the Feed name.
Select Amazon S3 V2as the Source type.
Select AWS ECS Metricsas the Log type.
Click Nextand then click Submit.
Specify values for the following fields:
- S3 URI: s3://ecs-metrics-to-secops/ecs-metrics/
- Source deletion option: Select the deletion option according to your preference
- Maximum File Age: Include files modified in the last number of days (default is 180 days)
- Access Key ID: User access key with access to the S3 bucket
- Secret Access Key: User secret key with access to the S3 bucket
- Asset namespace: The asset namespace
- Ingestion labels: The label to be applied to the events from this feed
Click Nextand then click Submit.

UDM mapping table

Log Field	UDM Mapping	Logic
`cluster_rtt_ClusterId_label`	`additional.fields`	Merged
`cluster_rtt_description_label`	`additional.fields`	Merged
`compressed_label`	`additional.fields`	Merged
`connectionID_localvalue_label`	`additional.fields`	Merged
`connectionID_serverValue_label`	`additional.fields`	Merged
`request_id_label`	`additional.fields`	Merged
`source_label`	`additional.fields`	Merged
`Time`	`metadata.event_timestamp`	Parsed as `RFC 3339`
`has_principal`	`metadata.event_type`	Mapped: `true` → `STATUS_UPDATE` , `true` → `NETWORK_CONNECTION`
`connection`	`metadata.product_event_type`	Directly mapped
`sent_bytes`	`network.sent_bytes`	Directly mapped
`hostname`	`principal.asset.hostname`	Directly mapped
`principal_ip`	`principal.asset.ip`	Merged
`hostname`	`principal.hostname`	Directly mapped
`principal_ip`	`principal.ip`	Merged
`principal_port`	`principal.port`	Directly mapped
`action`	`security_result.action`	Merged
`severity`	`security_result.severity`	Mapped: `"ALERT", "EMERGENCY"` → `HIGH` , `"INFO", "NOTICE"` → `INFORMATIONAL` , `DEBUG` → `LO...
`target_port`	`target.port`	Directly mapped
`url`	`target.url`	Directly mapped
N/A	`metadata.event_type`	Constant: `STATUS_UPDATE`
N/A	`security_result.severity`	Constant: `CRITICAL`

Change Log

View the Change Log for this parser

Need more help? Get answers from Community members and Google SecOps professionals.