Console
    Contact Us Start free

    Collect AWS EC2 Hosts logs

    Supported in:

    This document explains how to integrate AWS EC2 Hosts logs into Google Security Operations for monitoring and analysis. The integration involves parsing and mapping EC2 host logs to the Unified Data Model (UDM), performing data transformation, and creating relationships between EC2 hosts and instances. The logs provide valuable information about the instances, host properties, instance types, and performance metrics that can be used for security monitoring, audit, and compliance.

    Before you begin

    • Ensure that you have a Google SecOps instance.
    • Ensure that you have privileged access to AWS.

    Configure AWS IAM and S3

    1. Create an Amazon S3 bucketfollowing this user guide: Creating a bucket .
    2. Save the bucket Nameand Regionfor later use.
    3. Create a user following this user guide: Creating an IAM user .
    4. Select the created User.
    5. Select the Security credentialstab.
    6. Click Create Access Keyin the Access Keyssection.
    7. Select Third-party serviceas the Use case.
    8. Click Next.
    9. Optional: add a description tag.
    10. Click Create access key.
    11. Click Download CSV fileto save the Access Keyand Secret Access Keyfor later use.
    12. Click Done.
    13. Select the Permissionstab.
    14. Click Add permissionsin the Permissions policiessection.
    15. Select Add permissions.
    16. Select Attach policies directly
    17. Search for and select the AmazonS3FullAccesspolicy.
    18. Click Next.
    19. Click Add permissions.

    Configure CloudTrail for AWS KMS

    1. Sign in to the AWS Management Console .
    2. In the search bar, type and select CloudTrailfrom the services list.
    3. Click Create trail.
    4. Provide a Trail name; for example, EC2-Activity-Trail .
    5. Select the Enable for all accounts in my organizationcheckbox.
    6. Type the S3 bucket URI created earlier (the format should be: s3://your-log-bucket-name/ ), or create a new S3 bucket.
    7. If SSE-KMS is enabled, provide a name for AWS KMS alias, or choose an existing AWS KMS Key.
    8. You can leave the other settings as default.
    9. Click Next.
    10. Select Management eventsand Data eventsunder Event Typesto capture EC2 host activity.
    11. Click Next.
    12. Review the settings in Review and create.
    13. Click Create trail.
    14. Optional: if you created a new bucket, continue with the following process:
      • Go to S3.
      • Identify and select the newly created log bucket.
      • Select the folder AWSLogs.
      • Click Copy S3 URIand save it.

    Configure a feed in Google SecOps to ingest AWS EC2 Hosts

    1. Go to SIEM Settings > Feeds.
    2. Click Add new.
    3. In the Feed namefield, enter a name for the feed; for example, AWS EC2 Hosts Logs.
    4. Select Amazon S3 V2as the Source type.
    5. Select AWS EC2 Hostsas the Log type.
    6. Click Next.

    7. Specify values for the following input parameters:

      • S3 URI: the bucket URI.
        • s3://your-log-bucket-name/
          • Replace your-log-bucket-name with the actual name of the bucket.

      • Source deletion options: select the deletion option according to your preference.

      • Maximum File Age: Include files modified in the last number of days. Default is 180 days.

      • Access Key ID: the User access key with access to the s3 bucket.

      • Secret Access Key: the User secret key with access to the s3 bucket.

      • Asset namespace: the asset namespace .

      • Ingestion labels: the label to be applied to the events from this feed.

    8. Click Next.

    9. Review your new feed configuration in the Finalizescreen, and then click Submit.

    UDM Mapping Table

    Log Field UDM Mapping Logic
    AllocationTime
    		entity.metadata.creation_timestamp The AllocationTime field is parsed as a timestamp and mapped to the creation_timestamp field. The parser attempts various formats (yyyy-MM-dd HH:mm:ss, RFC 3339, UNIX, ISO8601).
    AllowsMultipleInstanceTypes
    		entity.entity.asset.attribute.labels.value The value of AllowsMultipleInstanceTypes from the raw log is used as the value of a label. The key for this label is set to allows_multiple_instance_types .
    AutoPlacement
    		entity.entity.asset.attribute.labels.value The value of AutoPlacement from the raw log is used as the value of a label. The key for this label is set to auto_placement .
    AvailabilityZone
    		entity.entity.asset.attribute.cloud.availability_zone The AvailabilityZone field is directly mapped to the availability_zone field.
    AvailabilityZoneID
    		entity.entity.asset.attribute.labels.value The value of AvailabilityZoneID from the raw log is used as the value of a label. The key for this label is set to availability_zone_id .
    AvailableCapacity.AvailableInstanceCapacity.AvailableCapacity
    		entity.entity.asset.attribute.labels.value The value of AvailableCapacity.AvailableInstanceCapacity.AvailableCapacity (or AvailableCapacity.Instance.Available after renaming) is converted to a string and used as the value of a label. The key is set to available_instance_capacity_available_capacity .
    AvailableCapacity.AvailableInstanceCapacity.InstanceType
    		entity.entity.asset.attribute.labels.value The value of AvailableCapacity.AvailableInstanceCapacity.InstanceType (or AvailableCapacity.Instance.Type after renaming) is used as the value of a label. The key is set to available_instance_capacity_instance_type .
    AvailableCapacity.AvailableInstanceCapacity.TotalCapacity
    		entity.entity.asset.attribute.labels.value The value of AvailableCapacity.AvailableInstanceCapacity.TotalCapacity (or AvailableCapacity.Instance.Total after renaming) is converted to a string and used as the value of a label. The key is set to total_capacity .
    AvailableCapacity.AvailableVCpus
    		entity.entity.asset.attribute.labels.value The value of AvailableCapacity.AvailableVCpus (or AvailableCapacity.VCPUs after renaming) is converted to a string and used as the value of a label. The key is set to available_v_cpus .
    ClientToken
    		entity.entity.asset.attribute.labels.value The value of ClientToken from the raw log is used as the value of a label. The key for this label is set to client_token .
    HostID
    		entity.metadata.product_entity_id The HostID (or ID after renaming) field is directly mapped to the product_entity_id field.
    HostID
    		entity.entity.asset.asset_id The HostID (or ID after renaming) field is directly mapped to the asset_id field.
    HostMaintenance
    		entity.entity.asset.attribute.labels.value The value of HostMaintenance (or Maintenance after renaming) from the raw log is used as the value of a label. The key for this label is set to host_maintenance .
    HostProperties.Cores
    		entity.entity.asset.hardware.cpu_number_cores The value of HostProperties.Cores is converted to an unsigned integer and mapped to cpu_number_cores .
    HostProperties.InstanceFamily
    		entity.entity.asset.attribute.labels.value The value of HostProperties.InstanceFamily from the raw log is used as the value of a label. The key for this label is set to host_properties_instance_family .
    HostProperties.InstanceType
    		entity.entity.asset.attribute.labels.value The value of HostProperties.InstanceType from the raw log is used as the value of a label. The key for this label is set to host_properties_instance_type .
    HostProperties.Sockets
    		entity.entity.asset.attribute.labels.value The value of HostProperties.Sockets is converted to a string and used as the value of a label. The key is set to host_properties_sockets .
    HostProperties.TotalVCpus
    		entity.entity.asset.attribute.labels.value The value of HostProperties.TotalVCpus (or HostProperties.TotalVCPUs after renaming) is converted to a string and used as the value of a label. The key is set to host_properties_total_v_cpus .
    HostRecovery
    		entity.entity.asset.attribute.labels.value The value of HostRecovery (or Recovery after renaming) from the raw log is used as the value of a label. The key for this label is set to host_recovery .
    HostReservationID
    		entity.entity.asset.attribute.labels.value The value of HostReservationID (or ReservationID after renaming) from the raw log is used as the value of a label. The key for this label is set to host_reservation_id .
    MemberOfServiceLinkedResourceGroup
    		entity.entity.asset.attribute.labels.value The value of MemberOfServiceLinkedResourceGroup is converted to a string and used as the value of a label. The key is set to member_of_service_linked_resource_group .
    OwnerID
    		entity.entity.asset.attribute.labels.value The value of OwnerID from the raw log is used as the value of a label. The key for this label is set to owner_id .
    ReleaseTime
    		entity.entity.asset.attribute.labels.value The value of ReleaseTime from the raw log is used as the value of a label. The key for this label is set to release_time .
    State
    		entity.entity.asset.attribute.labels.value The value of State from the raw log is used as the value of a label. The key for this label is set to state .
    TagSet
    		entity.entity.asset.attribute.labels The TagSet array is iterated over, and each tag's Key and Value are used as the key and value of a label, respectively. The value AMAZON_WEB_SERVICES is assigned to this field by the parser. The collection_time from the raw log is mapped to the collected_timestamp field. The value ASSET is assigned to this field by the parser. The value AWS EC2 HOSTS is assigned to this field by the parser. The value AWS is assigned to this field by the parser. Relations are derived from Instances and OutpostArn fields, but these fields themselves are not directly mapped to the IDM object. The parser logic creates relation objects based on these fields and merges them into the relations array.
    collection_time
    		entity.metadata.collected_timestamp The log's collection_time is used as the event's collected_timestamp .

    Need more help? Get answers from Community members and Google SecOps professionals.

    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: