Console
    Start free

    Storage Insights datasets tables and schemas

    This document describes the tables and their schemas included in Storage Insights datasets , which you can use to analyze operational patterns, monitor resource usage, and understand activity across your storage environment.

    Dataset schema of metadata

    The following sections describe the metadata fields included in datasets. For more information about BigQuery column modes, see Modes . The column modes determine how BigQuery stores and queries the data.

    Bucket metadata

    The bucket metadata schema includes the bucket_attributes_view and bucket_attributes_latest_snapshot_view tables. The bucket metadata tables have the following fields:

    Metadata field Mode Type Description
    snapshotTime
    		NULLABLE TIMESTAMP The snapshotTime field stores the time of the bucket metadata snapshot refresh in RFC 3339 format .
    name
    		NULLABLE STRING The name of the bucket.
    location
    		NULLABLE STRING The location of the bucket. Object data for objects in the bucket resides in physical storage within this location.
    project
    		NULLABLE INTEGER The project number of the project the bucket belongs to.
    storageClass
    		NULLABLE STRING The bucket's default storage class .
    public
    		NULLABLE RECORD Deprecated. This field indicates whether a bucket is publicly accessible. Use iamConfiguration instead.
    public.bucketPolicyOnly
    		NULLABLE BOOLEAN Deprecated. This field, part of the public record, indicates whether uniform bucket-level access is enabled, which prevents access through object-level ACLs.
    public.publicAccessPrevention
    		NULLABLE STRING Deprecated. This field, part of the public record, indicates whether public access to the bucket is prevented.
    autoclass
    		NULLABLE RECORD The bucket's Autoclass configuration , which, when enabled, controls the storage class of objects based on how and when the objects are accessed.
    autoclass.enabled
    		NULLABLE BOOLEAN Whether or not Autoclass is enabled.
    autoclass.toggleTime
    		NULLABLE TIMESTAMP The time at which Autoclass was last enabled or disabled for this bucket, in RFC 3339 format.
    versioning
    		NULLABLE BOOLEAN Whether or not the bucket has versioning enabled. For more information, see Object Versioning .
    lifecycle
    		NULLABLE BOOLEAN Whether or not the bucket has a lifecycle configuration. See lifecycle management for more information.
    metageneration
    		NULLABLE INTEGER The metadata generation of this bucket.
    timeCreated
    		NULLABLE TIMESTAMP The creation time of the bucket in RFC 3339 format.
    tags
    		NULLABLE RECORD Deprecated. This field contains user-defined key-value pairs associated with the bucket. Use resourceTags instead.
    tags.lastUpdatedTime
    		NULLABLE TIMESTAMP Deprecated. This field indicates the last time the tags were updated.
    tags.tagMap
    		REPEATED RECORD Deprecated. This field, part of the tags record, contains the map of tag keys and values.
    tags.tagMap.key
    		NULLABLE STRING Deprecated. This field, part of the tags.tagMap record, represents the key of a tag.
    tags.tagMap.value
    		NULLABLE STRING Deprecated. This field, part of the tags.tagMap record, represents the value of a tag.
    labels
    		REPEATED RECORD User-provided bucket labels , in key-value pairs.
    labels.key
    		NULLABLE STRING An individual label entry.
    labels.value
    		NULLABLE STRING The label's value.
    softDeletePolicy
    		NULLABLE RECORD The bucket's soft delete policy , which defines the period of time during which objects in the bucket are retained in a soft-deleted state after being deleted. Objects in a soft-deleted state cannot be permanently deleted, and are restorable until their hardDeleteTime .
    softDeletePolicy.effectiveTime
    		NULLABLE DATETIME

    The datetime at which the soft delete policy becomes effective, in RFC 3339 format.

    softDeletePolicy.effectiveTime is updated whenever softDeletePolicy.retentionDurationSeconds is increased.
    softDeletePolicy.retentionDurationSeconds
    		NULLABLE LONG The period of time during which a soft-deleted object is retained and cannot be permanently deleted, in seconds. The value must be greater than or equal to 604800 (7 days) and less than 7776000 (90 days). The value can also be set to 0 , which disables the soft delete policy.
    iamConfiguration
    		NULLABLE RECORD The IAM configuration for a bucket.
    iamConfiguration.uniformBucketLevelAccess
    		NULLABLE RECORD The bucket's uniform bucket-level access configuration.
    iamConfiguration.uniformBucketLevelAccess.enabled
    		NULLABLE BOOLEAN Whether or not the bucket uses uniform bucket-level access.
    iamConfiguration.publicAccessPrevention
    		NULLABLE STRING The bucket's public access prevention status, which is either "inherited" or "enforced" .
    resourceTags
    		REPEATED RECORD The bucket's tags . For more information, see Cloud Resource Manager API .
    resourceTags.key
    		NULLABLE STRING The resourceTags key.
    resourceTags.value
    		NULLABLE STRING The resourceTags value.

    Object metadata

    The object metadata schema includes the object_attributes_view and object_attributes_latest_snapshot_view tables. The object metadata tables have the following fields:

    Metadata field Mode Type Description
    snapshotTime
    		NULLABLE TIMESTAMP The snapshotTime field stores the time of the object metadata snapshot refresh in RFC 3339 format .
    bucket
    		NULLABLE STRING The name of the bucket containing this object.
    location
    		NULLABLE STRING The location of the bucket. Object data for objects in the bucket resides in physical storage within this location.
    componentCount
    		NULLABLE INTEGER Returned for composite objects only. Number of non-composite objects in the composite object. componentCount includes non-composite objects that were part of any composite objects used to compose the current object.
    contentDisposition
    		NULLABLE STRING Content-Disposition of the object data.
    contentEncoding
    		NULLABLE STRING Content-Encoding of the object data.
    contentLanguage
    		NULLABLE STRING Content-Language of the object data.
    contentType
    		NULLABLE STRING Content-Type of the object data.
    crc32c
    		NULLABLE INTEGER CRC32c checksum, as described in RFC 4960, Appendix B ; encoded using base64 in big-endian byte order.
    customTime
    		NULLABLE TIMESTAMP A user-specified timestamp for the object in RFC 3339 format.
    etag
    		NULLABLE STRING HTTP 1.1 Entity tag for the object.
    eventBasedHold
    		NULLABLE BOOLEAN Whether or not the object is subject to an event-based hold .
    generation
    		NULLABLE INTEGER The content generation of this object.
    md5Hash
    		NULLABLE STRING MD5 hash of the data, encoded using base64 . This field is not present for composite objects .
    metadata
    		REPEATED RECORD User-provided metadata, in key-value pairs.
    metadata.key
    		NULLABLE STRING An individual metadata entry.
    metadata.value
    		NULLABLE STRING The metadata value.
    metageneration
    		NULLABLE INTEGER The version of the metadata for this object at this generation.
    name
    		NULLABLE STRING The name of the object.
    size
    		NULLABLE INTEGER Content-Length of the data in bytes.
    storageClass
    		NULLABLE STRING Storage class of the object.
    temporaryHold
    		NULLABLE BOOLEAN Whether or not the object is subject to a temporary hold .
    timeCreated
    		NULLABLE TIMESTAMP The creation time of the object in RFC 3339 format.
    timeDeleted
    		NULLABLE TIMESTAMP The deletion time of the object in RFC 3339 format.
    updated
    		NULLABLE TIMESTAMP The modification time of the object metadata in RFC 3339 format.
    timeStorageClassUpdated
    		NULLABLE TIMESTAMP The time at which the object's storage class was last changed.
    retentionExpirationTime
    		NULLABLE TIMESTAMP The earliest time that the object can be deleted, which depends on any retention configuration set for the object and any retention policy set for the bucket that contains the object. The value for retentionExpirationTime is given in RFC 3339 format.
    softDeleteTime
    		NULLABLE DATETIME The time at which the object was soft deleted . Only available for objects in buckets with a soft delete policy.
    hardDeleteTime
    		NULLABLE DATETIME The time at which a soft-deleted object is permanently deleted and can no longer be restored. The value is the sum of the softDeleteTime value and the softDeletePolicy.retentionDurationSeconds value of the bucket. Only available for objects in buckets with a soft delete policy.
    project
    		NULLABLE INTEGER The project number of the project the bucket belongs to.

    Latest bucket and object metadata snapshot

    The linked dataset exposes the latest snapshot of the bucket and object metadata through the following dedicated views:

    • The bucket_attributes_latest_snapshot_view table provides the latest metadata for your Cloud Storage buckets. Its structure matches the Bucket metadata schema .

    • The object_attributes_latest_snapshot_view table provides the latest metadata for your Cloud Storage objects. Its structure matches the Object metadata schema .

    Project metadata

    The linked dataset exposes project metadata as a view named project_attributes_view :

    Metadata field Mode Type Description
    snapshotTime
    		NULLABLE TIMESTAMP The snapshotTime field stores the time of the project metadata snapshot refresh in RFC 3339 format .
    name
    		NULLABLE STRING The name of the project.
    id
    		NULLABLE STRING The unique identifier for the project.
    number
    		NULLABLE INTEGER A numeric value associated with the project.

    Dataset schema for activity data tables

    Storage Insights datasets provide activity data, including operational metrics, error logs, and aggregated views with customizable scopes to support diverse analysis needs. The following sections describe the activity data schemas and the data available in them.

    Object activity data

    The object_events_view table in your Storage Insights dataset provides queryable records of create, update, and delete object operations, metadata, and details about the errors encountered during these operations. You can use the data to track and troubleshoot changes to your storage assets by querying the object_events_view view. The data in the object_events_view view is scoped to your Storage Intelligence subscription and lists all the actively enabled objects within it.

    The object_events_view table includes the following fields:

    Field Mode Type Description
    requestCompletionTimestamp
    		NULLABLE TIMESTAMP The timestamp when the request was completed, in RFC 3339 format .
    project
    		NULLABLE INTEGER The project number that owns the bucket resource.
    bucketName
    		NULLABLE STRING The name of the bucket specified in the request.
    objectName
    		NULLABLE STRING The name of the object specified in the request. This field can be null.
    requestOperation
    		NULLABLE STRING The Cloud Storage operation performed. This field is limited to write, update, and delete operations and can be null.
    requestBytes
    		NULLABLE INTEGER The number of bytes sent in the request.
    responseBytes
    		NULLABLE INTEGER The number of bytes sent in the response.
    requestProcessingTimeMicros
    		NULLABLE INTEGER The time in microseconds the server took to process the request. This time is measured from when the first byte is received until the response is sent. For resumable uploads , the response to the final upload request determines the endpoint.
    requestId
    		NULLABLE STRING The identifier for the request.
    requestHost
    		NULLABLE STRING The host specified in the original request.
    requestHttpMethod
    		NULLABLE STRING The HTTP method used for this request.
    responseStatus
    		NULLABLE INTEGER The HTTP status code sent in the server's response.
    requestReferrer
    		NULLABLE STRING The HTTP referrer for the request.
    size
    		NULLABLE INTEGER The size of the object in bytes.
    generation
    		NULLABLE INTEGER The content generation of the object, used for object versioning.
    timeCreated
    		NULLABLE TIMESTAMP The object's creation time, in RFC 3339 format .
    storageClass
    		NULLABLE STRING The storage class of the object.
    errorReason
    		NULLABLE STRING The reason for the error (limited to 429 errors).
    location
    		NULLABLE STRING The location of the object.

    Project activity view

    The project_activity_view table offers aggregated operational insights across your projects. The table summarizes total activity by counting operation types, bytes transferred, and response codes and errors. You can use the project activity insights for monitoring, tracking, and internal cost attribution. The data in the view is determined by your Storage Intelligence subscription scope. The view displays aggregated data only for the buckets and objects that are actively enabled under your subscription. For example, if you enable Storage Intelligence only for ten specific buckets, the view displays data exclusively for those ten buckets. Conversely, if you enable Storage Intelligence at a higher level (organization, folder, or project), all buckets and objects within that scope are automatically included, and the view displays the aggregated data for them.

    The project_activity_view table includes the following fields:

    Field Mode Type Description
    snapshotStartTime
    		NULLABLE TIMESTAMP The snapshot's start time, in RFC 3339 format .
    snapshotEndTime
    		NULLABLE TIMESTAMP The snapshot's end time, in RFC 3339 format .
    project
    		NULLABLE INTEGER The project number.
    totalRequests
    		NULLABLE INTEGER The total number of requests made across the project.
    totalReads
    		NULLABLE INTEGER The total number of read operations.
    totalWrites
    		NULLABLE INTEGER The total number of write operations.
    totalUpdates
    		NULLABLE INTEGER The total number of update operations.
    totalDeletes
    		NULLABLE INTEGER The total number of delete operations.
    totalLists
    		NULLABLE INTEGER The total number of list operations.
    totalRequestBytes
    		NULLABLE INTEGER The total number of request bytes transferred.
    totalResponseBytes
    		NULLABLE INTEGER The total number of response bytes transferred.
    count4xx
    		NULLABLE INTEGER The total number of 4xx errors.
    count5xx
    		NULLABLE INTEGER The total number of 5xx errors.
    count2xx
    		NULLABLE INTEGER The total number of 2xx success responses.
    count400
    		NULLABLE INTEGER The total number of 400 errors for bad requests.
    count401
    		NULLABLE INTEGER The total number of 401 errors for unauthorized requests.
    count403
    		NULLABLE INTEGER The total number of 403 errors for forbidden requests.
    count429
    		NULLABLE INTEGER The total number of 429 errors for usage limit exceeded requests.

    Bucket activity view

    The bucket_activity_view table offers aggregated operational insights across your buckets. The table summarizes total activity by counting operation types, bytes transferred, response codes, errors, and includes details about top prefixes. You can use the bucket activity insights for bucket-level monitoring, troubleshooting, metering, and analyzing your most and least active buckets. The data in bucket_activity_view table is scoped to your Storage Intelligence subscription and includes records for all actively enabled buckets.

    The bucket_activity_view table includes the following fields:

    Field Mode Type Description
    snapshotStartTime
    		NULLABLE TIMESTAMP The snapshot's start time, in RFC 3339 format .
    snapshotEndTime
    		NULLABLE TIMESTAMP The snapshot's end time, in RFC 3339 format .
    name
    		NULLABLE STRING The name of the bucket.
    generation
    		NULLABLE INTEGER The bucket's generation number.
    project
    		NULLABLE INTEGER The project number.
    location
    		NULLABLE STRING The location of the bucket.
    totalRequests
    		NULLABLE INTEGER The total number of requests made on the bucket.
    totalReads
    		NULLABLE INTEGER The total number of read operations.
    totalWrites
    		NULLABLE INTEGER The total number of write operations.
    totalUpdates
    		NULLABLE INTEGER The total number of update operations.
    totalDeletes
    		NULLABLE INTEGER The total number of delete operations.
    totalLists
    		NULLABLE INTEGER The total number of list operations.
    totalRequestBytes
    		NULLABLE INTEGER The total number of request bytes transferred.
    totalResponseBytes
    		NULLABLE INTEGER The total number of response bytes transferred.
    count4xx
    		NULLABLE INTEGER The total number of 4xx errors.
    count5xx
    		NULLABLE INTEGER The total number of 5xx errors.
    count2xx
    		NULLABLE INTEGER The total number of 2xx success responses.
    count400
    		NULLABLE INTEGER The total number of 400 errors for bad requests.
    count401
    		NULLABLE INTEGER The total number of 401 errors for unauthorized requests.
    count403
    		NULLABLE INTEGER The total number of 403 errors for forbidden requests.
    count429
    		NULLABLE INTEGER The total number of 429 errors for usage limit exceeded requests.
    topPrefixes429Errors
    		REPEATED RECORD The top 5 bucket prefixes with the most 429 errors, returned as a RECORD containing the prefix as a STRING and the error count as an INTEGER .
    topPrefixesRequestBytes
    		REPEATED RECORD The top 5 bucket prefixes with the highest number of requested bytes, returned as a RECORD containing the prefix as a STRING and the count as an INTEGER .
    topPrefixesResponseBytes
    		REPEATED RECORD The top 5 bucket prefixes with the highest number of response bytes, returned as a RECORD containing the prefix as a STRING and the count as an INTEGER .
    topPrefixesHighestOperations
    		REPEATED RECORD The top 5 bucket prefixes with the highest number of operations, returned as a RECORD containing the prefix as a STRING and the count as an INTEGER .

    Bucket region activity view

    The bucket_region_activity_view table summarizes the total bytes sent and received for a bucket, aggregated by destination region. Use the bucket activity regional data to identify the most active regions, helping you determine the optimal bucket location, analyze regional egress, and consider bucket relocation as needed to optimize performance, cost, or resilience. The data in bucket_region_activity_view table is scoped to your Storage Intelligence subscription and includes records for all actively enabled buckets.

    Field Mode Type Description
    snapshotStartTime
    		NULLABLE TIMESTAMP The snapshot's start time, in RFC 3339 format .
    snapshotEndTime
    		NULLABLE TIMESTAMP The snapshot's end time, in RFC 3339 format .
    project
    		NULLABLE INTEGER The project number for the bucket.
    name
    		NULLABLE STRING The name of the bucket.
    generation
    		NULLABLE INTEGER The bucket's generation number.
    requestLocation
    		NULLABLE STRING The Google Cloud location where the request originated.
    bucketLocation
    		NULLABLE STRING The location of the bucket.
    requestBytes
    		NULLABLE INTEGER The total number of bytes transferred for requests between the specified bucket and location.
    responseBytes
    		NULLABLE INTEGER The total number of bytes transferred for responses between the specified bucket and location.

    Dataset schema for events and errors

    In the linked dataset, you can also view the snapshot processing events and errors in the events_view and error_attributes_view views. To troubleshoot the snapshot processing errors, see Troubleshoot dataset errors .

    Events log

    You can view event logs in the events_view view in the linked dataset:

    Metadata field
    Mode
    Type
    Description
    manifest.snapshotTime
    NULLABLE
    TIMESTAMP
    The time, in RFC 3339 format , when the snapshot of the events is refreshed.
    manifest.viewName
    NULLABLE
    STRING
    The name of the view that refreshes.
    manifest.location
    NULLABLE
    STRING
    The source location of the data that refreshes.
    globalManifest.snapshotTime
    NULLABLE
    TIMESTAMP
    The ingestion completion time, in RFC 3339 format , for the bucket and object attributes tables across all source locations.
    eventTime
    NULLABLE
    TIMESTAMP
    The time the event occurred.
    eventCode
    NULLABLE
    STRING
    The event code associated with the corresponding entry. The eventCode values are as follows:
    • eventCode 1 indicates that the manifest.viewName view is refreshed with all entries for the source location (manifest.location) within the snapshot (manifest.snapshotTime) .
    • eventCode 2 indicates that the dataset is refreshed with the bucket and object entries for all source locations. The refresh occurs within the snapshot (globalManifest.snapshotTime) .
    • eventCode 3 indicates that the dataset is published each time data processing is complete for the activity views. The corresponding eventMessage is DATA_REFRESH_ACTIVITY_JOURNAL . The event code 3 refers to the activity manifest.viewName view being refreshed with all entries for the source location manifest.location within the log window.

    Error codes

    You can view error codes in the error_attributes_view table in the linked dataset:

    Metadata field Mode Type Description
    errorCode
    		NULLABLE INTEGER The error code associated with this entry. For a list of valid values and how to resolve these errors, see Troubleshoot dataset errors .
    errorSource
    		NULLABLE STRING The source of the error. Valid value: CONFIGURATION_PREPROCESSING .
    errorTime
    		NULLABLE TIMESTAMP The time the error occurs.
    sourceGcsLocation
    		NULLABLE STRING The source Cloud Storage location of the error. For projects, this field is null because projects are not associated with a location.
    bucketErrorRecord.bucketName
    		NULLABLE STRING The name of the bucket involved in the error. You can use this information to debug a bucket error.
    bucketErrorRecord.serviceAccount
    		NULLABLE STRING The service account that needs permission to ingest objects from the bucket. You can use this information to debug a bucket error.
    projectErrorRecord.projectNumber
    		NULLABLE INTEGER The number of the project involved in the error. You can use this information to debug a project error.
    projectErrorRecord.organizationName
    		NULLABLE STRING The name of the organization the project must belong to for processing. A value of 0 indicates that the project is not in the organization. You can use this information to debug a project error.

    Troubleshoot dataset errors

    To troubleshoot the snapshot processing errors that are logged in the error_attributes_view view in the linked dataset, see the following table:

    Error code Error case Error message Troubleshooting
    1
    		 Source project doesn't belong to the organization Source project projectErrorRecord.projectNumber doesn't belong to the organization projectErrorRecord.organizationName . Add source project projectErrorRecord.projectNumber to organization projectErrorRecord.organizationName . For instructions to migrate a project between organizations, see Migrate projects between organizations .
    2
    		 Bucket authorization error Permission denied for ingesting objects for bucket bucketErrorRecord.bucketName . Give service account bucketErrorRecord.serviceAccount Identity and Access Management (IAM) permissions to allow ingestion of objects for bucket bucketErrorRecord.bucketName . For more information, see Grant required permissions to the service agent .
    3
    		 Destination project doesn't belong to the organization Destination project projectErrorRecord.projectNumber not in organization projectErrorRecord.organizationName . Add destination project projectErrorRecord.projectNumber to organization projectErrorRecord.organizationName . For instructions to migrate a project between organizations, see Migrate projects between organizations .
    4
    		 Source project doesn't have Storage Intelligence configured. Source project projectErrorRecord.projectNumber doesn't have Storage Intelligence configured. Configure Storage Intelligence for the source project projectErrorRecord.projectNumber . For more information, see Configure and manage Storage Intelligence .
    5
    		 Bucket doesn't have Storage Intelligence configured. Bucket bucketErrorRecord.bucketName doesn't have Storage Intelligence configured. Configure Storage Intelligence for the bucket bucketErrorRecord.bucketName . For more information, see Configure and manage Storage Intelligence .
    6
    		 Bucket access authentication error ACTIVITY_BUCKET_ACCESS_AUTHORIZATION_ERROR for bucket bucketErrorRecord.bucketName . Grant service account bucketErrorRecord.serviceAccount Identity and Access Management (IAM) permissions to allow ingestion of activity for bucket bucketErrorRecord.bucketName . For more information, see Grant required permissions to the service agent .

    What's next
    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: