Getting Files
The ingestion process for DTv2 is similar to DTv1 but there are some key things that you will need to change during your migration.
Processing Files
DTv2 processes hourly files (impression, click, and rich media) and daily files (activity and match tables). Hourly files (file name contains YYYYMMDDHH) are processed in UTC time. Daily files (file name contains YYYYMMDD) are processed in your local reporting time zone see file name format . The offset between your local time and UTC will differ depending on your location.
Files are also processed independently, and their processing time can vary. It's common for a later hour file to finish processing before an earlier file (like hour 6 before hour 5). Don't rely on file order for your ingestion process, otherwise your process may stall.
Events can appear in earlier or later processed files relative to their event time. Events are not always processed in the hour in which they occurred.
Column Order
Please do not rely on column order as a mechanism to consume your files. We want your processing to be resilient to change; if you request extra columns or if we make changes in the future then the order and number of columns in your reports can change. Read the header row of each file and map this to fields in your data warehouse before attempting to write data.
Duplicate files
Sometimes duplicate files are written for the same date and hour. If more than one file has the same date/hour stamp, use the one with the latest minutes / seconds, based on the filename timestamp. Duplicate files are created because back end processes determined there was an issue with the original file.
Fields
The field names and field name format all changed but there is a mapping from old to new (where available) at DCM Field migration including Match Tables. If you are not a DBM user, these will be empty and you can ignore the field.
You may also see DBM Fields in your file. If you are a DBM user these fields will populate only when the relevant permission is granted on the DBM advertiser level where advertisers are linked. There is a mapping from old to new (where available) at DBM Field migration . DBM entity read files will still be used for mapping purposes.
Bucket Names
The naming standards for DTv1 and DTv2 are different, specifically you can’t change the prefix on your existing bucket name to work out the DTv2 bucket name. The bucket name will be given to you by your support representative when your account is setup.
Generally DTv2 bucket names look like gs://dcdt_-dcm_account1234
You can see the contents of your bucket in a browser by looking at https://cloud.google.com/console#/storage/browser/dcdt_-dcm_account1234
File Name Format
Each file name will have a string of numbers, for example:
dcm_account1234_impression_2016022601_20160225_234912_218211994.csv.gz
2016022601
is in YYYYMMDDHH format. This is the UTC hour for events in that
file (hours are numbered 0 to 23).
20160225_234912
is in YYYYMMDD_HHMMSS format. This is the time at which the report was
generated.
218211994
is the file ID.
Activity Files
You get one Activity file per day and the filename looks like this:
dcm_account1234_activity_20160727_20160728_035750_268669761.csv.gz
This file contains data for 27 July 2016 and was generated at 3:57:50 on 28 July 2016.
Click Files
You get twenty four Click files per day and the filename looks like this:
dcm_account1234_click_2016072717_20160728_012331_268381796.csv.gz
Take note of the UTC Hour after the date string in the filename; hours are numbered from 0 to 23 so
a 17 here indicates that the events in this file are for 17:00 to 17:59 (24 hour clock time) or 5:00pm to 6:00pm.
This file contains data for 5:00pm to 6:00pm, 27 July 2016 and was generated at 1:23:31 on
28 July 2016.
Impression Files
You get twenty four Impression files per day and the filename looks like this:
dcm_account7312_impression_2016072717_20160728_012355_268381795.csv.gz
Other than the filename this behaves the same as Click Files.
Match Table Files
Match Table files are generated once daily, and filenames look like this:
dcm_account1234_match_table_activity_cats_20160727_20160728_032226_268648829.csv.gz
This file contains data for 27 July 2016 and was generated at 3:22:26 on 28 July 2016.
Note: For some new match tables the data is static and no daily downloadable file is produced; you
can get the data for these files from the reference pages e.g. Rich Media standard event types
.
Data Transfer Fields
| 1.0 field name | DT v2.0 field name |
|---|---|
| Time | Deprecated |
| User-ID | User ID |
| Advertiser-ID | Advertiser ID |
| Buy-ID | Deprecated |
| Order-ID | Campaign ID |
| Ad-ID | Ad ID |
| Creative-ID | Rendering ID |
| Creative-Version | Creative Version |
| Creative-Size-ID | Deprecated (Retrieved from Match Table as Creative Pixel Size) |
| Site-ID | Site ID (DCM) |
| Page-ID | Placement ID |
| Keyword | Deprecated |
| Country-ID | Country Code |
| State/Province | State/Region |
| Areacode | Deprecated |
| Browser-ID | Browser/Platform ID |
| Browser-Version | Browser/Platform Version |
| OS-ID | Operating System ID |
| DMA-ID | Designated Market Area (DMA) ID |
| City-ID | City ID |
| Zip-Code | ZIP/Postal Code |
| Time-UTC-Sec | Deprecated |
| Local-User-ID | Deprecated |
| Activity-Type | Deprecated (Retrieved from 'activity_cats' Match Table) |
| Activity-Sub-Type | Deprecated (Retrieved from 'activity_cats' Match Table) |
| Other-Data | Other Data |
| Ordinal | ORD Value |
| Click-Time | Interaction Time |
| Event-ID | Conversion ID |
| SV1 | Segment Value 1 |
| PartnerID1 | Partner 1 ID |
| PartnerID2 | Partner 2 ID |
| Quantity | Total Conversions |
| Revenue | Total Revenue |
Match table file names
| 1.0 file name | 2.0 file name |
|---|---|
activity_cat
|
activity_cats
|
activity_type
|
activity_types
|
ad
|
ads
|
ad_page_assignments
|
ad_placement_assignments
|
advertiser
|
advertisers
|
browser
|
browsers
|
campaign
|
campaigns
|
cities
|
cities
|
countries
|
countries
|
creative
|
creatives
|
creative_ad_assignments
|
creative_ad_assignments
|
custom_creative_fields
|
custom_creative_fields
|
dartsearch
|
paid_search
|
designated_market_area
|
designated_market_areas
|
keyval
|
keyword_values
|
met_event
|
rich_media_events
|
operating_system
|
operating_systems
|
page
|
placements
|
page_cost/page_flight_cost
|
placement_cost
|
site
|
sites
|
states
|
states
|
user-defined_spot_vars
|
custom_floodlight_variables
|
Match table fields
activity_cats
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Spot-ID
|
Floodlight Configuration
|
integer |
Activity-Type-ID
|
Activity Group ID
|
integer |
Activity-Type
|
Activity Type
|
string |
Activity-ID
|
Activity ID
|
integer |
Activity Sub-Type
|
Activity Sub-Type
|
string |
Report-Name
|
Activity
|
string |
Tag-Method-ID
|
Tag Counting Method ID
|
integer |
activity_types
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Spot-ID
|
Floodlight Configuration
|
integer |
Activity-Group-ID
|
Activity Group ID
|
integer |
Activity-Type
|
Activity Type
|
string |
Report-Name
|
Activity Group
|
string |
ads
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Advertiser-ID
|
Advertiser ID
|
integer |
Buy-ID
|
Campaign ID
|
integer |
Ad-ID
|
Ad ID
|
integer |
Ad-Description
|
Ad
|
string |
Click-URL
|
Ad Click URL
|
string |
Ad-Type
|
Ad Type
|
string |
Creative-Size-ID
|
Creative Pixel Size
|
string |
Ad-Comments
|
Ad Comments
|
string |
ad_placement_assignments
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Ad-ID
|
Ad ID
|
integer |
Page-ID
|
Placement ID
|
integer |
advertisers
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Spot-ID
|
Floodlight Configuration
|
integer |
Advertiser-ID
|
Advertiser ID
|
integer |
Advertiser
|
Advertiser
|
string |
Advertiser-Group-ID
|
Advertiser Group ID
|
integer |
Advertiser-Group
|
Advertiser Group
|
string |
browsers
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Browser-ID
|
Browser/Platform ID
|
integer |
Browser
|
Browswer/Platform
|
string |
campaigns
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Advertiser-ID
|
Advertiser ID
|
integer |
| |
Campaign ID
|
integer |
Order
(formerly Buy
) |
Campaign
|
string |
Start-Date
|
Campaign Start Date
|
date |
End-Date
|
Campaign End Date
|
date |
Billing-Invoice-Code
|
Billing Invoice Code
|
string |
cities
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
City-ID
|
City ID
|
integer |
City
|
City
|
string |
Creative-Library-Enabled
|
Deprecated (now always 1)
|
integer |
countries
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Country-ID
|
Deprecated
|
integer |
Country
|
Deprecated
|
string |
creatives
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Advertiser-ID
|
Advertiser ID
|
integer |
Rendering-ID
|
Rendering ID
|
integer |
UI-Creative-ID
|
deprecated | integer |
Creative
|
Creative
|
string |
Last-Modified-Date
|
Creative Last Modified Date
|
date |
Creative-Type
|
Creative Type
|
string |
Creative-Sub-Type
|
Deprecated
|
string |
Creative-Size-ID
|
Creative Pixel Size
|
string |
Image-URL
|
Creative Image URL
|
string |
Creative-Version
|
Creative Version
|
integer
|
creative_ad_assignments
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Ad-ID
|
Ad ID
|
integer |
Creative-ID
|
Creative ID
|
integer |
Start-Date
|
Creative Start Date
|
date |
End-Date
|
Creative End Date
|
date |
Rotation-Type
|
Creative Rotation Type
|
string |
Creative-Group-1
|
Creative Groups 1
|
string |
Creative-Group-2
|
Creative Groups 2
|
string |
Click-URL
|
Ad Click URL
|
string |
custom_creative_fields
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Advertiser-ID
|
Advertiser ID
|
integer |
Creative-ID
|
Creative ID
|
integer |
Custom-Field-Number
|
Creative Custom Field Number
|
integer |
Custom-Field-Name
|
Creative Custom Field Name
|
string |
Custom-Field-Value
|
Creative Custom Field Value
|
string |
custom_rich_media
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Advertiser-ID
|
Advertiser ID
|
integer |
Met-Event-ID
|
Rich Media Event ID
|
integer |
Met-Event-Name
|
Rich Media Event
|
string |
Met-Event-Type-ID
|
Rich Media Event Type ID
|
integer |
Met-Event-Type-Name
|
Rich Media Event Type
|
string |
designated_market_areas
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
DMA-ID
|
Designated Market Area (DMA) ID
|
integer |
DMA
|
Designated Market Area
|
string |
keyword_values
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Ad-ID
|
Ad ID
|
integer |
Keyword-Expression
|
Keyword
|
string |
paid_search
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Ad-ID
|
Ad ID
|
integer |
Advertiser-ID
|
Advertiser ID
|
integer |
Buy-ID
|
Campaign ID
|
integer |
DS-Ad-ID
|
Paid Search Ad ID
|
integer |
Extended-Keyword-ID
|
Paid Search Keyword ID
|
integer |
Keyword-ID
|
Paid Search Legacy Keyword ID
|
integer |
DS-Campaign-Name
|
Paid Search Campaign
|
string |
KW-Ad-Group-Name
|
Paid Search Ad Group
|
string |
Bid-Strategy-Name
|
Paid Search Bid Strategy
|
string |
Landing-Page-URL
|
Paid Search Landing Page URL
|
string |
Keyword-Name
|
Paid Search Keyword
|
string |
Match-Type
|
Paid Search Match Type
|
string |
operating_systems
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
OS-ID
|
Operating System ID
|
integer |
OS
|
Operating System
|
string |
placements
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Buy-ID
|
Campaign ID
|
integer |
Site-ID
|
Site ID (DCM)
|
integer |
Page-ID
|
Placement ID
|
integer |
Page
|
Site keyname
|
string |
Site-Placement
|
Placement
|
string |
Content-Category
|
Content Category
|
string |
Strategy
|
Placement Strategy
|
string |
Start-Date
|
Placement Start Date
|
date |
End-Date
|
Placement End Date
|
date |
Group-Type
|
Placement Group Type | string |
Group-Parent-ID
|
Package/Roadblock ID | integer |
Price-Type
|
Placement Cost Structure
|
string |
Cap-Cost
|
Placement Cap Cost Option
|
float |
CPA-Activity-ID
|
Activity ID
|
integer |
placement_cost
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Page-ID
|
Placement ID
|
integer |
Effective-Date
|
Placement Start Date
|
date |
End-Date
|
Placement End Date
|
date |
Units
|
Placement Total Booked Units
|
integer |
Rate
|
Placement Rate
|
float |
Comments
|
Placement Comments
|
string |
sites
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Site-ID
|
Site ID (DCM)
|
integer |
Site
|
Site (DCM)
|
string |
Directory-Site-ID
|
Site ID (Site Directory)
|
integer |
Directory-Site
|
Site (Site Directory)
|
string |
states
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
State/Province
|
State/Region
|
string |
State/Province Name
|
State/Region Full Name
|
string |
custom_floodlight_variables
| 1.0 field name | 2.0 field name | Type |
|---|---|---|
Spot-ID
|
Floodlight Configuration ID
|
integer |
Variable-ID
|
Floodlight Variable ID
|
string |
Variable-Name
|
Floodlight Variable Name
|
string |
