Prepare your data for import

Keep your data fresh

You must ensure your data is refreshed in the data source before a manual or scheduled data source connection runs in order for that data to be imported for use in the intended destination or use case. For example, if you choose a daily schedule, you need to refresh your data on a daily basis, before the scheduled start time.

Make your data available

When connecting to a data source, each use case you want to implement should have its own dedicated table or a filtered subset of table data within that source. You can reuse the same table from a connected data source for multiple use cases by using different filtered subsets of that table.

Some data sources require you to have proper credentials in order to make the data source connection. See the specific guide for the data source you're using for specific guidance.

Format your data

The following sections show you how to properly format your data, to ensure it can be imported without error.

About file formats

If you are uploading a file, such as a CSV file, the first line of the file must contain the headers.

Ensure that the file has an extension, as files without extensions are rejected.

About date and time formats

Converting dates and times of various formats into a logical timestamp is supported and should be based on six templates using three format sets: DATE, TIME, and TIMEZONE. The DATETIME type is not supported. DATETIME fields need to be converted to the STRING type in the data source, using the supported formats described in this article.

The following are examples of timestamps in supported formats:

• 2012-08-15T00:01:54Z (UTC ISO 8601 Standard)
• 2012-08-14T17:01:54-07:00 (ISO 8601 Standard with Offset)
• Aug 14, 2012 17:01:54
• 08/14/2012T5:01:54 PM
• 2012-08-14 5:01:54 PM
• 08/14/2012 17:01:54
• 2012-08-14 17:01:54
• 08/14/2012 17:01:54*123
• 2012-08-14T17:01:54-07
• 08/14/2012T17:01:54-0700
• 2012-08-14T17:01:54-070000
• 2012-08-14T17:01:54-07:00:00
• 2012-08-14T17:01:54 America/Los_Angeles
• Aug 14, 2012 17:01:54PST
• 2012-08-14 17:01:54 PST
• 2012-08-14 17:01:54 Pacific Standard Time
• 2012-08-14 17:01:54 GMT-07:00
• 08/14/2012 17:01:54 GMT-07:00:00

About hashing private customer data

To keep your data secure, private customer data that you import should be hashed. The data will be hashed for you using the SHA256 algorithm, which is the industry standard for one-way hashing. The result is hex encoded. You don’t need to pre-format your data. Your relevant PII fields will be normalized, hashed, and encoded for you, and data will be pushed to the API for your use cases.

If you prefer to hash private customer data yourself, see Format your customer data file to ensure it’s formatted correctly. If you upload a hashed data file, don’t hash non-private customer data. Your hashed data will be pushed to the API.

Note that smart hashing is automatic, meaning you don't need to select anything from the Actions menu.

Define the scope of your data import using filters

You can set filter conditions directly in the UI, eliminating the need to create custom data pipelines or write complicated SQL queries within your data source. When you create a filter, only data from your data source (for all use cases) that satisfies all of the filter conditions will be imported.

You can apply one filter with up to 25 conditions per connected data source.

Create a filter

To create filters for a new data source, add them during the Select datastage of setup:

1. From the Select datastep of setup, click Filterto expand it.
2. Select the field to use to filter your data.
3. Select an operator.
4. Enter a value.
5. Optional: Create additional conditions, by clicking Andor Or.
6. Continue the setup process.

To create and edit filters for an existing connected data source:

1. Click the data source name to edit it.
2. Under Filter, click Edit.
3. Make your changes and click Save.

Supported operators

• AND
• OR
• Doesn’t contain (strings, integer, date, time, Boolean)
• Contains (strings, integer, date, time, Boolean)
• Greater than (integer)
• Equal to (integer, string, date)
• Less than (integer) 
• Before (date, time)
• After (date, time)
• Does not equal
• Starts with
• Ends with
• Does not start with
• Does not end with

Supported data types

• Currency
• Date
• Time
• Boolean
• Integer
• Dropdown
• String