Redact confidential data

Create the pipeline

Create a pipeline that redacts sensitive customer data. The pipeline you build does the following:

• Reads the input data using the Cloud Storage source plugin.
• Deploys the Cloud DLP plugin from the Hub.
• Writes the output data using a Cloud Storage sink plugin.

Load the customer data

This tutorial uses the input dataset, CallCenterRecords.csv , provided in a publicly available Cloud Storage bucket.

1. Open your Cloud Data Fusion instance and click menu Menu > Studio.

2. In the Sourcemenu, click the Cloud Storageplugin.

   

3. On the Cloud Storagenode, click Properties.

4. In the Reference namefield, enter a name.

5. In the Pathfield, enter gs://datafusion-sample-datasets/CallCenterRecords.csv .

6. In the Formatfield, select CSV .

7. For the Output Schema, delete the offsetand bodyfields. Click add Addand enter the following fields:

   • Date
   • Bank
   • State
   • Zip
   • Notes

   

8. Click Validateto check for errors.

9. Click close Close.

Redact sensitive data

The Cloud DLP Redact plugin identifies sensitive records in your input stream of data and applies transformations that you define to those records. A record of data is considered sensitive if it matches predefined Cloud DLP filters you choose or a custom template you define.

In this tutorial, you want to redact customer phone numbers that some support technicians on your team accidentally took note of. They entered the sensitive information in the Notessection of the support tickets, which appears as the Notescolumn in the CSV file. You create a custom Cloud DLP template, and then provide the template ID in the properties menu of the plugin.

Deploy the Cloud DLP plugin

1. In your Cloud Data Fusion instance, click Hub.

2. Click the Cloud DLPplugin.

3. Click Deploy.

4. Click Finish.

5. Click close Closeto exit the Cloud DLP dialog.

6. Click close Closeto exit the Hub.

Create a custom template

1. In the Google Cloud console, go to the Cloud DLP page.

   Go to Cloud DLP

2. From the Createmenu, choose Template.

3. In the Template IDfield, enter an ID for your template.

4. Click Continue.

5. In the Configure detectionfield, click Manage infotypes.

6. In the Built-intab, use the filter to search for "phone number".

   

7. Select PHONE_NUMBER.

8. Click Done > Create.

Learn more about creating Cloud DLP templates .

Apply the Cloud DLP Redact transformation

1. Go to the Cloud Data Fusion Studiopage and click to expand the Transformmenu.

2. Click the Cloud DLP Redactplugin.

   

3. Drag a connection arrow from the Cloud Storagenode to the Redactnode.

   

4. Hold the pointer over the Redactnode and click Properties.

   1. Set Custom Templateto Yes .

   2. In the Template IDfield, enter the template ID of the custom template you created .

   3. In the Matchingfield , apply Maskingon Custom templatewithin Notes.

   4. In the Masking Characterfield, enter # .

      

   5. Click Validateto check for errors.

   6. Click close Close.

Store the output data

Store the results of your pipeline in a Cloud Storage file.

1. From the Studiopage, click to expand the Sinkmenu.

2. Click Cloud Storage.

3. Drag a connection arrow from the Redactnode to the Cloud Storage2node.

   

4. Hold the pointer over the Cloud Storage2node and click Properties.

   1. In the Reference namefield, enter a name.

   2. In the Pathfield, enter the path of a Cloud Storage bucket where you'd like to store the pipeline results. Cloud Data Fusion creates the bucket for you. Be sure to follow the bucket naming guidelines .

   3. In the Formatfield, select CSV.

   4. Click Validateto ensure that there are no errors.

   5. Click close Close.

Run the pipeline in preview mode

Run the pipeline in preview mode before you deploy it.

1. Click Preview, and then click Run.

   

   Clicking Rundisplays the pipeline status, which starts with Starting, then turns to Stop, and then to Run.

2. When the preview run completes, on the Redactnode, click Preview Datato see a side-by-side comparison of the input and output data. Check that phone numbers have been masked with the # character.

   

Redact another data type

While examining the preview run results, you notice that there's still sensitive information that appears in the Notescolumn: email addresses. You go back and edit the Cloud DLP template to redact email addresses as well.

1. In the Google Cloud console, go to the Cloud DLP page.

   Open the Cloud DLP page

2. On the Configurationtab, select your template.

3. Click Edit.

4. Click Manage infotypes.

5. In the Built-intab, use the filter to search for "OR" "email address".

   

6. Select all and click Done.

7. Click Save.

8. Once again, run your pipeline in preview mode . Cloud Data Fusion will automatically use the updated Cloud DLP template.

9. Check that both phone numbers and email addresses have been masked with the # character.

   

Deploy and run the pipeline

1. Make sure Previewmode is unchecked.

2. Click Save. Clicking Saveprompts you to name your pipeline. Then, click OK.

3. Click Deploy.

4. When deployment completes, click Run. Running your pipeline can take a few minutes. While you wait, you can observe the Statusof the pipeline transition from Provisioningto Startingto Runningto Deprovisioningto Succeeded.

View the results

1. In the Google Cloud console, go to the Cloud Storage page.

   Go to Cloud Storage

2. In the Storage browser, navigate to the sink Cloud Storage bucket you specified in the sink Cloud Storage plugin properties .

3. In Link URL, click the link to download the CSV file with the results. Check that the phone numbers and email addresses have been masked with the # character.