Introducing PII knowledge identification and dealing with utilizing AWS Glue DataBrew


AWS Glue DataBrew, a visible knowledge preparation instrument, can now establish and deal with delicate knowledge by making use of advance transformations like redaction, alternative, encryption, and decryption in your personally identifiable data (PII) knowledge. With exponential progress of information, corporations are dealing with big volumes and all kinds of information coming into their platform, together with PII knowledge. Figuring out and defending delicate knowledge at scale has develop into more and more complicated, costly, and time-consuming. Organizations have to stick to knowledge privateness, compliance, and regulatory wants comparable to GDPR and CCPA. They should establish delicate knowledge, together with PII comparable to identify, SSN, tackle, e-mail, driver’s license, and extra. Even after identification, it’s cumbersome to implement redaction, masking, or encryption of delicate private data at scale.

To allow knowledge privateness and safety, DataBrew has launched PII statistics, which identifies PII columns and supply their knowledge statistics while you run a profile job in your dataset. Moreover, DataBrew has launched PII knowledge dealing with transformations, which allow you to use knowledge masking, encryption, decryption, and different operations in your delicate knowledge.

On this publish, we stroll by means of an answer through which we run a knowledge profile job to establish and recommend potential PII columns current in a dataset. Subsequent, we goal PII columns in a DataBrew undertaking and apply varied transformations to deal with the delicate columns present within the dataset. Lastly, we run a DataBrew job to use the transformations on the whole dataset and retailer the processed, masked, and encrypted knowledge securely in Amazon Easy Storage Service (Amazon S3).

Answer overview

We use a public dataset that’s out there for obtain at Artificial Affected person Information with COVID-19. The information hosted inside SyntheticMass has been generated by SyntheaTM, an open-source affected person inhabitants simulation made out there by The MITRE Company.

Obtain the zipped file for this answer and unzip it domestically. The answer makes use of the dummy knowledge within the file affected person.csv to exhibit knowledge redaction and encryption functionality. The file incorporates 10,000 artificial affected person data in CSV format, together with PII columns like driver’s license, beginning date, tackle, SSN, and extra.

The next diagram illustrates the structure for our answer.

The steps on this answer are as follows:

  1. The delicate knowledge is saved in an S3 bucket. You create a DataBrew dataset by connecting to the information in Amazon S3.
  2. Run a DataBrew profile job to establish the PII columns current within the dataset by enabling PII statistics.
  3. After identification of PII columns, apply transformations to redact or encrypt column values as part of your recipe.
  4. A DataBrew job runs the recipe steps on the whole knowledge and generates output information with delicate knowledge redacted or encrypted.
  5. After the output knowledge is written to Amazon S3, we create an exterior desk on high in Amazon Athena. Information shoppers can use Athena to question the processed and cleaned knowledge.


For this walkthrough, you want an AWS account. Use us-east-1 as your AWS Area to implement this answer.

Arrange your supply knowledge in Amazon S3

Create an S3 bucket known as databrew-clean-pii-data-<Your-Account-ID> in us-east-1 with the next prefixes:

  • sensitive_data_input
  • cleaned_data_output
  • profile_job_output

Add the affected person.csv file to the sensitive_data_input prefix.

Create a DataBrew dataset

To create a DataBrew dataset, full the next steps:

  1. On the DataBrew console, within the navigation pane, select Datasets.
  2. Select Join new dataset.
  3. For Dataset identify, enter a reputation (for this publish, Sufferers).
  4. Beneath Hook up with new dataset, choose Amazon S3 as your supply.
  5. For Enter your supply from S3, enter the S3 path to the affected person.csv file. In our case, that is s3://databrew-clean-pii-data-<Account-ID>/ sensitive_data_input/sufferers.csv.
  6. Scroll to the underside of the web page and select Create dataset.

Run a knowledge profile job

You’re now able to create your profile job.

  1. Within the navigation pane, select Datasets.
  2. Choose the Sufferers dataset.
  3. Select Run knowledge profile and select Create profile job.
  4. Identify the job Sufferers - Information Profile Job.
  5. We run the information profile on the whole dataset, so for Information pattern, choose Full dataset.
  6. Within the Job output settings part, level to the profile_job_output S3 prefix the place the information profile output is saved when the job is full.
  7. Increase Information profile configurations, and choose Allow PII statistics to establish PII columns when operating the information profile job.

This selection is disabled by default; it’s essential to allow it manually earlier than operating the information profile job.

  1. For PII classes, choose All classes.
  2. Maintain the remaining settings at their default.
  3. Within the Permissions part, create a brand new AWS Identification and Entry Administration (IAM) function that’s utilized by the DataBrew job to run the profile job, and use PII-DataBrew-Position because the function suffix.
  4. Select Create and run job.

The job runs on the pattern knowledge and takes a couple of minutes to finish.

Now that we’ve run our profile job, we will assessment knowledge profile insights about our dataset by selecting View knowledge profile. We are able to additionally assessment the outcomes of the profile by means of the visualizations on the DataBrew console and look at the PII widget. This part supplies a listing of recognized PII columns mapped to PII classes with column statistics. Moreover, it suggests potential PII knowledge which you could assessment.

Create a DataBrew undertaking

After we establish the PII columns in our dataset, we will give attention to dealing with the delicate knowledge in our dataset. On this answer, we carry out redaction and encryption in our DataBrew undertaking utilizing the Delicate class of transformations.

To create a DataBrew undertaking for dealing with our delicate knowledge, full the next steps:

  1. On the DataBrew console, select Tasks.
  2. Select Create undertaking.
  3. For Venture identify, enter a reputation (for this publish, patients-pii-handling).
  4. For Choose a dataset, choose My datasets.
  5. Choose the Sufferers dataset.
  6. Beneath Permissions, for Position identify, select the IAM function that we created beforehand for our DataBrew profile job AWSGlueDataBrewServiceRole-PII-DataBrew-Position.
  7. Select Create undertaking.

The dataset takes couple of minutes to load. When the dataset is loaded, we will begin performing redactions. Allow us to begin with the column SSN.

  1. For the SSN column, on the Delicate menu, select Redact knowledge.
  2. Beneath Apply redaction, choose Full string worth.
  3. We redact all of the non-alphanumeric characters and substitute them with #.
  4. Select Preview modifications to match the redacted values.
  5. Select Apply.

On the Delicate menu, all the information masking transformations—redact, substitute, and hash knowledge—are irreversible. After we finalize our recipe and run the DataBrew job, the job output to Amazon S3 is completely redacted and we will’t get well it.

  1. Now, let’s apply redaction to a number of columns, assuming the next columns should not be consumed by any downstream customers like knowledge analyst, BI engineer, and knowledge scientist:
    1. DRIVERS
    4. ADDRESS
    5. LAT
    6. LON

In particular instances, when we have to get well our delicate knowledge, as a substitute of masking, we will encrypt our column values and when wanted, decrypt the information to carry it again to its unique format. Let’s assume we require a column worth to be decrypted by a downstream software; in that case, we will encrypt our delicate knowledge.

We have now two encryption choices: deterministic and probabilistic. To be used instances once we need to be a part of two datasets on the identical encrypted column, we should always apply deterministic encryption. It makes certain that the encrypted worth of all of the distinct values is identical throughout DataBrew initiatives so long as we use the identical AWS secret key. Moreover, understand that while you apply deterministic encryption in your PII columns, you may solely use DataBrew to decrypt these columns.

For our use case, let’s assume we need to carry out deterministic encryption on a number of of our columns.

  1. On the Delicate menu, select Deterministic encryption.
  2. For Supply columns, choose BIRTHDATE, DEATHDATE, FIRST, and LAST.
  3. For Encryption choice, choose Deterministic encryption.
  4. For Choose secret, select the databrew!default AWS secret.
  5. Select Apply.
  6. After you end making use of all of your transformations, select Publish.
  7. Enter an outline for the recipe model and select Publish.

Create a DataBrew job

Now that our recipe is prepared, we will create a job to use the recipe steps to the Sufferers dataset.

  1. On the DataBrew console, select Jobs.
  2. Select Create a job.
  3. For Job identify, enter a reputation (for instance, Affected person PII Making and Encryption).
  4. Choose the Sufferers dataset and select patients-pii-handling-recipe as your recipe.
  5. Beneath Job output settings¸ for File kind, select your ultimate storage format to be Parquet.
  6. For S3 location, enter your S3 output as s3://databrew-clean-pii-data-<Account-ID>/cleaned_data_output/.
  7. For Compression, select None.
  8. For File output storage, choose Substitute output information for every job run.
  9. Beneath Permissions, for Position identify¸ select the identical IAM function we used beforehand.
  10. Select Create and run job.

Create an Athena desk

You may create tables by writing the DDL assertion within the Athena question editor. When you’re not accustomed to Apache Hive, you need to assessment Creating Tables in Athena to discover ways to create an Athena desk that references the information residing in Amazon S3.

To create an Athena desk, use the question editor and enter the next DDL assertion:

CREATE EXTERNAL TABLE patient_masked_encrypted_data (
   `id` string, 
  `birthdate` string, 
  `deathdate` string, 
  `ssn` string, 
  `drivers` string, 
  `passport` string, 
  `prefix` string, 
  `first` string, 
  `final` string, 
  `suffix` string, 
  `maiden` string, 
  `marital` string, 
  `race` string, 
  `ethnicity` string, 
  `gender` string, 
  `birthplace` string, 
  `tackle` string, 
  `metropolis` string, 
  `state` string, 
  `county` string, 
  `zip` int, 
  `lat` string, 
  `lon` string, 
  `healthcare_expenses` double, 
  `healthcare_coverage` double 
LOCATION 's3://databrew-clean-pii-data-<Account-ID>/cleaned_data_output/'

Let’s validate the desk output in Athena by operating a easy SELECT question. The next screenshot reveals the output.

We are able to clearly see the encrypted and redacted column values in our question output.

Cleansing up

To keep away from incurring future expenses, delete the assets created throughout this walkthrough.


As demonstrated on this publish, you should use DataBrew to assist establish, redact, and encrypt PII knowledge. With these new PII transformations, you may streamline and simplify buyer knowledge administration throughout industries comparable to monetary companies, authorities, retail, and rather more.

Now which you could defend your delicate knowledge workloads to satisfy regulatory and compliance greatest practices, you should use this answer to construct de-identified knowledge lakes in AWS. Delicate knowledge fields stay protected all through their lifecycle, whereas non-sensitive knowledge fields stay within the clear. This strategy can permit analytics or different enterprise features to function on knowledge with out exposing delicate knowledge.

In regards to the Authors

Harsh Vardhan Singh Gaur is an AWS Options Architect, specializing in Analytics. He has over 5 years of expertise working within the subject of massive knowledge and knowledge science. He’s enthusiastic about serving to clients undertake greatest practices and uncover insights from their knowledge.

Navnit Shukla is an AWS Specialist Answer Architect, Analytics, and is enthusiastic about serving to clients uncover insights from their knowledge. He has been constructing options to assist organizations make data-driven choices.