Data is necessary for testing, research, and algorithm training. However, privacy regulations and data security protocols mean that companies can’t just run these on their gathered data. Businesses risk high financial losses and reputational damage if such data leaks. This is where de-identified data comes into play.
By de-identifying data, you remove direct and indirect identifiers from the datasets, making it impossible to track the information back to individuals. However, manual de-identification is time-consuming and error-prone. Companies can use automation tools, but not all offer equal levels of privacy. Even worse, certain techniques decrease data usability, making it less fit for purpose.
Our article will discuss the most popular techniques of data de-identification and explain how to maintain the quality of the datasets. But first, let’s start with the definition of the de-identified data. re-identification re-identification re-identification
Table of Contents
- What is data de-identification?
- Watch our webinar on Test Data Management
- Why should you de-identify data?
- Regulatory requirements for data de-identification
- How to de-identify data
- Smart data de-identification
- Smart de-identification process
- Automated data de-identification at scale
- FAQs About Data De-Identification
- About the author
What is data de-identification?
Data de-identification means removing, masking, or replacing sensitive personally identifiable information (PII) from data. This process allows companies to comply with privacy regulations when they use data for testing, analytics, research, and so on. re identify
PII can reveal sensitive data directly and indirectly via identifiers. Direct identifiers point to an individual and can include the following:
- Full names
- Unique identifying numbers (Social Security number, certificate, license number, etc.)
- Passport scans
- Location data
- Biometric identifiers (fingerprint, voice sample, face ID, etc.)
- Trade union membership cards
- Protected health information (medical records, treatment history, etc.)
Indirect identifiers can be used to identify an individual when combined with other information. Examples of this data include:
- Date of birth
- ZIP code
- Contact information (email address, telephone or fax number, web URL, etc.)
- IP addresses
- Vehicle information
- Gender identity
- Educational information
- Product serial numbers
- Transaction history
- Employment data
- Private communications (correspondence)
Dealing with identifiers is quite a challenge, given the volume of information organizations collect daily.
Sensitive information is found across industries
Businesses in all sectors accumulate indirect information that could reveal someone’s identity and, consequently, break privacy laws. Here are a few examples:
- Finance companies store account identifiers, credit card numbers, and customers’ spending patterns.
- Healthcare providers collect information about health conditions, treatments, and insurance details.
- Marketers work with information about purchase histories and a wealth of demographic information.
- Manufacturers record employee details, supplier information, production output, and maintenance logs.
- Logistics and transportation companies keep customer delivery addresses, payment data, and driver details.
The abundance of PII data increases the risks of non-compliance, which is just one of the reasons businesses invest in de-identification tools.
Watch our webinar on Test Data Management
Why should you de-identify data?
Data de-identification allows companies to use, share, and sell high-quality data freely. Let’s break down all the benefits:
- Adherence to data privacy laws. Data privacy regulations mandate the rules for collecting, storing, sharing, and managing PII. Data de-identification and anonymization are necessary to comply with strict data privacy regulations.
- Reduced costs of compliance. With automated de-identification software, you can apply standardized techniques consistently across your datasets. This further reduces the costs of privacy compliance and risks of financial penalties for regulatory violations.
- Lowered impact of data breaches. The cost of an average data breach has risen from $3.62 million in 2017 to $4.45 million in 2023 (according to an IBM report). By de-identifying your datasets, you will greatly reduce the potential harm if attackers gain access to your databases.
- Extra protection of PII. You can prevent unauthorized access to and malicious misuse of data by using only de-identified datasets. It’s important to note that the risk is from malicious agents and those who handle data daily — such as software developers, testers, data analysts, and service providers — who can unintentionally compromise data.
- Improved operational efficiency. By reducing the need for extensive protection measures, you facilitate access to the data for your employees. A reliable process actually speeds up product development cycles, research, and business operations.
- Elevated customer trust. Customers are more likely to use your services and share their personal information if your reputation isn’t tarnished by data leaks, civil lawsuits, and compliance fines.
- Enhanced sharing and collaboration. De-identification allows you to safely share data with your employees, business partners, and other third parties without breaking privacy regulations.
- Additional revenue streams. You can use de-identification tools to build a data marketplace for monetization purposes. Many businesses will pay for high-quality data for testing, AI algorithm training, or research. For example, thanks to our platform, Erasmus Medical Center sells synthetic data to healthcare and medical research companies.
Because privacy requirements differ in certain jurisdictions and sectors, de-identification tools must meet several regulations.
Regulatory requirements for data de-identification
Most privacy regulations contain requirements similar to those in GDPR, CCPA, CPRA, and HIPAA. The data protection laws with which you must comply will depend on the location of your business and the residency of your users:
- General Data Protection Regulation: The GDPR is a comprehensive data protection law in the European Union (the United Kingdom follows the UK-GDPR). Companies must comply with it if they collect, store, share, and process the personal data of EU (or UK) citizens.
- Health Insurance Portability and Accountability Act: HIPAA regulates health information management for US healthcare providers, clearinghouses, and business providers. Specifically, the HIPAA (Privacy Rule) sets forth the conditions for using, disclosing, and retaining individual information. It also sets out permissible methods of de-identifying data, such as the Safe Harbor or Expert Determination methods.
- California Privacy Laws: The California Consumer Privacy Act (CCPA) and the California Privacy Rights Act (CPRA) establish additional privacy rules regarding collecting and using California residents’ PII.
All businesses and entrepreneurs must comply with these strict obligations when dealing with any information. However, GDPR, HIPAA, and California Privacy Laws exclude de-identified data. That’s right. Any dataset that lacks information that is traceable back to an individual falls outside the scope of these regulations. To make your data non-regulated, you must remove PII in a way that makes it impossible to re-identify the data.
How to de-identify data
We follow the Safe Harbor method to guarantee compliance with de-identified datasets. This method requires you to remove or modify all direct and indirect identifiers—18 types in total. Here’s how organizations can establish a continuous data de-identification process:
1. Organize the data
De-identification begins with thoroughly auditing all applications, databases, and tables. You should understand what data is collected, how it’s stored, and how long it’s retained. Create a map of all data sources and their flow within your organization.
At this point, stakeholders should be assigned ownership of specific types of data to ensure accountability. Conduct audits regularly to maintain compliance.
2. Detect PII in datasets
Identify all datasets that contain PII and other sensitive data. Next, you should classify this data into different groups, such as non-sensitive data, direct and indirect identifiers, corporate information, and compliant data.
To streamline the management, companies also establish policies for identifying and handling PII. For extra security, apply access control rules to different types of data based on regulatory requirements and business needs.
3. Tag identifiers
Once data is classified, it should be tagged with appropriate metadata to indicate its sensitivity and type. Implement standardized tagging conventions to ensure uniformity across all datasets and streamline the de-identification process.
4. Select the de-identification method
Select the de-identification technique based on your needs, such as the data utility requirements and regulatory rules. The techniques vary in terms of privacy protection and have different impacts on usability.
For instance, the pseudonymization technique replaces PII with pseudonyms or codes while barely affecting data structure. However, in experienced hands, this information can be re-identified. More advanced tools can replace sensitive data without compromising privacy or usability.
You may de-identify data on database and column levels.
Database-level de-identification
For database-level de-identification, simply drag tables from your relational database into the de-identify section in the workspace.
Column-level de-identification
To apply de-identification on a more granular level or column level, open a table, choose the specific column you want to de-identify, and effortlessly apply a mocker. Streamline your data protection process with our intuitive configuration features.
5. De-identify datasets
Apply the selected de-identification techniques to the selected datasets. De-identification should be viewed as an iterative process rather than a one-time task. We recommend picking a few sample datasets. After the initial de-identification, you should review the results before proceeding.
6. Validate the results
You should assess the de-identified data to ensure it meets your business requirements. It’s necessary to engage the data owners and other relevant stakeholders in the review. The validation process itself should involve several steps:
- Verify that all identifiers were removed or replaced.
- Evaluate the possibility of re-identification based on a combination of remaining data points.
- Confirm that the de-identified information retains the original level of detail and accuracy.
- Make sure no critical records or files are lost or corrupted.
- Check if the relationships and patterns within the data are preserved.
As you can expect, doing all this manually is tedious, long, and expensive. Besides, doing this by hand will result in occasional errors and inconsistencies, which increase the identification risk. That’s why organizations use automated de-identification methods.
Smart data de-identification
The truth is that most techniques of PII removal leave vulnerabilities that malicious actors can exploit to trace the data back to the individuals. Other methods decrease the statistical accuracy of data to the point that they can’t be used for advanced research and AI training.
Syntho’s smart de-identification technology is made to automate manual work without privacy or quality trade-offs. Our advanced AI-powered scanner identifies PII across tables, databases, and other sources.
Once identified, the platform replaces the sensitive information with mock data. At the same time, our engine maintains a consistent mapping to preserve referential integrity and business patterns.
That isn’t all. Our software has extra features that might enhance the de-identification process:
- Data enrichment allows adding rows and columns to the de-identified datasets, making it easier to create larger and more comprehensive testing datasets.
- Subsetting helps create smaller datasets for testing, reducing the burden on storage and processing resources.
- Rule-based flexibility makes the data adaptable to different data formats, structures, and scenarios.
- Data cleansing corrects inconsistencies, fills in missing values, and removes corrupted data.
Syntho automates most of the manual work, lowers the chances of missing sensitive data, and maintains the quality of the original data.
Smart de-identification process
- Automatically scan for personally identifiable information (PII) in your databases
- Protect sensitive information by removing or modifying personally identifiable information (PII)
- Apply the selected mocker to a column through either the PII or Job Configuration tab
1. Identify PII
Scan PII automatically with our AI powered PII Scanner or identify columns that you would like to mock.
2. Select Mockers
Select the suggested mocker by our PII scanner or configure mockers on column level.
3. Apply Mocker
Confirm to apply the selected mocker on a column via the PII or Job Configuration tab.
Automated data de-identification at scale
De-identification is necessary to comply with privacy regulations, protect sensitive information, and maintain data usability. Removing or masking identifiers can improve operational efficiency, lower security risks, and even decrease operational costs. However, manual de-identification is far too inefficient for most businesses.
Syntho’s smart de-identification technology automates the de-identification of PII across datasets. It uses AI to detect sensitive information and replaces it with mock data based on your business rules, all while maintaining the original quality of the data.
Do you want to improve your de-identification process and ensure compliance? Contact us to get a demo.
FAQs About Data De-Identification
What are the main data de-identifying techniques (methods)?
Data de-identifying techniques include redaction, removal, pseudonymization, perturbation, and subsampling. Redaction involves obscuring sensitive information, removal deletes identifiable data, pseudonymization replaces identifiers with codes, perturbation adds noise to data to mask values, and subsampling involves using only a subset of data.
What is the difference between de-identified, anonymized, and synthetic data?
In de-identified data, direct and indirect identifiers are deleted or replaced to ensure individuals cannot be identified. Anonymized data involves altering or removing confidential information using advanced algorithms to ensure individuals cannot be re-identified. Synthetic data is newly generated data that replicates the structure and properties of the original dataset without links to real individuals.
What are the differences between de-identified and limited data sets?
A limited dataset under HIPAA includes identifiable healthcare information that can be shared for research, public health, and healthcare operations, but only with entities that have signed a data use agreement. In contrast, de-identified data lacks identifiers and is not regulated by HIPAA, GDPR, or other privacy laws so you can share it freely.
About the author
Shahin Huseyngulu
Customer Service Engineer & Data Scientist
Shahin Huseyngulu has a strong academic foundation in Computer Science and Data Science and is an experienced Customer Service Engineer and Data Scientist. Shahin has held key roles in customer service, cloud solutions, and machine learning research, showcasing expertise in Python, SQL, and data analytics. Currently, Shahin excels as a Customer Service Engineer at Syntho, building and optimizing customer service operations while bringing a unique blend of technical and customer service skills to drive innovation and customer satisfaction in the tech industry.
What is Data De-Identification, and Why do I Need it? – Syntho
Explore what is data de-identification and why it’s crucial for privacy. Learn how Syntho’s solutions safeguard your data while maintaining usability.