Fraud and financial crimes remain a significant challenge for businesses and institutions worldwide. Access to large, clean, and privacy-compliant datasets is critical for building effective fraud detection models. However, obtaining real-world fraud data can be difficult due to privacy concerns, sensitive information, and data-sharing limitations. This is where synthetic data comes in, offering a secure and powerful alternative.
In this research together with SAS Institute, we have explored how Syntho’s synthetic data solution addresses these challenges and delivers superior performance for fraud detection and anti-money laundering (AML) use cases, based on a real-world fraud case study.
Why Use Synthetic Data in Fraud and Financial Crimes?
Data in fraud and financial crimes detection often presents unique challenges, including:
- Inaccessibility: Real-world fraud data is sensitive and often protected by stringent regulations, making it difficult to share for collaborative purposes.
- Imbalanced Data: Fraudulent activities make up only a small fraction of most datasets, leading to unbalanced data that can affect the performance of machine learning models.
- Privacy Concerns: Sharing customer or transaction data can expose businesses to privacy risks and regulatory violations.
- Biased Data: Real-world data often contains biases that can skew fraud detection models.
Synthetic data provides a solution by generating artificial datasets that mimic the statistical patterns of the original data, without using any actual sensitive information. This ensures data privacy and security while still maintaining the integrity and usefulness of the data for training models.
How Syntho Helps Detect Fraud Using Synthetic Data
Syntho offers an advanced platform for generating high-quality synthetic data that can be applied to fraud detection and AML scenarios. In a recent fraud reasearch, a dataset consisting of 100,000 observations with seven features was used to simulate fraudulent transactions. The goal was to create a synthetic version of the dataset that could be used to train machine learning models for fraud detection.
In our exploration of synthetic data for fraud detection, it’s essential to discuss the Synthetic Data Vault (SDV), an open-source Python library designed to generate high-quality synthetic data using various models and techniques.
What is the Synthetic Data Vault (SDV)?
The SDV is a flexible framework that allows users to create synthetic datasets while preserving the statistical properties of the original data. It includes several synthesizers, and in our fraud detection research, we used the Gaussian Copula synthesizer to mimic real-world data patterns effectively.
This tool is particularly beneficial for:
- Maintaining the structure and relationships between features in complex datasets.
- Generating synthetic datasets that can be safely shared across teams without privacy concerns.
Gaussian Copula: A Key Synthesizer
The Gaussian Copula synthesizer, a model within the SDV framework, was instrumental in generating our fraud detection dataset. By estimating parameters for each column’s distribution and capturing the covariance matrix, the Gaussian Copula creates synthetic data that retains relationships between columns, such as those between transaction amounts and times in fraud detection.
The use of Gaussian Copula in our research led to highly accurate synthetic data that closely mirrored the original dataset, particularly in preserving the correlation structure critical for fraud detection.
Source: Kalyan VeeramachaneniNeha Patki, Roy Wedge. The synthetic data vault. In International Conference on Data Science and Advance Analytics, 2016.
Evaluating the Quality of Synthetic Data: Key Techniques
To assess the effectiveness of synthetic data for fraud detection, several evaluation techniques were employed, focusing on how well the synthetic data mimicked the original dataset. This ensures that the generated data is reliable for training machine learning models and maintaining data utility. Below are the main evaluation techniques used:
Main Evaluation Techniques
- Machine Learning Models:
The synthetic data was tested by training machine learning models and comparing the performance with models trained on the original data. - Correlations:
The relationships between features in the dataset were analyzed to ensure that the synthetic data preserved the correlations found in the original data. - Distributions:
Key data distributions, such as transaction amounts and times, were compared between the synthetic and original datasets to verify similarity. - Upsampling:
Synthetic data was used to balance datasets where fraud cases are rare, improving the performance of fraud detection models.
Fraud Indicator Distributions
Both the SDV and Syntho solutions managed to reproduce the fraud indicators accurately, reflecting the original data patterns. This was crucial in ensuring that the synthetic data could be used effectively for fraud detection without compromising accuracy.
Box Plot of Transaction Amounts
Syntho’s solution proved to be more effective than SDV at mimicking the distribution of transaction amounts, as shown in the box plot comparison. This demonstrated that Syntho’s synthetic data could closely reflect real-world transaction behavior.
Transaction Time Distributions
The distribution of transaction times was another critical metric. Syntho’s generated data mirrored the original data patterns better than SDV, reinforcing the platform’s capability to maintain temporal relationships in the dataset.
Dimensionality Reduction Using PCA
Dimensionality reduction was performed using Principal Component Analysis (PCA) to visualize and analyze the high-dimensional data. This technique allowed the dataset to be re-expressed while preserving the maximum amount of information. The PCA scatter plots confirmed that Syntho’s synthetic data closely aligned with the structure of the original data, ensuring data consistency across all dimensions.
Key Highlights from the research
Machine Learning Models: Decision Tree
Synthetic data generated by Syntho can effectively be used for scoring and assessing fraud detection models. In the research, a Decision Tree model was applied to both real and synthetic datasets, and the results indicated that the model trained on synthetic data (Syntho) performed comparably to those trained on real data.
- KS Youden (Goodness-of-Fit):
- Syntho achieved a KS score of 0.8751, closely matching the real data’s score of 0.8730 and performing better than SDV (0.4385).
- Confusion Matrix and Variable Importance:
The confusion matrix and decision tree visualizations show the accuracy and key variables that contributed to the model’s decision-making process. Variables like Point of Sale and Country Code were particularly important in detecting fraud.
Machine Learning Model Comparison
The models trained on Syntho’s synthetic data produced high accuracy levels across multiple evaluation metrics:
- AUC-ROC (Area Under the Curve – Receiver Operating Characteristic):
Decision Tree models trained on Syntho’s synthetic data produced AUC-ROC scores of 0.9571 (training) and 0.9591 (testing), outperforming those trained on real data. - Misclassification Rate (MCE):
The misclassification rate for models trained on Syntho’s synthetic data was extremely close to those trained on the real dataset, showing negligible differences in performance.
This demonstrates that synthetic data can reliably be used to train machine learning models without sacrificing performance.
Upsampling for Fraud Detection
Fraud detection datasets are often imbalanced, with very few instances of fraud compared to legitimate transactions. Upsampling with synthetic data solves this issue by generating additional fraudulent cases to balance the dataset.
- Experiment Setup:
Synthetic datasets were used alongside real data to train models, with the upsampled data enhancing the model’s ability to detect rare fraud cases. - Best Score (AUC):
The Decision Tree model trained on the upsampled synthetic dataset achieved the highest AUC score of 0.78, highlighting the effectiveness of synthetic data for augmenting limited real-world data in fraud detection.
Benefits of Synthetic Data for Fraud Use Cases
Syntho’s synthetic data solution provides numerous advantages for fraud detection and anti-money laundering (AML):
- Privacy Concerns:
Synthetic data offers lower re-identification risks, ensuring compliance with privacy regulations while maintaining utility for training models. - Sharing and Collaboration:
Organizations can safely share synthetic data across teams, facilitating collaboration without exposing sensitive customer information. - Mimicking Real-World Data:
Syntho’s synthetic data closely mimics the features and patterns of the real-world data, making it ideal for fraud detection and model training. - Machine Learning Models:
It can be used for Fraud and AML detection models. - Handling Imbalanced Data:
Synthetic data is highly useful for upsampling limited fraud data, enabling more effective training of machine learning models.
Business Benefits of Synthetic Data
From a business perspective, using synthetic data in fraud detection leads to:
- Increased Sales Opportunities:
By improving fraud detection accuracy, companies can offer more reliable and secure services to their clients. - Ease of Project Opportunities:
Synthetic data allows for faster project execution by eliminating privacy concerns and simplifying data acquisition processes.
Future Steps
The research also outlines key areas for future research and development in synthetic data:
- Metrics on Privacy Risks:
Developing more robust privacy metrics to ensure synthetic data remains secure. - Testing with Advanced Machine Learning Models:
Exploring more complex models and evaluating their performance with synthetic data. - Benchmarking and Balancing Techniques:
Establishing benchmarks for different synthetic data generation techniques and exploring more advanced balancing methods for highly imbalanced datasets. - Time Series Data:
Expanding the use of synthetic data to cover time series fraud detection, which is critical for transaction data.
At Syntho, we believe that synthetic data is the future of fraud detection and AML. By providing a privacy-compliant and scalable way to generate realistic data, we empower organizations to combat financial crime without compromising on security or performance.
If you’d like to learn more about how Syntho can enhance your fraud detection models, feel free to contact us or request a demo!