Test Data Engineering for Multi-Use Case Analytics: Beyond Simple Synthetic Data Generation

We evaluated three synthetic healthcare datasets (CMS PUFS, NCQA certification data, Synthea) and found each had critical limitations—format mismatch, narrow scope, or insufficient measure coverage. This post documents our hybrid approach combining CMS PUFS for format compatibility with NCQA datasets for coverage, and the specialized engineering pipeline we built to adjust synthetic data distributions to match expected client patterns while maintaining statistical validity for hierarchical rollup testing.

Why this matters: Most synthetic data content focuses on privacy-preserving generation or basic format matching, but rarely addresses the challenge of test data that must simultaneously support certification validation, forecast simulation, client demos, and multi-tier aggregation testing. When each use case requires different statistical properties and your datasets have complementary limitations, standard synthetic data approaches fall short—yet practical hybrid strategies are underrepresented in current literature.

Synthetic data in healthcare is usually pitched as a solution to privacy, availability and access concerns. Generating artificial patient records sidesteps HIPAA, IT timelines, and data-sharing agreements that can stall a project for months, and it can significantly increase the amount of data available to the project. But teams actually building healthcare analytics products quickly encounter the tricky problem of producing data that's simultaneously format-correct, statistically valid, clinically realistic, and useful across multiple downstream tasks such as testing, demos, certification validation, and forecasting simulation. Most publicly available synthetic datasets do one of those things well and fail at the rest.

What's Out There (and Where It Falls Short)

If you've worked with healthcare data in a product development context, you've probably encountered some combination of these sources. Public reference datasets like CMS Public Use Files (PUFS) give you real-world format compatibility and reasonable distributions, but they're scoped to Medicare populations and lack the clinical detail needed for measure-level testing. Certification datasets from bodies like NCQA give you measure coverage for validation workflows, but the underlying patient records can be clinically unusual such as 80-year-old pregnant patients and 18-year-olds with dementia and osteoporosis  diagnoses. They were never designed to be realistic, just structurally complete. Synthetic generators like Synthea produce longitudinal patient records with clinical plausibility, but the output doesn't map cleanly onto the formats, measure definitions and distributions that real analytics platforms require.

On a project building a healthcare analytics platform that needed to support HEDIS certification testing, client demos, and forecast simulation, we evaluated all three and found that none could serve as a standalone source. Each had complementary strengths—CMS for format, NCQA for measure coverage, Synthea for clinical logic—but critical gaps that required a hybrid approach: merging datasets, engineering target distributions, and building a pipeline to adjust proportions to match expected client patterns.

That hybrid worked. It got us through certification testing and initial demos. But it had real limitations. We could control aggregate distributions such as the proportion of members in each age band, the prevalence of certain conditions, but we couldn't cheaply generate the underlying clinical detail that makes individual records believable. Adjusting proportions is engineering. Making each record internally consistent across demographics, diagnoses, medications, and utilization history is a different kind of problem that requires encoding hundreds of implicit clinical rules.

Where GenAI Changes the Equation

This is where generative AI becomes genuinely useful, and not in the way most synthetic data content frames it. The standard narrative is that GenAI can generate entire synthetic datasets from scratch for privacy preservation. That's true but not very interesting since dedicated tools like Synthea and commercial platforms like Tonic.ai already handle that workflow well with more control. The more compelling use case is GenAI as a clinical realism layer on top of engineered statistical scaffolding.

The idea: you engineer the distributions you need (age bands, condition prevalence, measure eligibility rates, hierarchical rollup proportions), then use a large language model to generate individual patient records that fit those distributions while remaining clinically plausible. The LLM implicitly understands that 80-year-olds are rarely pregnant, that an 18-year-old with a dementia diagnosis needs an exceptional clinical rationale, that a patient on metformin probably has a diabetes-related diagnosis somewhere in their history. You don't need to enumerate every rule because the model has internalized them from its training corpus of clinical literature, coding guidelines, and medical documentation.

Where This Still Breaks

Despite the gains, there are still some limits to this approach. Synthetic data, whether rule-based, GAN-generated, or LLM-produced, cannot replace real-world data for building predictive models. The subtle reason is that when you define the distributions and clinical profiles that go into your synthetic dataset, any model trained on that data will learn your assumptions back at you. You specified the condition prevalence, the demographic breakdowns, the utilization patterns—all things a model would typically encode from real data so the model "predicts" what you already told it. 

This is particularly dangerous because synthetic data generated by modern tools looks convincing. Individual records are clinically plausible. Aggregate statistics match your targets which may also match what you’ve seen in the wild. Standard validation metrics come back clean. But the data carries no information the generating process didn't already contain. A Lancet Digital Health paper from late 2025 coined the term "synthetic trust" to describe exactly this trap: unwarranted confidence in models trained on artificial datasets that fail to preserve the relationships and edge cases present in real clinical populations.

The model-collapse literature adds another dimension. Research published in Nature (Shumailov et al., 2024) showed that models trained recursively on synthetic data lose tail distributions first—rare events, edge cases, atypical presentations—before eventually degrading entirely. In healthcare, those tails are often where the most consequential clinical decisions happen. Rare conditions, atypical presentations, intersectional risk factors—synthetic data systematically smooths over exactly the cases where AI tools are most needed and most dangerous when wrong.

The honest position is this: the engineering-plus-GenAI approach is powerful for bounded use cases where you control the scenario parameters and need data that's realistic enough to validate systems, run demos, and simulate outcomes. It is not a path to training predictive models, and the fact that the data looks more realistic than ever makes the temptation to try and the risk of doing so uncritically, higher than before.

What This Means in Practice

If you're building a healthcare product that needs synthetic data, the practical framework is:

Use the hybrid approach for test data, QA pipelines, certification validation, sandbox environments, client demos, and scenario-based simulation where you define the parameters. Engineer distributions to match your target populations, then use GenAI to generate clinically plausible records that fit those distributions. Validate individual records against clinical logic and aggregate statistics against your targets. Do not use it for training predictive models, risk stratification, or any application where the goal is to discover patterns in the data rather than verify system behavior against known patterns. For those applications, real-world data—with all its access constraints, messiness, and regulatory overhead—remains irreplaceable. Develop on synthetic, validate and deploy on real. Watch out for the confidence gap: modern synthetic data looks better than ever, which makes it easier to skip the step where you ask whether the data can actually support the use case you're applying it to.