# Synthetic-to-real validation: evidence gaps | SourceX

Canonical: https://refer.sourcex.si/questions/synthetic-to-real-validation-evidence-gaps
Last updated: 2026-11-04

Synthetic-to-real validation is hard to assess when the real side of the comparison is incomplete. Four gaps do most of the damage: only the current policy is kept, nobody recorded who could approve exceptions, case statuses were overwritten as they moved, and final outcomes sit in a retired tool. Each one turns an expert's judgment about a generated case from a checked finding into an opinion. None ends the conversation, but each narrows what can be validated.

This gap analysis is part of the guide to [synthetic-to-real validation](/guides/concept-synthetic-to-real-validation), which follows one generated product return through an expert check. The question here is what happens when the records behind such a check are missing, and how an introducer should describe that honestly.

## What each verdict needs behind it

In this kind of check, a person who did the job marks each decision in a generated case as right, partly right or wrong. A reviewer accepts that mark only when a company record settles it. Five kinds of record do that work.

| Record a review expects | What it settles |
|---|---|
| Dated policy versions | Which rule applied on the simulated order date |
| Exception authority | Whether a departure from policy was a breach or an approved exception |
| Case status history | The real order of steps, not only the last status |
| Final disposition | What happened to the item or claim at the end |
| Comparable real cases with outcomes | That the operator's reading holds across more than one case |

A mark without a record is still useful as a pointer to where to look, but nobody outside the room can test it. The archive is what lets a stranger agree or disagree on the evidence.

## Expected against usual: the gap table

The pattern below reflects how returns and customer service systems are commonly run, not a survey of companies.

| Record | What companies usually keep | What usually goes missing | Verdicts left unsupported | Usual owner |
|---|---|---|---|---|
| Policy versions | The current returns policy | Earlier versions and their effective dates | Time limits and fees | Customer service or legal |
| Exception authority | The exception itself, in a note | Who was allowed to waive the rule | Checks on high-value items | Operations management |
| Case status history | The latest status | The sequence of statuses with dates | Order of steps | E-commerce or IT |
| Final disposition | Refund records | Grading and resale outcome | What happened to the item | Warehouse or finance |
| Comparable cases | Many routine cases | Tagged exception cases | Anything unusual | Customer service |

## Four gaps, traced through the simulated return

**Only today's policy.** If the company keeps only the current returns policy, a reviewer cannot tell whether the simulation's "45 days" was wrong in 2022 or only now. Versioned policy documents, or at least dated change notices, are the first thing to establish.

**No exception authority.** The operator says high-value refunds needed a serial check. If the archive does not record who could waive it, a reviewer cannot separate a policy breach from an authorized exception, and the serial-check verdict cannot be confirmed.

**Overwritten statuses.** Some order systems overwrite the return status as it moves. If only "Refunded" survives, the real sequence (claim filed, inspection, then refund) is lost, and the operator's verdict that the simulation refunded too early cannot be shown.

**Outcome in a retired tool.** If grading and resale were tracked in a separate warehouse tool that was switched off, the disposition field has nothing to check against.

The 2024 paper "Best Practices and Lessons Learned on Synthetic Data" names fidelity as a core requirement. Each of these gaps removes the evidence of fidelity for one field, which is why the gaps are worth naming field by field rather than as a general weakness. Rare branches suffer most: a 2024 paper in Nature found that indiscriminate training on model-generated content makes the tails of the original distribution disappear, and in returns the tails are the holiday extension, the high-value check and the damaged parcel.

## Six years of returns records, mapped (Illustrative)

**Illustrative and fictional.** The same 450-person consumer electronics retailer from the guide is asked one question per record type: from which year does it survive in a form someone could check?

| Record | Surviving coverage, 2020 to 2025 | Reason for the gap |
|---|---|---|
| Policy versions | 2023 to 2025 (3 of 6 years) | Older versions overwritten on the website |
| Exception authority | 2025 only (1 of 6 years) | Approval matrix written down last year |
| Case status history | 2022 to 2025 (4 of 6 years) | Order platform replaced in 2022 |
| Final disposition | 2024 to 2025 (2 of 6 years) | Grading moved from a retired warehouse tool |
| Refund outcomes | 2020 to 2025 (6 of 6 years) | Payment processor records kept throughout |

Read across, every field can be validated only for 2025. For 2024, every field except the serial check can be validated; for 2023, the window, fee and damage order, but not disposition; for 2022, only the damage order and the refund method. That is a precise, honest description: a narrow fully checkable slice and a wider partly checkable one. The graphic on this page draws the same five rows as bars, with the missing years outlined.

## How to describe the gaps to SourceX

| Instead of | Say |
|---|---|
| "We have all our returns history." | "Refund records since 2020; dated policies from 2023; grading from 2024." |
| "Our team knows the exceptions." | "Exception approvals were written down from 2025; before that, the returns lead remembers them." |
| "Every case is tracked." | "Full status history since the 2022 platform change; only final status before." |

Give the year each record type begins and who gave the answer. The live pages on [synthetic data limits](/proof/synthetic-data-limits) and [whether synthetic data will replace real data](/questions/will-synthetic-data-replace-real-data) explain why buyers care about exactly these boundaries, and [what synthetic data is](/questions/what-is-synthetic-data) covers the basics for an owner new to the term.

## Who owns the missing records, and what review settles

Customer service or legal owns policy versions; operations management owns exception authority; e-commerce or IT owns order history; the warehouse and finance own disposition and refunds. Authority to discuss licensing normally sits with the CEO, COO, CFO or owner.

The company test is unchanged by gaps: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Unknown until SourceX reviews it: SourceX's current data needs, the rights review result, the de-identification required and how much of each year is usable. Partners never request policy files or case exports; discovery stays at the level of systems, years and owners. What the company would prepare for a review is listed in the [synthetic-to-real validation review packet](/resources/synthetic-to-real-validation-review-packet); the trade-offs between generated and real records are compared in [synthetic data vs real data for AI agents](/compare/synthetic-data-vs-real-data-ai-agents), and other record families are under [data types](/data-types).

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the operations lead can say which year each record type starts, even with gaps, [register as a partner](/register), check [who qualifies](/who-qualifies), and read the [terms](/terms) before introducing the executive who can authorize the discussion.

## Sources

- https://www.nature.com/articles/s41586-024-07566-y
- https://arxiv.org/abs/2404.07503
- https://refer.sourcex.si/terms
