Synthetic-to-real validation evidence gaps: when the real side of the comparison is missing

Short answer

Synthetic-to-real validation is hard to assess when the real side of the comparison is incomplete. Four gaps do most of the damage: only the current policy is kept, nobody recorded who could approve exceptions, case statuses were overwritten as they moved, and final outcomes sit in a retired tool. Each one turns an expert's judgment about a generated case from a checked finding into an opinion. None ends the conversation, but each narrows what can be validated.

Visual briefFull-size diagram
Five horizontal timeline bars span 2020 to 2025 for a fictional electronics retailer's returns records; filled segments show the years each record type survives and orange outlined segments show missing years, shortest for exception authority and warehouse grading.
Illustrative coverage for a fictional 450-person electronics retailer. Missing years are where operator verdicts cannot be checked.

This gap analysis is part of the guide to synthetic-to-real validation, which follows one generated product return through an expert check. The question here is what happens when the records behind such a check are missing, and how an introducer should describe that honestly.

What each verdict needs behind it

In this kind of check, a person who did the job marks each decision in a generated case as right, partly right or wrong. A reviewer accepts that mark only when a company record settles it. Five kinds of record do that work.

Record a review expectsWhat it settles
Dated policy versionsWhich rule applied on the simulated order date
Exception authorityWhether a departure from policy was a breach or an approved exception
Case status historyThe real order of steps, not only the last status
Final dispositionWhat happened to the item or claim at the end
Comparable real cases with outcomesThat the operator's reading holds across more than one case

A mark without a record is still useful as a pointer to where to look, but nobody outside the room can test it. The archive is what lets a stranger agree or disagree on the evidence.

Expected against usual: the gap table

The pattern below reflects how returns and customer service systems are commonly run, not a survey of companies.

RecordWhat companies usually keepWhat usually goes missingVerdicts left unsupportedUsual owner
Policy versionsThe current returns policyEarlier versions and their effective datesTime limits and feesCustomer service or legal
Exception authorityThe exception itself, in a noteWho was allowed to waive the ruleChecks on high-value itemsOperations management
Case status historyThe latest statusThe sequence of statuses with datesOrder of stepsE-commerce or IT
Final dispositionRefund recordsGrading and resale outcomeWhat happened to the itemWarehouse or finance
Comparable casesMany routine casesTagged exception casesAnything unusualCustomer service

Four gaps, traced through the simulated return

Only today's policy. If the company keeps only the current returns policy, a reviewer cannot tell whether the simulation's "45 days" was wrong in 2022 or only now. Versioned policy documents, or at least dated change notices, are the first thing to establish.

No exception authority. The operator says high-value refunds needed a serial check. If the archive does not record who could waive it, a reviewer cannot separate a policy breach from an authorized exception, and the serial-check verdict cannot be confirmed.

Overwritten statuses. Some order systems overwrite the return status as it moves. If only "Refunded" survives, the real sequence (claim filed, inspection, then refund) is lost, and the operator's verdict that the simulation refunded too early cannot be shown.

Outcome in a retired tool. If grading and resale were tracked in a separate warehouse tool that was switched off, the disposition field has nothing to check against.

The 2024 paper "Best Practices and Lessons Learned on Synthetic Data" names fidelity as a core requirement. Each of these gaps removes the evidence of fidelity for one field, which is why the gaps are worth naming field by field rather than as a general weakness. Rare branches suffer most: a 2024 paper in Nature found that indiscriminate training on model-generated content makes the tails of the original distribution disappear, and in returns the tails are the holiday extension, the high-value check and the damaged parcel.

Six years of returns records, mapped (Illustrative)

Illustrative and fictional. The same 450-person consumer electronics retailer from the guide is asked one question per record type: from which year does it survive in a form someone could check?

RecordSurviving coverage, 2020 to 2025Reason for the gap
Policy versions2023 to 2025 (3 of 6 years)Older versions overwritten on the website
Exception authority2025 only (1 of 6 years)Approval matrix written down last year
Case status history2022 to 2025 (4 of 6 years)Order platform replaced in 2022
Final disposition2024 to 2025 (2 of 6 years)Grading moved from a retired warehouse tool
Refund outcomes2020 to 2025 (6 of 6 years)Payment processor records kept throughout

Read across, every field can be validated only for 2025. For 2024, every field except the serial check can be validated; for 2023, the window, fee and damage order, but not disposition; for 2022, only the damage order and the refund method. That is a precise, honest description: a narrow fully checkable slice and a wider partly checkable one. The graphic on this page draws the same five rows as bars, with the missing years outlined.

How to describe the gaps to SourceX

Instead ofSay
"We have all our returns history.""Refund records since 2020; dated policies from 2023; grading from 2024."
"Our team knows the exceptions.""Exception approvals were written down from 2025; before that, the returns lead remembers them."
"Every case is tracked.""Full status history since the 2022 platform change; only final status before."

Give the year each record type begins and who gave the answer. The live pages on synthetic data limits and whether synthetic data will replace real data explain why buyers care about exactly these boundaries, and what synthetic data is covers the basics for an owner new to the term.

Who owns the missing records, and what review settles

Customer service or legal owns policy versions; operations management owns exception authority; e-commerce or IT owns order history; the warehouse and finance own disposition and refunds. Authority to discuss licensing normally sits with the CEO, COO, CFO or owner.

The company test is unchanged by gaps: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Unknown until SourceX reviews it: SourceX's current data needs, the rights review result, the de-identification required and how much of each year is usable. Partners never request policy files or case exports; discovery stays at the level of systems, years and owners. What the company would prepare for a review is listed in the synthetic-to-real validation review packet; the trade-offs between generated and real records are compared in synthetic data vs real data for AI agents, and other record families are under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the operations lead can say which year each record type starts, even with gaps, register as a partner, check who qualifies, and read the terms before introducing the executive who can authorize the discussion.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

If an experienced operator is available, do record gaps still matter?

Yes. The operator's judgment is the starting point, but a reviewer needs a record that settles each verdict. Without it, a disagreement with the simulation is an opinion, however well informed.

Are gaps in old years worse than gaps in recent ones?

Not necessarily. Recent years are usually more complete, and a clean recent slice can be enough. What matters is that the company can say where coverage starts for each record type.

Should the company recreate missing policy versions from memory?

No. A reconstructed policy is a new document, not a record from the time. Describe the gap; SourceX decides in review how to scope around it.

Sources

  1. nature.com/articles/s41586-024-07566-y
  2. arxiv.org/abs/2404.07503
  3. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment