The synthetic-to-real validation review packet: eight documents, in order, with who prepares each

Short answer

A synthetic-to-real validation review packet holds eight documents, assembled in this order: a scope note, the generated cases exactly as received, an index of dated policy versions, an index of comparable real cases, the expert's verdict sheet, a list of branches the generator missed, the reviewer's profile, and rights notes. Each describes or cites records rather than containing them. Together they let a reviewer test whether an expert's verdicts on generated cases rest on real evidence.

Visual briefFull-size diagram
A box file with eight index tabs fills from front to back as labelled documents drop in, from the scope note and the unedited simulated cases to the expert verdict sheet and rights notes, each tagged with the role that prepares it.
Illustrative packet for a fictional electronics retailer's returns workflow. Descriptions and verdicts only, no customer records.

The packet belongs to the guide to synthetic-to-real validation, which walks through one generated return checked by a long-serving member of staff. Partners do not build it; it shows what the company's records would need to support if SourceX and the company later agree a scope, so an introducer can describe the effort honestly.

The eight documents, front to back

The order follows the logic of a review: what is being checked, what it is checked against, what the expert concluded, and what may be licensed.

TabDocumentWhat it showsWho prepares it
1Scope noteThe workflow, years and systems the validation coversOperations or customer service lead
2Generated cases, uneditedThe generated cases exactly as receivedFiled by the returns lead, as supplied under the agreed scope
3Policy version indexEach policy version with its effective datesCustomer service or legal
4Comparable case indexCounts and types of real cases available per fieldE-commerce operations
5Expert verdict sheetA verdict per field, each citing a record typeA long-serving team member
6Missing-branch listBranches found in real cases but absent from the generated setThe same expert
7Reviewer profileThe expert's role and tenure, so the judgment can be weighedOperations manager
8Rights and de-identification notesPersonal data present and contract limitsCounsel or the CFO

Documents 3 and 4 come before the verdicts because the expert should cite from them. A verdict sheet written first and justified afterwards is the most common way a packet goes wrong.

Inside each document, for one returns workflow (Illustrative)

Illustrative and fictional. A 450-person online retailer receives a set of generated return requests for a 65-inch television delivered with a cracked screen.

1. Scope note. "Returns and carrier damage claims for large electronics, online orders 2022 to 2025; order platform, carrier claims portal and warehouse grading log."

2. Generated cases, unedited. Twelve generated cases, filed as received. Nothing is corrected, even where the expert sees errors; the errors are the point.

3. Policy version index. Four versions of the returns policy, each with a start date, and two dated notices for seasonal extensions.

4. Comparable case index. "About 140 carrier-damage cases a year on large screens, tagged with a damage reason code since 2022." Counts and tags only, no case content.

5. Expert verdict sheet. For each generated field, a verdict and the record type that settles it. The task specification for this sheet, in full: "For each simulated return, compare every decision field with the company's policy in force on the simulated order date and with at least three real cases of the same type. Mark each field Match, Partial or Diverges, cite the record that settles it, and note any branch the simulation omitted."

6. Missing-branch list. "Carrier inspection appointment before refund; partial refund when the customer keeps the item at a discount."

7. Reviewer profile. "Returns lead, nine years in the role; handled carrier claims throughout."

8. Rights and de-identification notes. Customer names and addresses in cases; carrier contract terms on claim data to be checked.

Why insist on unedited cases and cited verdicts? The 2024 paper "Best Practices and Lessons Learned on Synthetic Data" stresses factuality and fidelity, and a 2024 Nature study found that the tails of a distribution vanish when generated data is used without careful filtering. The packet is a structured way of showing where fidelity breaks, which only works if neither side has been tidied up.

The exact pass test, applied to two entries (Illustrative)

A validation entry passes review when all four conditions hold:

  1. Every field has a verdict.
  2. Every Diverges or Partial verdict cites a real record type that settles it.
  3. The policy version used matches the simulated order date.
  4. At least one real case per field confirms the expert's reading.
ConditionEntry 4Entry 9
Every field has a verdictPass: six of sixPass: six of six
Divergences cite a record typePass: claims portal record cited for the refund orderFail: "refund timing looks wrong" with no record cited
Policy version matches order datePass: version 3, in force on the simulated dateFail: current policy used for a 2023 order
A real case confirms each fieldPassPass
ResultPassesFails, though the expert is probably right

Entry 9 fails even though the expert's instinct matches entry 4. An entry that says "looks wrong" without a citing record fails, however experienced the reviewer; that is what makes a passed entry worth something.

Where packets usually come up short

Three documents cause most delays. The policy version index (tab 3) stalls when old versions were overwritten; the synthetic-to-real validation evidence gaps page shows how to describe that. The comparable case index (tab 4) stalls when exception cases are not tagged. And the rights notes (tab 8) stall when carrier or marketplace contracts govern claim data. None of these is a reason to delay an introduction; they are the questions SourceX's review is designed to answer. An owner preparing for that conversation can start with questions to ask before licensing company data.

What the packet cannot decide

The packet cannot show SourceX's current data needs, the result of the rights review, the de-identification required, or the company's offer. The comparison of synthetic data and real data for AI agents and the page on synthetic data limits explain why buyers look for real reference records; the outlook for that demand is discussed in the future of AI training data, without any promise for a particular company.

The company must meet the basics: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the operations lead confirms the company keeps dated policy versions and tagged real cases that could fill tabs 3 and 4, register as a partner, check who qualifies, and read the terms before introducing the executive who can authorize the discussion.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Who writes the simulated cases in the packet?

Not the company and not the introducer. Simulated cases come from whoever is testing an agent, and they enter the packet unedited, under a scope SourceX agrees with the company. The company's contribution is the reality to check them against.

Can the expert's verdict sheet be done in a workshop?

It can be drafted that way, but every Partial or Diverges verdict still needs a cited record type. A workshop consensus without citations fails the acceptance check.

Does the introducer see the finished packet?

No. The packet is prepared under SourceX's agreement with the company. The introducer's role ends with a qualified introduction to the person who can authorize the discussion.

Sources

  1. arxiv.org/abs/2404.07503
  2. nature.com/articles/s41586-024-07566-y
  3. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment