Synthetic-to-real validation: checking a simulated task against real company records

Short answer

Synthetic-to-real validation means checking a simulated task, such as a generated customer return, against the real thing: the company's policy in force at the time, what actually happened in comparable cases, and the judgment of an experienced operator. For an introducer, the useful question is whether the company holds the versioned policies, the case history with outcomes and the people who can say where a simulation is wrong, and whether someone can authorize a licensing discussion.

Visual briefFull-size diagram
A six-row matrix compares a simulated product return with the operator's verdict: refund method matches, the restocking fee partly matches, and the return window, serial check, damage claim order and disposition diverge from real policy.
Illustrative: one simulated return, six fields, and what an experienced returns lead found.

What is synthetic-to-real validation, in plain terms?

AI developers often train and test agents on generated scenarios because they are cheap and plentiful. The open question is whether an agent that handles the simulated version will handle the real one. Validation is the step that answers it: take simulated cases, compare them field by field with real policy and real records, and have someone who did the job mark every divergence.

Two research findings explain why the step matters. A 2024 paper in Nature showed that indiscriminate training on model-generated content causes defects in which the tails of the original distribution disappear, and that generated data needs careful filtering. In business terms, the tails are the rare branches: the damaged parcel, the fraud check, the holiday exception. A COLM 2024 paper on best practices for synthetic data stresses factuality, fidelity and unbiasedness. Fidelity to a specific company's rules is exactly what an outside generator cannot know. The live comparison of synthetic data and real data for AI agents covers when each wins; this page is about the check that connects them.

One simulated return, checked by a returns lead (Illustrative)

A 450-person online retailer of consumer electronics is asked to review a simulated return generated for an agent test. The returns lead, with eight years on the team, compares it with the policy in force and with how comparable real cases were handled. All details are fictional.

FieldSimulated caseOperator's verdictWhat settles it in the real records
Return window45 days from deliveryDiverges: 30 days, extended to January 31 for holiday ordersPolicy document, version in force on the order date
Restocking fee15% on every opened itemPartly matches: only on opened large appliances, waived for defectsFee rules plus refund lines on past cases
Serial number checkNot performedDiverges: required on items above a price threshold before refundWarehouse receiving scans
Carrier damageRefund issued at onceDiverges: carrier claim filed first, refund after inspectionClaim records linked to the order
Refund methodOriginal payment methodMatchesPayment processor refund records
DispositionRestockDiverges: graded A, B or C for resale, refurbishment or recyclingGrading log in the warehouse system

One field matches, one partly matches and four diverge. Three of the four divergences sit on rare or conditional branches (holiday extension, high-value serial checks, damage claims), which is what the Nature finding would lead you to expect from generated data. Without the company's records, nobody outside could have marked them.

What should an introducer ask about synthetic-to-real validation?

Ask five questions of the operations or customer service lead and score each answer 0, 1 or 2. This is a rough sorting aid for you, not a SourceX acceptance test.

Question2 points1 point0 points
Are past versions of the policy kept, with dates?Every version, datedCurrent plus some oldCurrent only
Can a case be traced from request to final outcome?One case ID across systemsPossible with effortOutcome not recorded
Are exception cases tagged (damage, fraud, warranty)?Tagged reason codesFree-text notesNot distinguishable
Are experienced operators still with the company?Several with five-plus yearsOne or twoTeam turned over
Who could authorize a licensing discussion?Named executive with authorityNeeds parent or board inputUnknown

A score of 7 or more out of 10 suggests the records exist to validate simulations in this workflow. Lower scores point to what the owner should check first; the guide to questions to ask before licensing company data helps with that.

Who owns the records, and who can say yes?

Returns policies usually sit with customer service or legal; case history with e-commerce or customer service; inspection and grading with the warehouse; refunds with finance. Authority to discuss a license normally rests with the CEO, COO, CFO or owner. Customer personal data will need de-identification, which SourceX settles with the company before any delivery. Escalation-heavy workflows, like those in the support escalation process overview, often sit right next to returns.

The company must have 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. See who qualifies for detail.

What stays unknown until SourceX reviews it: SourceX's current data needs for these records, the rights review result, the de-identification required, the company's offer and any reward. More record families are listed under data types.

Go deeper on synthetic-to-real validation

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client scores 7 or more on the five questions, register as a partner, confirm the basics on who qualifies, and read the program terms before introducing the executive who can authorize the discussion.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does the company need to have built simulations itself?

No. The simulation usually comes from an AI developer. What the company contributes is the reality to check it against: its policies over time, its case history and the judgment of people who did the work.

Can the operator's review notes be licensed too?

Only if the company commissions or owns them and the agreement covers them. An introducer should not arrange reviews; SourceX scopes any such work with the company directly.

Is a returns archive with no damage claims still useful?

It may be, but rare branches are where simulations most often go wrong. Ask how the company handled exceptions and whether those cases are tagged, rather than assuming the archive covers them.

Sources

  1. refer.sourcex.si/terms
  2. nature.com/articles/s41586-024-07566-y
  3. openreview.net/forum

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment