Benchmark contamination controls evidence gaps: what a review expects, what companies hold, and how to say it

Short answer

Four gaps make contamination controls hard to assess: records with no reliable creation date, an unknown list of outside recipients, overwritten versions of the policy or document a task depended on, and tasks that never reached a final outcome. None of them rules a company out. Each weakens the claim that a task is unseen by a model, so the honest introduction names the gap, says who could answer it and leaves the judgment to the review.

Visual briefFull-size diagram
Five horizontal bars compare the evidence a review expects with what a fictional insurance broker holds: creation dates 70%, final outcomes 85%, recipient list 50%, version history 40% and rights notes 60%, with each missing segment outlined in orange.
Illustrative gap analysis for a fictional 180-person commercial insurance broker. Percentages are invented for this page.

This page is part of benchmark contamination in LLMs. Its job is a gap analysis: set what a review would expect next to what companies usually hold, then show how to describe the difference in plain words.

What an exposure review expects to see

A buyer considering business records as held-out evaluation material wants to know two things about each task: could a model already have seen it, and can an answer be scored? Six kinds of evidence speak to that. The list echoes the questions a dataset datasheet asks about composition, collection and distribution, applied to one archive.

EvidenceWhy a review wants it
Creation date per recordPlaces each task before or after a model's data cutoff
Source system per recordShows where the full answer lives and whether it was migrated
Outside-recipient listVendors, administrators or partners who received copies are routes to exposure
Public-page inventoryHelp articles, case studies or posted disputes can be near-twins of a task
Version historyThe policy, price list or procedure in force when the task was decided
Final outcomeThe recorded decision that makes a task scorable

The working assumption behind the list comes from benchmark practice: the BIG-bench documentation says anything accessible online without its canary marker should be treated as potentially part of a model's training data. Business records carry no canary, so the documentation has to do that job.

Expected versus typically held: the gap table

Companies rarely hold all six cleanly. This is the pattern seen in ordinary operating archives, not a statistic.

EvidenceWhat companies typically haveWhat is usually missingWhat breaks without it
Creation dateDates in the current systemOriginal dates for records imported from a retired systemThe task cannot be placed against a cutoff
Source systemThe current system's nameWhich records came from where after a merger or migrationNobody can say where the full answer lives
Outside-recipient listVendor contracts in a legal folderA mapping of which record types each vendor sawExposure cannot be ruled out for those records
Public-page inventoryA help center and a marketing siteAnyone having compared them to the archiveNear-twin tasks stay undetected
Version historyThe current policy or procedureEarlier versions, overwritten on each revisionThe right answer at decision time is uncertain
Final outcomeClosed records with a decisionAbandoned or long-pending recordsNo answer to score

A gap analysis worked through (Illustrative)

Illustrative and fictional. A 180-person commercial insurance broker keeps a log of coverage questions from clients, each answered by an account manager and closed with a coverage position. It has about 3,400 logged questions from 2019 to 2025. An introducer asks the operations director where the key kinds of evidence stand, adds one question on rights, and records the answers as rough shares of the log.

EvidenceExpectedHeld (rough share of the log)The gap in one line
Creation datesAll records70%Questions logged before a 2021 CRM move carry only the import date
Final outcomesAll records85%Some questions closed as "client to confirm" and were never resolved
Outside-recipient listAll records mapped50%Two wholesale partners received some questions; nobody knows which
Version historyAll records40%Coverage guides are edited in place; older wordings are gone
Rights notesAll records60%Some carrier agreements restrict sharing of submission material

Read across, the broker has a likely usable core: dated, resolved questions from 2021 onward, once someone maps which ones the two partners received. It also has a long tail that a buyer may treat as training-only material. That is a perfectly good introduction, provided nobody calls the whole log uncontaminated.

How to describe a gap honestly in an introduction

Gaps get worse when they are rounded up. The right column below is what a review can work with.

Instead ofSay
"All their records are private.""Records sit in their CRM; two partners received some, and the mapping is unknown."
"Everything is dated.""Records since 2021 carry creation dates; older ones carry an import date."
"Every question has an answer.""Most close with a coverage position; some were left pending."
"They have full history.""Current guides only; earlier versions were overwritten."
"Rights are fine.""The broker owns the log; some carrier agreements need review."

Buyers weigh these gaps differently, and some run their own contamination checks. Researchers have shown, for instance, that a statistical test can detect a benchmark in a model's pretraining data without access to that data. That is the buyer's work, not the introducer's, and it is one more reason never to promise that an archive is clean.

Gaps an introducer should not try to close

Some gaps tempt a well-meaning introducer to dig. Don't. Searching the web for a client's records, asking for samples to check dates, or reading vendor contracts all move beyond metadata and into material the company controls. Note the gap, note who could answer it, and stop.

Before you introduce, confirm the basics: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and a named contact who can authorize a discussion. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. Records that went into an industry pool raise their own questions; see data shared with an industry benchmark consortium. Records inherited in a deal are covered in who owns startup data after an acquisition.

To decide whether the company is worth the introduction at all, use the qualification scorecard. For what the company assembles later, see the review packet contents list; for how buyers use benchmarks, AI agent benchmarks.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If you can describe a company's gaps in the plain terms above and name who owns the records, register as a partner, check who qualifies and read the program terms before you make the introduction.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Is a company with evidence gaps still worth introducing?

Often, yes. Gaps limit which records can be offered as held-out evaluation material, but records with known outcomes can still suit training use, where exposure matters less. The honest move is to name the gap rather than leave it for the review to discover.

Should the introducer try to close a gap, for example by searching the web for a company's records?

No. Any search of record content happens inside the company, under its own control, after a licensing discussion starts. The introducer records which gaps exist and who could answer them.

Which gap matters most?

It depends on the buyer's use. For held-out evaluation, missing creation dates and unknown outside recipients do the most damage. For training, missing final outcomes matter more, because a task with no recorded answer cannot be scored.

Sources

  1. arxiv.org/abs/1803.09010
  2. github.com/google/BIG-bench/blob/main/docs/doc.md
  3. arxiv.org/abs/2310.17623
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment