Private archive benchmark contamination: a provenance and exposure worksheet that never assumes private means unseen

Short answer

No. A private archive puts a record in scope for clean testing, but only a documented exposure check keeps it there. Records leave companies in ordinary ways: a help article that describes the same case, a vendor that received copies, a customer who posted the dispute, an acquired business that shared data before the deal. The worksheet below records, task by task, where each record has been and what a search found, and it starts every task as unchecked rather than assuming that private means unseen.

Visual briefFull-size diagram
A worksheet grid of five candidate tasks against six exposure checks; cells fill in one by one, two tasks pick up an exposure flag, and the verdict column marks three as held-out candidates and two as training only.
Illustrative worksheet for a fictional HVAC service contractor. Every task starts unchecked, however private the archive.

This worksheet is part of the benchmark contamination guide. The workflow example shows the same controls running inside one business process, and the review packet shows what a company assembles later.

Why private is a starting point, not a verdict

Benchmark builders work from a cautious default. The BIG-bench documentation asks task authors to embed a canary string so their data can be filtered out of training corpora, and it states that anything accessible online and not marked with the canary should be treated as potentially part of a model's training data. The burden is on showing a record is absent, not on showing it is present.

The cost of getting this wrong is measurable. When researchers built GSM1k, a fresh set of grade-school math problems matched to the public GSM8k benchmark, several model families scored lower on the new problems, which the authors read as overfitting to the public set. Fresh material revealed what familiar material had hidden.

Detection methods help buyers, but they point one way. Oren and colleagues showed that a statistical test can flag contamination of a known dataset without access to a model's training data, by checking whether the model prefers the dataset's original ordering to shuffled versions. A test like that needs the dataset to exist publicly in some form. For a never-published business record, there is nothing to test against, so the owner's documentation of where the record has been is the main evidence available.

The worksheet: one line per candidate task

Each candidate task gets one line. A task is a single case with a checkable outcome: a service call with a confirmed diagnosis, a claim with a final decision, an order exception with a recorded fix.

ColumnWhat to enterWhy it is there
Task IDThe record's own ID in the system of recordLets every later check be audited
CreatedThe date the case openedPlaces the task against a model's data cutoff
SystemsEvery system that held any part of the caseShows where copies could exist
Outside recipientsVendors, outsourcers or partners that received itEach is a route to check
Six exposure checksClear, flagged or unknown for each check belowThe core of the worksheet
Outcome recordedYes or no, and whereNo outcome means no checkable right answer
BandHeld-out candidate, training only, or excludedThe decision the line supports

Every check starts as unknown. A task moves to clear only when someone has looked and recorded what they found.

Six exposure checks, and what each can and cannot show

CheckHow a private record escapesWhat a check can showWhat it cannot show
Public twinThe company's own help article, manual or case study walks through the same scenarioA phrase search of the company's public pages finds or misses a close matchWhether an old, since-deleted page was captured by a crawler
Customer postThe other party describes the case on a forum or review siteA search on distinctive phrases or part numbers turns it upPosts in closed communities or under different wording
Vendor copyAn outsourced provider or software vendor received the recordsThe vendor list and each contract's data-use termsWhat the vendor actually did with the copies
Shared poolRecords went into an industry benchmarking consortiumMembership and contribution records; see data shared with an industry benchmark consortiumHow the consortium's members used the pooled data
Prior ownerRecords came with an acquired business that had shared themDeal files and the seller's disclosures; see who owns startup data after an acquisitionSharing the seller never disclosed
Date knownThe case cannot be dated, so its timing against a cutoff is unknownA creation date in the system of recordWhether the date survived a past migration intact

The right-hand column is the honest part. Each check narrows the uncertainty without closing it, which is why the worksheet records findings rather than conclusions.

A worked line (Illustrative)

Fictional and invented for this page. A 230-person commercial HVAC service contractor considers its service-call diagnoses as possible evaluation material. One candidate: "Rooftop unit trips on high head pressure on hot afternoons only; find the cause."

ColumnEntry
Task IDService call from the field service system
CreatedJuly 2025
SystemsField service tool, technician photos, parts order in the ERP
Outside recipientsEquipment maker's warranty portal received the failure code
Public twinClear: the company blog covers seasonal maintenance, not this fault
Customer postClear: no match on the unit model and symptom wording
Vendor copyFlagged: the maker's warranty portal received the code and a short description
Shared poolClear: the company joins no benchmarking pool
Prior ownerClear: the branch was not acquired
Date knownClear: creation date kept in the field service tool
Outcome recordedYes: condenser fan motor replaced, callback-free for 90 days
BandTraining only, pending the rights review of the warranty portal's terms

One flag moved this task out of the held-out band. It may still be valuable for training, and the flag may fall away if the portal's terms bar reuse. The worksheet keeps the record of why.

Qualification: who answers, who decides, what stays open

The partner's role is to find out whether the company could fill in a worksheet like this, not to fill it in. Ask at metadata level: which workflow produces cases with a recorded outcome, whether case material has ever been published or shared, which outside parties received copies, whether records can be dated, and who can authorize a licensing discussion. The operations lead usually knows the first four; the CEO, or the owner-CEO in an owner-run company, decides the last.

The company needs 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

What stays unknown until SourceX reviews it: vendor contract terms, how each buyer weighs a flag, whether a buyer runs its own overlap checks, demand for the record type and any offer. Nobody at the introduction stage should promise that an archive is uncontaminated. For how evaluation tasks are used on the buyer side, see AI agent benchmarks and what GDPval is. SourceX funds any referral reward separately; it never comes out of the company's payout.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client's operations lead could answer the worksheet's columns for a workflow with recorded outcomes, register as a partner, check the company against who qualifies and read the program terms before you introduce its decision-maker.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Who fills in the worksheet, the partner or the company?

The company, under its own control and only once a licensing discussion has started. The partner only asks whether the company could answer the columns, never sees the entries, and never runs searches on record content.

Does one exposure flag disqualify a task?

It usually removes the task from the held-out set, but it may still suit training, where prior exposure matters less. The worksheet records the flag; the buyer and SourceX decide what it means.

Can the company promise buyers its records are uncontaminated?

No. It can show what it checked and what it found. A clean worksheet lowers uncertainty; it does not remove it.

Sources

  1. github.com/google/BIG-bench/blob/main/docs/doc.md
  2. arxiv.org/abs/2405.00332
  3. arxiv.org/abs/2310.17623
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment