Benchmark contamination controls review packet: the seven metadata-only documents, in order

Short answer

Seven documents, assembled in this order: a cover sheet, a task specification, a provenance worksheet, an exposure register, an exact outcome check, an exclusions log and rights notes. Together they describe a candidate set of evaluation tasks without sending a single record. The introducer can help with the cover sheet from conversation; the company prepares the other six under its own control once a discussion with SourceX starts.

Visual briefFull-size diagram
Seven labelled documents drop one by one into an open review folder: cover sheet, task specification, provenance worksheet, exposure register, outcome check, exclusions log and rights notes, with the first marked as the only one an introducer helps assemble.
The packet in assembly order. Only the cover sheet is built from answers an introducer can collect.

This contents list is part of benchmark contamination in LLMs. The main guide explains why a private archive is not automatically clean; this page is the practical inventory of what a company puts in front of a review to show how clean its candidate tasks are.

The packet at a glance

OrderDocumentWhat it showsWho prepares itWhen
1Cover sheetRecord type, years, rough volume, systems, ownerCompany, from answers an introducer can collectBefore or at the introduction
2Task specificationWhat a model would receive and what counts as the right outputOperations leadAfter a discussion starts
3Provenance worksheetOne line per candidate task: dates, systems, recipients, public-twin resultRecords or operations teamAfter a discussion starts
4Exposure registerEvery route by which records left the companyLegal or operationsAfter a discussion starts
5Outcome checkThe exact rule a task must pass to stay in the setOperations leadAfter a discussion starts
6Exclusions logTasks removed, with reasonsRecords teamDuring preparation
7Rights notesOwnership, vendor and client restrictionsLegalBefore the rights review

The structure borrows from dataset documentation practice. The Datasheets for Datasets proposal asks creators to record a dataset's composition, collection process, recommended uses and distribution so that others can judge it; this packet asks the same of a candidate task set before anyone sees it.

Document by document

1. Cover sheet

One page, metadata only: which workflow produced the tasks (for example, escalated support cases closed with a resolution), the years covered, a rough count, the system of record and the named owner. This is the only document an introducer can help with, because every field is an answer, not a record.

2. Task specification

The input a model would receive, such as the case as first submitted, and the expected output, such as the resolution code and the fix applied. It also names the reference material in force on the case date: the product version, the policy or the runbook revision. Without that version, the right answer may have changed since.

3. Provenance worksheet

One row per candidate task, with five columns: creation date, systems that held it, outside recipients, public-twin search result and an exposure band (no twin found, near-topic page, shared with a named recipient, quoted publicly). The bands follow the benchmark rule of thumb in the BIG-bench documentation: assume anything online without a canary marker may have been seen.

4. Exposure register

A list of routes rather than tasks: outsourced support vendors, resellers, industry data pools, acquired businesses, public help pages and community forums. Each line says which record types the route touched and since when. Pools deserve their own line; see data shared with an industry benchmark consortium.

5. Outcome check, stated exactly

One sentence a reviewer can apply mechanically. For example: a task passes if its recorded resolution code matches the case system by case ID, its creation date is known, and a phrase search of the company's public pages and named recipients' copies finds no matching case text, product serial or customer wording. Objective, checkable answers are what make tasks usable for evaluation; LiveBench, for instance, scores answers automatically against objective ground-truth values.

6. Exclusions log

Every task removed and why: no creation date, outcome never recorded, near-twin found, shared with a vendor. The log lets a reviewer audit what remains instead of trusting it.

7. Rights notes

Who owns the records, which vendor contracts allow reuse and which client agreements restrict sharing, with the CEO named as the signer. These feed the rights review; they are not a legal opinion.

An Illustrative cover sheet

Illustrative and fictional. A 300-person B2B software company considers its escalated support cases as evaluation material.

FieldEntry
WorkflowEscalated support cases, each closed with a resolution code and a fix
Years2018 to 2025; creation dates kept throughout
Rough volumeAbout 9,000 escalations
System of recordHelp desk, with fixes linked to engineering tickets
Known exposure routesAn outsourced tier-1 vendor since 2022; a public knowledge base
Version referenceProduct release number recorded on each case
OwnerVP of Customer Support; the CEO signs

Nothing on that sheet is a record. It tells SourceX enough to decide whether the next six documents are worth the company's time.

What the packet does not settle

The packet shows how carefully the company has described its tasks. It does not prove that no model has seen them, it does not say whether a buyer wants them, and it does not set a price. How evaluation-only use is licensed, and how it differs from training use, is covered in evaluation-only data licenses. For what professional-task evaluations look like from the buyer side, see what GDPval is.

Before assembly even starts, the company has to qualify: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The qualification scorecard helps decide that, and the evidence gaps analysis shows how to describe a document the company cannot complete. Other record types are listed under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a company you know could fill in the cover sheet today and name who would prepare the rest, register as a partner, check who qualifies and read the program terms before you introduce its owner.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does the introducer see the finished packet?

No. The introducer may help shape the cover sheet from answers given in conversation. Everything after it is prepared by the company under its own control, once a discussion with SourceX has started, and goes to the review rather than to the partner.

Does the packet contain the records themselves?

No. It describes them: counts, dates, systems, exposure routes, the exact pass rule and the exclusions. Any record content is shared later, only if the company decides to proceed under a signed agreement.

What if the company cannot fill in one of the documents?

Send the packet with that document marked incomplete and a one-line reason. A missing version history or recipient map limits what the records can be used for; hiding it only delays the same conclusion.

Sources

  1. arxiv.org/abs/1803.09010
  2. github.com/google/BIG-bench/blob/main/docs/doc.md
  3. arxiv.org/abs/2406.19314
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment