# Evaluation holdouts review packet contents | SourceX

Canonical: https://refer.sourcex.si/resources/evaluation-holdouts-review-packet
Last updated: 2026-10-18

An evaluation holdouts review packet is a set of nine metadata-only documents that lets SourceX judge whether a company's archive could supply a sealed test set, without anyone outside the company seeing a case or an answer. In order: a cover sheet, the task statement, a cutoff memo, the outcome mix, a description of the answer key, a custody statement, an access-history summary, the scoring rule and rights notes. Each is written by the person who already holds that fact.

This contents list is part of the [evaluation holdouts guide for introducers](/guides/concept-evaluation-holdouts), which covers the discovery questions to ask first. This page picks up after those answers come back positive: it lists what the company assembles, in what order, and who holds the pen. It is a working document for the owner's team, not a script for the introducer.

## Nine documents, in assembly order

The order follows the logic a reviewer uses. First, what is being tested. Then, which cases are in scope and what their answers look like. Then, the proof that nobody has seen those answers. Last, how a test is scored and who may license it.

| # | Document | What it shows | Who prepares it |
|---|---|---|---|
| 1 | Cover sheet | Headcount at peak, case system, years of history, named owner and contact | Owner or COO, with the introducer helping only on these metadata fields |
| 2 | Task statement | The single decision a system under test would be asked to make | Head of the audit or operations team |
| 3 | Cutoff memo | The date that splits a possible training pool from the reserved window, and why | Operations lead, with IT confirming close dates are locked |
| 4 | Outcome mix | Counts of each final outcome inside the reserved window | Reporting analyst |
| 5 | Answer key description | Which fields make up the answer, where each is stored, how reversals are captured | Operations lead and system administrator |
| 6 | Custody statement | Every place the archive has been copied: vendors, pilots, outsourcers, prior deals | IT and procurement, reviewed by counsel |
| 7 | Access-history summary | Whether the system logs who opened case records, and for how long | IT or security |
| 8 | Scoring rule | What counts as a correct answer, written before any case is drawn | Audit lead, reviewed by SourceX |
| 9 | Rights and sensitivity notes | Customer contract limits, personal data present, carrier or vendor terms | Counsel or CFO |

Every entry is a description, a count, a date or a name. None is a record. If a document cannot be completed, the honest move is to write "not known yet" and say who would know.

## The test card behind the packet (Illustrative)

Documents 2 to 5 together describe one candidate test set. Here they are for a fictional 280-person freight audit and payment company that checks carrier invoices against contracted rates and shipment records before paying them for shippers. The live overview of [freight audit exceptions](/data-types/freight-audit-exceptions) describes the record family in general; every number below is invented.

| Packet item | Entry on the test card |
|---|---|
| Task statement | Audit a carrier invoice against the rate agreement and shipment record; approve as billed, short-pay with a corrected amount, or dispute with a reason code |
| Reserved window | Invoices closed January 1 to June 30, 2026 |
| Earlier pool | Invoices closed 2018 to 2025, which could be discussed separately for training and never mixed with the window |
| Candidate holdout size | 300 invoices, drawn to match the real outcome mix |
| Outcome mix | 210 approved as billed, 66 short-paid, 24 disputed |
| Answer key fields | Auditor's final decision, amount paid, reason code, and whether the carrier accepted or contested |
| Isolation rule | Nothing closed on or after the cutoff enters any training set; key stored apart from case inputs |
| Assessor access (proposed) | Left to the agreement; the company's preference is that an independent assessor runs the test and reports only scores |

The test card is the heart of the packet because it turns "we have an archive" into a specific, checkable claim: this window, this many cases, this answer, kept this way. A company that can fill every row in plain language is ready for review. A company that cannot fill the answer key row has found its first gap, which the companion page on [evaluation holdout evidence gaps](/questions/evaluation-holdouts-evidence-gaps) helps describe.

## Sealing the answer key: documents 5 to 7

A holdout is only worth anything if the system being tested has never seen the cases or their answers. Research shows why reviewers insist on proof. When researchers built GSM1k, a fresh held-out set mirroring the widely used GSM8k math benchmark, some models scored up to 8% lower on it, with signs of systematic overfitting in several model families, while frontier models showed minimal signs of it. LiveBench takes another route to the same goal, refreshing its questions from recent sources and scoring them automatically against objective answers. Both depend on knowing when material was created and who has had it. Documents 5 to 7 carry that knowledge for a private archive.

**Document 5, answer key description.** Lists the fields, not their values. For the freight case: decision code, paid amount, reason code, carrier response, and the date of the last change. It must say how a later reversal is stored. If a short-pay was contested and the carrier won, the key has to hold the settled amount, or the test will mark a correct system wrong.

**Document 6, custody statement.** A plain list of every copy that ever left the company, with dates. "A 2022 pilot with an invoice-scanning vendor received 2021 to 2022 invoices" is a useful sentence. "Probably nothing" is not. A past share usually narrows the window rather than ending the discussion.

**Document 7, access-history summary.** Says whether the case system records who opened records and how long those logs are kept. It does not attach the logs. The point is to show that "unseen" could be demonstrated later, under an agreement, not merely asserted.

## Scoring rule and the exact outcome check

Document 8 states what a correct answer is, written before anyone draws a case. A rule drafted after looking at outcomes tends to fit whatever the archive happens to contain.

For the freight test card, the rule reads:

- The decision (approve, short-pay, dispute) matches the auditor's final decision.
- For short-pays, the corrected amount is within one dollar of the amount finally paid.
- For disputes, the reason code matches the key.
- Where the carrier contested and the outcome changed, the settled result is the answer, not the first decision.

The last line is why carrier responses belong in document 5. Scoring is easy here because answers are close to binary; where answers are judgment calls, the guide to [rubric-based evaluation](/guides/rubric-based-evaluation-ai) shows how scoring criteria are written instead. The live explainer on [what model evaluation is](/proof/what-is-model-evaluation) covers why developers pay attention to tests like this.

A holdout is often licensed for evaluation only, a narrower use than training. The comparison of an [evaluation-only data license](/compare/evaluation-only-data-license) with a training license sets out the usual differences, which document 9 should anticipate.

## Who signs off, and what the packet leaves open

Ownership usually splits three ways: the audit or operations team owns cases and decisions, IT owns the system and its logs, and finance or counsel owns customer and carrier contracts. Authority to discuss a license, for evaluation-only or training use, sits with the CEO (in an owner-run company, the owner); the CFO or COO can sponsor the introduction. Document 1 names the CEO as the authorized contact.

The company itself must qualify: 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 90% of the data is in English.

Partners never assemble the packet, and never receive cases, exports, logs or screenshots. Their part ends at the metadata on the cover sheet: what exists, how much, how long, which system, who owns it. What the packet cannot settle: whether the custody statement holds up, how many cases survive de-identification, whether customer contracts permit the use, whether a buyer wants this task, and therefore any offer or reward. Those are decided in SourceX's review. For a sense of how one archive is split in practice, see the [evaluation holdouts workflow example](/guides/evaluation-holdouts-workflow-example).

## Next step

> **Refer a company. Earn up to $100,000.** Default reward: 25% of Actual Supplier Payout, capped at $100,000 cumulatively per referred company. Eligibility and payment conditions apply, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the owner's team could complete the test card and custody statement above from what they already know, [register as a partner](/register), confirm the company against [who qualifies](/who-qualifies), and read the [terms](/terms) before introducing the CEO, who can approve evaluation-only use.

## Sources

- https://proceedings.neurips.cc/paper_files/paper/2024/hash/53384f2090c6a5cac952c598fd67992f-Abstract-Datasets_and_Benchmarks_Track.html
- https://proceedings.iclr.cc/paper_files/paper/2025/hash/e4a46394ba5378b3f9a186a5b4c650d1-Abstract-Conference.html
- https://refer.sourcex.si/terms