What should an introducer ask about evaluation holdouts?
An evaluation holdout is a set of real workflow cases deliberately kept out of any training data and reserved for testing. Its value rests on one claim: the model being tested has never seen these cases or their answers. The five questions below test whether a company's records could back that claim, not whether the idea is interesting. Ask them of an operations director or CFO, in this order.
- "Do your closed cases have a locked close date?" Without reliable dates, nobody can draw a clean cutoff line between cases that could be used for training and cases kept for testing.
- "Is the final outcome stored with each case, including any later reversal?" The answer key is only as good as the last recorded outcome. A case marked approved and later reversed in another system has two answers.
- "Has this archive ever left the company: a vendor pilot, an outsourcing provider, a previous data deal?" A yes does not end the discussion, but it narrows which years can serve as a holdout.
- "Can IT show who accessed these records, and when?" Access logs are what make "unseen" provable rather than asserted.
- "Who could authorize a conversation about licensing these records, including for evaluation only?" The answer should name the CEO, who can authorize evaluation-only use as well as training use.
The order matters. Questions 1 and 2 establish that a test set could exist. Questions 3 and 4 establish that it could stay clean. Question 5 establishes that someone can say yes.
The discovery card: real records or a theoretical opportunity
Each question has an answer that points to records the company actually holds, and an answer that leaves the opportunity theoretical. This card is the asset to keep in front of you during the conversation.
| Question | Answer pointing to real records | Answer that stays theoretical | Who usually knows |
|---|---|---|---|
| 1. Locked close dates | "Cases close in the system and the close date cannot be edited afterwards" | "We could work out roughly when each case finished" | Operations manager, system administrator |
| 2. Final outcome stored | "Outcome is a coded field, and reversals are logged against the same case" | "The outcome is in the notes somewhere" or "finance has the real answer" | Operations or finance lead |
| 3. Never left the company | "No vendor, outsourcer or pilot has had a copy" or "only 2021 to 2023 went to a vendor" | "Probably not, but nobody would know" | IT, procurement, legal |
| 4. Access logged | "The system keeps an access history we can report on" | "Only our team uses it" | IT or security |
| 5. Authorizer named | "The CEO approves any use of these records" | "We would need to find out" | CEO, CFO, owner |
A theoretical answer is not a dead end. It tells you what the owner would need to confirm before an introduction is worth making, and it is honest information to pass on.
Reading the answers (Illustrative)
Two fictional companies answer the same five questions. Neither is a real client.
| Question | 380-person freight audit firm | 210-person staffing agency |
|---|---|---|
| 1. Locked close dates | Yes, close date locks on payment | No, placements are edited after start |
| 2. Final outcome stored | Yes, approved, short-paid or disputed, with reversals logged | Partly, in recruiter notes |
| 3. Never left the company | Yes, except one 2022 vendor pilot | Unknown |
| 4. Access logged | Yes, an access report exists | No |
| 5. Authorizer named | Yes, the CEO | Not yet |
| Record-backed answers | 5 of 5, with one year excluded | 0 of 5, one partial |
Four or five clear, record-backed answers suggest the archive could support a holdout and that an introduction is worth making. The freight audit firm's archive, for instance, belongs to a record family covered on the freight audit exceptions page. One or two suggest waiting until the owner has confirmed more. Mixed answers are worth passing on to the owner along with the live overview of how AI labs and data users evaluate datasets.
Why "unseen" has to be provable
A holdout that leaked is worse than useless, because it makes a model look better than it is. Research shows how much this matters. When researchers built GSM1k, a fresh held-out set designed to mirror the widely used GSM8k math benchmark, some models' accuracy dropped by up to 8%, with evidence of systematic overfitting in several model families, while frontier models showed minimal signs of it. LiveBench takes a different route, limiting contamination with frequently updated questions from recent sources, scored automatically against objective answers. Both approaches depend on the same thing the discovery card checks: knowing when material was created and who has seen it. Our explainers on benchmark contamination and what model evaluation is cover the background, and rubric-based evaluation covers how answers are scored once a test set exists.
A holdout is often licensed differently from training data, because the cases are used to test models rather than shape them. The comparison of an evaluation-only data license with a training license sets out the usual differences. For an owner, that can make a first conversation easier: a narrower use, a smaller set and clear isolation rules. It does not make demand certain. The live page on AI evaluation data demand describes the market in general.
Who qualifies, and what stays unknown
The company must have 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 90% of the data is in English. Customer contracts may restrict reuse of case data, so a rights review is part of the process. The company qualification checklist covers the basics.
Partners stay at metadata level: they never ask for case files, exports or access reports, only whether these things exist and who owns them. What stays unknown until SourceX reviews it: SourceX's current data needs, the rights review result, what de-identification is needed, the company's offer and therefore any reward. Industry interest in holdouts does not mean every archive qualifies. More record families are listed under data types.
Go deeper on evaluation holdouts
- Evaluation holdouts in an ordinary business workflow: an Illustrative worked example of one archive split into a training pool and a sealed test set.
- Record gaps that make evaluation holdouts hard to assess: a guide to the editable dates, missing reversals and absent access logs that spoil a holdout.
- Evaluation holdouts review packet: a list of the metadata a company prepares before SourceX reviews a candidate test set.
Next step
Refer a company. Earn up to $100,000. Default reward: 25% of Actual Supplier Payout, capped at $100,000 cumulatively per referred company. Eligibility and payment conditions apply, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
If a client gives four or five record-backed answers on the discovery card, register as a partner, check who qualifies, and read the program terms before introducing the CEO, who can authorize an evaluation-only discussion.