# Evaluation holdouts evidence gap analysis | SourceX

Canonical: https://refer.sourcex.si/questions/evaluation-holdouts-evidence-gaps
Last updated: 2026-10-20

Four gaps make evaluation holdouts hard to assess: no proof the cases stayed unseen, such as a missing export history or access log; dates that can be edited; superseded rules, such as old rate agreements, that were overwritten; and cases with no final, recorded outcome. Each weakens a different property a holdout needs: unseen, dated, answerable and final. None ends a conversation. Each narrows which cases could be sealed, and the honest introduction says which.

This analysis is part of [evaluation holdouts](/guides/concept-evaluation-holdouts). The [worked freight audit example](/guides/evaluation-holdouts-workflow-example) shows a holdout where every record survived; this page looks at what usually does not survive, and how to say so.

## Four properties a holdout has to prove

A holdout is credible only if a reviewer can believe four things about every sealed case. Each property maps to evidence a company either has or lacks.

| Property | What the reviewer must believe | Evidence that proves it |
|---|---|---|
| Unseen | No model or vendor has had the case | Export history, vendor list, access log |
| Dated | The case was created and closed when the record says | System-generated, locked close dates |
| Answerable | The task can be posed exactly as it stood | The rule, price list or agreement version in force at the time |
| Final | The case has one recorded right answer | Closed status, plus the counterparty's acceptance or dispute |

The first property is where business archives have an advantage and also their biggest gap. Public benchmarks leak because they are published; the BIG-bench documentation tells users to treat anything online without its canary marker as potentially part of a model's training data. A private archive starts unseen, but it has no canary either, so its privacy has to be shown by records.

## What a review expects against what archives usually hold

The pattern below describes ordinary operating archives, not measured rates.

| Property | Typically held | Typically missing | What breaks | Metadata question |
|---|---|---|---|---|
| Unseen | Vendor contracts in a legal folder | A list of which record types and years each vendor, pilot or outsourcer received | Exposure cannot be ruled out for those years | "Has this archive ever been exported to a third party?" |
| Unseen | Named user accounts | An access log the company can report on | Privacy is asserted rather than shown | "Can IT show who accessed these records?" |
| Dated | Close dates in the current system | Locked dates; dates reset by a migration | Cases cannot be placed on the right side of a cutoff | "Is the close date system-generated and locked?" |
| Answerable | The current rate agreement or policy | Superseded versions, kept with their effective dates | The test asks about a rule the case was never decided under | "Are superseded agreements kept?" |
| Final | A closed flag | The counterparty's response, and later reversals | The case has two answers, or none | "Is the carrier's response stored with the case?" |

Documentation of this kind has a recognized form. Gebru and colleagues' Datasheets for Datasets proposes recording a dataset's composition, collection process, distribution and maintenance, and the distribution questions in particular, who has received the data and under what terms, are exactly what the unseen property needs answered.

## Illustrative gap profile: a 190-person parcel and LTL audit firm

The firm is fictional and its figures are invented. Each score is out of ten, estimated in conversation with the operations director and IT lead for the last four years of closed cases.

| Property | Score | What is missing | Effect on a holdout |
|---|---|---|---|
| Unseen | 4 of 10 | Cases from 2022 to mid-2023 went to an outsourced audit vendor; no access log before 2025 | Only cases created after the vendor contract ended are candidates |
| Dated | 8 of 10 | Close dates for one shipper were reset during a platform move | That shipper's cases need dates rebuilt from payment records, or exclusion |
| Answerable | 5 of 10 | Rate agreements before 2024 were overwritten at renewal | Older cases cannot be posed as they stood |
| Final | 6 of 10 | Disputes handled by email; carrier responses not filed with cases | Many cases close with an amount but no confirmation |

The overlap is what matters. Cases created after the vendor contract ended, under a kept rate agreement, with a carrier response on file, form the realistic candidate pool, probably the last twelve to eighteen months for this firm. That sentence, not the four scores, is what goes into an introduction.

## How each gap changes what a test can say

- **Unseen gaps** turn a score into a question. GSM1k, a fresh set built to mirror the GSM8k math benchmark, showed accuracy drops of up to 8% for some models, with evidence of overfitting in several model families. If a holdout might have leaked, a high score could mean memory rather than skill, and nobody can tell which.
- **Dating gaps** blur the cutoff. A case dated after a model's training data was collected, but actually created before, can be sitting in the training data.
- **Answerability gaps** test the wrong rule. A short-pay judged against today's agreement instead of the one in force marks a correct decision wrong.
- **Finality gaps** remove the answer key, or supply two. An open dispute has no answer; a decision reversed in another system has two.

Why buyers care about these properties is covered in [how AI labs and data users evaluate datasets](/proof/how-ai-buyers-evaluate-datasets). Why a smaller, cleaner set can be worth more than a large leaky one ties back to [why AI needs so much data](/questions/why-does-ai-need-so-much-data), which explains the training side of the same trade-off.

## Saying it plainly in an introduction

One line per gap, with the window that survives. For the fictional firm:

- "Cases from 2022 to mid-2023 went to an outsourced audit vendor; the company reports no export since then, and its access log starts in 2025."
- "Superseded rate agreements are kept from 2024 onward."
- "Carrier responses are filed with cases since the dispute module went live; earlier ones sit in email."

Phrasings to avoid: "the archive has never been shared" (unless someone has checked), "every case is final" (open disputes say otherwise) and "it's perfect test data" (the review decides that). The usual gates still apply first: 50 or more full-time employees at peak, documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 90% of the data is in English.

Freight audit records as a family are described on the [freight audit exceptions](/data-types/freight-audit-exceptions) page. Holdouts are often licensed for evaluation only; the [evaluation-only data license comparison](/compare/evaluation-only-data-license) explains the difference. Once the owner agrees to proceed, the [review packet](/resources/evaluation-holdouts-review-packet) lists what the company prepares. More record families are under [data types](/data-types).

## Next step

> **Refer a company. Earn up to $100,000.** Default reward: 25% of Actual Supplier Payout, capped at $100,000 cumulatively per referred company. Eligibility and payment conditions apply, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the gaps still leave a window of unseen, dated, answerable and final cases, [register as a partner](/register), check [who qualifies](/who-qualifies) and read the [terms](/terms), then introduce the owner with that window written down.

## Sources

- https://proceedings.neurips.cc/paper_files/paper/2024/hash/53384f2090c6a5cac952c598fd67992f-Abstract-Datasets_and_Benchmarks_Track.html
- https://github.com/google/BIG-bench/blob/main/docs/doc.md
- https://arxiv.org/abs/1803.09010
- https://refer.sourcex.si/terms