# Evaluation holdouts: freight audit example | SourceX

Canonical: https://refer.sourcex.si/guides/evaluation-holdouts-workflow-example
Last updated: 2026-10-18

An evaluation holdout is a set of real, closed cases kept out of any training data and used only to test whether an AI system makes the right decision. In an ordinary freight audit workflow, that means sealing a batch of carrier invoices whose short-pay decisions have already been confirmed or reversed by the carrier, so an assessor can compare a model's call with the recorded outcome. The records each audit step leaves behind are what make the seal credible.

This worked example is part of [evaluation holdouts](/guides/concept-evaluation-holdouts), which sets out the five questions to ask an owner. Here the job is different: follow one fictional company's workflow from start to finish and show which of its records carry the signal an evaluation review looks for.

## The company and its archive (Illustrative)

Everything in this example is fictional. A 300-person freight audit and payment company checks carrier invoices for shippers before paying them. Each invoice is compared with the rate agreement in force on the ship date. When the billed amount is higher than the agreement allows, the auditor short-pays the invoice and records why. The carrier then accepts the short-pay or disputes it, and the case closes as paid, credited or written off.

The company keeps six years of cases in its audit platform. Its closed cases have never been published or shared with a vendor, which is the property public benchmarks lose over time. The BIG-bench documentation tells its users to treat anything accessible online without its canary marker as potentially part of a model's training data; this company's closed cases have never been published anywhere.

## Six steps from invoice to sealed test case

Each step involves a decision and leaves a record. The fourth column says whether that record carries signal for an evaluation, meaning it is needed either to pose the task or to score the answer.

| Step | What happens | Record it leaves | Signal for evaluation |
|---|---|---|---|
| 1. Invoice received | Carrier bills for a shipment | Invoice with billed amount, ship date, lane, accessorial charges | Poses the task |
| 2. Rate check | Auditor finds the agreement in force on the ship date | Rate agreement version ID and the expected charge | Poses the task; must be the version in force, not today's |
| 3. Short-pay decision | Auditor pays less than billed and codes the reason | Original billed amount, corrected amount, reason code, auditor ID | The decision a model would be asked to make |
| 4. Carrier response | Carrier accepts the short-pay or disputes it | Acceptance or dispute note, any revised invoice | Confirms or overturns the decision |
| 5. Case close | Final amount settled | Locked close date, final status: paid, credited or written off | The answer key |
| 6. Holdout seal | Closed cases in a date range are reserved for testing | Case list, seal date, access history | Proves the cases stayed unseen |

Steps 1 to 3 describe the task. Steps 4 and 5 supply the answer. Step 6 is not part of the audit at all; it happens when a license is scoped, and it relies entirely on records the earlier steps already left. The general family of these records is described on the [freight audit exceptions](/data-types/freight-audit-exceptions) page, and other worked workflows sit in the [logistics workflow example pack](/resources/logistics-workflow-example-pack).

## One case, annotated (Illustrative)

A single fictional invoice shows how the records line up.

| Field | Value | Why it matters |
|---|---|---|
| Ship date | March 11, 2026 | Selects the rate agreement version |
| Rate agreement version | Version 7, effective January 1, 2026 | Version 8 took effect in May; using it would change the answer |
| Billed amount | Line-haul plus a liftgate fee | The carrier's claim |
| Audit decision | Short-pay: liftgate fee not in version 7 | The decision under test |
| Carrier response | Accepted on April 2, 2026 | Confirms the short-pay was right |
| Close date | April 9, 2026, locked on payment | Places the case in the holdout window |
| Final status | Paid at the corrected amount | The answer a model's call is scored against |

A model given the invoice and version 7 of the agreement should reach the same short-pay. If it does, it scores; if it pays the liftgate fee or applies version 8, it does not. That is the whole test, and it works only because the archive kept version 7 after version 8 replaced it, and kept the original billed amount after the correction.

## Where the line is drawn: training pool against sealed holdout

The company's archive splits along close dates. In this example, cases closed through December 31, 2025 form the pool that could be licensed for training. Invoices dated January 1 to June 30, 2026 that have since closed form the holdout. The window stops at June 30, 2026 to leave time for late disputes to settle, so every sealed case has a final answer.

Why the line has to be clean is shown by published research. When researchers built GSM1k, a fresh held-out set designed to mirror the GSM8k math benchmark, some models' accuracy dropped by up to 8%, with evidence of systematic overfitting in several model families, while frontier models showed minimal signs of it. LiveBench addresses the same risk by adding questions from recent sources and scoring them against objective answers. The freight archive offers both properties: cases created after a known date, with recorded outcomes. Our explainers on [benchmark contamination](/guides/benchmark-contamination-llm) and [what model evaluation is](/proof/what-is-model-evaluation) cover the background.

Evaluation use is often licensed separately from training use; the [evaluation-only data license comparison](/compare/evaluation-only-data-license) sets out the usual differences. Why business archives matter for this at all is covered in [why enterprise AI needs enterprise data](/proof/why-enterprise-ai-needs-enterprise-data).

## What this example does not show

- **It does not prove demand.** A buyer may want a different lane mix, a different period or no freight cases at all. That is assessed in SourceX's review.
- **It does not settle rights.** Shipper contracts may limit reuse of invoice data, and carriers appear by name in every case. Both are part of a rights review.
- **It assumes the gaps are absent.** Real archives often have overwritten rate versions, editable close dates or past vendor exports. The [evidence gaps analysis](/questions/evaluation-holdouts-evidence-gaps) covers how those break a holdout, and the [review packet](/resources/evaluation-holdouts-review-packet) lists what a company prepares.

A company like this one would also need to meet the basics: 50 or more full-time employees at peak, documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 90% of the data is in English. Partners stay at metadata level throughout. Other record families are listed under [data types](/data-types).

## Next step

> **Refer a company. Earn up to $100,000.** Default reward: 25% of Actual Supplier Payout, capped at $100,000 cumulatively per referred company. Eligibility and payment conditions apply, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client's workflow leaves the same records as steps 2 to 5 here, with superseded versions and locked close dates kept, [register as a partner](/register), check [who qualifies](/who-qualifies) and read the [terms](/terms) before introducing the executive who can authorize an evaluation-only discussion.

## Sources

- https://proceedings.neurips.cc/paper_files/paper/2024/hash/53384f2090c6a5cac952c598fd67992f-Abstract-Datasets_and_Benchmarks_Track.html
- https://proceedings.iclr.cc/paper_files/paper/2025/hash/e4a46394ba5378b3f9a186a5b4c650d1-Abstract-Conference.html
- https://github.com/google/BIG-bench/blob/main/docs/doc.md
- https://refer.sourcex.si/terms