# Contamination controls: a workflow example | SourceX

Canonical: https://refer.sourcex.si/guides/benchmark-contamination-controls-workflow-example
Last updated: 2026-11-04

In an ordinary business workflow, benchmark contamination controls are a short run of record-keeping steps: choose a process with checkable outcomes, list candidate cases by ID and date, trace where each case has been, check outside recipients, search for public copies, confirm the final outcome, and band every case with a written reason. Each step leaves a record. Four of those records carry the signal an AI-data review looks for. The Illustrative example below follows one accounts payable archive through all seven steps.

This example is part of the [benchmark contamination guide](/guides/benchmark-contamination-llm). The per-task [provenance and exposure worksheet](/guides/private-archive-benchmark-contamination) is the tool used in steps three to five; this page shows the whole run.

## The company and the workflow (Illustrative)

Fictional and invented for this page. A 310-person regional food distributor keeps three years of invoice price-variance exceptions: invoices where the supplier's price did not match the purchase order, so the three-way match failed. Each exception was investigated by an AP specialist, resolved by a buyer, and closed with a credit memo, a corrected invoice or an approved variance.

That makes the workflow a good candidate. Each case has an input (the invoice and the order as first received), a decision (accept, dispute or partly accept) and a recorded outcome that can be checked against the ledger. Benchmark builders look for the same property: LiveBench, for example, includes only questions with an objectively correct answer so they can be scored automatically, and it adds new questions from recent sources to limit contamination. A closed invoice exception has both qualities: a checkable answer and a known date.

## Step by step: the decision and the record at each stage

| Step | Decision | Who decides | Record it leaves | Carries the signal? |
|---|---|---|---|---|
| 1. Choose the workflow | Use price-variance exceptions from the last three years | Controller | Scope note naming the process and years | No, context only |
| 2. List candidates | Include every closed exception with an ID and a creation date | AP manager | Candidate list: 900 exception IDs with dates | Yes: dates place each task against a model's data cutoff |
| 3. Trace systems | Record every system that held part of each case | AP manager with the ERP administrator | System trace: ERP, AP mailbox, supplier portal | Partly: shows where copies exist |
| 4. Check outside recipients | Flag cases processed by the outsourced invoice-scanning vendor | Controller | Vendor register entry and the contract's data-use clause | Yes: names every outside route |
| 5. Search for public copies | Flag cases whose correct price was published on a supplier's public website | AP specialist, under company control | Search log per supplier with date and result | Yes: the exposure evidence |
| 6. Confirm the outcome | Keep only cases closed with a credit memo, corrected invoice or approved variance | AP manager | Outcome match by exception ID against the ledger | Yes: the checkable right answer |
| 7. Band and log | Sort every case into a band with a reason | Controller | Band tally and exclusions log | No, but it makes the set auditable |

The four records marked yes are the ones a reviewer will ask for first: the dated candidate list, the outside-recipient record, the search log and the outcome match. The others make the run understandable and auditable but do not by themselves say anything about exposure.

## What the run produced

| Band | Cases | Reason |
|---|---|---|
| Held-out candidate | 745 | Dated, no outside recipient, no public copy found, outcome confirmed |
| Public price | 80 | The correct price appeared on a supplier's public price list, so the answer is findable online |
| Vendor processed | 45 | Scanned by the outsourced vendor; its contract allows use of documents to improve its service |
| No outcome | 30 | Disputes still open or abandoned, so there is no right answer to score |
| Total | 900 | Every candidate accounted for |

Only the first band would be offered as held-out evaluation material. The other 155 are set aside, not deleted: the public-price and vendor cases may still suit training, where prior exposure matters less.

## Why the public-price band matters

Step five is the one most companies would skip, because the invoices themselves were never published. But the answer to the task, the correct contract price, was. A model that has read a supplier's public price list could solve those 80 cases without reasoning through the exception. That is contamination in the practical sense: the test no longer measures what it claims to.

Benchmark practice treats this risk seriously. The BIG-bench documentation asks authors to embed a canary string so their data can be filtered from training corpora, and states that anything accessible online without the canary should be treated as potentially part of a model's training data. When researchers built GSM1k as a fresh counterpart to the public GSM8k set, several model families scored lower on the fresh problems, which the authors read as overfitting to what had been public. Step five is the business version of building fresh problems: removing the cases where the answer was already out there.

## Where this workflow breaks in practice

- **Dates lost in a migration.** If the ERP was replaced, imported exceptions may carry the load date instead of the creation date. Step two then cannot place those tasks.
- **Unclear vendor terms.** If nobody can find the scanning vendor's contract, step four cannot rule the route in or out, and the cases stay flagged.
- **Outcomes outside the ledger.** Some variances are settled by phone and recorded only in email. Step six then needs the mailbox, which raises its own rights questions.
- **Searches nobody logged.** A search done but not recorded is no evidence. The log with date and result is the record.

The [evidence gaps page](/questions/benchmark-contamination-controls-evidence-gaps) covers these failure points in more depth. The [logistics workflow example pack](/resources/logistics-workflow-example-pack) shows the same record-chain approach on shipping claims.

## Qualification: what a partner asks and what stays open

The partner does not run any of these steps. Ask at metadata level whether the company could: does a workflow produce cases with IDs, creation dates and recorded outcomes; which outside vendors handle those documents; has any related material been published; and who can authorize a licensing discussion. The AP manager or controller can usually answer the first three; the CEO, usually the owner in an owner-run company, decides the last.

The company needs 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

What stays unknown until SourceX reviews it: supplier and vendor contract terms, how buyers weigh each band, demand for this record type and any offer. Why company records matter to enterprise AI at all is covered in [why enterprise AI needs enterprise data](/proof/why-enterprise-ai-needs-enterprise-data); what professional-task evaluations look like is covered in [what GDPval is](/questions/what-is-gdpval); more record types are under [data types](/data-types). SourceX funds any referral reward separately and never deducts it from the company's payout.

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client's finance or operations team keeps dated cases with recorded outcomes and knows which vendors touched them, [register as a partner](/register), check the company against [who qualifies](/who-qualifies) and read the [program terms](/terms) before you make the introduction.

## Sources

- https://github.com/google/BIG-bench/blob/main/docs/doc.md
- https://arxiv.org/abs/2405.00332
- https://arxiv.org/abs/2406.19314
- https://refer.sourcex.si/terms
