# Contamination controls data fit scorecard | SourceX

Canonical: https://refer.sourcex.si/questions/does-benchmark-contamination-controls-data-qualify
Last updated: 2026-11-04

Ask eight things, and score each answer strong, borderline or weak: headcount, operating history, rights to license, record depth, linkage, dating, exposure notes and who can authorize a discussion. Outside the U.S., Canada and the EU, also ask what share of the records is in English. A company scores well when its records carry a checkable right answer, a creation date and a short note of where each record has been. Collect answers from the owner or operations lead, never the records themselves.

This scorecard is part of [benchmark contamination in LLMs](/guides/benchmark-contamination-llm), which explains why a private archive is not automatically clean. Here the job is narrower: deciding whether a particular company is worth introducing for contamination-controlled material, and saying how strong the fit looks.

## How the scorecard works

Each criterion gets 2 points for a strong answer, 1 for borderline and 0 for weak, so the maximum is 16. Three criteria are gates rather than points: if headcount, rights or authorization scores 0, stop and fix that first, whatever the total.

| Total | Band | What it means for the introduction |
|---|---|---|
| 13 to 16 | Strong | Records likely carry dated tasks with known outcomes and some exposure notes; introduce with confidence about the basics |
| 8 to 12 | Borderline | Real records, but dating, exposure or rights need work; introduce and name the open items |
| 0 to 7 | Weak | The opportunity may be theoretical; resolve the gaps before taking anyone's time |

Why contamination controls change the usual screen: a buyer that wants held-out evaluation material cares about whether a task could already be in a model's training data. The BIG-bench project tells its users to treat anything accessible online without its canary marker as potentially seen by a model. Fresh, dated material is the remedy benchmark builders use themselves; LiveBench, for example, relies on frequently updated questions from recent sources. A business archive can play the same role only if its records can be dated and their exposure described.

## The eight criteria and what each band looks like

The general screen lives in the [company qualification checklist](/resources/company-qualification-checklist). This table applies it to contamination-controlled records.

| Criterion | Strong (2) | Borderline (1) | Weak (0) | How to learn it at metadata level |
|---|---|---|---|---|
| Company size (gate) | Well above 50 full-time employees at peak | Close to 50; headcount needs confirming | Under 50 full-time employees, or mostly contractors | Ask for peak full-time headcount; contractors do not count |
| Operating history | Several years of records in a system of record | History exists but split across a migration | Recent start-up or records rebuilt from memory | Ask which years each system covers |
| Rights to license (gate) | Company owns the records; vendor and client terms known | Some client or vendor contracts unreviewed | A client, parent or acquirer controls the records | Ask who owns the records and whether contracts restrict use |
| Record depth | Each task closes with a recorded decision or result | Outcomes recorded for some record types only | Records are notes or messages with no final answer | Ask which workflow ends in a recorded right answer |
| Linkage | Task, decision and outcome share an ID | Joined by typed references | No shared key | Ask what number ties the request to its outcome |
| Dating | Creation dates kept in the system of record | Dates lost for one period, such as an import | Only load or export dates | Ask whether any records were migrated, and when |
| Exposure notes | Vendor list, public pages and any shared pools known | Partly known; nobody has checked public pages | "It's private, so no" with nothing checked | Ask who outside the company has received copies |
| Authorized contact (gate) | Named executive with authority over the records | Senior contact, authority unconfirmed | Nobody identified | Ask who signs data agreements today |

English share applies outside the U.S., Canada and the EU: Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. For a company in those other regions, treat English share as a gate alongside the three above.

## Three Illustrative companies, scored

All three are fictional. Their scores show how the same questions separate a ready introduction from a long shot.

| Criterion | 260-person freight audit firm | 140-person managed IT provider | 70-person marketing agency, Singapore |
|---|---|---|---|
| Company size | 2 | 2 | 2 |
| Operating history | 2 | 1 | 1 |
| Rights to license | 2 | 1 | 0 |
| Record depth | 2 | 1 | 0 |
| Linkage | 2 | 1 | 1 |
| Dating | 2 | 1 | 1 |
| Exposure notes | 1 | 1 | 0 |
| Authorized contact | 2 | 2 | 1 |
| Total | 15 of 16, strong | 10 of 16, borderline | 5 of 16, weak; rights gate failed |

**The freight audit firm** closes every disputed carrier invoice with a recorded adjustment, keyed by invoice number, with creation dates intact since 2017. It loses a point on exposure notes because carriers see the disputes through a shared portal, which the owner could name but had not mapped.

**The managed IT provider** has rich ticket histories, but a 2022 help desk migration kept only load dates for older tickets, and several client contracts say client data stays with the client. Both are fixable questions for the rights review, so this is an introduction with named open items.

**The marketing agency** has campaign files, but its clients own the work, much of it has been published, and about a fifth of its records are in languages other than English, which misses the 70% English rule for a Singapore company. Even with an energetic founder, the opportunity is theoretical until the rights question has a different answer.

## Questions that move a score up or down

Five questions do most of the work. Ask them in this order, because the early answers decide whether the later ones matter.

1. **Which workflow produces tasks with a recorded right answer?** Moves record depth. "Disputed invoices, each closed with an adjustment" scores higher than "we have lots of data".
2. **Has any of it been published, shared or posted?** Moves exposure notes. Help articles, case studies and forum posts are all routes. Industry pools count too; see [data shared with an industry benchmark consortium](/questions/data-shared-with-industry-benchmark-consortium).
3. **Who outside the company has received copies?** Moves exposure notes and rights. A named vendor list with contracts on file scores strong; "unknown" scores weak.
4. **Can each record be dated?** Moves dating. Ask about migrations directly; they are where creation dates disappear.
5. **Who can authorize a licensing discussion?** Moves the authorization gate. If the records came with an acquisition, read [who owns startup data after an acquisition](/questions/who-owns-startup-data-after-acquisition) first.

Write the answer next to the score, including "unknown". A scorecard with honest blanks is more useful to SourceX than a tidy one that guesses.

## What the score cannot tell you

The score describes what the company says it holds. It cannot show whether a given task has a public twin, because that check means searching record content, which happens inside the company under its own control after a discussion starts. It cannot show whether a buyer wants the record type, what de-identification a buyer would require, or what the company would be paid. Researchers have shown how much fresh material can matter: on a new problem set matched to a public math benchmark, several model families scored lower. That is the demand-side reason this kind of record can be interesting. Supply-side, only SourceX's review settles fit.

The partner's role ends at metadata: what the records are, how many, which years, which system, who owns them. The [evidence gaps page](/questions/benchmark-contamination-controls-evidence-gaps) shows how to describe the weak rows honestly, and the [review packet contents list](/resources/benchmark-contamination-controls-review-packet) shows what the company prepares later. Other record types are listed under [data types](/data-types).

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If your scorecard lands in the strong or borderline band with all three gates passed, [register as a partner](/register), confirm the basics on [who qualifies](/who-qualifies) and read the [program terms](/terms) before you introduce the decision-maker.

## Sources

- https://github.com/google/BIG-bench/blob/main/docs/doc.md
- https://arxiv.org/abs/2406.19314
- https://arxiv.org/abs/2405.00332
- https://refer.sourcex.si/terms
