Does a company's benchmark contamination controls data qualify? A strong, borderline and weak fit scorecard

Short answer

Ask eight things, and score each answer strong, borderline or weak: headcount, operating history, rights to license, record depth, linkage, dating, exposure notes and who can authorize a discussion. Outside the U.S., Canada and the EU, also ask what share of the records is in English. A company scores well when its records carry a checkable right answer, a creation date and a short note of where each record has been. Collect answers from the owner or operations lead, never the records themselves.

Visual briefFull-size diagram
A semicircle gauge with weak, borderline and strong bands; as eight criteria are ticked off, the needle settles at 15 of 16 for a fictional 260-person freight audit firm, in the strong band.
Illustrative score for a fictional company. The scorecard sorts introductions; it does not predict acceptance or a payout.

This scorecard is part of benchmark contamination in LLMs, which explains why a private archive is not automatically clean. Here the job is narrower: deciding whether a particular company is worth introducing for contamination-controlled material, and saying how strong the fit looks.

How the scorecard works

Each criterion gets 2 points for a strong answer, 1 for borderline and 0 for weak, so the maximum is 16. Three criteria are gates rather than points: if headcount, rights or authorization scores 0, stop and fix that first, whatever the total.

TotalBandWhat it means for the introduction
13 to 16StrongRecords likely carry dated tasks with known outcomes and some exposure notes; introduce with confidence about the basics
8 to 12BorderlineReal records, but dating, exposure or rights need work; introduce and name the open items
0 to 7WeakThe opportunity may be theoretical; resolve the gaps before taking anyone's time

Why contamination controls change the usual screen: a buyer that wants held-out evaluation material cares about whether a task could already be in a model's training data. The BIG-bench project tells its users to treat anything accessible online without its canary marker as potentially seen by a model. Fresh, dated material is the remedy benchmark builders use themselves; LiveBench, for example, relies on frequently updated questions from recent sources. A business archive can play the same role only if its records can be dated and their exposure described.

The eight criteria and what each band looks like

The general screen lives in the company qualification checklist. This table applies it to contamination-controlled records.

CriterionStrong (2)Borderline (1)Weak (0)How to learn it at metadata level
Company size (gate)Well above 50 full-time employees at peakClose to 50; headcount needs confirmingUnder 50 full-time employees, or mostly contractorsAsk for peak full-time headcount; contractors do not count
Operating historySeveral years of records in a system of recordHistory exists but split across a migrationRecent start-up or records rebuilt from memoryAsk which years each system covers
Rights to license (gate)Company owns the records; vendor and client terms knownSome client or vendor contracts unreviewedA client, parent or acquirer controls the recordsAsk who owns the records and whether contracts restrict use
Record depthEach task closes with a recorded decision or resultOutcomes recorded for some record types onlyRecords are notes or messages with no final answerAsk which workflow ends in a recorded right answer
LinkageTask, decision and outcome share an IDJoined by typed referencesNo shared keyAsk what number ties the request to its outcome
DatingCreation dates kept in the system of recordDates lost for one period, such as an importOnly load or export datesAsk whether any records were migrated, and when
Exposure notesVendor list, public pages and any shared pools knownPartly known; nobody has checked public pages"It's private, so no" with nothing checkedAsk who outside the company has received copies
Authorized contact (gate)Named executive with authority over the recordsSenior contact, authority unconfirmedNobody identifiedAsk who signs data agreements today

English share applies outside the U.S., Canada and the EU: Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. For a company in those other regions, treat English share as a gate alongside the three above.

Three Illustrative companies, scored

All three are fictional. Their scores show how the same questions separate a ready introduction from a long shot.

Criterion260-person freight audit firm140-person managed IT provider70-person marketing agency, Singapore
Company size222
Operating history211
Rights to license210
Record depth210
Linkage211
Dating211
Exposure notes110
Authorized contact221
Total15 of 16, strong10 of 16, borderline5 of 16, weak; rights gate failed

The freight audit firm closes every disputed carrier invoice with a recorded adjustment, keyed by invoice number, with creation dates intact since 2017. It loses a point on exposure notes because carriers see the disputes through a shared portal, which the owner could name but had not mapped.

The managed IT provider has rich ticket histories, but a 2022 help desk migration kept only load dates for older tickets, and several client contracts say client data stays with the client. Both are fixable questions for the rights review, so this is an introduction with named open items.

The marketing agency has campaign files, but its clients own the work, much of it has been published, and about a fifth of its records are in languages other than English, which misses the 70% English rule for a Singapore company. Even with an energetic founder, the opportunity is theoretical until the rights question has a different answer.

Questions that move a score up or down

Five questions do most of the work. Ask them in this order, because the early answers decide whether the later ones matter.

  1. Which workflow produces tasks with a recorded right answer? Moves record depth. "Disputed invoices, each closed with an adjustment" scores higher than "we have lots of data".
  2. Has any of it been published, shared or posted? Moves exposure notes. Help articles, case studies and forum posts are all routes. Industry pools count too; see data shared with an industry benchmark consortium.
  3. Who outside the company has received copies? Moves exposure notes and rights. A named vendor list with contracts on file scores strong; "unknown" scores weak.
  4. Can each record be dated? Moves dating. Ask about migrations directly; they are where creation dates disappear.
  5. Who can authorize a licensing discussion? Moves the authorization gate. If the records came with an acquisition, read who owns startup data after an acquisition first.

Write the answer next to the score, including "unknown". A scorecard with honest blanks is more useful to SourceX than a tidy one that guesses.

What the score cannot tell you

The score describes what the company says it holds. It cannot show whether a given task has a public twin, because that check means searching record content, which happens inside the company under its own control after a discussion starts. It cannot show whether a buyer wants the record type, what de-identification a buyer would require, or what the company would be paid. Researchers have shown how much fresh material can matter: on a new problem set matched to a public math benchmark, several model families scored lower. That is the demand-side reason this kind of record can be interesting. Supply-side, only SourceX's review settles fit.

The partner's role ends at metadata: what the records are, how many, which years, which system, who owns them. The evidence gaps page shows how to describe the weak rows honestly, and the review packet contents list shows what the company prepares later. Other record types are listed under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If your scorecard lands in the strong or borderline band with all three gates passed, register as a partner, confirm the basics on who qualifies and read the program terms before you introduce the decision-maker.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Can a company with a weak score still be introduced?

Yes, if it clears the three gates: 50+ full-time employees at peak, rights to license and a named person who can authorize a discussion. A low score on exposure notes or dating narrows what the records can be used for; it does not stop a conversation. Say so plainly in the introduction.

Who fills in the scorecard?

The introducer, from answers the owner or operations lead gives in conversation. Nobody opens a record to score it. Where the answer is unknown, score it as borderline and write unknown next to it.

Does a high score mean SourceX will accept the company?

No. The score only organizes what you learned. SourceX's review, the rights check and buyer interest decide what happens next, and none of that is known at the introduction stage.

Sources

  1. github.com/google/BIG-bench/blob/main/docs/doc.md
  2. arxiv.org/abs/2406.19314
  3. arxiv.org/abs/2405.00332
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment