When evaluation data changes which companies you call: ten training-versus-evaluation questions that re-rank a fund's call list, and the evidence that would undo the re-ranking

Short answer

Evaluation data changes your call list when a company's strongest asset is a set of cases with known, recorded outcomes rather than sheer volume. Training rewards breadth and years of records; evaluation rewards cases where the right answer was checked, failures were kept and nothing was ever published. Re-rank the companies you already know with ten questions that separate the two, call first those whose records carry verified outcomes, and keep the fit tests unchanged: 50+ full-time employees at peak, own records, rights and a CEO who wants to talk.

Visual briefFull-size diagram
A slope chart of five Illustrative companies ranked under a training lens on the left and an evaluation lens on the right: the specialty insurer climbs from 4th to 1st and the distributor falls from 1st to 5th, while a sixth company drops off both lists because its records belong to clients.
Illustrative re-ranking. Generic companies and invented record counts; the ranks follow the ten questions on this page.

This is a thesis about where AI development effort may move, turned into a network decision. Whether any current SourceX program wants evaluation sets in a given domain is decided in SourceX's review and is not something a fund can know in advance.

What changes when a model is tested rather than taught

Training changes a model; evaluation measures it. SourceX's own guide to converting an evaluation license into a training deal draws the same line from the contract side: the grant has to be rewritten, because training changes a model permanently and evaluation does not.

Three findings show why the companies worth calling can differ under each use:

  • The outcome is the test. Anthropic's engineering guide to agent evals describes a task as one test with its own defined inputs and success criteria, and judges the outcome by the final state: a flight-booking agent passes only if a reservation actually exists in the database, whatever the agent claims. A record is evaluation material when its final state was captured.
  • Small sets count. The same guide says teams wait too long because they think they need hundreds of tasks, when 20-50 simple tasks drawn from real failures is a great start. A company with a modest archive of well-documented cases, including its failures, can matter.
  • Published tests leak. Zhang and colleagues wrote GSM1k, 1,205 new math problems matched to the public GSM8k benchmark, and found accuracy drops of up to 8% for some models, with several model families showing systematic overfitting. Cases that were never public are harder for a model to have seen.

Translate the evaluation thesis into a company question

The training question a fund usually asks is about volume: how many records of this kind, across how many years, and how varied? The evaluation question is narrower:

Does this company keep cases whose final outcome was recorded and checked by someone accountable, in a system whose contents were never published or shared with vendors?

Record types where the answer is often yes include closed claims with final settlement and adjuster notes, support tickets with a confirmed resolution and a reopen flag, invoice exceptions with the approver's decision, inspection reports with pass, fail and rework, and dispatch exceptions with what actually happened. For the vocabulary, see what a golden dataset is; for explaining the two uses to an owner in two lines, training versus evaluation, briefly.

Ten training-versus-evaluation referral questions

Ask them of the relationship owner first, then of the CEO if the CEO wants the conversation. Every question stays at metadata level.

#Question (metadata only)Points to training valuePoints to evaluation value
1How many records of the core type exist, across how many years?Large counts over many yearsCount matters less than question 2
2Is the final outcome stored with each case?Helpful, not requiredYes, as a field rather than a buried note
3Who confirmed the outcome?Anyone, or no oneA named accountable role, such as an approver or inspector
4Were failures, reversals and reopened cases kept?Useful varietyEssential; evaluation sets are often built from real failures
5Has any of it been published, given to a vendor or used in a public benchmark?Matters lessNever shared is the point
6Is expert judgment written down as reasons, not only a code?Adds richnessLets a grader check the reasoning
7When did the process last change?Variety across eras helpsStable periods give comparable cases
8Is each case a self-contained record of inputs and result?Not requiredNeeded to build a task with defined inputs and success criteria
9Do customer contracts limit reuse of these records?A rights question under either lensA rights question under either lens
10Could a set stay held out, with access logged?Not relevantShows the cases stayed unseen

Method note. Questions 2, 4 and 8 rest on Anthropic's description of evaluation tasks, outcomes and starting sets; question 5 on the GSM1k contamination finding; questions 9 and 10 on SourceX's rights criteria and the held-out records discussed in its license guide, all read on 2026-10-26. The split between the two columns is our framework, not SourceX data and not any AI developer's stated requirement. It ranks companies for calling order only; it does not measure demand, record quality or value.

One call list, re-ranked (Illustrative)

Illustrative. Six generic companies a fund's network already knows, with invented record counts.

Company (Illustrative)Records described by the ownerTraining-lens rankEvaluation-lens rankWhy it moved
1,200-person industrial distributor20 million order lines over 15 years, no case outcomes15Volume without recorded outcomes
800-person B2B software company2.4 million support tickets over 9 years; resolution code stored, customer confirmation not24Outcomes recorded but not confirmed
400-person freight brokerage1.1 million loads, plus 30,000 exceptions with logged outcomes32The exception log is a ready set of outcome-checked cases
260-person specialty insurer40,000 closed claims over 12 years with final settlement and adjuster notes41Every case has a checked final outcome and written reasons
150-person inspection firm18,000 inspection reports with pass, fail and rework53Clear outcomes; whether reports belong to clients needs checking
300-person accounting outsourcerClose workpapers prepared for clientsOff the listOff the listRecords belong to clients under either lens

Same network, same relationships, a different order of calls. The accounting outsourcer shows what the lens never changes: client-owned material is usually not a fit.

Relationship and ownership facts that stay fixed under either lens

The lens reorders the list; it does not loosen anything. The CEO is the person who authorizes a licensing discussion, and the relationship owner, not the platform team, asks first. A company needs 50+ full-time employees at peak (contractors excluded), years of records it created itself, the rights to license them and a location the program covers. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. Funding status plays no part.

Credit is the same too: one partner per company, for the introduction that was first, valid and made within the 90-day window, counted only after the company submits a verified application, and any earlier lead, account or email exchange between SourceX and the company makes it a duplicate. Personal consumer data is usually not a fit under either lens. What only SourceX can settle: whether a current program needs evaluation sets in the company's domain, whether customer contracts allow reuse, and what any license would cover. The comparison of evaluation-only and training licenses covers the terms that follow.

A two-week test of the re-ranking

Test the lens before trusting it. Take the companies that moved two or more places (the distributor, the insurer and the inspection firm in the example) and ask their relationship owners three questions only: 2, 3 and 5. Record yes, no or unknown for each.

  • If the evaluation-lens leaders answer yes to questions 2 and 3 more often than the training-lens leaders, call them first, through their owners, and only with each CEO's consent.
  • If they do not, the owner's first description overstated the outcomes; return to the fund's default order.
  • Either way, report the counts and the reasons. No reward estimate belongs in the note.

Whoever submits an introduction must first sign the SourceX Channel Partner Agreement; partners who signed it as the Channel Partner Non-Circumvention and Non-Disclosure Agreement stay bound by that signed text. A sector-specific version of this test is laid out in a referral pilot for a fund exploring vertical AI, and the portfolio view in the page for venture platform teams.

What would undo the evaluation thesis

  • Experts can write the tests. OpenAI's GDPval builds its tasks from the representative work of industry professionals averaging about 14 years of experience, across 44 occupations, and grades them by head-to-head comparison with expert work. If developers commission expert-written tasks, they may not need company archives for evaluation at all.
  • The frontier may not be contaminated. The same GSM1k study found that frontier models showed minimal signs of overfitting. If the strongest models are not memorizing public tests, the premium on never-published cases may be smaller than the thesis assumes.
  • A small set needs a small slice. If 20-50 well-chosen tasks can start an evaluation, a developer may want only a narrow part of a company's archive. That is our inference, and SourceX has not published how evaluation use affects any license.
  • Rights bind both uses. Client-owned records and personal consumer data are usually not a fit whether a model is trained or tested.
  • Demand is decided in review. SourceX checks each company against its current programs, and those change.

Methods note for this section. Each risk comes from the cited research or SourceX's published criteria as read on 2026-10-26. Nothing here measures how much evaluation demand exists, which sectors it favors or how often a re-ranked call leads to a credited introduction; those figures are not published.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

When a re-ranked company's owner confirms stored, checked outcomes and the CEO wants the conversation, register for a partner link, confirm the fit facts against who qualifies and read the program terms; related guides sit in the guides library.

Re-ordering a call list is a referral task, and weighing SourceX as an investment is another; anyone doing the second can ask for investor materials, and the specific diligence items they need, through the SourceX investor overview.

This article is for information only and is not an offer to sell or a solicitation of an offer to buy securities. Any offering by SourceX is made only to verified accredited investors through its offering documents.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does a company need millions of records to matter for evaluation?

Not necessarily. Anthropic's guidance on agent evals says 20-50 simple tasks drawn from real failures is a great start for a team building an evaluation. What matters is that each case has defined inputs and a recorded, checked outcome, though SourceX still applies its size, history and rights tests to the company.

Should a fund tell a CEO that its records would suit evaluation rather than training?

No. The partner's questions only decide whom to call first. Which use, if any, fits a company's records is settled in SourceX's review and the signed agreement, and the two uses carry different license terms.

Do the fit facts change under an evaluation lens?

No. A company still needs 50+ full-time employees at peak, years of records it created, rights to license and a CEO who authorizes the discussion. The lens changes the order of calls, not who can be introduced.

Sources

  1. refer.sourcex.si/terms
  2. refer.sourcex.si/who-qualifies
  3. anthropic.com/engineering/demystifying-evals-for-ai-agents
  4. arxiv.org/abs/2405.00332
  5. arxiv.org/abs/2510.04374
  6. sourcex.si/insights/ai-dataset-evaluation-to-training-c…
  7. sourcex.si/investors

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment