Does benchmark freshness data qualify? A scorecard for private code changes with tests that still run

Short answer

To check whether a company could supply fresh benchmark material, ask the CTO or head of engineering about seven things: how often recent private changes add a test, whether that code has ever been public, whether a year-old build still runs, whether pull requests link to the reported problem, how new changes keep arriving, who wrote and owns the code, and how far history goes back. Pass four gates first: size, rights, authority and, where it applies, English share.

Visual briefFull-size diagram
A horizontal gauge running from weak through borderline to strong; as four gates and seven freshness signals are checked, the needle slides and settles in the strong zone for a fictional 520-person logistics software firm.
Illustrative score for a fictional company. The scorecard organizes an introduction; it does not predict acceptance or a payout.

This scorecard is one of three working pages under the guide to benchmark freshness in an ordinary workflow, which compares three fictional code changes against a model's training cutoff. Here the job is narrower: deciding whether a particular software team is worth introducing, and saying how strong the fit looks, using only what people tell you.

Four gates before any scoring

Gates are pass or fail. If one fails, the score below does not matter yet; fix the gate or wait.

GatePasses whenAsk
Size50 or more full-time employees at peak, contractors excluded"What was full-time headcount at its highest?"
RightsThe company owns its repositories, including code written by contractors"Do contractor agreements assign code to the company?"
AuthorityA named executive can authorize a licensing discussion"Who signs agreements about the codebase today?"
English shareOnly outside the U.S., Canada and the EU: at least 70% of the material in English"Are commit messages, tickets and comments written in English?"

The geography rule in full: Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The company also needs documented operating history, which the seventh signal below checks.

Seven freshness signals, from strong to weak

Benchmark builders define freshness by two things: a date that places the task after a model's training cutoff, and a check that can still score a solution. LiveCodeBench tags each problem with its release date so a model is tested only on later problems. SWE-bench keeps only tasks where a test fails before the fix and passes after it. The signals translate those ideas into questions a software team can answer from memory.

SignalStrongBorderlineWeak
1. Tests with fixesMost bug-fix pull requests add or change a testSome teams do, others do notTests are rare or written separately later
2. Never publicRepositories private, never mirrored or shared; exceptions namedMostly private; one past vendor pilot or escrow unconfirmedLarge parts open-sourced or shared outside the company
3. RebuildableA change from a year ago can be rebuilt and its tests runRecent changes rebuild; older ones uncertain"Probably not"; dependencies were never pinned
4. Linked task statementPull requests reference the issue or ticket that prompted themLinking is a habit, not a ruleFixes carry no reference to a reported problem
5. Steady flowDozens of tested fixes merged every month, continuingA few per month, or one active productCodebase frozen or in maintenance only
6. Clear authorshipEmployees or contractors under assignment; no shared accountsSome contractor terms unreviewedShared logins, or an outsourcer owns parts of the code
7. History depthSeveral years in one code host, history never rewrittenHistory split by a migrationRecent start, or history squashed during a move

Signal 5 matters more for freshness than for most record types. Freshness wears off as models are retrained, and SWE-bench-Live exists because earlier benchmarks were not updated after release. A team that keeps producing tested changes keeps producing fresh material; an archive alone goes stale. The why data freshness matters page covers the demand side of that point.

Reading the result. A strong fit passes every gate and rates at least five signals strong, with signals 2 and 3 never weak. A borderline fit passes every gate with three or four strong signals, or a weak rating on one of signals 4 to 7. Anything weaker is a weak fit for now: the opportunity may be real but cannot yet be described.

Three software teams, scored (Illustrative)

All three companies are fictional.

Signal520-person logistics software firm210-person insurance claims platform90-person dev agency, India
GatesAll passAll passRights gate fails
1. Tests with fixesStrongStrongBorderline
2. Never publicStrongBorderlineWeak
3. RebuildableStrongWeakBorderline
4. Linked task statementStrongStrongWeak
5. Steady flowStrongBorderlineStrong
6. Clear authorshipBorderlineStrongWeak
7. History depthStrongStrongBorderline
ResultStrong: six strong, none weakWeak for now: signal 3 is weakNot yet scorable: rights gate fails

The logistics firm merges tested fixes to its private routing engine every week and pins dependencies in lock files, so year-old changes still build. One contractor team's assignment terms are unreviewed, which is a question for review, not a blocker.

The claims platform links every pull request to a ticket and has eight years of history, but a 2023 runtime upgrade broke builds for older changes. Its recent changes may still score well; the honest description is "fresh from 2024 onward, older work not rebuildable."

The agency writes code for clients who own it, much of it later published by those clients. However active its engineers are, it cannot license what its clients own, so the gate decides the outcome before any signal is scored.

When a borderline answer is still worth an introduction

Borderline does not mean wait. It means introduce with the open items named. A useful note to SourceX reads like the claims platform's summary above: what is strong, what is uncertain, and from which date the material is likely usable. Three borderline patterns come up often:

  • "Mostly private." A past code escrow arrangement or vendor pilot is common. Name it; SourceX checks it in review.
  • "Older builds might not run." Record the date from which the team is confident. Freshness favors recent work anyway.
  • "Linking is a habit." Ask roughly what share of fixes reference a ticket. Half is still a meaningful supply.

Some buyers want fresh material only to test models, never to train them, which suits a narrower agreement; see evaluation-only data licenses.

What the scorecard leaves to SourceX's review

The scorecard records what the company says. It cannot confirm that code was never exposed, that tests are strong enough to separate a right fix from a wrong one, or that customer data in test fixtures can be removed. It cannot show buyer interest, the company's offer or any reward. Industry interest in fresh benchmarks does not mean every private repository qualifies.

Partners never request code, repository access, screenshots or exports. Discovery stays at metadata level: which systems, how many changes, how many years, who owns them. When a team scores borderline, the benchmark freshness evidence gaps page helps phrase the weak rows, and the benchmark freshness review packet lists what the company prepares later. For contamination as the reverse problem, read benchmark contamination in LLMs; the general screen is the company qualification checklist.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the team passes all four gates and lands strong or borderline with signals 2 and 3 intact, register as a partner, confirm the basics on who qualifies, and read the terms before you introduce the CEO or CTO.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Who should answer the scorecard questions?

The CTO or head of engineering, ideally in one short call. They know whether pull requests add tests, whether old builds still run and whether any code has been published. The owner or CEO answers the gate questions on headcount and authority.

Is a company with mostly open-source work a weak fit?

For freshness specifically, yes, because public code may already be in model training data. The private parts of its codebase can still score well if they meet the other signals, so ask which repositories have never left the company.

Does a strong score mean the company will be paid?

No. The scorecard only sorts what you learned. SourceX's review, the rights check and buyer interest decide what happens next, and none of that is known when you make the introduction.

Sources

  1. arxiv.org/abs/2403.07974
  2. arxiv.org/abs/2310.06770
  3. arxiv.org/abs/2505.23419
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment