# Benchmark freshness qualification scorecard | SourceX

Canonical: https://refer.sourcex.si/questions/does-benchmark-freshness-data-qualify
Last updated: 2026-11-04

To check whether a company could supply fresh benchmark material, ask the CTO or head of engineering about seven things: how often recent private changes add a test, whether that code has ever been public, whether a year-old build still runs, whether pull requests link to the reported problem, how new changes keep arriving, who wrote and owns the code, and how far history goes back. Pass four gates first: size, rights, authority and, where it applies, English share.

This scorecard is one of three working pages under the guide to [benchmark freshness in an ordinary workflow](/guides/concept-benchmark-freshness), which compares three fictional code changes against a model's training cutoff. Here the job is narrower: deciding whether a particular software team is worth introducing, and saying how strong the fit looks, using only what people tell you.

## Four gates before any scoring

Gates are pass or fail. If one fails, the score below does not matter yet; fix the gate or wait.

| Gate | Passes when | Ask |
|---|---|---|
| Size | 50 or more full-time employees at peak, contractors excluded | "What was full-time headcount at its highest?" |
| Rights | The company owns its repositories, including code written by contractors | "Do contractor agreements assign code to the company?" |
| Authority | A named executive can authorize a licensing discussion | "Who signs agreements about the codebase today?" |
| English share | Only outside the U.S., Canada and the EU: at least 70% of the material in English | "Are commit messages, tickets and comments written in English?" |

The geography rule in full: Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The company also needs documented operating history, which the seventh signal below checks.

## Seven freshness signals, from strong to weak

Benchmark builders define freshness by two things: a date that places the task after a model's training cutoff, and a check that can still score a solution. LiveCodeBench tags each problem with its release date so a model is tested only on later problems. SWE-bench keeps only tasks where a test fails before the fix and passes after it. The signals translate those ideas into questions a software team can answer from memory.

| Signal | Strong | Borderline | Weak |
|---|---|---|---|
| 1. Tests with fixes | Most bug-fix pull requests add or change a test | Some teams do, others do not | Tests are rare or written separately later |
| 2. Never public | Repositories private, never mirrored or shared; exceptions named | Mostly private; one past vendor pilot or escrow unconfirmed | Large parts open-sourced or shared outside the company |
| 3. Rebuildable | A change from a year ago can be rebuilt and its tests run | Recent changes rebuild; older ones uncertain | "Probably not"; dependencies were never pinned |
| 4. Linked task statement | Pull requests reference the issue or ticket that prompted them | Linking is a habit, not a rule | Fixes carry no reference to a reported problem |
| 5. Steady flow | Dozens of tested fixes merged every month, continuing | A few per month, or one active product | Codebase frozen or in maintenance only |
| 6. Clear authorship | Employees or contractors under assignment; no shared accounts | Some contractor terms unreviewed | Shared logins, or an outsourcer owns parts of the code |
| 7. History depth | Several years in one code host, history never rewritten | History split by a migration | Recent start, or history squashed during a move |

Signal 5 matters more for freshness than for most record types. Freshness wears off as models are retrained, and SWE-bench-Live exists because earlier benchmarks were not updated after release. A team that keeps producing tested changes keeps producing fresh material; an archive alone goes stale. The [why data freshness matters](/proof/why-data-freshness-matters) page covers the demand side of that point.

**Reading the result.** A strong fit passes every gate and rates at least five signals strong, with signals 2 and 3 never weak. A borderline fit passes every gate with three or four strong signals, or a weak rating on one of signals 4 to 7. Anything weaker is a weak fit for now: the opportunity may be real but cannot yet be described.

## Three software teams, scored (Illustrative)

All three companies are fictional.

| Signal | 520-person logistics software firm | 210-person insurance claims platform | 90-person dev agency, India |
|---|---|---|---|
| Gates | All pass | All pass | Rights gate fails |
| 1. Tests with fixes | Strong | Strong | Borderline |
| 2. Never public | Strong | Borderline | Weak |
| 3. Rebuildable | Strong | Weak | Borderline |
| 4. Linked task statement | Strong | Strong | Weak |
| 5. Steady flow | Strong | Borderline | Strong |
| 6. Clear authorship | Borderline | Strong | Weak |
| 7. History depth | Strong | Strong | Borderline |
| Result | Strong: six strong, none weak | Weak for now: signal 3 is weak | Not yet scorable: rights gate fails |

**The logistics firm** merges tested fixes to its private routing engine every week and pins dependencies in lock files, so year-old changes still build. One contractor team's assignment terms are unreviewed, which is a question for review, not a blocker.

**The claims platform** links every pull request to a ticket and has eight years of history, but a 2023 runtime upgrade broke builds for older changes. Its recent changes may still score well; the honest description is "fresh from 2024 onward, older work not rebuildable."

**The agency** writes code for clients who own it, much of it later published by those clients. However active its engineers are, it cannot license what its clients own, so the gate decides the outcome before any signal is scored.

## When a borderline answer is still worth an introduction

Borderline does not mean wait. It means introduce with the open items named. A useful note to SourceX reads like the claims platform's summary above: what is strong, what is uncertain, and from which date the material is likely usable. Three borderline patterns come up often:

- **"Mostly private."** A past code escrow arrangement or vendor pilot is common. Name it; SourceX checks it in review.
- **"Older builds might not run."** Record the date from which the team is confident. Freshness favors recent work anyway.
- **"Linking is a habit."** Ask roughly what share of fixes reference a ticket. Half is still a meaningful supply.

Some buyers want fresh material only to test models, never to train them, which suits a narrower agreement; see [evaluation-only data licenses](/compare/evaluation-only-data-license).

## What the scorecard leaves to SourceX's review

The scorecard records what the company says. It cannot confirm that code was never exposed, that tests are strong enough to separate a right fix from a wrong one, or that customer data in test fixtures can be removed. It cannot show buyer interest, the company's offer or any reward. Industry interest in fresh benchmarks does not mean every private repository qualifies.

Partners never request code, repository access, screenshots or exports. Discovery stays at metadata level: which systems, how many changes, how many years, who owns them. When a team scores borderline, the [benchmark freshness evidence gaps](/questions/benchmark-freshness-evidence-gaps) page helps phrase the weak rows, and the [benchmark freshness review packet](/resources/benchmark-freshness-review-packet) lists what the company prepares later. For contamination as the reverse problem, read [benchmark contamination in LLMs](/guides/benchmark-contamination-llm); the general screen is the [company qualification checklist](/resources/company-qualification-checklist).

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the team passes all four gates and lands strong or borderline with signals 2 and 3 intact, [register as a partner](/register), confirm the basics on [who qualifies](/who-qualifies), and read the [terms](/terms) before you introduce the CEO or CTO.

## Sources

- https://arxiv.org/abs/2403.07974
- https://arxiv.org/abs/2310.06770
- https://arxiv.org/abs/2505.23419
- https://refer.sourcex.si/terms
