Benchmark freshness: why a recent private code change with a reproducible test is worth more than an old public fix

Short answer

Benchmark freshness means an evaluation task is new enough, and private enough, that the AI model being tested cannot have seen it or its answer during training, while still carrying a test that can check a solution. In an ordinary software team, the freshest material is a recent change to a private codebase that fixes a reported problem and adds a test that fails before the fix and passes after it, in a build that can still be run.

Visual briefFull-size diagram
A timeline with a model training cutoff line and three code changes passing through three gates: an old public fix fails the after-cutoff and never-public gates, a private change from last year fails the after-cutoff and reproducible-test gates, and a private change from September 2026 passes all three.
Illustrative changes at a fictional software company. Freshness needs all three: recent, never public, and a test that still runs.

What does benchmark freshness mean in an ordinary business workflow?

A benchmark task is fresh when the model being tested could not have learned it in advance. Freshness wears off: every public task eventually leaks into the data later models learn from, and a high score then measures memory more than skill. Our explainer on benchmark contamination covers how that leak happens. This page is about the other side: what a company's records need to show for a task to count as fresh in the first place.

Benchmark builders handle freshness in two ways that map directly onto business records. LiveCodeBench collects new problems over time and tags each one with its release date, so a newer model can be tested only on problems released after its training cutoff. SWE-bench-Live builds tasks from issues created since 2024 and refreshes continuously, because, as its authors note, earlier benchmarks were not updated after release. Both treat the date as evidence, not a detail.

The second requirement is a valid check. SWE-bench, the benchmark these code tasks descend from, keeps only tasks where at least one test fails before the original fix and passes after it, and counts a task solved only when those tests and the previously passing ones all pass. SWE-bench-Live ships each task with its own container image so the test runs the same way every time. A fresh task with no runnable test cannot score anyone.

In ordinary software work, the closest match is a pull request in a private repository that fixes a reported problem and adds a test proving the fix. Companies with internal software teams produce these every week. The field guide to companies with pull requests and code reviews covers where they sit; this page covers which of them are fresh.

Three changes, one freshness test (Illustrative)

These changes are fictional. A 450-person freight software company keeps its rating engine in a private repository. Three changes from its history are compared against the training cutoff of a model a buyer wants to test.

ChangeWhenEver public?Test still runs?Verdict
A. Date-parsing fix contributed to an open-source library the company usesThree years before the cutoffYes, on a public code hostYesNot fresh: old and public, so it may be in training data
B. Fuel surcharge rounding fix in the private rating engineOne year before the cutoffNoNo: a package it depends on was withdrawn and the build failsNot usable: private, but before the cutoff and nothing can check a solution
C. Fix for weekend delivery fees applied twice on multi-stop loadsSeptember 2026, after the cutoffNoYes: a new test fails on the old code and passes on the fixed code; existing tests still passFresh and checkable

Change C is the shape a buyer looks for. Its acceptance check is exact: on the code as it stood before the fix, the new multi-stop weekend test fails and the existing rating tests pass; with the fix applied, all of them pass, in an environment rebuilt from the recorded dependency versions. Change B shows the trap that catches many archives. The company kept the code and the test but not a way to run them, so the task can no longer separate a right answer from a wrong one.

Freshness is also a moving target. Change C is fresh for models trained before September 2026. For a model trained in 2027 on data that somehow included it, it would not be, which is why the never-public column matters as much as the date. The why data freshness matters page explains why a steady supply of new material counts for more than a single archive.

The fields that make change C datable and checkable

Change C passes only because ordinary engineering records answer three questions: when, where and how to check. These are the fields behind each answer in the example.

QuestionField in the exampleWhy it matters for freshness
When did the problem appear?Issue creation date in the tracker, September 2026Places the task relative to a model's training cutoff
When was it fixed?Merge date of the pull requestConfirms the solution, not just the problem, is after the cutoff
Was it ever public?Repository visibility and any mirror or vendor-sharing historyA private date proves nothing if the code was copied somewhere public
What was the code before the fix?Parent commit of the mergeGives the starting point a model must work from
What proves the fix?New multi-stop weekend test added in the same changeThe fail-before, pass-after check
What must not break?Existing rating tests run in continuous integrationThe pass-to-pass check that stops a fix from breaking other fees
Can it still run?Lock file with pinned dependency versions, plus a build logShows the environment can be rebuilt, the failure that sank change B

None of these fields is exotic. Most teams using a code host and continuous integration already record them. What varies is whether they survive: whether old build logs are kept, whether dependency versions were pinned, and whether repository history was rewritten during a migration.

What freshness does not prove

Freshness shows a task could not have been memorized. It does not show the task is any good, and the example should not be read as more than it is.

  • Fresh is not hard. Change C fixes a fee applied twice. Whether that is a demanding test or a trivial one is a separate judgment.
  • Fresh is not representative. One rating-engine bug says little about the rest of the codebase or about what a buyer wants to measure.
  • Private is a claim until checked. The company believes the repository never left its control. Contractor access, vendor pilots and past code escrow arrangements are verified in review, not assumed.
  • A passing test can be weak. A test that checks only one load shape could pass a wrong fix. How strong each test is gets examined in review.

Who owns these records, and what stays unknown

Engineering owns the repositories, tracker and build system; the CTO or head of engineering knows how far back history runs. Customer contracts and any code escrow arrangements sit with legal. Authority to discuss licensing usually rests with the CEO or CTO.

The company needs 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license its records, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The company qualification checklist covers the basics, and an evaluation-only license is one structure owners ask about for test material like this.

Unknown until SourceX reviews it: how many changes pass the acceptance check, how customer data in fixtures and commit messages is handled, whether any contracts restrict use, and whether buyers want this kind of task. Interest in fresh benchmarks across the industry does not mean every private repository qualifies. Partners never request code, screenshots or repository access. Related record families include test case data and the rest of the data types hub.

Go deeper on benchmark freshness

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the head of engineering confirms that recent private changes ship with tests and that old builds can still be rebuilt, register as a partner, check the company against who qualifies, and read the terms before you introduce the CEO or CTO.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

How recent does a change need to be to count as fresh?

There is no single date. Freshness is relative to the training cutoff of the models being tested, which keeps moving, so a change that is fresh today becomes stale for later models. That is why buyers value a steady flow of new changes over a one-time archive.

Does an open-source project's code count?

It can still be useful, but public code may already sit in training data, so it is not fresh in the benchmark sense. The strongest material for this purpose has never been published or shared outside the company and its bound contractors.

Should I look at the repository to check whether tests exist?

No. Partners never access client code. Ask whether pull requests add tests, whether old builds still run, and who owns the repositories, and leave inspection to SourceX under an agreement with the company.

Sources

  1. arxiv.org/abs/2403.07974
  2. arxiv.org/abs/2310.06770
  3. arxiv.org/abs/2505.23419
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment