# Benchmark freshness: what makes a task fresh | SourceX

Canonical: https://refer.sourcex.si/guides/concept-benchmark-freshness
Last updated: 2026-11-04

Benchmark freshness means an evaluation task is new enough, and private enough, that the AI model being tested cannot have seen it or its answer during training, while still carrying a test that can check a solution. In an ordinary software team, the freshest material is a recent change to a private codebase that fixes a reported problem and adds a test that fails before the fix and passes after it, in a build that can still be run.

## What does benchmark freshness mean in an ordinary business workflow?

A benchmark task is fresh when the model being tested could not have learned it in advance. Freshness wears off: every public task eventually leaks into the data later models learn from, and a high score then measures memory more than skill. Our explainer on [benchmark contamination](/guides/benchmark-contamination-llm) covers how that leak happens. This page is about the other side: what a company's records need to show for a task to count as fresh in the first place.

Benchmark builders handle freshness in two ways that map directly onto business records. LiveCodeBench collects new problems over time and tags each one with its release date, so a newer model can be tested only on problems released after its training cutoff. SWE-bench-Live builds tasks from issues created since 2024 and refreshes continuously, because, as its authors note, earlier benchmarks were not updated after release. Both treat the date as evidence, not a detail.

The second requirement is a valid check. SWE-bench, the benchmark these code tasks descend from, keeps only tasks where at least one test fails before the original fix and passes after it, and counts a task solved only when those tests and the previously passing ones all pass. SWE-bench-Live ships each task with its own container image so the test runs the same way every time. A fresh task with no runnable test cannot score anyone.

In ordinary software work, the closest match is a pull request in a private repository that fixes a reported problem and adds a test proving the fix. Companies with internal software teams produce these every week. The [field guide to companies with pull requests and code reviews](/guides/a-partner-field-guide-to-finding-companies-with-pull-requests-and-code-reviews) covers where they sit; this page covers which of them are fresh.

## Three changes, one freshness test (Illustrative)

These changes are fictional. A 450-person freight software company keeps its rating engine in a private repository. Three changes from its history are compared against the training cutoff of a model a buyer wants to test.

| Change | When | Ever public? | Test still runs? | Verdict |
|---|---|---|---|---|
| A. Date-parsing fix contributed to an open-source library the company uses | Three years before the cutoff | Yes, on a public code host | Yes | Not fresh: old and public, so it may be in training data |
| B. Fuel surcharge rounding fix in the private rating engine | One year before the cutoff | No | No: a package it depends on was withdrawn and the build fails | Not usable: private, but before the cutoff and nothing can check a solution |
| C. Fix for weekend delivery fees applied twice on multi-stop loads | September 2026, after the cutoff | No | Yes: a new test fails on the old code and passes on the fixed code; existing tests still pass | Fresh and checkable |

Change C is the shape a buyer looks for. Its acceptance check is exact: on the code as it stood before the fix, the new multi-stop weekend test fails and the existing rating tests pass; with the fix applied, all of them pass, in an environment rebuilt from the recorded dependency versions. Change B shows the trap that catches many archives. The company kept the code and the test but not a way to run them, so the task can no longer separate a right answer from a wrong one.

Freshness is also a moving target. Change C is fresh for models trained before September 2026. For a model trained in 2027 on data that somehow included it, it would not be, which is why the never-public column matters as much as the date. The [why data freshness matters](/proof/why-data-freshness-matters) page explains why a steady supply of new material counts for more than a single archive.

## The fields that make change C datable and checkable

Change C passes only because ordinary engineering records answer three questions: when, where and how to check. These are the fields behind each answer in the example.

| Question | Field in the example | Why it matters for freshness |
|---|---|---|
| When did the problem appear? | Issue creation date in the tracker, September 2026 | Places the task relative to a model's training cutoff |
| When was it fixed? | Merge date of the pull request | Confirms the solution, not just the problem, is after the cutoff |
| Was it ever public? | Repository visibility and any mirror or vendor-sharing history | A private date proves nothing if the code was copied somewhere public |
| What was the code before the fix? | Parent commit of the merge | Gives the starting point a model must work from |
| What proves the fix? | New multi-stop weekend test added in the same change | The fail-before, pass-after check |
| What must not break? | Existing rating tests run in continuous integration | The pass-to-pass check that stops a fix from breaking other fees |
| Can it still run? | Lock file with pinned dependency versions, plus a build log | Shows the environment can be rebuilt, the failure that sank change B |

None of these fields is exotic. Most teams using a code host and continuous integration already record them. What varies is whether they survive: whether old build logs are kept, whether dependency versions were pinned, and whether repository history was rewritten during a migration.

## What freshness does not prove

Freshness shows a task could not have been memorized. It does not show the task is any good, and the example should not be read as more than it is.

- **Fresh is not hard.** Change C fixes a fee applied twice. Whether that is a demanding test or a trivial one is a separate judgment.
- **Fresh is not representative.** One rating-engine bug says little about the rest of the codebase or about what a buyer wants to measure.
- **Private is a claim until checked.** The company believes the repository never left its control. Contractor access, vendor pilots and past code escrow arrangements are verified in review, not assumed.
- **A passing test can be weak.** A test that checks only one load shape could pass a wrong fix. How strong each test is gets examined in review.

## Who owns these records, and what stays unknown

Engineering owns the repositories, tracker and build system; the CTO or head of engineering knows how far back history runs. Customer contracts and any code escrow arrangements sit with legal. Authority to discuss licensing usually rests with the CEO or CTO.

The company needs 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license its records, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The [company qualification checklist](/resources/company-qualification-checklist) covers the basics, and an [evaluation-only license](/compare/evaluation-only-data-license) is one structure owners ask about for test material like this.

Unknown until SourceX reviews it: how many changes pass the acceptance check, how customer data in fixtures and commit messages is handled, whether any contracts restrict use, and whether buyers want this kind of task. Interest in fresh benchmarks across the industry does not mean every private repository qualifies. Partners never request code, screenshots or repository access. Related record families include [test case data](/data-types/test-case-data-ai-training) and the rest of the [data types hub](/data-types).

## Go deeper on benchmark freshness

- [Questions to ask a client about benchmark freshness](/questions/does-benchmark-freshness-data-qualify): a short discovery script for checking whether a company's code history could qualify.
- [Record gaps that make benchmark freshness hard to assess](/questions/benchmark-freshness-evidence-gaps): a guide to the missing dates, rewritten history and broken builds that weaken a freshness claim.
- [Benchmark freshness review packet](/resources/benchmark-freshness-review-packet): a list of the metadata a company prepares before SourceX reviews its recent changes.

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the head of engineering confirms that recent private changes ship with tests and that old builds can still be rebuilt, [register as a partner](/register), check the company against [who qualifies](/who-qualifies), and read the [terms](/terms) before you introduce the CEO or CTO.

## Sources

- https://arxiv.org/abs/2403.07974
- https://arxiv.org/abs/2310.06770
- https://arxiv.org/abs/2505.23419
- https://refer.sourcex.si/terms
