Benchmark freshness evidence gaps: what a review expects from a code archive, and what is usually missing

Short answer

Benchmark freshness is hard to assess when a code archive cannot prove one of seven things about a change: what problem it fixed, when it merged, that it was never public, which test proves it, which tests must keep passing, how to rebuild its environment, and whether the fix held. Teams usually keep dates and tests. Exposure history, pinned environments and later outcomes are what tend to go missing.

Visual briefFull-size diagram
Seven horizontal bars compare the evidence a freshness review expects with what a typical software team holds; merge dates and fail-to-pass tests are mostly filled, while exposure history, pinned environments and later outcomes show large outlined missing segments.
Illustrative pattern, not measured data: where code archives tend to be complete and where they tend to fall short.

This gap analysis is one of three working pages under the guide to benchmark freshness in an ordinary workflow. The guide shows what a fresh change looks like. This page does the opposite: it lists the evidence a review expects, compares it with what software teams usually hold, and shows how to describe the shortfall in an introduction without overstating anything.

What a freshness review expects for each change

A review asks of every candidate change whether it is datable, private and checkable. Benchmark builders use the same three tests. LiveCodeBench dates each problem so it can be matched against a model's training cutoff. SWE-bench keeps a task only if a test fails before the original fix and passes after it, and counts it solved only when that test and the previously passing ones all pass. SWE-bench-Live adds a dedicated container image per task so the check runs the same way every time.

For a private business codebase, those tests break down into seven pieces of evidence per change:

  1. Task statement: the reported problem, in the reporter's words, linked to the change.
  2. Merge date: when the fix landed, from the code host.
  3. Exposure history: confirmation that the code never left the company and its bound contractors.
  4. Fail-to-pass test: a new or changed test that fails on the old code and passes on the fix.
  5. Pass-to-pass tests: the existing tests that must keep passing.
  6. Pinned environment: lock files, container definitions or build logs from the time of the change.
  7. Final outcome: whether the fix held, or whether the ticket was reopened or a follow-up fix followed.

Expected, typically held, usually missing

The table below is a pattern from how software teams commonly work, not a measurement; any given company will differ. Use it to know where to aim your questions.

EvidenceWhat review expectsWhat teams typically holdWhat usually goes missing
Task statementIssue or ticket linked from the pull requestTickets exist in a trackerThe link; the pull request names no ticket
Merge dateMerge timestamp in the original code hostPresent while history is intactDates after a migration that rewrote history
Exposure historyA named list of every place the code wentA belief that the code is privateMirrors, vendor pilots, escrow deposits, open-sourced modules
Fail-to-pass testTest added in the same change as the fixCommon in teams with code reviewTests added weeks later, or never
Pass-to-pass testsContinuous integration results at merge timeCurrent test suitesResults from the time of the change
Pinned environmentExact dependency versions and build recipeLock files in recent yearsOlder changes built against floating versions
Final outcomeReopen events or follow-up fixes tied to the changeTicket status in the trackerThe tie back to the code change, held in a different system

The shape is consistent: what lives in the code host survives; what lives elsewhere, or only in people's memory, does not.

Five gaps, and how each one hides a stale or unusable task

Each gap below can make a change that looks fresh impossible to tell apart from an old public fix or a change that no longer runs. They are questions for SourceX's review, not reasons to stop.

No link from change to reason. Without a ticket reference, a reviewer has a code diff and no task statement. The model being tested would have to guess what problem it is solving. The guide to private code without issue history covers this gap in depth.

Unknown exposure. Freshness assumes the code never left the company. Open-sourced modules, public mirrors, code shared with a vendor or partner, and contributions to a shared test pool can all break that assumption for the parts involved. Our page on data shared with an industry benchmark consortium covers one version of this.

Unpinned environments. If dependencies floated, a year-old change may no longer build, and an unbuildable change cannot score a solution. This is the gap SWE-bench-Live's per-task container images exist to close, and private archives rarely close it on their own.

Unclear authorship and permissions. Contractor-written code belongs to whoever the contract says. Commits made under shared accounts cannot show who wrote or reviewed a change.

Unrecorded final outcome. A passing test is not proof the fix held. A reopened ticket or a follow-up fix tells a reviewer the original change was incomplete, and that history usually sits in a tracker that nobody connects to the merge.

Wording a gap honestly in the introduction

The goal is a note SourceX can act on, with no gap hidden and none exaggerated. Three illustrative examples, all fictional:

SituationOverstatedHonest
Lock files only since 2024"All history is reproducible""Builds are pinned from early 2024; older changes may not rebuild"
One module open-sourced"Fully private codebase""Private except the export library, open-sourced in 2022"
Ticket links inconsistent"Every fix is documented""About half of fixes link to a ticket, by the CTO's estimate"

Keep estimates labeled as estimates, and say who gave them. "Unknown" is an acceptable answer and more useful than a guess.

Gaps that are SourceX's to resolve

Some gaps cannot be judged from a conversation, and an introducer should not try. Whether a given module was ever exposed, whether a test is strong enough to reject a wrong fix, and whether test fixtures contain customer data that can be replaced all need inspection under an agreement with the company. So does the question of buyer interest. Interest in fresh benchmarks across the industry, and buyers' general preference for ongoing data over a one-time archive, say nothing about demand for one company's code.

Ownership sits with engineering for the repositories, tracker and build system, and with legal for contractor terms. The CEO or CTO usually authorizes a licensing discussion. The company needs 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Before describing gaps, score the company with the benchmark freshness qualification scorecard. After an introduction, the company prepares the benchmark freshness review packet. For the reverse problem, see benchmark contamination in LLMs.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If you can describe the company's gaps as plainly as the honest column above, register as a partner, confirm the basics on who qualifies, and read the terms before you introduce the CTO.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does one gap mean the company should not be introduced?

No. Most code archives have at least one gap. A gap narrows which changes can be used, often to a date range, and SourceX's review decides how much remains. Introduce with the gap stated.

Can the company fix a gap before the introduction?

Some gaps can be described more precisely, such as the date from which dependencies were pinned. Others, like code that was once published, cannot be undone. Rebuilding history or editing records to look complete would make things worse.

Should I check the repository myself to find gaps?

No. Partners never access client code. Ask the questions on this page and pass on the answers, including the ones that are unknown.

Sources

  1. arxiv.org/abs/2310.06770
  2. arxiv.org/abs/2505.23419
  3. arxiv.org/abs/2403.07974
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment