This gap analysis is one of three working pages under the guide to benchmark freshness in an ordinary workflow. The guide shows what a fresh change looks like. This page does the opposite: it lists the evidence a review expects, compares it with what software teams usually hold, and shows how to describe the shortfall in an introduction without overstating anything.
What a freshness review expects for each change
A review asks of every candidate change whether it is datable, private and checkable. Benchmark builders use the same three tests. LiveCodeBench dates each problem so it can be matched against a model's training cutoff. SWE-bench keeps a task only if a test fails before the original fix and passes after it, and counts it solved only when that test and the previously passing ones all pass. SWE-bench-Live adds a dedicated container image per task so the check runs the same way every time.
For a private business codebase, those tests break down into seven pieces of evidence per change:
- Task statement: the reported problem, in the reporter's words, linked to the change.
- Merge date: when the fix landed, from the code host.
- Exposure history: confirmation that the code never left the company and its bound contractors.
- Fail-to-pass test: a new or changed test that fails on the old code and passes on the fix.
- Pass-to-pass tests: the existing tests that must keep passing.
- Pinned environment: lock files, container definitions or build logs from the time of the change.
- Final outcome: whether the fix held, or whether the ticket was reopened or a follow-up fix followed.
Expected, typically held, usually missing
The table below is a pattern from how software teams commonly work, not a measurement; any given company will differ. Use it to know where to aim your questions.
| Evidence | What review expects | What teams typically hold | What usually goes missing |
|---|---|---|---|
| Task statement | Issue or ticket linked from the pull request | Tickets exist in a tracker | The link; the pull request names no ticket |
| Merge date | Merge timestamp in the original code host | Present while history is intact | Dates after a migration that rewrote history |
| Exposure history | A named list of every place the code went | A belief that the code is private | Mirrors, vendor pilots, escrow deposits, open-sourced modules |
| Fail-to-pass test | Test added in the same change as the fix | Common in teams with code review | Tests added weeks later, or never |
| Pass-to-pass tests | Continuous integration results at merge time | Current test suites | Results from the time of the change |
| Pinned environment | Exact dependency versions and build recipe | Lock files in recent years | Older changes built against floating versions |
| Final outcome | Reopen events or follow-up fixes tied to the change | Ticket status in the tracker | The tie back to the code change, held in a different system |
The shape is consistent: what lives in the code host survives; what lives elsewhere, or only in people's memory, does not.
Five gaps, and how each one hides a stale or unusable task
Each gap below can make a change that looks fresh impossible to tell apart from an old public fix or a change that no longer runs. They are questions for SourceX's review, not reasons to stop.
No link from change to reason. Without a ticket reference, a reviewer has a code diff and no task statement. The model being tested would have to guess what problem it is solving. The guide to private code without issue history covers this gap in depth.
Unknown exposure. Freshness assumes the code never left the company. Open-sourced modules, public mirrors, code shared with a vendor or partner, and contributions to a shared test pool can all break that assumption for the parts involved. Our page on data shared with an industry benchmark consortium covers one version of this.
Unpinned environments. If dependencies floated, a year-old change may no longer build, and an unbuildable change cannot score a solution. This is the gap SWE-bench-Live's per-task container images exist to close, and private archives rarely close it on their own.
Unclear authorship and permissions. Contractor-written code belongs to whoever the contract says. Commits made under shared accounts cannot show who wrote or reviewed a change.
Unrecorded final outcome. A passing test is not proof the fix held. A reopened ticket or a follow-up fix tells a reviewer the original change was incomplete, and that history usually sits in a tracker that nobody connects to the merge.
Wording a gap honestly in the introduction
The goal is a note SourceX can act on, with no gap hidden and none exaggerated. Three illustrative examples, all fictional:
| Situation | Overstated | Honest |
|---|---|---|
| Lock files only since 2024 | "All history is reproducible" | "Builds are pinned from early 2024; older changes may not rebuild" |
| One module open-sourced | "Fully private codebase" | "Private except the export library, open-sourced in 2022" |
| Ticket links inconsistent | "Every fix is documented" | "About half of fixes link to a ticket, by the CTO's estimate" |
Keep estimates labeled as estimates, and say who gave them. "Unknown" is an acceptable answer and more useful than a guess.
Gaps that are SourceX's to resolve
Some gaps cannot be judged from a conversation, and an introducer should not try. Whether a given module was ever exposed, whether a test is strong enough to reject a wrong fix, and whether test fixtures contain customer data that can be replaced all need inspection under an agreement with the company. So does the question of buyer interest. Interest in fresh benchmarks across the industry, and buyers' general preference for ongoing data over a one-time archive, say nothing about demand for one company's code.
Ownership sits with engineering for the repositories, tracker and build system, and with legal for contractor terms. The CEO or CTO usually authorizes a licensing discussion. The company needs 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.
Before describing gaps, score the company with the benchmark freshness qualification scorecard. After an introduction, the company prepares the benchmark freshness review packet. For the reverse problem, see benchmark contamination in LLMs.
Next step
Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
If you can describe the company's gaps as plainly as the honest column above, register as a partner, confirm the basics on who qualifies, and read the terms before you introduce the CTO.