Benchmark freshness review packet: seven metadata documents that describe recent private code changes

Short answer

A benchmark freshness review packet describes one class of recent private code changes well enough for SourceX to judge whether they are datable, never public and checkable, using metadata the company already has. It holds seven documents: a repository inventory, an exposure statement, a task sheet, a test and environment note, the acceptance check, a volume and span sheet, and a rights and fixture note. It contains no source code until an agreement exists.

Visual briefFull-size diagram
Seven labelled documents, from repository inventory to rights and fixture note, slide in from the right and stack into a review folder; a tag on the folder reads no source code.
Illustrative packet for a fictional software company. The folder describes changes; it contains no code until an agreement exists.

The packet belongs to the guide to benchmark freshness in an ordinary workflow, which explains why a recent private fix with a reproducible test counts for more than an old public one. This page is the practical counterpart: the documents, their order, and who in the engineering organization fills in each one.

Contents list, in assembly order

The order follows the questions a reviewer asks: where the code lives, whether it stayed private, what one task looks like, whether it can be checked, how many exist, and who may license them.

#DocumentWhat it showsPrepared by
1Repository inventoryWhich repositories, which code host, how far back history runs, any migrationsHead of engineering
2Exposure statementWhether any code was published, mirrored, deposited in escrow or shared with outsidersCTO with legal counsel
3Task sheetOne representative change described field by fieldA senior engineer on the team that made it
4Test and environment noteHow tests run and how an old build is rebuiltBuild or platform engineering lead
5Acceptance checkThe exact pass condition for a solutionEngineering lead, reviewed by a second engineer
6Volume and span sheetHow many similar changes per month, over how many monthsEngineering manager
7Rights and fixture noteAuthorship, contractor terms, customer data in test fixturesLegal counsel or the CFO

Each document is short. The whole packet is usually a few pages.

What each document must answer

1. Repository inventory. Name the repositories in scope, the code host, the earliest commit date, and any migration that rewrote or squashed history. A migration date matters because merge dates before it may be load dates, not real dates.

2. Exposure statement. Freshness depends on the code never having left the company and its bound contractors. List the exceptions: modules open-sourced, public mirrors, code shared in a vendor pilot, escrow deposits, or contributions to a shared test pool. "None known" is a valid answer if it is true; "probably none" should be written as such. For one common route, see data shared with an industry benchmark consortium.

3. Task sheet. One change, described in words, as in the filled example below. It shows a reviewer the shape of the class.

4. Test and environment note. Explain how a change from a year ago would be rebuilt: lock files, container definitions, build logs kept from the time. SWE-bench-Live ships a dedicated container image per task so the check runs the same way every time; the note says how close the company's own setup comes to that.

5. Acceptance check. State the pass condition exactly. SWE-bench's version is the model to follow: a task counts as solved only when the tests that failed before the fix now pass and the tests that passed before still pass.

6. Volume and span sheet. Rough monthly counts of tested fixes, months of history, and whether new changes keep arriving. Freshness wears off, so a continuing flow is worth recording; the why data freshness matters page explains why.

7. Rights and fixture note. Who wrote the code (employees, contractors under assignment, an outsourcer), any customer contract that restricts use, and whether test fixtures hold real customer data.

A filled task sheet for one change (Illustrative)

The company and change are fictional: a 450-person freight software company, describing a fix to its private rating engine.

FieldWhat it recordsEntry
Task statementThe problem as reportedWeekend delivery fee charged twice on multi-stop loads
Change dateMerge date from the code hostSeptember 2026
ExposureEver published or shared outside the companyNever; private repository, employees only
Fail-to-pass testNew test that fails before and passes afterMulti-stop weekend fee test
Pass-to-pass testsExisting tests that must keep passingRating engine regression suite
EnvironmentHow to rebuild the code as it wasLock file and container definition committed with the change
Acceptance checkThe exact pass conditionAll listed tests pass on the fixed code, and the new test fails on the old code
Volume and spanHow many similar changes, over what periodAbout 40 tested fixes a month; 30 months since the last repository migration
RightsOwnership and limitsEmployee authorship; one contractor team, assignment terms on file

The September 2026 date is what makes the change useful for testing models trained before then, the same logic LiveCodeBench applies by dating every problem. The entries describe the change; none of them reproduces it.

The fixture problem

The last row of most task sheets hides the most common open question. Test fixtures in business software often contain real customer records: names, addresses, contract rates, shipment details. Developers copy them in because realistic data catches real bugs.

The rights and fixture note should say, at metadata level, whether fixtures contain customer data, roughly how much, and in which repositories. Whether they can be removed or replaced with synthetic values without breaking the tests is decided in SourceX's review. For test material more generally, see test case data for AI training.

Who signs off, and what the packet leaves open

Engineering prepares most of the packet; legal reviews the exposure statement and the rights note; the CEO or CTO signs off as the person who can authorize a licensing discussion. Introducers do not assemble or hold the packet. Their role ends at the metadata conversation, and they never request code, repository access or exports.

The company must meet the basics: 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

The packet leaves open how many changes pass the acceptance check, how fixtures will be handled, whether contracts restrict use, and whether any buyer wants this kind of task. Some buyers want fresh material only for testing, which suits an evaluation-only license. To check fit before any of this, use the benchmark freshness qualification scorecard; for weak spots, see benchmark freshness evidence gaps.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the head of engineering could fill in a task sheet like the one above for a steady flow of recent changes, register as a partner, check the company against who qualifies, and read the terms before you introduce the CEO or CTO.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does the packet describe every change in the codebase?

No. It describes one class of changes, such as bug fixes that added a test, well enough to judge it. One filled task sheet stands in for the class, and the volume sheet says how many similar changes exist.

Can the illustrative task sheet use a real change?

Inside the company, yes; that is how the sheet gets filled in. But the sheet goes to SourceX, not to the introducer, and it describes the change in words. No code, diff or test file is attached.

What happens if test fixtures contain customer records?

Say so in the rights and fixture note. Whether fixtures can be removed or replaced is decided in SourceX's review, and it is one of the most common open questions for business software.

Sources

  1. arxiv.org/abs/2310.06770
  2. arxiv.org/abs/2505.23419
  3. arxiv.org/abs/2403.07974
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment