What can Terminal-Bench-style tasks teach introducers about software workflow evidence?
Mainly this: a task is only useful for evaluation when success can be checked. Terminal-Bench was designed around that idea, and the same test separates strong company records from weak ones.
According to its paper, Terminal-Bench 2.0 is a set of 89 hard tasks in computer terminal environments, inspired by problems from real workflows. Each task has its own environment, a human-written solution and tests for verification, and the authors report that frontier models and agents score below 65%. The paper was published in the ICLR 2026 proceedings, and the project site at tbench.ai lists versions 2.0 and 1.0 as active, with 1.0 described as 80 tasks.
The paper describes each task as five parts: an instruction, a Docker image, a set of tests, an example solution, and a time limit. You do not need to know what a Docker image is to use the lens. It is simply a saved, reproducible copy of the computer setup the task runs in.
For a wider survey of how agents are measured, see our guide to AI agent benchmarks.
The five task parts, translated into company records
The mapping below is our illustration, not part of the benchmark. It shows which ordinary operations records share each part's role.
| Task part | Its role in the benchmark | Company record with a similar role | Question to ask the company |
|---|---|---|---|
| Instruction | States the job in plain words | Change request, ticket or runbook trigger | Are requests written down before work starts? |
| Environment | Fixes the system the task runs on | Infrastructure-as-code, configuration history, environment inventories | Can you tell which configuration a past change applied to? |
| Reference solution | Shows one correct way to do it | Runbooks, change records listing the steps, scripts in version control | Do engineers record the steps they actually took? |
| Tests | Decide pass or fail without a human | CI results, post-change checks, monitoring alerts that cleared | Is success confirmed by a check, or by someone saying it looks fine? |
| Time limit | Bounds the attempt | Maintenance windows, service-level timers, change start and end times | Are start and finish times recorded? |
The strongest companies keep all five and link them by a change or ticket number. Many keep three. The test column is usually the deciding one: a change with a recorded verification check is evidence; a change marked "done" with no check is a claim.
Two adjacent record types are covered elsewhere: infrastructure-as-code history for the environment part, and CI build failure logs for the tests part.
A change record read through the lens (Illustrative)
A fictional 260-person hosting company renews expiring TLS certificates on its own internal load balancers every quarter. One quarter's record looks like this:
- Instruction. A change ticket: "Renew certificates expiring within 30 days on internal load balancers," raised automatically by a monitor.
- Environment. The ticket names the load balancer group, and the configuration is held in version control at a known commit.
- Reference solution. The engineer follows the runbook and pastes the commands run, plus one deviation: a legacy host needed a manual restart.
- Tests. A post-change check confirms each endpoint serves the new certificate, and the expiry monitor clears.
- Time limit. The change ran inside an approved two-hour window, with start and end times logged.
That record has the anatomy of a benchmark task, including the kind of detail that is hard to invent: the deviation, and how the engineer handled it. It concerns the company's own systems, which keeps ownership simple. Records that describe a client's environment belong to the client.
Company qualification questions
Ask these at metadata level only. You never need to see a ticket or a command.
- Which systems hold change requests, runbooks and configuration history, and how many years back?
- Roughly what share of changes record a verification step?
- Are records about the company's own systems, or its clients'?
- Have the records ever been published, for example in public repositories or blog posts?
- Who can authorize a licensing discussion?
The company itself needs 50 or more full-time employees at peak (contractors excluded), a documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. Check who qualifies for the full list.
What a reward could look like (Illustrative)
Rewards are a share of the compensation SourceX actually pays the company you referred (the Actual Supplier Payout): on a company's first qualifying transaction, the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap, subject to the program terms. SourceX pays rewards separately; they never reduce the company's payout. Two illustrative cases, neither a forecast nor a quote:
| Illustrative Actual Supplier Payout | Reward on a first qualifying transaction (2026 terms) |
|---|---|
| $250,000 | $62,500 (USD 50,000 plus 5%) |
| $800,000 | $120,000 (15%) |
A further qualifying transaction with the same company, first agreed within 12 months of the first Closing, pays 15% of its Actual Supplier Payout; the USD 50,000 floor applies once per referred company.
A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details. For more scenarios, see how much you can make from referrals.
Limits and unanswered questions
- A company record is not a benchmark task. Benchmarks are curated, cleaned and tested by researchers. Company records are messier, and whether they suit a buyer is decided in SourceX's review.
- Public exposure matters. Material that has appeared online may already be in training data, the problem described in our note on benchmark contamination.
- Security-sensitive content. Commands and configurations can reveal infrastructure details; what is excluded is the company's decision in its rights review.
- No numbers yet on how often companies qualify. SourceX has not published conversion figures for this category, and this page does not estimate one.
More guides are in the guides hub.
Next step
Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
If an engineering or operations leader tells you their changes carry a request, a recorded procedure and a verification check, register, confirm the company on who qualifies, and read the terms before introducing the owner.