Terminal-Bench for introducers: recognizing realistic software tasks

Short answer

Terminal-Bench tasks show what a realistic, checkable software task looks like: a written instruction, a defined environment, a human-written solution, and tests that decide whether the job was done, all within a time limit. Introducers can use that anatomy as a lens. A company whose operations records keep the request, the environment it applied to, the steps taken and the check that confirmed success has records of the same shape, and that is worth raising with its owner.

Visual briefFull-size diagram
A task card split into five bands, instruction, environment, reference solution, tests and time limit, each annotated with the company record that most resembles it.
Terminal-Bench's task parts, as described in its paper, mapped to everyday operations records. The mapping is our illustration.

What can Terminal-Bench-style tasks teach introducers about software workflow evidence?

Mainly this: a task is only useful for evaluation when success can be checked. Terminal-Bench was designed around that idea, and the same test separates strong company records from weak ones.

According to its paper, Terminal-Bench 2.0 is a set of 89 hard tasks in computer terminal environments, inspired by problems from real workflows. Each task has its own environment, a human-written solution and tests for verification, and the authors report that frontier models and agents score below 65%. The paper was published in the ICLR 2026 proceedings, and the project site at tbench.ai lists versions 2.0 and 1.0 as active, with 1.0 described as 80 tasks.

The paper describes each task as five parts: an instruction, a Docker image, a set of tests, an example solution, and a time limit. You do not need to know what a Docker image is to use the lens. It is simply a saved, reproducible copy of the computer setup the task runs in.

For a wider survey of how agents are measured, see our guide to AI agent benchmarks.

The five task parts, translated into company records

The mapping below is our illustration, not part of the benchmark. It shows which ordinary operations records share each part's role.

Task partIts role in the benchmarkCompany record with a similar roleQuestion to ask the company
InstructionStates the job in plain wordsChange request, ticket or runbook triggerAre requests written down before work starts?
EnvironmentFixes the system the task runs onInfrastructure-as-code, configuration history, environment inventoriesCan you tell which configuration a past change applied to?
Reference solutionShows one correct way to do itRunbooks, change records listing the steps, scripts in version controlDo engineers record the steps they actually took?
TestsDecide pass or fail without a humanCI results, post-change checks, monitoring alerts that clearedIs success confirmed by a check, or by someone saying it looks fine?
Time limitBounds the attemptMaintenance windows, service-level timers, change start and end timesAre start and finish times recorded?

The strongest companies keep all five and link them by a change or ticket number. Many keep three. The test column is usually the deciding one: a change with a recorded verification check is evidence; a change marked "done" with no check is a claim.

Two adjacent record types are covered elsewhere: infrastructure-as-code history for the environment part, and CI build failure logs for the tests part.

A change record read through the lens (Illustrative)

A fictional 260-person hosting company renews expiring TLS certificates on its own internal load balancers every quarter. One quarter's record looks like this:

  1. Instruction. A change ticket: "Renew certificates expiring within 30 days on internal load balancers," raised automatically by a monitor.
  2. Environment. The ticket names the load balancer group, and the configuration is held in version control at a known commit.
  3. Reference solution. The engineer follows the runbook and pastes the commands run, plus one deviation: a legacy host needed a manual restart.
  4. Tests. A post-change check confirms each endpoint serves the new certificate, and the expiry monitor clears.
  5. Time limit. The change ran inside an approved two-hour window, with start and end times logged.

That record has the anatomy of a benchmark task, including the kind of detail that is hard to invent: the deviation, and how the engineer handled it. It concerns the company's own systems, which keeps ownership simple. Records that describe a client's environment belong to the client.

Company qualification questions

Ask these at metadata level only. You never need to see a ticket or a command.

  • Which systems hold change requests, runbooks and configuration history, and how many years back?
  • Roughly what share of changes record a verification step?
  • Are records about the company's own systems, or its clients'?
  • Have the records ever been published, for example in public repositories or blog posts?
  • Who can authorize a licensing discussion?

The company itself needs 50 or more full-time employees at peak (contractors excluded), a documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. Check who qualifies for the full list.

What a reward could look like (Illustrative)

Rewards are a share of the compensation SourceX actually pays the company you referred (the Actual Supplier Payout): on a company's first qualifying transaction, the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap, subject to the program terms. SourceX pays rewards separately; they never reduce the company's payout. Two illustrative cases, neither a forecast nor a quote:

Illustrative Actual Supplier PayoutReward on a first qualifying transaction (2026 terms)
$250,000$62,500 (USD 50,000 plus 5%)
$800,000$120,000 (15%)

A further qualifying transaction with the same company, first agreed within 12 months of the first Closing, pays 15% of its Actual Supplier Payout; the USD 50,000 floor applies once per referred company.

A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details. For more scenarios, see how much you can make from referrals.

Limits and unanswered questions

  • A company record is not a benchmark task. Benchmarks are curated, cleaned and tested by researchers. Company records are messier, and whether they suit a buyer is decided in SourceX's review.
  • Public exposure matters. Material that has appeared online may already be in training data, the problem described in our note on benchmark contamination.
  • Security-sensitive content. Commands and configurations can reveal infrastructure details; what is excluded is the company's decision in its rights review.
  • No numbers yet on how often companies qualify. SourceX has not published conversion figures for this category, and this page does not estimate one.

More guides are in the guides hub.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If an engineering or operations leader tells you their changes carry a request, a recorded procedure and a verification check, register, confirm the company on who qualifies, and read the terms before introducing the owner.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does SourceX build benchmarks like Terminal-Bench?

No. SourceX licenses approved business data directly from companies for AI training and evaluation, and manages the agreement, rights review, data preparation, delivery and payout. Benchmarks are built by researchers and AI developers.

Does a company need command-line records to be interesting?

No. Terminal-Bench is one lens. The same pattern of a stated task, a known environment, a worked solution and a check appears in finance, support and operations records too.

Why does it matter that the records are private?

Public material can leak into model training and blur evaluation results, a problem known as benchmark contamination. Records that never appeared on the public web avoid that particular issue, though they still need rights review.

Sources

  1. arxiv.org/abs/2601.11868
  2. tbench.ai/benchmarks
  3. proceedings.iclr.cc/paper_files/paper/2026/hash/444a3737adaee10d…
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment