This page is part of the guide to long-horizon tasks and AI agents. Its job is a gap analysis: what a review would expect, what companies typically keep, where the evidence usually runs out, and how to describe that honestly in an introduction. For a case where every link held, see the explainer on long-horizon task data.
The evidence a review expects from one long case
Long-horizon work matters to AI developers because agents are being asked to complete longer sequences of dependent steps. METR measures this as a time horizon: the length of task, timed by how long skilled people take, that a model completes about half the time, and its research reports that the horizon has grown steadily across model releases (METR). The METR task-length guide explains what that trend does and does not show. What it implies for records is simple: a long case is useful only if each step can be checked against the steps before it.
For a procurement dispute, a review would expect six kinds of evidence:
- Request reason. Why the purchase was raised: the need, the trigger and any constraint such as a deadline.
- Approval limit in force. Who approved, and whether that person was allowed to approve that amount under the rules on that date. Approval routing often depends on limits; Microsoft's documentation describes a requisition workflow that sends a line to a higher manager only when its cost exceeds the department manager's limit (Microsoft Learn).
- Order revisions. Each version of the purchase order that a later step relied on, not only the latest one. Some systems keep these; confirming a purchase order in Dynamics 365 stores a journal copy, and each reconfirmation adds another (Microsoft Learn).
- Goods receipt. What actually arrived, when, and against which order line.
- Settlement outcome. How the dispute ended: a credit note, a replacement, a write-off or a price adjustment.
- Shared case ID. One identifier, or a reliable set of keys, that joins all of the above across systems.
Expected versus available: one fictional company's estimate
Illustrative and fictional. A 410-person food manufacturer receives a shipment of packaging film that fails a seal test. The case runs eleven weeks: a buyer raises a rush request for replacement film, a plant director approves it, the supplier ships a partial order against a revised purchase order, and accounts payable settles a credit for the failed batch. The partner asks the finance lead, at metadata level only, how many of every ten disputes like this keep each piece of evidence. The finance lead gives rough estimates.
| Evidence expected | Where it would live | Kept in about | Usually missing because |
|---|---|---|---|
| Request reason | Request form, chat, email | 4 of 10 cases | The reason was discussed in chat that has since been deleted |
| Approval limit in force | Finance policy file | 3 of 10 cases | The limit table was edited in place when limits changed |
| Order revisions | ERP purchasing journals | 5 of 10 cases | The previous system kept only the current version |
| Goods receipt | Warehouse receiving log | 8 of 10 cases | Partial deliveries were sometimes received against the wrong line |
| Settlement outcome | Accounts payable, supplier statement | 4 of 10 cases | Credits were agreed by email and netted off later invoices |
| Shared case ID | Carried across all of the above | 6 of 10 cases | Requests and supplier claims used separate numbering |
Read the table by row, not as an average. The goods receipt is usually there; the approval limit in force is usually not. That pattern is common because operational systems keep what they need to run today, and policy documents get overwritten when the policy changes.
Why each gap weakens more than its own step
In a short task, a missing record removes one data point. In a long case, it removes the basis for judging everything that depended on it.
| Missing evidence | The check a reviewer can no longer make | Steps affected downstream |
|---|---|---|
| Request reason | Was the rush order necessary, and was the quantity right? | Approval, order and settlement all inherit the doubt |
| Approval limit in force | Was the approval valid on that date? | Every action taken on the strength of that approval |
| Order revisions | Did the shipment match the instruction in force when it shipped? | Receipt matching and the size of the credit |
| Settlement outcome | Did the sequence end in a correct result? | The whole case, since success cannot be confirmed |
| Shared case ID | Do these records belong to the same case at all? | All steps, which must be matched by date and amount |
This is why the share of complete cases matters more than the share of records kept. Using the estimates above, a case needs all six pieces to be fully assessable, and the finance lead may only be able to say that "a minority" of disputes are complete. That is an honest answer and a useful one.
How to describe the gaps honestly in an introduction
The aim is a description the owner would sign: accurate, at metadata level, and neither rounded up nor talked down.
| Tempting phrasing | Honest phrasing |
|---|---|
| "They have complete procurement histories." | "Receipts and final orders are kept for most disputes; reasons and settlement are kept for fewer than half, by the finance lead's estimate." |
| "Ten years of approval data." | "Approvals are recorded for ten years; the limit table in force at the time survives for about three in ten cases." |
| "Every case links end to end." | "About six in ten cases share an ID from request to settlement; the rest would need matching." |
| "Gaps can be fixed." | "Some settlement detail may exist in email and supplier statements; the company has not checked." |
Credit each estimate to the person who gave it and the date, and note the systems named. Do not attach samples, screenshots or exports, and do not offer to count cases yourself.
What the partner records and what waits for SourceX
Before introducing, the partner confirms only what the owner can state:
- Records that exist: the systems named (purchasing, receiving, accounts payable, email), the years covered and the owner's estimate of completeness for each evidence type.
- Who owns them: the company, subject to any supplier or customer contract limits the owner flags.
- Who can authorize a licensing discussion: the CEO, not the system administrator; the CFO or chief operating officer can sponsor the introduction.
- Eligibility: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.
What stays unknown until SourceX reviews it: how many cases are actually complete, whether gaps can be filled from other systems, whether the records fit a dataset need, the rights review result and any offer. Interest in long-horizon data across the AI industry does not mean every procurement archive qualifies. For how complete cases become training examples, see task and outcome pairs; for a wider screen of purchasing records, see assessing procurement approvals and exception workflows; other record types are listed under data types.
Next step
Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
If the owner can describe its gaps as plainly as the table above and wants to explore a license, register as a partner, confirm the company on who qualifies, and read the program terms before you introduce the decision-maker.