Long-horizon task data evidence gaps: expected versus available records in one procurement dispute

Short answer

Long-horizon task data is hard to assess when a case loses the evidence that ties later steps to earlier ones. In a procurement dispute that runs from request to approval, shipment and settlement, the usual gaps are the reason for the request, the approval limit in force that day, the order revisions, and the final settlement, plus a shared case ID that joins them. Each gap leaves a reviewer unable to judge part of the sequence. None rules a company out; each narrows what a review can use.

Visual briefFull-size diagram
Six horizontal bars, each out of ten cases, compare expected and available evidence at a fictional food manufacturer: request reason 4, approval limit in force 3, order revisions 5, goods receipt 8, settlement outcome 4 and shared case ID 6, with the missing segments outlined in orange.
Illustrative gap analysis for a fictional 410-person food manufacturer, based on an owner's rough estimates. Figures are invented for this page.

This page is part of the guide to long-horizon tasks and AI agents. Its job is a gap analysis: what a review would expect, what companies typically keep, where the evidence usually runs out, and how to describe that honestly in an introduction. For a case where every link held, see the explainer on long-horizon task data.

The evidence a review expects from one long case

Long-horizon work matters to AI developers because agents are being asked to complete longer sequences of dependent steps. METR measures this as a time horizon: the length of task, timed by how long skilled people take, that a model completes about half the time, and its research reports that the horizon has grown steadily across model releases (METR). The METR task-length guide explains what that trend does and does not show. What it implies for records is simple: a long case is useful only if each step can be checked against the steps before it.

For a procurement dispute, a review would expect six kinds of evidence:

  1. Request reason. Why the purchase was raised: the need, the trigger and any constraint such as a deadline.
  2. Approval limit in force. Who approved, and whether that person was allowed to approve that amount under the rules on that date. Approval routing often depends on limits; Microsoft's documentation describes a requisition workflow that sends a line to a higher manager only when its cost exceeds the department manager's limit (Microsoft Learn).
  3. Order revisions. Each version of the purchase order that a later step relied on, not only the latest one. Some systems keep these; confirming a purchase order in Dynamics 365 stores a journal copy, and each reconfirmation adds another (Microsoft Learn).
  4. Goods receipt. What actually arrived, when, and against which order line.
  5. Settlement outcome. How the dispute ended: a credit note, a replacement, a write-off or a price adjustment.
  6. Shared case ID. One identifier, or a reliable set of keys, that joins all of the above across systems.

Expected versus available: one fictional company's estimate

Illustrative and fictional. A 410-person food manufacturer receives a shipment of packaging film that fails a seal test. The case runs eleven weeks: a buyer raises a rush request for replacement film, a plant director approves it, the supplier ships a partial order against a revised purchase order, and accounts payable settles a credit for the failed batch. The partner asks the finance lead, at metadata level only, how many of every ten disputes like this keep each piece of evidence. The finance lead gives rough estimates.

Evidence expectedWhere it would liveKept in aboutUsually missing because
Request reasonRequest form, chat, email4 of 10 casesThe reason was discussed in chat that has since been deleted
Approval limit in forceFinance policy file3 of 10 casesThe limit table was edited in place when limits changed
Order revisionsERP purchasing journals5 of 10 casesThe previous system kept only the current version
Goods receiptWarehouse receiving log8 of 10 casesPartial deliveries were sometimes received against the wrong line
Settlement outcomeAccounts payable, supplier statement4 of 10 casesCredits were agreed by email and netted off later invoices
Shared case IDCarried across all of the above6 of 10 casesRequests and supplier claims used separate numbering

Read the table by row, not as an average. The goods receipt is usually there; the approval limit in force is usually not. That pattern is common because operational systems keep what they need to run today, and policy documents get overwritten when the policy changes.

Why each gap weakens more than its own step

In a short task, a missing record removes one data point. In a long case, it removes the basis for judging everything that depended on it.

Missing evidenceThe check a reviewer can no longer makeSteps affected downstream
Request reasonWas the rush order necessary, and was the quantity right?Approval, order and settlement all inherit the doubt
Approval limit in forceWas the approval valid on that date?Every action taken on the strength of that approval
Order revisionsDid the shipment match the instruction in force when it shipped?Receipt matching and the size of the credit
Settlement outcomeDid the sequence end in a correct result?The whole case, since success cannot be confirmed
Shared case IDDo these records belong to the same case at all?All steps, which must be matched by date and amount

This is why the share of complete cases matters more than the share of records kept. Using the estimates above, a case needs all six pieces to be fully assessable, and the finance lead may only be able to say that "a minority" of disputes are complete. That is an honest answer and a useful one.

How to describe the gaps honestly in an introduction

The aim is a description the owner would sign: accurate, at metadata level, and neither rounded up nor talked down.

Tempting phrasingHonest phrasing
"They have complete procurement histories.""Receipts and final orders are kept for most disputes; reasons and settlement are kept for fewer than half, by the finance lead's estimate."
"Ten years of approval data.""Approvals are recorded for ten years; the limit table in force at the time survives for about three in ten cases."
"Every case links end to end.""About six in ten cases share an ID from request to settlement; the rest would need matching."
"Gaps can be fixed.""Some settlement detail may exist in email and supplier statements; the company has not checked."

Credit each estimate to the person who gave it and the date, and note the systems named. Do not attach samples, screenshots or exports, and do not offer to count cases yourself.

What the partner records and what waits for SourceX

Before introducing, the partner confirms only what the owner can state:

  • Records that exist: the systems named (purchasing, receiving, accounts payable, email), the years covered and the owner's estimate of completeness for each evidence type.
  • Who owns them: the company, subject to any supplier or customer contract limits the owner flags.
  • Who can authorize a licensing discussion: the CEO, not the system administrator; the CFO or chief operating officer can sponsor the introduction.
  • Eligibility: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

What stays unknown until SourceX reviews it: how many cases are actually complete, whether gaps can be filled from other systems, whether the records fit a dataset need, the rights review result and any offer. Interest in long-horizon data across the AI industry does not mean every procurement archive qualifies. For how complete cases become training examples, see task and outcome pairs; for a wider screen of purchasing records, see assessing procurement approvals and exception workflows; other record types are listed under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the owner can describe its gaps as plainly as the table above and wants to explore a license, register as a partner, confirm the company on who qualifies, and read the program terms before you introduce the decision-maker.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Should the introducer count cases with gaps by opening the purchasing system?

No. The introducer records the owner's estimate and who gave it. Any counting against real records happens inside the company, under its own controls and any agreement it later signs with SourceX.

Is a company with mostly incomplete cases still worth introducing?

Often, yes. A minority of complete cases across many years can still be a meaningful set, and some gaps can be filled from email, finance or supplier records. State the share honestly and let SourceX's review decide.

Why does the shared case ID matter so much?

Without one identifier carried from request to settlement, each step has to be matched by date, supplier and amount. That is slower and less certain, so a review can trust fewer of the cases.

Sources

  1. arxiv.org/abs/2503.14499
  2. learn.microsoft.com/en-us/dynamics365/supply-chain/procurement/p…
  3. learn.microsoft.com/en-us/dynamics365/supply-chain/procurement/p…
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment