This scorecard belongs to the guide on long-horizon task data in an ordinary workflow, which traces one replacement-parts order across 41 days and four systems. That guide shows what a long case is and which fields hold it together. This page turns that into a rating an introducer can complete in one call with an operations or finance lead, using answers rather than files.
Company criteria: size, history, rights, location
These decide whether the company can be introduced at all. A weak mark on any of them stops the rating.
| Criterion | Strong | Borderline | Weak |
|---|---|---|---|
| Size | 50 or more full-time employees at peak, contractors excluded | Near 50, headcount unconfirmed | Clearly under 50 full-time |
| Operating history | Documented history across several years of trading | A recent merger or carve-out complicates the record | No documented history |
| Rights to license | A named executive could authorize a discussion; no known restriction on supplier records | A named executive, but supplier terms unreviewed | Records controlled by a parent or customer |
| Location and English share | U.S., Canada or EU company, or another covered country with at least 70% of records in English | Covered country, mixed languages, share unknown | Not in a covered country, or mostly another language outside the U.S., Canada and the EU |
Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.
Case criteria: depth, reasons and outcome
These decide whether the records hold whole cases with something to check at the end.
| Criterion | Strong | Borderline | Weak |
|---|---|---|---|
| Record depth | Five or more years of complete cases, closed and disputed ones included | Two to four years, or only open cases survived a migration | Under two years |
| Exception reasons | Match exceptions, returns and rejections carry a written cause in the system | Reasons live in email | Exceptions fixed with no note |
| Recorded outcome | Settlement, credit and replacement are all recorded and dated | Settlement recorded; replacements raised separately | Nobody can say how a case ended |
Exception reasons matter more than they seem. ERP matching catches the symptom automatically: Microsoft's documentation describes three-way matching as comparing invoiced quantity with received quantity and treating a difference as a matching error. What the match cannot record is why the quantities differ. That explanation is what turns a flag into a step a reviewer can follow.
The trace test: count the joins in one case
This is the heart of the scorecard. Ask the lead to picture an ordinary order that went wrong, then walk through eight steps and ask, for each link, "Which number on the later record points back to the earlier one?" Count a join only if the answer names a reference the system stores, not one a person would have to look up.
| Join | From step | To step | Reference that confirms it |
|---|---|---|---|
| 1 | Work order | Requisition | Work order number on the requisition |
| 2 | Requisition | Approval | Requisition number in the approval log |
| 3 | Approval | Purchase order | Requisition number on the PO |
| 4 | Purchase order | Shipment and receipt | PO number on receipts and inspection results |
| 5 | Receipt | Return | Receipt or PO number on the return authorization |
| 6 | Purchase order | Invoice and match exception | PO number on the invoice |
| 7 | Exception and return | Settlement and credit | Invoice or return number on the payment and credit memo |
Read the count this way:
| Joins confirmed | Fit | What it usually means |
|---|---|---|
| 6 or 7 | Strong | A reviewer could follow a case end to end |
| 4 or 5 | Borderline | Cases exist, but one or two joins need matching by hand |
| 0 to 3 | Weak | The records are many short fragments, not long cases |
The result is the lower of the trace test and the weakest criterion above. A company with seven joins and a weak rights mark is not a strong fit.
Why so strict? Long cases are where AI agents still struggle. METR measures capability as the length of tasks, in human expert time, that agents complete half the time, and reports that this length has been doubling roughly every seven months; the background is in our explainer on METR's task-length measure. A long task can only be checked end to end if every join between its steps holds.
Three manufacturers rated (Illustrative)
All three companies are fictional, rated from a conversation.
| Criterion | 320-person food manufacturer | 450-person packaging converter | 180-person machine shop |
|---|---|---|---|
| Company criteria | All strong | All strong | Rights borderline: supplier terms unreviewed |
| Record depth | Strong: seven years on one ERP | Borderline: ERP cutover three years ago kept open items only | Strong: six years |
| Exception reasons | Strong | Strong | Weak: fixed by phone |
| Recorded outcome | Strong | Strong | Borderline |
| Trace test | 6 of 7: credits quote the invoice but not the return number | 7 of 7 | 3 of 7: returns by email, quality log keyed by date |
| Result | Strong | Borderline: limited by record depth | Weak |
The food manufacturer's missing join is worth naming in the introduction rather than hiding: credits can still be matched to returns through the invoice, but not directly. The converter has perfect joins and a short usable history, which only SourceX's review can weigh.
What a rating cannot settle
The scorecard decides whether to make an introduction. It cannot tell anyone how many cases survive a full review, whether supplier contracts limit use of pricing, what de-identification is needed, or whether any buyer wants this task type. Those are measured by SourceX with the company. Partners stay at metadata level: which systems hold each step, roughly how many orders a year, how many years, whether returns and credits carry references, and who owns the records. Operations or finance leads know the systems; the CFO or owner decides on a licensing discussion.
For what the company's team prepares once a review starts, see the long-horizon task data review packet. The record family is covered more broadly in procurement and vendor workflows and the guide to assessing procurement approvals and exception workflows; long-horizon tasks and project records gives a project-based example. Before any of this, the company qualification checklist and the questions to ask before licensing company data still apply.
Next step
Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
If a client's operations lead confirms six or more joins on the trace test and no criterion rates weak, register as a partner, check who qualifies, and read the terms before introducing the CFO or owner.