# Rubric failure review packet contents | SourceX

Canonical: https://refer.sourcex.si/resources/rubric-failure-review-packet
Last updated: 2026-11-04

A rubric failure review packet holds eight metadata-only documents, in this order: a task specification, a rubric version register, a case index by criterion score, an attachment storage note, a map of later evidence, an exact outcome check, a volume and span summary, and a rights note. Together they show SourceX whether a company can pair a graded case with proof that the grade was wrong. The AP manager usually leads. No invoices, scorecards or vendor documents go in before an agreement.

This list belongs to [rubric failure](/guides/concept-rubric-failure). It follows the introducer's work: the [rubric failure scorecard](/questions/does-rubric-failure-data-qualify) decides whether to introduce, and the [rubric failure evidence gaps analysis](/questions/rubric-failure-evidence-gaps) explains which weaknesses the packet should state. Partners do not build or handle packets.

## Contents at a glance

| Order | Document | What it shows | Prepared by |
|---|---|---|---|
| 1 | Task specification | What correct approval requires, in full | AP manager |
| 2 | Rubric version register | Each checklist version, its criteria and its dates | QA lead |
| 3 | Case index by criterion score | How graded cases are logged and how many exist | QA lead |
| 4 | Attachment storage note | Where supporting documents live and how they join to invoices | ERP or document system administrator |
| 5 | Later-evidence map | Where credit memos, disputes and audit findings are recorded | AP manager with purchasing |
| 6 | Outcome check | The exact test a correct approval passes, fixed before review | Controller |
| 7 | Volume and span | Counts, years and system changes | AP manager |
| 8 | Rights note | Vendor terms, identifiers present and the authorizer | CFO or counsel |

## Documents 1 to 3: the task, the rule and the graded cases

**1. Task specification (illustrative).** "Approve vendor invoices for payment only for goods received in usable condition, at PO price, after reviewing every attached document. Where any document shows a shortage or damage, short-pay and request a credit memo." It is written broader than the checklist on purpose, so the gap between them is visible.

**2. Rubric version register.** One line per version: effective date, criteria list, reason for change. In the main guide's example, version 1 had four criteria and a later version adds a fifth: "each attachment is listed and any quantity or condition note is reconciled to the payment amount." The date of that change is the most useful fact in the packet, because every case graded before it is a candidate.

**3. Case index by criterion score.** How sampled invoices are logged (per criterion, or pass or fail only), how many are sampled per month, and how far back the log goes. The index is described, not exported.

## Documents 4 and 5: where the contradiction lives

**4. Attachment storage note.** Whether supporting documents received with each invoice are stored with the invoice record or in a separate inbox or drive, and what key joins them. A system match, which Microsoft's documentation describes as comparing invoice price with PO price and invoiced quantity with receipt quantity, can pass even when a receiving note shows a shortfall. Say plainly where such notes would be found.

**5. Later-evidence map.** Every place a passed invoice can later be shown wrong: credit memos, vendor statements, disputes, internal audit findings, returns. For each, state whether it references the original invoice number. Unlinked credits are common and worth naming.

## Document 6: the exact outcome check

The check is written by whoever evaluates the work, before cases are examined. For the main guide's fictional invoice of 40 units at $1,150, of which 3 arrived unusable:

| Condition | Correct value | Passes? |
|---|---|---|
| Amount paid equals price times usable quantity | $1,150 times 37, or $42,550 | Required |
| Credit memo requested for the shortfall | $3,450 for 3 units | Required |
| Approval at the full invoice amount | $46,000 | Fails, whatever its rubric score |
| Revised checklist includes the attachment criterion | Fifth criterion present | Shows the gap was closed |

The rubric gave the $46,000 approval 4 of 4; the outcome check fails it. That disagreement is the case. AI developers audit their own rubrics for the same kind of hole: the "Rubrics as Rewards" paper uses checklist rubrics as a training reward, and Appen's agentic AI page (checked 2026-10-18) lists testing task environments for ambiguous success criteria among its agent data services. SourceX does not write checks for AI developers; it licenses approved business records directly from companies and manages the agreement, rights review, data preparation, delivery and payout.

## Documents 7 and 8: volume, span and rights

**7. Volume and span.** Invoices per year, QA sample rate, number of checklist revisions, years of supporting documents retained, and the date of any ERP or AP system change. Ranges are fine.

**8. Rights note.** Vendor names, pricing, signatures and sometimes employee names appear on supporting documents. List any vendor agreements that restrict reuse of their documents, and name who can authorize a licensing discussion. How such clauses are read is the subject of [MCP for M&A contract review](/resources/mcp/mcp-manda-contract-review); owners asking the basic question can start with [whether a company can license its data to AI companies](/questions/can-a-company-license-its-data-to-ai-companies).

## Illustrative cover sheet

Fictional, for a 350-person industrial fasteners distributor.

| Field | Entry |
|---|---|
| Task | Approve vendor invoices after reviewing all attachments; short-pay on shortage or damage |
| Rubric versions | Three, dated; document-review criterion added in the most recent |
| Case log | About 120 invoices sampled monthly, scored per criterion for four years |
| Supporting documents | Stored with invoices in the AP system since its rollout |
| Later evidence | Credit memos reference invoice numbers; disputes tracked in email |
| Authorizer | CEO (the owner), with the CFO preparing the case |

Why real graded cases differ from expert-written examples is covered in [expert data versus real-world data](/compare/expert-data-vs-real-world-data-ai). Eligibility still applies: 50 or more full-time employees at peak, documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. Other record families are under [data types](/data-types).

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the AP manager could fill the rubric register and later-evidence map from records already kept, [register as a partner](/register), confirm the company against [who qualifies](/who-qualifies) and read the [terms](/terms), then introduce the CFO and let the company prepare the packet with SourceX.

## Sources

- https://arxiv.org/abs/2507.17746
- https://www.appen.com/agentic-ai
- https://learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/three-way-matching-policies
- https://refer.sourcex.si/terms
