Rubric failure review packet: eight metadata documents an AP team prepares, in order

Short answer

A rubric failure review packet holds eight metadata-only documents, in this order: a task specification, a rubric version register, a case index by criterion score, an attachment storage note, a map of later evidence, an exact outcome check, a volume and span summary, and a rights note. Together they show SourceX whether a company can pair a graded case with proof that the grade was wrong. The AP manager usually leads. No invoices, scorecards or vendor documents go in before an agreement.

Visual briefFull-size diagram
Eight labelled cards drop from above into an open folder in order: task specification, rubric register, case index, attachment note, later-evidence map, outcome check, volume and span, and rights note; the outcome check card shows $42,550 payable and a $3,450 credit.
Illustrative packet order for an accounts payable team. Figures come from the fictional invoice on the main guide.

This list belongs to rubric failure. It follows the introducer's work: the rubric failure scorecard decides whether to introduce, and the rubric failure evidence gaps analysis explains which weaknesses the packet should state. Partners do not build or handle packets.

Contents at a glance

OrderDocumentWhat it showsPrepared by
1Task specificationWhat correct approval requires, in fullAP manager
2Rubric version registerEach checklist version, its criteria and its datesQA lead
3Case index by criterion scoreHow graded cases are logged and how many existQA lead
4Attachment storage noteWhere supporting documents live and how they join to invoicesERP or document system administrator
5Later-evidence mapWhere credit memos, disputes and audit findings are recordedAP manager with purchasing
6Outcome checkThe exact test a correct approval passes, fixed before reviewController
7Volume and spanCounts, years and system changesAP manager
8Rights noteVendor terms, identifiers present and the authorizerCFO or counsel

Documents 1 to 3: the task, the rule and the graded cases

1. Task specification (illustrative). "Approve vendor invoices for payment only for goods received in usable condition, at PO price, after reviewing every attached document. Where any document shows a shortage or damage, short-pay and request a credit memo." It is written broader than the checklist on purpose, so the gap between them is visible.

2. Rubric version register. One line per version: effective date, criteria list, reason for change. In the main guide's example, version 1 had four criteria and a later version adds a fifth: "each attachment is listed and any quantity or condition note is reconciled to the payment amount." The date of that change is the most useful fact in the packet, because every case graded before it is a candidate.

3. Case index by criterion score. How sampled invoices are logged (per criterion, or pass or fail only), how many are sampled per month, and how far back the log goes. The index is described, not exported.

Documents 4 and 5: where the contradiction lives

4. Attachment storage note. Whether supporting documents received with each invoice are stored with the invoice record or in a separate inbox or drive, and what key joins them. A system match, which Microsoft's documentation describes as comparing invoice price with PO price and invoiced quantity with receipt quantity, can pass even when a receiving note shows a shortfall. Say plainly where such notes would be found.

5. Later-evidence map. Every place a passed invoice can later be shown wrong: credit memos, vendor statements, disputes, internal audit findings, returns. For each, state whether it references the original invoice number. Unlinked credits are common and worth naming.

Document 6: the exact outcome check

The check is written by whoever evaluates the work, before cases are examined. For the main guide's fictional invoice of 40 units at $1,150, of which 3 arrived unusable:

ConditionCorrect valuePasses?
Amount paid equals price times usable quantity$1,150 times 37, or $42,550Required
Credit memo requested for the shortfall$3,450 for 3 unitsRequired
Approval at the full invoice amount$46,000Fails, whatever its rubric score
Revised checklist includes the attachment criterionFifth criterion presentShows the gap was closed

The rubric gave the $46,000 approval 4 of 4; the outcome check fails it. That disagreement is the case. AI developers audit their own rubrics for the same kind of hole: the "Rubrics as Rewards" paper uses checklist rubrics as a training reward, and Appen's agentic AI page (checked 2026-10-18) lists testing task environments for ambiguous success criteria among its agent data services. SourceX does not write checks for AI developers; it licenses approved business records directly from companies and manages the agreement, rights review, data preparation, delivery and payout.

Documents 7 and 8: volume, span and rights

7. Volume and span. Invoices per year, QA sample rate, number of checklist revisions, years of supporting documents retained, and the date of any ERP or AP system change. Ranges are fine.

8. Rights note. Vendor names, pricing, signatures and sometimes employee names appear on supporting documents. List any vendor agreements that restrict reuse of their documents, and name who can authorize a licensing discussion. How such clauses are read is the subject of MCP for M&A contract review; owners asking the basic question can start with whether a company can license its data to AI companies.

Illustrative cover sheet

Fictional, for a 350-person industrial fasteners distributor.

FieldEntry
TaskApprove vendor invoices after reviewing all attachments; short-pay on shortage or damage
Rubric versionsThree, dated; document-review criterion added in the most recent
Case logAbout 120 invoices sampled monthly, scored per criterion for four years
Supporting documentsStored with invoices in the AP system since its rollout
Later evidenceCredit memos reference invoice numbers; disputes tracked in email
AuthorizerCEO (the owner), with the CFO preparing the case

Why real graded cases differ from expert-written examples is covered in expert data versus real-world data. Eligibility still applies: 50 or more full-time employees at peak, documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. Other record families are under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the AP manager could fill the rubric register and later-evidence map from records already kept, register as a partner, confirm the company against who qualifies and read the terms, then introduce the CFO and let the company prepare the packet with SourceX.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Who inside the company usually leads the packet?

The AP manager or the QA lead, because the checklist, its revisions and the graded cases sit with them. The ERP administrator supplies storage facts, and the CFO or counsel prepares the rights note, which names the CEO as the person who can authorize licensing.

Does the company have to write a corrected rubric?

No. If the company already revised its checklist after a miss, the packet records that revision and its date. Writing checks for model training is the evaluator's work, not the company's.

Is the introducer involved in preparing it?

No. The introducer's part ends with the introduction. Showing the owner this list helps them see the effort before deciding.

Sources

  1. arxiv.org/abs/2507.17746
  2. appen.com/agentic-ai
  3. learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/t…
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment