Rubric failure evidence gaps: the four missing records that make a wrong pass look clean

Short answer

Rubric failure is shown by contrast: the checklist said pass, and something later said fail. Four gaps remove one side of that contrast: the checklist version in force was not dated, the attachments the checker should have opened were not kept, the procedure that defined the checker's duty is unknown, and the later evidence, such as a credit memo, never links back to the case. Each makes a wrong pass look clean. All four can be checked by asking about storage and versioning, never by collecting documents.

Visual briefFull-size diagram
Paired bars for five evidence items at a fictional plumbing supply wholesaler: each pair shows the full expected bar beside the kept bar, with checklist versions 35%, per-criterion scores 60%, attachments 50%, procedure in force 25% and linked credit memos 40%; the missing share is outlined in orange.
Illustrative gap profile for a fictional 280-person plumbing supply wholesaler. Percentages are invented for this page.

This analysis is part of rubric failure, which walks through one invoice approval that scored perfectly and was still wrong. Here the job is to set what a review would expect beside what companies usually keep, and to describe the shortfall honestly.

The contrast a review has to rebuild

A reviewer needs three things side by side: the rule the case was graded against, the full work item the rule should have covered, and later proof that the pass was wrong. Losing any one leaves an ordinary graded case.

EvidenceSide of the contrastWhat it proves
Checklist version, datedThe ruleWhich criteria applied when the case was graded
Per-criterion scores for the caseThe rule appliedThat the case passed every criterion it was asked
The invoice with every supporting documentThe full taskWhat the checker could have seen
Procedure in force, with rolesThe checker's dutyWhether reading every document was expected
Credit memo or dispute linked to the invoiceThe later failThat the pass was wrong, and what correct looked like

This mirrors how AI developers audit their own rubrics. The "Rubrics as Rewards" paper trains models against checklist-style rubrics, and micro1's description of its flow platform (checked 2026-10-18) includes a QC step that looks for contradictions between a task and its rubric. A missing criterion is the thing those checks try to find; a company archive that keeps all five rows shows one found in the wild.

Expected against usually kept

The pattern below reflects common AP practice. It is not a statistic.

GapWhat the reviewer sees insteadUsually keptUsually missingMetadata question
Undated rubricA score with no stated criteriaThe current checklistEarlier versions and their dates"Are checklist revisions dated?"
Attachments droppedHeader data that matches perfectlyInvoice, PO and receipt in the ERPEmailed supporting documents such as receiving notes"Are attachments kept with the invoice record?"
Procedure unknownAn unexplained individual missToday's AP manualThe version that applied at the time"Is the AP procedure manual versioned?"
Outcome unlinkedA case that never failedCredit memos as separate transactionsA reference back to the original invoice"Do credit memos reference the original invoice?"

The document gap deserves a note. Microsoft's documentation describes three-way matching as comparing invoice price with PO price and invoiced quantity with receipt quantity. If the goods receipt was keyed at full quantity, the match passes and the system record looks clean. Only a supporting document outside the match, such as a receiving note, can show the shortfall.

Versioning sometimes depends on settings. Where a checklist or approval setup lives inside an ERP, a change log can date its edits: Microsoft Learn describes Business Central's change log as recording the user, time, and old and new values for tables and fields a company chooses to log. If it was never activated, the dates must come from the AP team's own records.

Illustrative gap profile: a 280-person plumbing supply wholesaler

Fictional company, invented figures. Each percentage is the share of QA-sampled invoices from the last three years for which that evidence exists, as an AP manager might estimate it.

EvidenceShare keptWhat is missingWhat it hides
Checklist versions, dated35%Revisions before the QA lead started dating themWhich rule graded older cases
Per-criterion scores60%Earlier samples logged as pass or fail onlyWhich criterion a case satisfied
Attachments with invoice50%Emailed documents before the AP inbox was connected to the ERPThe document that contradicts the pass
Procedure in forcethe greater of $50,000 plus 5% and 15%Superseded AP manuals not archivedWhether the miss was a rule gap or an individual slip
Linked credit memos40%Credits booked against vendor statements, not invoicesThe later fail

The procedure row is lowest, but the document and credit memo rows decide usability. A case without its contradicting document or its later fail is a pass and nothing more. The cases worth describing are the recent ones where all five rows exist together.

How each gap changes the answer

  • An undated rubric mixes cases graded under different rules. A case that passed before a new criterion was added looks the same as one that passed after it.
  • Dropped supporting documents erase the requirement that was missed. An approval stripped of its contradicting document reads as a correct payment.
  • An unknown procedure turns a rule gap into apparent human error, which is a different lesson for anyone studying the case.
  • An unlinked outcome leaves the case looking like a clean pass forever, even if a credit was eventually requested from the vendor.

Failure records are valuable for exactly this reason, as the guide to failure data in AI training explains. Warranty departments face a similar linkage problem between the original sale and the later claim; see warranty claims data.

Wording the gap in an introduction

One sentence per gap, naming the complete window. For the fictional wholesaler:

  • "QA checklists are dated since the current QA lead started; earlier versions are not."
  • "Supporting documents are stored with invoices since the AP inbox was connected to the ERP."
  • "Credit memos reference the original invoice only for some vendors."

Avoid "we check everything" (the rubric is the point, and it was incomplete), "our error rate is low" (not the question) and "all documents are in the system" (emailed documents usually say otherwise). Stay with metadata: system names, years, counts, owners.

Eligibility still comes first: 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The rubric failure scorecard turns these answers into a fit rating, and the rubric failure review packet shows what the company would prepare next. Tracing a summary figure back to source documents is also the theme of MCP for CFO dashboards. More record families are under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the AP manager can name a window where dated checklists, supporting documents and linked credit memos all survive, register as a partner, check the company against who qualifies and read the terms, then introduce the CFO with the gaps stated plainly.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Is a missing checklist version fatal to an introduction?

No. It narrows which cases can be used. Cases graded after the first dated revision may still be complete; the introduction should name that window.

Why do attachments go missing so often?

Many AP teams store invoice header data in the ERP and leave emailed documents in inboxes or shared drives. The match result survives; the document that contradicts it does not.

Should the introducer check whether credit memos link to invoices?

Only by asking. The AP manager can say whether credit memos carry the original invoice number. Nobody should pull records to prove it.

Sources

  1. learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/t…
  2. arxiv.org/abs/2507.17746
  3. micro1.ai/blog/introducing-flow
  4. learn.microsoft.com/en-us/dynamics365/business-central/across-lo…
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment