# Rubric failure evidence gaps analysis | SourceX

Canonical: https://refer.sourcex.si/questions/rubric-failure-evidence-gaps
Last updated: 2026-11-04

Rubric failure is shown by contrast: the checklist said pass, and something later said fail. Four gaps remove one side of that contrast: the checklist version in force was not dated, the attachments the checker should have opened were not kept, the procedure that defined the checker's duty is unknown, and the later evidence, such as a credit memo, never links back to the case. Each makes a wrong pass look clean. All four can be checked by asking about storage and versioning, never by collecting documents.

This analysis is part of [rubric failure](/guides/concept-rubric-failure), which walks through one invoice approval that scored perfectly and was still wrong. Here the job is to set what a review would expect beside what companies usually keep, and to describe the shortfall honestly.

## The contrast a review has to rebuild

A reviewer needs three things side by side: the rule the case was graded against, the full work item the rule should have covered, and later proof that the pass was wrong. Losing any one leaves an ordinary graded case.

| Evidence | Side of the contrast | What it proves |
|---|---|---|
| Checklist version, dated | The rule | Which criteria applied when the case was graded |
| Per-criterion scores for the case | The rule applied | That the case passed every criterion it was asked |
| The invoice with every supporting document | The full task | What the checker could have seen |
| Procedure in force, with roles | The checker's duty | Whether reading every document was expected |
| Credit memo or dispute linked to the invoice | The later fail | That the pass was wrong, and what correct looked like |

This mirrors how AI developers audit their own rubrics. The "Rubrics as Rewards" paper trains models against checklist-style rubrics, and micro1's description of its flow platform (checked 2026-10-18) includes a QC step that looks for contradictions between a task and its rubric. A missing criterion is the thing those checks try to find; a company archive that keeps all five rows shows one found in the wild.

## Expected against usually kept

The pattern below reflects common AP practice. It is not a statistic.

| Gap | What the reviewer sees instead | Usually kept | Usually missing | Metadata question |
|---|---|---|---|---|
| Undated rubric | A score with no stated criteria | The current checklist | Earlier versions and their dates | "Are checklist revisions dated?" |
| Attachments dropped | Header data that matches perfectly | Invoice, PO and receipt in the ERP | Emailed supporting documents such as receiving notes | "Are attachments kept with the invoice record?" |
| Procedure unknown | An unexplained individual miss | Today's AP manual | The version that applied at the time | "Is the AP procedure manual versioned?" |
| Outcome unlinked | A case that never failed | Credit memos as separate transactions | A reference back to the original invoice | "Do credit memos reference the original invoice?" |

The document gap deserves a note. Microsoft's documentation describes three-way matching as comparing invoice price with PO price and invoiced quantity with receipt quantity. If the goods receipt was keyed at full quantity, the match passes and the system record looks clean. Only a supporting document outside the match, such as a receiving note, can show the shortfall.

Versioning sometimes depends on settings. Where a checklist or approval setup lives inside an ERP, a change log can date its edits: Microsoft Learn describes Business Central's change log as recording the user, time, and old and new values for tables and fields a company chooses to log. If it was never activated, the dates must come from the AP team's own records.

## Illustrative gap profile: a 280-person plumbing supply wholesaler

Fictional company, invented figures. Each percentage is the share of QA-sampled invoices from the last three years for which that evidence exists, as an AP manager might estimate it.

| Evidence | Share kept | What is missing | What it hides |
|---|---|---|---|
| Checklist versions, dated | 35% | Revisions before the QA lead started dating them | Which rule graded older cases |
| Per-criterion scores | 60% | Earlier samples logged as pass or fail only | Which criterion a case satisfied |
| Attachments with invoice | 50% | Emailed documents before the AP inbox was connected to the ERP | The document that contradicts the pass |
| Procedure in force | the greater of $50,000 plus 5% and 15% | Superseded AP manuals not archived | Whether the miss was a rule gap or an individual slip |
| Linked credit memos | 40% | Credits booked against vendor statements, not invoices | The later fail |

The procedure row is lowest, but the document and credit memo rows decide usability. A case without its contradicting document or its later fail is a pass and nothing more. The cases worth describing are the recent ones where all five rows exist together.

## How each gap changes the answer

- **An undated rubric** mixes cases graded under different rules. A case that passed before a new criterion was added looks the same as one that passed after it.
- **Dropped supporting documents** erase the requirement that was missed. An approval stripped of its contradicting document reads as a correct payment.
- **An unknown procedure** turns a rule gap into apparent human error, which is a different lesson for anyone studying the case.
- **An unlinked outcome** leaves the case looking like a clean pass forever, even if a credit was eventually requested from the vendor.

Failure records are valuable for exactly this reason, as the guide to [failure data in AI training](/guides/failure-data-ai-training) explains. Warranty departments face a similar linkage problem between the original sale and the later claim; see [warranty claims data](/data-types/warranty-claims-data).

## Wording the gap in an introduction

One sentence per gap, naming the complete window. For the fictional wholesaler:

- "QA checklists are dated since the current QA lead started; earlier versions are not."
- "Supporting documents are stored with invoices since the AP inbox was connected to the ERP."
- "Credit memos reference the original invoice only for some vendors."

Avoid "we check everything" (the rubric is the point, and it was incomplete), "our error rate is low" (not the question) and "all documents are in the system" (emailed documents usually say otherwise). Stay with metadata: system names, years, counts, owners.

Eligibility still comes first: 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The [rubric failure scorecard](/questions/does-rubric-failure-data-qualify) turns these answers into a fit rating, and the [rubric failure review packet](/resources/rubric-failure-review-packet) shows what the company would prepare next. Tracing a summary figure back to source documents is also the theme of [MCP for CFO dashboards](/resources/mcp/mcp-cfo-dashboard). More record families are under [data types](/data-types).

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the AP manager can name a window where dated checklists, supporting documents and linked credit memos all survive, [register as a partner](/register), check the company against [who qualifies](/who-qualifies) and read the [terms](/terms), then introduce the CFO with the gaps stated plainly.

## Sources

- https://learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/three-way-matching-policies
- https://arxiv.org/abs/2507.17746
- https://www.micro1.ai/blog/introducing-flow
- https://learn.microsoft.com/en-us/dynamics365/business-central/across-log-changes
- https://refer.sourcex.si/terms
