What does rubric failure mean in an ordinary business workflow?
It means the scoring rule and the real task have drifted apart. Everything the rubric checks can be correct while the outcome is wrong, because the outcome depended on something the rubric never looked at.
AI developers increasingly grade model work with rubrics. A Scale AI paper presented at ICLR 2026, "Rubrics as Rewards", trains models using checklist-style rubrics as the reward signal, and treats a single pass or fail check as the simplest case of a rubric. If the checklist leaves out a requirement, a model trained against it has no reason to meet that requirement. This is why vendors describe auditing rubrics themselves: Appen's whitepaper on agent environments says each rubric is checked for atomicity, adversarial exploitation, scoring consistency and coverage mapping, and micro1's flow includes a QC step that looks for contradictions between a task and its rubric. Companies already own the human version of this problem, in their own QA checklists. The live guide on rubrics and QA scorecards as evaluation data explains why scorecards matter in general; this page is about the specific case where a scorecard was incomplete.
The coverage grid for one invoice (Illustrative)
Illustrative. A 210-person building materials distributor receives an invoice for 40 pallets of floor tile at $1,150 per pallet, $46,000 in total. The invoice email carries three attachments: a packing slip, a freight memo, and a delivery receipt signed by the warehouse lead with a handwritten note that 3 pallets arrived damaged. The goods receipt had already been posted in the ERP at 40 pallets before the note was read. All details are fictional.
| Requirement of the real task | Checked by the AP quality rubric? | Result for this invoice |
|---|---|---|
| Invoice references a valid PO | Yes | Pass |
| Unit price matches the PO | Yes | Pass |
| Invoiced quantity matches the posted goods receipt | Yes | Pass, because the receipt said 40 |
| Approver is within their approval limit | Yes | Pass |
| Every attachment was opened and reconciled to the invoice | No | Not checked: damage note on the delivery receipt missed |
Score: 4 of 4. Correct outcome: pay $42,550 for 37 pallets and request a credit memo for 3, or $3,450. The clerk paid $46,000. The ERP's own matching did its job: Microsoft's documentation describes three-way matching as comparing invoice price with PO price and invoiced quantity with the product receipt quantity. The match passed because the receipt was wrong, and only the attachment showed it.
Evidence fields that expose the missing criterion
A rubric failure is invisible in the scorecard itself, because the scorecard is the thing that was incomplete. It shows up only when the graded case can be laid next to later evidence. For the tile invoice, these are the fields that make that possible.
| Field | What it shows | In the example (Illustrative) |
|---|---|---|
| Rubric version and criteria list | What the check claimed to cover at the time | Four criteria: PO, price, quantity, approver |
| Score per criterion for the case | That the case passed everything it was asked | 4 of 4 |
| The work item with all attachments | What the real task contained | Invoice email with packing slip, freight memo and signed delivery receipt |
| Posted goods receipt | What the system believed | 40 pallets received |
| Later contradicting evidence | That the pass was wrong | Damage note for 3 pallets; vendor credit request |
| Corrected outcome | What should have happened | $42,550 payable, $3,450 credit |
| Rubric revision, dated | That the gap was recognized and closed | Fifth criterion added: every attachment opened and reconciled |
The last row is the strongest signal. A dated change to the checklist, made after a missed case, is the company's own admission that the old rubric left something out, and it lets a reviewer find every case graded under the old version.
What can make the gap impossible to see
- Attachments stored apart from invoices. If scans sit in a shared drive with no invoice number, the contradicting evidence cannot be joined to the case.
- Scorecards kept only as totals. A monthly pass rate without per-case, per-criterion scores cannot show which criterion was missing.
- Corrections booked as new transactions. A credit memo that does not reference the original invoice breaks the link between the wrong pass and its fix.
- Undated checklist edits. Without a version date, nobody can tell which rule a past case was graded against.
How many cases survive these problems is measured in SourceX's review with the company, not from one example.
Does the company qualify, and what stays unknown?
The introduced company needs 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license the records, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.
Stay at metadata level: which AP or ERP system (plain names such as SAP or Dynamics 365), how many invoices per year, how many years of attachments, whether QA scorecards and their revisions exist, and who owns the records. Do not collect invoices, scorecards or screenshots. Vendor documents can carry third-party terms, so legal review may narrow what is licensable, much as the M&A contract review guide describes for agreements in a data room. What stays unknown until SourceX reviews the opportunity: SourceX's current data needs, the rights review result, de-identification needs, the company's offer and any reward. For the owner's first question, see whether a company can license its data to AI companies; other record families are under data types.
Go deeper on rubric failure
- Questions to ask a client about rubric failure: a discovery list for an AP manager or finance lead, kept at metadata level.
- Record gaps that make rubric failure hard to assess: the storage and versioning problems that most often hide an incomplete checklist, and how to spot them early.
- Rubric failure review packet: what an evidence packet for SourceX's review should contain, item by item.
Next step
Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
If a client can point to a checklist revision made after a missed case, and still holds the case, register as a partner, confirm the basics on who qualifies, and read the program terms before introducing its finance lead.