Rubric failure: when an invoice check scores 4 out of 4 and misses the damage note

Short answer

Rubric failure is when the checklist used to grade work leaves out part of what the work actually requires, so wrong work can score as perfect. A business example: an accounts payable quality check confirms the PO, price, quantity and approver, but never asks whether every invoice attachment was opened, so an approval that missed a damage note passes. For an introducer, the question is whether a company kept its checklists, the graded cases and the later evidence that exposed the gap, and who can authorize a license discussion.

Visual briefFull-size diagram
A coverage grid lists five requirements of an invoice approval; the rubric checks four of them and passes the invoice, while the unchecked requirement, inspecting all attachments, hides a damage note that should have triggered a short payment.
Illustrative: a rubric that covers four of five requirements scores a wrong approval as perfect.

What does rubric failure mean in an ordinary business workflow?

It means the scoring rule and the real task have drifted apart. Everything the rubric checks can be correct while the outcome is wrong, because the outcome depended on something the rubric never looked at.

AI developers increasingly grade model work with rubrics. A Scale AI paper presented at ICLR 2026, "Rubrics as Rewards", trains models using checklist-style rubrics as the reward signal, and treats a single pass or fail check as the simplest case of a rubric. If the checklist leaves out a requirement, a model trained against it has no reason to meet that requirement. This is why vendors describe auditing rubrics themselves: Appen's whitepaper on agent environments says each rubric is checked for atomicity, adversarial exploitation, scoring consistency and coverage mapping, and micro1's flow includes a QC step that looks for contradictions between a task and its rubric. Companies already own the human version of this problem, in their own QA checklists. The live guide on rubrics and QA scorecards as evaluation data explains why scorecards matter in general; this page is about the specific case where a scorecard was incomplete.

The coverage grid for one invoice (Illustrative)

Illustrative. A 210-person building materials distributor receives an invoice for 40 pallets of floor tile at $1,150 per pallet, $46,000 in total. The invoice email carries three attachments: a packing slip, a freight memo, and a delivery receipt signed by the warehouse lead with a handwritten note that 3 pallets arrived damaged. The goods receipt had already been posted in the ERP at 40 pallets before the note was read. All details are fictional.

Requirement of the real taskChecked by the AP quality rubric?Result for this invoice
Invoice references a valid POYesPass
Unit price matches the POYesPass
Invoiced quantity matches the posted goods receiptYesPass, because the receipt said 40
Approver is within their approval limitYesPass
Every attachment was opened and reconciled to the invoiceNoNot checked: damage note on the delivery receipt missed

Score: 4 of 4. Correct outcome: pay $42,550 for 37 pallets and request a credit memo for 3, or $3,450. The clerk paid $46,000. The ERP's own matching did its job: Microsoft's documentation describes three-way matching as comparing invoice price with PO price and invoiced quantity with the product receipt quantity. The match passed because the receipt was wrong, and only the attachment showed it.

Evidence fields that expose the missing criterion

A rubric failure is invisible in the scorecard itself, because the scorecard is the thing that was incomplete. It shows up only when the graded case can be laid next to later evidence. For the tile invoice, these are the fields that make that possible.

FieldWhat it showsIn the example (Illustrative)
Rubric version and criteria listWhat the check claimed to cover at the timeFour criteria: PO, price, quantity, approver
Score per criterion for the caseThat the case passed everything it was asked4 of 4
The work item with all attachmentsWhat the real task containedInvoice email with packing slip, freight memo and signed delivery receipt
Posted goods receiptWhat the system believed40 pallets received
Later contradicting evidenceThat the pass was wrongDamage note for 3 pallets; vendor credit request
Corrected outcomeWhat should have happened$42,550 payable, $3,450 credit
Rubric revision, datedThat the gap was recognized and closedFifth criterion added: every attachment opened and reconciled

The last row is the strongest signal. A dated change to the checklist, made after a missed case, is the company's own admission that the old rubric left something out, and it lets a reviewer find every case graded under the old version.

What can make the gap impossible to see

  • Attachments stored apart from invoices. If scans sit in a shared drive with no invoice number, the contradicting evidence cannot be joined to the case.
  • Scorecards kept only as totals. A monthly pass rate without per-case, per-criterion scores cannot show which criterion was missing.
  • Corrections booked as new transactions. A credit memo that does not reference the original invoice breaks the link between the wrong pass and its fix.
  • Undated checklist edits. Without a version date, nobody can tell which rule a past case was graded against.

How many cases survive these problems is measured in SourceX's review with the company, not from one example.

Does the company qualify, and what stays unknown?

The introduced company needs 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license the records, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Stay at metadata level: which AP or ERP system (plain names such as SAP or Dynamics 365), how many invoices per year, how many years of attachments, whether QA scorecards and their revisions exist, and who owns the records. Do not collect invoices, scorecards or screenshots. Vendor documents can carry third-party terms, so legal review may narrow what is licensable, much as the M&A contract review guide describes for agreements in a data room. What stays unknown until SourceX reviews the opportunity: SourceX's current data needs, the rights review result, de-identification needs, the company's offer and any reward. For the owner's first question, see whether a company can license its data to AI companies; other record families are under data types.

Go deeper on rubric failure

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client can point to a checklist revision made after a missed case, and still holds the case, register as a partner, confirm the basics on who qualifies, and read the program terms before introducing its finance lead.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

How is rubric failure different from reward hacking?

Rubric failure is a flaw in the check: it leaves out a requirement. Reward hacking is behavior that exploits a flaw like that to score well. A missing criterion can produce wrong passes even when nobody is trying to game anything, which is the case on this page.

Does the company need its own QA scorecards to qualify?

Scorecards help, because they show what the company thought it was checking. But the stronger evidence is the gap: cases the scorecard passed that a later step, such as a vendor dispute or an audit, showed were wrong.

Are invoice attachments too sensitive to license?

They can contain vendor names, pricing and personal signatures, so rights and de-identification are reviewed case by case. That review is SourceX's job under its agreement with the company; the partner only notes that attachments exist and where they are stored.

Sources

  1. refer.sourcex.si/terms
  2. arxiv.org/abs/2507.17746
  3. micro1.ai/blog/introducing-flow
  4. appen.com/whitepapers/reinforcement-learning-environme…
  5. learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/t…

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment