# Rubric failure data fit scorecard | SourceX

Canonical: https://refer.sourcex.si/questions/does-rubric-failure-data-qualify
Last updated: 2026-11-04

Rate eight things as strong, borderline or weak: headcount, operating history, rights, whether a written QA checklist exists with dated versions, whether graded cases keep a score per criterion, whether supporting documents stay with each case, whether later evidence such as credit memos links back to the case, and who can authorize a discussion. A company fits well when it holds both the checklist and the cases it got wrong. Ask the controller or AP manager; never request records.

This scorecard belongs to [rubric failure](/guides/concept-rubric-failure), which follows an invoice that scored 4 of 4 while a damage note sat in an unopened attachment. Here the task is deciding whether a specific company is worth introducing for this kind of record, and how confident to be.

## How the rating works

Count strong answers out of eight. Borderline answers do not count but are written down, because they become the open items in the introduction. Three criteria are gates: if size, rights or the authorized contact is weak, stop and resolve that first.

| Strong answers | Zone | What it means |
|---|---|---|
| 6 to 8 | Strong | Checklist, graded cases and later contradictions likely all survive; introduce with borderline items listed |
| 3 to 5 | Borderline | The rubric exists, but the miss may not be provable; introduce only with gaps named |
| 0 to 2 | Weak | The company has quality habits, not documented rubric failures; wait |

The reason the rubric itself counts so heavily: AI developers increasingly grade model work with checklists. The "Rubrics as Rewards" paper, by Scale AI authors and presented at ICLR 2026, uses checklist-style rubrics as the training reward, and Appen's agentic AI page (checked 2026-10-18) lists testing task environments for reward hacking opportunities and ambiguous success criteria among its agent data services. A company that can show a human checklist with a missing requirement, and the case that exposed it, holds exactly the failure those audits look for.

## Eight criteria, applied to QA checklists

The generic screen is in the [company qualification checklist](/resources/company-qualification-checklist). This version applies it to rubric failure.

| Criterion | Strong | Borderline | Weak | Who answers |
|---|---|---|---|---|
| Company size (gate) | Well over 50 full-time employees at peak | Near 50 | Under 50 full-time, or mostly contractors | CEO or HR |
| Operating history | Several years in one AP or ERP system | Years split by a system change | Recently formed | Controller |
| Rights (gate) | Company owns its records; vendor terms known | Some vendor contracts unreviewed | A parent or customer controls the records | CFO or counsel |
| Written rubric, dated | Named QA form with dated revisions | Form exists, revisions undated | "Managers spot check" | AP manager |
| Score per criterion | Each case scored criterion by criterion | Pass or fail only | Monthly pass rate only | QA lead |
| Documents kept with case | Attachments stored with the invoice record | Stored in a shared drive by date | Not kept | ERP administrator |
| Later evidence linked | Credit memos and disputes reference the original invoice | Linkable by vendor and date | Not recorded | AP manager |
| Authorized contact (gate) | Named executive over the records | Senior contact, authority unconfirmed | Nobody identified | CEO |

A note on what "strong" looks like for documents: Microsoft's documentation describes three-way matching as comparing invoice price with PO price and invoiced quantity with receipt quantity. A system can pass that match perfectly when the receipt itself was wrong, which is why the attachments, not the match result, carry the evidence.

On history, the IRS generally ties record retention to the period of limitations, three years in most cases and longer for some claims, so many companies keep AP records for at least that long. Ask what the actual policy is rather than assuming.

Outside the U.S., Canada and the EU, add English share as a gate. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

## Three Illustrative companies, rated

All three are fictional.

| Criterion | 380-person electrical distributor | 150-person dental supply wholesaler | 90-person apparel importer, Singapore |
|---|---|---|---|
| Company size | Strong | Strong | Strong |
| Operating history | Strong | Strong | Borderline |
| Rights | Borderline | Strong | Weak |
| Written rubric, dated | Strong | Borderline | Weak |
| Score per criterion | Strong | Weak | Weak |
| Documents kept with case | Strong | Strong | Borderline |
| Later evidence linked | Borderline | Borderline | Weak |
| Authorized contact | Strong | Strong | Borderline |
| Strong answers | 6 of 8, strong | 5 of 8, borderline | 1 of 8, weak; rights gate failed |

**The electrical distributor** revised its AP checklist twice after vendor disputes and scores every sampled invoice by criterion. Rights are borderline because two large vendor agreements have not been read for data-use terms; credit memos reference invoices only since a recent system upgrade.

**The dental supply wholesaler** keeps attachments well, but its QA log records pass or fail only, so a reviewer cannot see which criterion a case satisfied.

**The apparel importer** records invoices in English and Mandarin, so the 70% English rule is in doubt, and its parent company owns the AP system.

## Five questions that move a rating

Ask in this order and listen for form names, dates and counts.

1. **"Does your team grade its own work with a written checklist or QA scorecard?"** Moves the rubric row. A vague "managers spot check" means there is no rubric to study.
2. **"Has the checklist been changed after something slipped through?"** A dated revision, such as an attachments line added after a vendor dispute, is direct evidence of a rubric failure found and fixed.
3. **"When a passed item later turns out wrong, how do you find out?"** Listen for credit memos, vendor disputes or audit findings tied back to the original case number. Moves the linkage row.
4. **"Are the source documents the checker should have looked at stored with the case?"** If attachments were never saved, the missing requirement cannot be shown.
5. **"Who could approve a conversation about licensing these records?"** Moves both gates; usually the CEO, or the owner in an owner-run firm.

Clear answers to questions 2 and 3 are the strongest signal. A company with scorecards but no record of the cases they got wrong has the rubric but not the failure. The [questions to ask before licensing company data](/resources/questions-to-ask-before-licensing-company-data) cover the remaining basics.

## What the rating cannot settle

The rating records what the company says. It cannot show how many cases pass a fixed outcome check, how vendor names and signatures would be de-identified, whether vendor contracts permit use, or whether any buyer wants this task. SourceX settles those in its review with the company. The [rubric failure evidence gaps analysis](/questions/rubric-failure-evidence-gaps) shows how to word borderline rows, and the [rubric failure review packet](/resources/rubric-failure-review-packet) lists what the company prepares later. For why scorecards matter in general, see [rubric-based evaluation](/guides/rubric-based-evaluation-ai); the surrounding AP records are described under [accounts payable exception handling](/data-types/accounts-payable-exception-handling-process). More record families are under [data types](/data-types).

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the company reaches the strong or borderline zone with all three gates passed, [register as a partner](/register), check it against [who qualifies](/who-qualifies) and read the [terms](/terms) before you introduce the CFO.

## Sources

- https://arxiv.org/abs/2507.17746
- https://www.appen.com/agentic-ai
- https://learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/three-way-matching-policies
- https://www.irs.gov/businesses/small-businesses-self-employed/how-long-should-i-keep-records
- https://refer.sourcex.si/terms
