Does rubric failure data qualify? A fit scorecard for QA checklists and the cases they missed

Short answer

Rate eight things as strong, borderline or weak: headcount, operating history, rights, whether a written QA checklist exists with dated versions, whether graded cases keep a score per criterion, whether supporting documents stay with each case, whether later evidence such as credit memos links back to the case, and who can authorize a discussion. A company fits well when it holds both the checklist and the cases it got wrong. Ask the controller or AP manager; never request records.

Visual briefFull-size diagram
A semicircular gauge with weak, borderline and strong zones; eight criteria for a fictional 380-person electrical distributor are checked one by one on the left, and the needle settles at 6 of 8 strong answers, in the strong zone.
Illustrative rating for a fictional company. The scorecard organizes an introduction; it does not predict acceptance or a payout.

This scorecard belongs to rubric failure, which follows an invoice that scored 4 of 4 while a damage note sat in an unopened attachment. Here the task is deciding whether a specific company is worth introducing for this kind of record, and how confident to be.

How the rating works

Count strong answers out of eight. Borderline answers do not count but are written down, because they become the open items in the introduction. Three criteria are gates: if size, rights or the authorized contact is weak, stop and resolve that first.

Strong answersZoneWhat it means
6 to 8StrongChecklist, graded cases and later contradictions likely all survive; introduce with borderline items listed
3 to 5BorderlineThe rubric exists, but the miss may not be provable; introduce only with gaps named
0 to 2WeakThe company has quality habits, not documented rubric failures; wait

The reason the rubric itself counts so heavily: AI developers increasingly grade model work with checklists. The "Rubrics as Rewards" paper, by Scale AI authors and presented at ICLR 2026, uses checklist-style rubrics as the training reward, and Appen's agentic AI page (checked 2026-10-18) lists testing task environments for reward hacking opportunities and ambiguous success criteria among its agent data services. A company that can show a human checklist with a missing requirement, and the case that exposed it, holds exactly the failure those audits look for.

Eight criteria, applied to QA checklists

The generic screen is in the company qualification checklist. This version applies it to rubric failure.

CriterionStrongBorderlineWeakWho answers
Company size (gate)Well over 50 full-time employees at peakNear 50Under 50 full-time, or mostly contractorsCEO or HR
Operating historySeveral years in one AP or ERP systemYears split by a system changeRecently formedController
Rights (gate)Company owns its records; vendor terms knownSome vendor contracts unreviewedA parent or customer controls the recordsCFO or counsel
Written rubric, datedNamed QA form with dated revisionsForm exists, revisions undated"Managers spot check"AP manager
Score per criterionEach case scored criterion by criterionPass or fail onlyMonthly pass rate onlyQA lead
Documents kept with caseAttachments stored with the invoice recordStored in a shared drive by dateNot keptERP administrator
Later evidence linkedCredit memos and disputes reference the original invoiceLinkable by vendor and dateNot recordedAP manager
Authorized contact (gate)Named executive over the recordsSenior contact, authority unconfirmedNobody identifiedCEO

A note on what "strong" looks like for documents: Microsoft's documentation describes three-way matching as comparing invoice price with PO price and invoiced quantity with receipt quantity. A system can pass that match perfectly when the receipt itself was wrong, which is why the attachments, not the match result, carry the evidence.

On history, the IRS generally ties record retention to the period of limitations, three years in most cases and longer for some claims, so many companies keep AP records for at least that long. Ask what the actual policy is rather than assuming.

Outside the U.S., Canada and the EU, add English share as a gate. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Three Illustrative companies, rated

All three are fictional.

Criterion380-person electrical distributor150-person dental supply wholesaler90-person apparel importer, Singapore
Company sizeStrongStrongStrong
Operating historyStrongStrongBorderline
RightsBorderlineStrongWeak
Written rubric, datedStrongBorderlineWeak
Score per criterionStrongWeakWeak
Documents kept with caseStrongStrongBorderline
Later evidence linkedBorderlineBorderlineWeak
Authorized contactStrongStrongBorderline
Strong answers6 of 8, strong5 of 8, borderline1 of 8, weak; rights gate failed

The electrical distributor revised its AP checklist twice after vendor disputes and scores every sampled invoice by criterion. Rights are borderline because two large vendor agreements have not been read for data-use terms; credit memos reference invoices only since a recent system upgrade.

The dental supply wholesaler keeps attachments well, but its QA log records pass or fail only, so a reviewer cannot see which criterion a case satisfied.

The apparel importer records invoices in English and Mandarin, so the 70% English rule is in doubt, and its parent company owns the AP system.

Five questions that move a rating

Ask in this order and listen for form names, dates and counts.

  1. "Does your team grade its own work with a written checklist or QA scorecard?" Moves the rubric row. A vague "managers spot check" means there is no rubric to study.
  2. "Has the checklist been changed after something slipped through?" A dated revision, such as an attachments line added after a vendor dispute, is direct evidence of a rubric failure found and fixed.
  3. "When a passed item later turns out wrong, how do you find out?" Listen for credit memos, vendor disputes or audit findings tied back to the original case number. Moves the linkage row.
  4. "Are the source documents the checker should have looked at stored with the case?" If attachments were never saved, the missing requirement cannot be shown.
  5. "Who could approve a conversation about licensing these records?" Moves both gates; usually the CEO, or the owner in an owner-run firm.

Clear answers to questions 2 and 3 are the strongest signal. A company with scorecards but no record of the cases they got wrong has the rubric but not the failure. The questions to ask before licensing company data cover the remaining basics.

What the rating cannot settle

The rating records what the company says. It cannot show how many cases pass a fixed outcome check, how vendor names and signatures would be de-identified, whether vendor contracts permit use, or whether any buyer wants this task. SourceX settles those in its review with the company. The rubric failure evidence gaps analysis shows how to word borderline rows, and the rubric failure review packet lists what the company prepares later. For why scorecards matter in general, see rubric-based evaluation; the surrounding AP records are described under accounts payable exception handling. More record families are under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the company reaches the strong or borderline zone with all three gates passed, register as a partner, check it against who qualifies and read the terms before you introduce the CFO.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does the checklist have to be about invoices?

No. Any written QA checklist works the same way: claims audits, call scorecards, inspection forms. Accounts payable is the clearest case because the later evidence, a credit memo or vendor dispute, is usually recorded somewhere.

Can I ask to see a scorecard so I can rate it properly?

No. Rate every criterion from what the AP manager or controller tells you about forms, counts and retention. Records stay with the company until an agreement exists.

What if the company scores well but vendor contract terms are unknown?

Mark rights as borderline and introduce with that open item named. Vendor and customer terms are reviewed by SourceX with the company; the introducer only notes that they exist.

Sources

  1. arxiv.org/abs/2507.17746
  2. appen.com/agentic-ai
  3. learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/t…
  4. irs.gov/businesses/small-businesses-self-employed/ho…
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment