# Human adjudication records fit scorecard | SourceX

Canonical: https://refer.sourcex.si/questions/does-human-adjudication-records-data-qualify
Last updated: 2026-11-04

Rate a client's review program on eight criteria: headcount, operating history, rights, location and English share, how often items are judged twice, whether rulings carry a written reason, whether each ruling links to the exact work item, and whether the rubric is versioned with dates. Two are must-haves: the written reason and the work-item link. If both are strong and nothing is weak, the fit is strong. If either is weak, the records are not yet checkable, whatever else looks good.

This scorecard belongs to the guide on [human adjudication records in an ordinary workflow](/guides/concept-human-adjudication-records), which follows one escalated support ticket through two disagreeing QA reviews and a lead's ruling. That guide explains what an adjudication record is. This page is a rating sheet for a 20-minute call with a head of support quality, an operations director or whoever runs the review program, kept to answers rather than files.

## The scorecard: eight criteria for review rulings

Each row has three answers. Mark the one closest to what the client says.

| Criterion | Strong | Borderline | Weak | Who usually answers |
|---|---|---|---|---|
| 1. Company size | 50 or more full-time employees at peak, contractors excluded | Near 50 and not yet confirmed, or count includes contractors | Under 50 full-time at peak | Owner or HR |
| 2. Operating history and retention | Several years of reviews kept, with the items they judged | One to two years, or a QA tool change cut history | Reviews purged after months | QA lead, IT |
| 3. Rights to license | The company owns the reviews and the work items | Client or vendor terms not yet checked | Records belong to a client | COO, counsel |
| 4. Location and English share | Based in the U.S., Canada or the EU, or in another covered country with at least 70% of tickets, reviews and comments in English | Covered country outside the U.S., Canada and the EU; English share unknown | Not in a covered country, or mostly another language | Support director |
| 5. Double-review depth | Calibration rounds, disputes or a double-review sample every month, with a rough count | Occasional calibration sessions | No item is judged twice | QA lead |
| 6. Written reason (must-have) | Rulings carry a comment that is normally filled in | Comments exist but are often blank | Decided in a meeting, nothing kept | QA lead |
| 7. Work-item linkage (must-have) | Each review carries the ticket, claim or invoice ID, and that item's history is kept | IDs present, but item history partly purged | Reviews in a spreadsheet with no IDs | QA lead, help desk admin |
| 8. Rubric versioning | Rubric or policy has version numbers and change dates | One document with some dated edits | Edited in place, no history | Quality manager |

Criterion 4 uses the program's location rule. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

## The rule that decides strong, borderline or weak

Rather than adding points, read the two must-haves first.

| Result | Rule | What it means for an introduction |
|---|---|---|
| Strong fit | Criteria 6 and 7 strong, nothing weak | Introduce the COO or head of support |
| Borderline fit | Criteria 6 and 7 at least borderline, one or two others weak or borderline | Worth introducing with the weak criteria named plainly |
| Weak fit | Criterion 6 or 7 weak, or criterion 1, 3 or 4 weak | Hold; the rulings cannot yet be checked or licensed |

Why these two? A ruling is useful to an outsider because it records what a careful senior person decided when two competent reviewers disagreed, and why. Without the reason, it only shows who won. Without the link to the ticket or claim, nobody can check the reason against what actually happened. Zendesk's developer documentation describes ticket audits as a read-only history of every update to a ticket, which is the kind of trail that lets a reviewer check a ruling against timestamps rather than take it on trust.

QA tools make criterion 6 easier than owners expect. In Zendesk QA, a dispute asks for a second opinion, a comment is mandatory, and new scores can be proposed; the person responding can accept, partially accept or reject it with comments. A team using a tool like this may already have reasons on every disputed review.

## Scoring three review programs (Illustrative)

All three are fictional, and every rating came from a conversation.

| Criterion | 240-person software support desk | 520-person outsourced contact center, South Africa | 130-person insurance claims administrator |
|---|---|---|---|
| 1. Company size | Strong | Strong | Strong |
| 2. History and retention | Strong: four years | Borderline: QA tool changed last year | Borderline: two years |
| 3. Rights | Strong | Weak: reviews judge clients' customers | Strong |
| 4. English share | Strong (U.S.) | Strong: English-language accounts | Strong (Canada) |
| 5. Double-review depth | Strong: about 120 disputes a month | Strong: weekly calibration | Borderline: quarterly calibration |
| 6. Written reason | Strong: dispute comments required | Strong | Borderline: often blank |
| 7. Work-item linkage | Strong: ticket ID on each review | Strong | Weak: reviews in a spreadsheet |
| 8. Rubric versioning | Borderline: some dated edits | Strong | Weak |
| Result | Strong fit | Weak fit until client rights are confirmed | Weak fit: rulings cannot be tied to claims |

The contact center has the best review practice of the three and still cannot be introduced as-is, because its rights row is weak. The claims administrator may hold useful review data of another kind; the guide on [assessing human review records and correction workflows](/guides/how-to-assess-human-review-records-and-correction-workflows) covers that wider case.

## Borderline answers worth a follow-up question

Some borderline marks resolve with one more question. Ask these before deciding.

- **"Often blank" comments.** "Are comments blank on routine reviews, or on disputed ones too?" Blank comments on agreed reviews matter much less than blank comments on rulings.
- **A QA tool change.** "When you moved tools, did you keep the old reviews with their ticket IDs?" A migration that kept IDs is a short gap, not a cut.
- **Untested client terms.** "Who signs your client contracts, and do they mention reuse of quality data?" The answer tells you who the introduction needs to include.
- **Partly dated rubric.** "When the rubric changes, is the old version kept anywhere?" An archived file with dates can be enough.

Teams whose main asset is call scores and calibration statistics, rather than individual rulings, are better served by the page on [quality scoring disagreements](/guides/workflow-quality-scoring-disagreements).

## Who owns the answers, and what scoring cannot show

The QA lead knows criteria 5 to 8. The help desk administrator knows retention and IDs. The COO or CEO owns the rights question, and the CEO can authorize a licensing discussion. Appen's writing on human-in-the-loop design lists multi-judgment tasks and adjudication steps as parts of a review workflow, so the vocabulary will be familiar to most quality teams.

The scorecard cannot show how many rulings pass an acceptance check, how employee and customer names will be de-identified, whether a QA vendor's terms limit use, or whether any buyer wants this task type. SourceX measures those in review. Partners never request exports, screenshots or sample reviews; discovery stays at metadata level: which QA tool, roughly how many rulings a month, how far back, and who owns them. For what the company's team prepares once a review starts, see the [human adjudication records review packet](/resources/human-adjudication-records-review-packet). Related reading: [human-in-the-loop data](/proof/what-is-human-in-the-loop-data), [human feedback and review records](/data-types/human-feedback-and-review), and the [company qualification checklist](/resources/company-qualification-checklist).

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client's rulings carry written reasons and ticket IDs, and nothing on the scorecard is weak, [register as a partner](/register), check [who qualifies](/who-qualifies), and read the [terms](/terms) before introducing the head of support or COO.

## Sources

- https://support.zendesk.com/hc/en-us/articles/7043760048922-Disputing-a-review
- https://support.zendesk.com/hc/en-us/articles/9642557697306-Responding-to-disputes-in-Zendesk-QA
- https://developer.zendesk.com/api-reference/ticketing/tickets/ticket_audits/
- https://www.appen.com/blog/human-in-the-loop
- https://refer.sourcex.si/terms
