Do a client's human adjudication records qualify? An eight-criterion scorecard for QA and review programs

Short answer

Rate a client's review program on eight criteria: headcount, operating history, rights, location and English share, how often items are judged twice, whether rulings carry a written reason, whether each ruling links to the exact work item, and whether the rubric is versioned with dates. Two are must-haves: the written reason and the work-item link. If both are strong and nothing is weak, the fit is strong. If either is weak, the records are not yet checkable, whatever else looks good.

Visual briefFull-size diagram
A dial gauge on the left and eight criteria on the right; as each criterion is ticked strong or flagged borderline, the needle swings and settles in the strong zone for a fictional software support desk, with the two must-have criteria marked.
Illustrative rating for a fictional 240-person software company's support QA program. A conversation aid, not SourceX's review.

This scorecard belongs to the guide on human adjudication records in an ordinary workflow, which follows one escalated support ticket through two disagreeing QA reviews and a lead's ruling. That guide explains what an adjudication record is. This page is a rating sheet for a 20-minute call with a head of support quality, an operations director or whoever runs the review program, kept to answers rather than files.

The scorecard: eight criteria for review rulings

Each row has three answers. Mark the one closest to what the client says.

CriterionStrongBorderlineWeakWho usually answers
1. Company size50 or more full-time employees at peak, contractors excludedNear 50 and not yet confirmed, or count includes contractorsUnder 50 full-time at peakOwner or HR
2. Operating history and retentionSeveral years of reviews kept, with the items they judgedOne to two years, or a QA tool change cut historyReviews purged after monthsQA lead, IT
3. Rights to licenseThe company owns the reviews and the work itemsClient or vendor terms not yet checkedRecords belong to a clientCOO, counsel
4. Location and English shareBased in the U.S., Canada or the EU, or in another covered country with at least 70% of tickets, reviews and comments in EnglishCovered country outside the U.S., Canada and the EU; English share unknownNot in a covered country, or mostly another languageSupport director
5. Double-review depthCalibration rounds, disputes or a double-review sample every month, with a rough countOccasional calibration sessionsNo item is judged twiceQA lead
6. Written reason (must-have)Rulings carry a comment that is normally filled inComments exist but are often blankDecided in a meeting, nothing keptQA lead
7. Work-item linkage (must-have)Each review carries the ticket, claim or invoice ID, and that item's history is keptIDs present, but item history partly purgedReviews in a spreadsheet with no IDsQA lead, help desk admin
8. Rubric versioningRubric or policy has version numbers and change datesOne document with some dated editsEdited in place, no historyQuality manager

Criterion 4 uses the program's location rule. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

The rule that decides strong, borderline or weak

Rather than adding points, read the two must-haves first.

ResultRuleWhat it means for an introduction
Strong fitCriteria 6 and 7 strong, nothing weakIntroduce the COO or head of support
Borderline fitCriteria 6 and 7 at least borderline, one or two others weak or borderlineWorth introducing with the weak criteria named plainly
Weak fitCriterion 6 or 7 weak, or criterion 1, 3 or 4 weakHold; the rulings cannot yet be checked or licensed

Why these two? A ruling is useful to an outsider because it records what a careful senior person decided when two competent reviewers disagreed, and why. Without the reason, it only shows who won. Without the link to the ticket or claim, nobody can check the reason against what actually happened. Zendesk's developer documentation describes ticket audits as a read-only history of every update to a ticket, which is the kind of trail that lets a reviewer check a ruling against timestamps rather than take it on trust.

QA tools make criterion 6 easier than owners expect. In Zendesk QA, a dispute asks for a second opinion, a comment is mandatory, and new scores can be proposed; the person responding can accept, partially accept or reject it with comments. A team using a tool like this may already have reasons on every disputed review.

Scoring three review programs (Illustrative)

All three are fictional, and every rating came from a conversation.

Criterion240-person software support desk520-person outsourced contact center, South Africa130-person insurance claims administrator
1. Company sizeStrongStrongStrong
2. History and retentionStrong: four yearsBorderline: QA tool changed last yearBorderline: two years
3. RightsStrongWeak: reviews judge clients' customersStrong
4. English shareStrong (U.S.)Strong: English-language accountsStrong (Canada)
5. Double-review depthStrong: about 120 disputes a monthStrong: weekly calibrationBorderline: quarterly calibration
6. Written reasonStrong: dispute comments requiredStrongBorderline: often blank
7. Work-item linkageStrong: ticket ID on each reviewStrongWeak: reviews in a spreadsheet
8. Rubric versioningBorderline: some dated editsStrongWeak
ResultStrong fitWeak fit until client rights are confirmedWeak fit: rulings cannot be tied to claims

The contact center has the best review practice of the three and still cannot be introduced as-is, because its rights row is weak. The claims administrator may hold useful review data of another kind; the guide on assessing human review records and correction workflows covers that wider case.

Borderline answers worth a follow-up question

Some borderline marks resolve with one more question. Ask these before deciding.

  • "Often blank" comments. "Are comments blank on routine reviews, or on disputed ones too?" Blank comments on agreed reviews matter much less than blank comments on rulings.
  • A QA tool change. "When you moved tools, did you keep the old reviews with their ticket IDs?" A migration that kept IDs is a short gap, not a cut.
  • Untested client terms. "Who signs your client contracts, and do they mention reuse of quality data?" The answer tells you who the introduction needs to include.
  • Partly dated rubric. "When the rubric changes, is the old version kept anywhere?" An archived file with dates can be enough.

Teams whose main asset is call scores and calibration statistics, rather than individual rulings, are better served by the page on quality scoring disagreements.

Who owns the answers, and what scoring cannot show

The QA lead knows criteria 5 to 8. The help desk administrator knows retention and IDs. The COO or CEO owns the rights question, and the CEO can authorize a licensing discussion. Appen's writing on human-in-the-loop design lists multi-judgment tasks and adjudication steps as parts of a review workflow, so the vocabulary will be familiar to most quality teams.

The scorecard cannot show how many rulings pass an acceptance check, how employee and customer names will be de-identified, whether a QA vendor's terms limit use, or whether any buyer wants this task type. SourceX measures those in review. Partners never request exports, screenshots or sample reviews; discovery stays at metadata level: which QA tool, roughly how many rulings a month, how far back, and who owns them. For what the company's team prepares once a review starts, see the human adjudication records review packet. Related reading: human-in-the-loop data, human feedback and review records, and the company qualification checklist.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client's rulings carry written reasons and ticket IDs, and nothing on the scorecard is weak, register as a partner, check who qualifies, and read the terms before introducing the head of support or COO.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Why are only two criteria must-haves?

Because without a written reason and a link to the judged item, a ruling cannot be checked by anyone outside the team. Weakness elsewhere, such as a short history, reduces how much is usable but does not make the rulings uncheckable.

Does an outsourced contact center qualify?

It can, if it owns the review records or its client contracts allow licensing. Often the reviews judge the client's customers, so the rights criterion is where an outsourcer most often scores weak.

Should I ask for the QA tool's export settings?

No. Ask whether original scores survive a dispute, and let the QA lead answer from experience. Settings and exports are checked in SourceX's review under an agreement.

Sources

  1. support.zendesk.com/hc/en-us/articles/7043760048922-Disputing-a-…
  2. support.zendesk.com/hc/en-us/articles/9642557697306-Responding-t…
  3. developer.zendesk.com/api-reference/ticketing/tickets/ticket_audits
  4. appen.com/blog/human-in-the-loop
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment