Human adjudication records: how a support team settles two QA reviewers who disagree about an escalation

Short answer

Human adjudication records are the written trail left when a third person settles a disagreement between two reviewers who judged the same work. In an ordinary support team, that is two QA reviewers scoring one escalated ticket differently, a lead checking the ticket history and ruling, and the reason and rule version written down. For a referral, what matters is whether the company kept both scores, the ticket, the ruling and its reason, and who can authorize a licensing discussion.

Visual briefFull-size diagram
One support ticket's history runs down the middle; Reviewer A's Fail points at the public reply, Reviewer B's Pass points at an internal note at 10:12, and a QA lead's ruling at the bottom settles it as Pass and dates a rubric change from version 3 to version 4.
Illustrative ticket at a fictional software company. The ruling is only checkable because both scores and the ticket history were kept.

What does human adjudication records mean in an ordinary business workflow?

In practice it is the paperwork behind a tie-break. The record is not the disagreement itself. It is the ruling plus everything needed to check the ruling later.

Most companies with a quality program already produce them without using the name. A support team scores a sample of tickets. A claims team double-checks a share of decisions. An accounts payable team has a second person look at exceptions. When two people disagree, someone senior decides, and in a well-run team that decision is written down with a reason.

QA software makes the mechanics visible. In Zendesk QA, a reviewee who disagrees with a review can open a dispute to ask for a second opinion; a comment is required, new scores can be proposed, and the dispute can go to the original reviewer or to another team member. The person answering can accept, partially accept or reject it, with comments. Data vendors describe the same structure on their side: Appen's writing on human-in-the-loop design lists multi-judgment tasks and adjudication steps as parts of a review workflow.

Why does this interest AI developers? Because a ruling encodes what a careful senior person thinks "correct" means when two competent people read the same evidence differently. Our explainer on human-in-the-loop data covers the broader category, and why synthetic data rarely replaces real data explains why such judgment is hard to generate. Interest in the concept does not mean every QA archive qualifies; the questions below decide that.

One escalation, two reviewers, one ruling (Illustrative)

This case is fictional. A 240-person B2B software company runs a Tier 1 support desk. Its escalation policy, version 3, says an agent may escalate to Tier 2 after two documented troubleshooting steps from the knowledge base, or at once if the issue needs back-end access.

A customer reports that invoices have stopped syncing to their accounting tool. The ticket's history looks like this:

TimeTicket eventWho
10:02Ticket created from email: invoices not syncing since yesterdayCustomer
10:07Public reply: first step, customer asked to reconnect the integration; sync still failingTier 1 agent
10:12Internal note: second step, sync queue cleared from the admin view; sync still failingTier 1 agent
10:27Group changed to Tier 2 with tag "escalated"Tier 1 agent
14:40Solved: sync token had expired on the server sideTier 2 engineer

Two QA reviewers score the ticket the following week.

  • Reviewer A, "Valid escalation": Fail. Comment: only one troubleshooting step appears in the replies to the customer, so the escalation came one step early.
  • Reviewer B, "Valid escalation": Pass. Comment: the 10:12 internal note records a second step, so two steps were taken before 10:27.

The QA lead adjudicates. She opens the ticket history, confirms the 10:12 note exists and was written before the group change, and rules Pass. Her written reason: "Policy v3 requires documented steps, not customer-facing ones. Internal notes count." Because two experienced reviewers read the rule differently, she also amends the rubric to version 4, dated, adding the sentence "Steps recorded in internal notes count as documented."

That last move is what makes the record valuable. The ruling settles one ticket, and the dated rubric change shows how the standard itself moved because of it. Zendesk's developer documentation describes ticket audits as a read-only history of all updates to a ticket, which is the kind of trail that lets a third party check the lead's reasoning against the 10:12 timestamp rather than take it on trust.

Evidence fields in one ruling

A ruling is only useful to someone outside the team if each part of it can be found and joined. Here is the escalation above, broken into the fields that carry it.

FieldValue in the example (Illustrative)Where it usually lives
Work itemThe escalated invoice-sync ticket with its full change historyHelp desk ticket and its audit trail
First judgmentReviewer A, "Valid escalation": Fail, with commentQA tool review record
Second judgmentReviewer B, "Valid escalation": Pass, with commentQA tool review record
TriggerScores disagree on the same criterionQA tool dispute or calibration queue
AdjudicatorQA lead, a different person from both reviewersDispute response or calibration notes
Decision and reasonPass: "Internal notes count"Dispute comment or ruling log
Evidence citedInternal note at 10:12, written before the 10:27 group changeTicket audit timestamps
Rule versionEscalation policy v3, amended to v4 with a dateRubric or policy document history

The first three fields describe the disagreement. The next four describe the ruling. The last one dates the standard, so a later reader knows which rule the lead was applying. Without it, the same ticket scored after the amendment would look like a contradiction rather than a change in policy.

Failure and uncertainty checks

Some adjudication archives look complete until someone tries to check a ruling. The common failures:

  • Overwritten scores. If accepting a challenge simply replaces the original score, the disagreement disappears. Zendesk's documentation notes that a partially accepted dispute changes only some scores, and that scores cannot be changed after a rejection without opening a new dispute; ask whether reports or exports keep the before and after values.
  • Rulings without reasons. A final score with an empty comment says who won, not why.
  • Rubric drift. If the rubric was edited without dates, a ruling cannot be matched to the rule it applied.
  • Broken links to the work item. Reviews held in one tool and tickets in another, joined only by a pasted subject line.
  • Archived source items. The reviews survive but the ticket or claim history they judged was purged.
  • Client-owned material. An outsourced contact center may hold reviews of its clients' customers, which it may not have the right to license.

None of these settles the question alone. They change how many rulings are usable, and only SourceX's review with the company can measure that.

Does the company qualify, and what stays unknown?

The company needs 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license its records, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

What stays unknown until SourceX reviews it: how many rulings pass the acceptance check, how employee and customer names are de-identified, whether the QA vendor's or a client's contract limits use, and whether any buyer wants this task type. Partners never request exports, screenshots or sample tickets; discovery stays at metadata level: which QA tool, roughly how many rulings a month, how far back, and who owns them. The company qualification checklist covers the company-level basics.

Go deeper on human adjudication records

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the quality lead confirms that disagreements end in a written ruling tied to the ticket and a dated rubric, register as a partner, check the company against who qualifies, and read the terms before you introduce the COO or head of support.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Is a calibration session the same as adjudication?

Not quite. Calibration measures how far reviewers drift apart across many items; adjudication settles one specific disagreement and records the reason. A team that runs calibration often produces adjudication records along the way, but only if someone writes down the ruling.

Does an accepted dispute in a QA tool count as an adjudication record?

It can, if the original score, the challenge, the decision and the comment all survive. Ask whether the tool or an export keeps the score before the change, not just the final one.

Should I ask to see a ruling as a sample?

No. Partners stay at metadata level: which tool, how many rulings a month, how far back, who owns them. Any review of actual records happens between SourceX and the company under an agreement.

Sources

  1. support.zendesk.com/hc/en-us/articles/9642557697306-Responding-t…
  2. support.zendesk.com/hc/en-us/articles/7043760048922-Disputing-a-…
  3. developer.zendesk.com/api-reference/ticketing/tickets/ticket_audits
  4. appen.com/blog/human-in-the-loop
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment