# Human adjudication records, explained | SourceX

Canonical: https://refer.sourcex.si/guides/concept-human-adjudication-records
Last updated: 2026-11-04

Human adjudication records are the written trail left when a third person settles a disagreement between two reviewers who judged the same work. In an ordinary support team, that is two QA reviewers scoring one escalated ticket differently, a lead checking the ticket history and ruling, and the reason and rule version written down. For a referral, what matters is whether the company kept both scores, the ticket, the ruling and its reason, and who can authorize a licensing discussion.

## What does human adjudication records mean in an ordinary business workflow?

In practice it is the paperwork behind a tie-break. The record is not the disagreement itself. It is the ruling plus everything needed to check the ruling later.

Most companies with a quality program already produce them without using the name. A support team scores a sample of tickets. A claims team double-checks a share of decisions. An accounts payable team has a second person look at exceptions. When two people disagree, someone senior decides, and in a well-run team that decision is written down with a reason.

QA software makes the mechanics visible. In Zendesk QA, a reviewee who disagrees with a review can open a dispute to ask for a second opinion; a comment is required, new scores can be proposed, and the dispute can go to the original reviewer or to another team member. The person answering can accept, partially accept or reject it, with comments. Data vendors describe the same structure on their side: Appen's writing on human-in-the-loop design lists multi-judgment tasks and adjudication steps as parts of a review workflow.

Why does this interest AI developers? Because a ruling encodes what a careful senior person thinks "correct" means when two competent people read the same evidence differently. Our explainer on [human-in-the-loop data](/proof/what-is-human-in-the-loop-data) covers the broader category, and [why synthetic data rarely replaces real data](/questions/will-synthetic-data-replace-real-data) explains why such judgment is hard to generate. Interest in the concept does not mean every QA archive qualifies; the questions below decide that.

## One escalation, two reviewers, one ruling (Illustrative)

This case is fictional. A 240-person B2B software company runs a Tier 1 support desk. Its escalation policy, version 3, says an agent may escalate to Tier 2 after two documented troubleshooting steps from the knowledge base, or at once if the issue needs back-end access.

A customer reports that invoices have stopped syncing to their accounting tool. The ticket's history looks like this:

| Time | Ticket event | Who |
|---|---|---|
| 10:02 | Ticket created from email: invoices not syncing since yesterday | Customer |
| 10:07 | Public reply: first step, customer asked to reconnect the integration; sync still failing | Tier 1 agent |
| 10:12 | Internal note: second step, sync queue cleared from the admin view; sync still failing | Tier 1 agent |
| 10:27 | Group changed to Tier 2 with tag "escalated" | Tier 1 agent |
| 14:40 | Solved: sync token had expired on the server side | Tier 2 engineer |

Two QA reviewers score the ticket the following week.

- **Reviewer A, "Valid escalation": Fail.** Comment: only one troubleshooting step appears in the replies to the customer, so the escalation came one step early.
- **Reviewer B, "Valid escalation": Pass.** Comment: the 10:12 internal note records a second step, so two steps were taken before 10:27.

The QA lead adjudicates. She opens the ticket history, confirms the 10:12 note exists and was written before the group change, and rules **Pass**. Her written reason: "Policy v3 requires documented steps, not customer-facing ones. Internal notes count." Because two experienced reviewers read the rule differently, she also amends the rubric to version 4, dated, adding the sentence "Steps recorded in internal notes count as documented."

That last move is what makes the record valuable. The ruling settles one ticket, and the dated rubric change shows how the standard itself moved because of it. Zendesk's developer documentation describes ticket audits as a read-only history of all updates to a ticket, which is the kind of trail that lets a third party check the lead's reasoning against the 10:12 timestamp rather than take it on trust.

## Evidence fields in one ruling

A ruling is only useful to someone outside the team if each part of it can be found and joined. Here is the escalation above, broken into the fields that carry it.

| Field | Value in the example (Illustrative) | Where it usually lives |
|---|---|---|
| Work item | The escalated invoice-sync ticket with its full change history | Help desk ticket and its audit trail |
| First judgment | Reviewer A, "Valid escalation": Fail, with comment | QA tool review record |
| Second judgment | Reviewer B, "Valid escalation": Pass, with comment | QA tool review record |
| Trigger | Scores disagree on the same criterion | QA tool dispute or calibration queue |
| Adjudicator | QA lead, a different person from both reviewers | Dispute response or calibration notes |
| Decision and reason | Pass: "Internal notes count" | Dispute comment or ruling log |
| Evidence cited | Internal note at 10:12, written before the 10:27 group change | Ticket audit timestamps |
| Rule version | Escalation policy v3, amended to v4 with a date | Rubric or policy document history |

The first three fields describe the disagreement. The next four describe the ruling. The last one dates the standard, so a later reader knows which rule the lead was applying. Without it, the same ticket scored after the amendment would look like a contradiction rather than a change in policy.

## Failure and uncertainty checks

Some adjudication archives look complete until someone tries to check a ruling. The common failures:

- **Overwritten scores.** If accepting a challenge simply replaces the original score, the disagreement disappears. Zendesk's documentation notes that a partially accepted dispute changes only some scores, and that scores cannot be changed after a rejection without opening a new dispute; ask whether reports or exports keep the before and after values.
- **Rulings without reasons.** A final score with an empty comment says who won, not why.
- **Rubric drift.** If the rubric was edited without dates, a ruling cannot be matched to the rule it applied.
- **Broken links to the work item.** Reviews held in one tool and tickets in another, joined only by a pasted subject line.
- **Archived source items.** The reviews survive but the ticket or claim history they judged was purged.
- **Client-owned material.** An outsourced contact center may hold reviews of its clients' customers, which it may not have the right to license.

None of these settles the question alone. They change how many rulings are usable, and only SourceX's review with the company can measure that.

## Does the company qualify, and what stays unknown?

The company needs 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license its records, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

What stays unknown until SourceX reviews it: how many rulings pass the acceptance check, how employee and customer names are de-identified, whether the QA vendor's or a client's contract limits use, and whether any buyer wants this task type. Partners never request exports, screenshots or sample tickets; discovery stays at metadata level: which QA tool, roughly how many rulings a month, how far back, and who owns them. The [company qualification checklist](/resources/company-qualification-checklist) covers the company-level basics.

## Go deeper on human adjudication records

- [Questions to ask a client about human adjudication records](/questions/does-human-adjudication-records-data-qualify): a short discovery list an introducer can use with a quality or support lead.
- [Human adjudication records review packet](/resources/human-adjudication-records-review-packet): what an evidence packet for SourceX's review should contain, item by item.

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the quality lead confirms that disagreements end in a written ruling tied to the ticket and a dated rubric, [register as a partner](/register), check the company against [who qualifies](/who-qualifies), and read the [terms](/terms) before you introduce the COO or head of support.

## Sources

- https://support.zendesk.com/hc/en-us/articles/9642557697306-Responding-to-disputes-in-Zendesk-QA
- https://support.zendesk.com/hc/en-us/articles/7043760048922-Disputing-a-review
- https://developer.zendesk.com/api-reference/ticketing/tickets/ticket_audits/
- https://www.appen.com/blog/human-in-the-loop
- https://refer.sourcex.si/terms
