Human adjudication records: a gap analysis of expected versus available evidence

Short answer

Human adjudication records are hard to assess when the evidence a reviewer would expect has thinned out. Seven pieces matter: the case material both reviewers saw, the first score, the second score, each reviewer's comment, the rubric version in force, the ruling with its reason, and any rubric change that followed. Most companies keep the case and the final score. The second opinion, the comments and the dated rubric are what usually go missing. State the gap plainly.

Visual briefFull-size diagram
Seven horizontal bars compare expected and available evidence for a disputed escalation at a fictional IT service desk; the ticket thread, first score and ruling are mostly available, while the second score, reviewer comments, rubric version and rubric change show outlined missing segments.
Illustrative gap profile for a fictional 520-person managed IT services company. Bar lengths are a sketch, not measurements.

This analysis belongs to how to assess human review records and correction workflows, which walks one disputed case through four gates. This page does a different job: it compares what a review expects against what companies typically keep, piece by piece, so an introducer can describe an incomplete archive accurately.

Expected versus available: the gap table

The middle columns are typical patterns from QA and support operations, not measurements. Every company differs, which is why the last column is a question to ask, not a conclusion.

Expected evidenceWhy a review wants itWhat companies typically haveWhat is usually missingQuestion for the QA lead
Case materialThe input both reviewers judgedTicket thread while retention allowsClosed tickets past the deletion windowHow long are closed tickets kept?
First scoreOne reasonable readingStored in the QA toolOverwritten when a dispute succeedsIs the original score kept after a dispute?
Second scoreThe competing readingSometimes, in a calibration or appeal viewNever exported with the main QA dataWhere do second opinions live?
Reviewer commentsThe reasoning behind each scoreFree-text fields, unevenly usedBlank or one-word commentsAre comments required on disputed reviews?
Rubric versionThe rule in force that dayCurrent rubric onlyOld wording after in-place editsCan last year's rubric be shown with its date?
Ruling and reasonHow the disagreement was settledFinal scoreThe reason, given verbally or in chatWhere is a lead's ruling written down?
Rubric changeWhether the ruling changed the ruleRarely linkedNo reference from rubric change to caseDo rubric edits cite the case that prompted them?

The pattern is consistent: evidence near the start of the chain survives better than evidence at the end. A company often holds thousands of scored tickets and very few rulings with reasons, and that difference decides how the archive should be described.

Worked gap profile: a disputed escalation at an IT service desk

Illustrative and fictional. A 520-person managed IT services company runs a service desk for client firms. An agent escalates a password-reset request to the security team instead of resetting it on the call, because the caller could not answer the second verification question.

  • Reviewer A scores the escalation unnecessary: the knowledge base allowed a reset after one failed question if the caller's phone number matched.
  • Reviewer B scores it correct: the client's contract required escalation for any failed verification on administrator accounts, and this was one.
  • The QA lead rules for Reviewer B and adds a line to the rubric: check the account type before applying the general reset rule.

Now the gap profile, as the company's QA lead might describe it at metadata level:

EvidenceStatus at this company
Case materialKept: tickets retained five years
First scoreKept, but replaced by the ruling in reports
Second scorePartial: in the calibration view, not in the main export
Reviewer commentsPartial: Reviewer B wrote two lines, Reviewer A none
Rubric versionMissing: rubric edited in a shared document with no dated history kept
Ruling and reasonKept: lead's note on the review
Rubric changeMissing: the new line does not cite the case

This archive still has value: case, scores and rulings with reasons. What it cannot show is the rule as it read before the change. That is a fair, specific sentence for an introduction.

Where the gaps come from in ordinary tools

Most gaps are produced by normal settings, not neglect.

Case material and deletion schedules. In Zendesk, a ticket deletion schedule continuously deletes archived tickets that were closed and not modified for a set period, and deleted tickets cannot be restored. A company with a short schedule may have rich QA data with no source to check it against.

Second scores in a separate view. In Zendesk QA, calibration reviews, comments and scores appear in a separate Calibration dashboard, not on the main dashboard. A team that reports only from the main view may not realize it holds paired scores at all.

Rubric versions. Where a rubric lives decides whether old wording survives. Confluence Cloud, for example, keeps page history and lets a user select two versions and compare them or restore an earlier one. A rubric kept in a wiki with page history may be reconstructable; one pasted into slides each quarter usually is not.

Agent-data vendors describe this kind of evidence too: Appen's agentic AI page, checked on October 18, 2026, lists failure-mode taxonomies and golden trajectories among its agent data services. Disagreements settled with a reason are close to that material. A vendor's service list is not evidence that any buyer wants a given archive, or that it qualifies.

Writing the gap into the introduction

The honest sentence is short and names what is missing. Three patterns cover most cases:

  1. Strong core, missing rules: "Disputed reviews with both scores and written rulings for four years; rubric changes were not versioned."
  2. Scores without reasoning: "Paired scores exist in the calibration view; comments and rulings are sparse."
  3. Outcomes only: "Final QA scores with ticket history; second opinions were not kept."

Avoid words that imply completeness, such as "full audit trail", unless the QA lead has confirmed every piece. An overstated introduction costs the owner time when SourceX's review finds the gap. Related record types are described under human feedback and review records and call center QA scorecards, and the wider idea is explained in what human-in-the-loop data is.

What the partner never asks for, and what SourceX decides

A partner records metadata: which tools, how many years, rough volumes of disputes, who owns the QA program and who can authorize a licensing discussion. A partner never asks for tickets, reviews, rubric files or reviewer names.

What stays unknown until SourceX reviews it: whether client contracts allow ticket content to be licensed, how agent and customer identities would be removed, whether reviewer comments can be used, and whether any buyer wants this record type. If the company has cases where the expected outcome is clear, the guide to quality assurance examples with an expected outcome shows a related pattern. Other record families are listed under data types.

The company also needs 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the QA lead can confirm case material, both scores and written rulings for most disputes, register as a partner, check the company against who qualifies and read the terms, then introduce the executive with the gap stated plainly.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Is a company with only one score per review worth introducing?

It may be, as a source of ordinary QA reviews, but it is not an adjudication archive. Describe it as QA scores with outcomes, and let SourceX decide whether that record type fits.

Can the partner estimate how many disputes the company has?

Ask the QA lead for an order of magnitude, such as disputes per month, and write it down as their estimate. Do not count records yourself or ask for an export.

Does a rubric kept in a shared document count as versioned?

Only if old versions can be shown with dates. Many wiki and document tools keep page history, but retention settings and manual deletions vary, so record what the company says and mark it unconfirmed.

Sources

  1. support.zendesk.com/hc/en-us/articles/6388012977306-Creating-tic…
  2. support.zendesk.com/hc/en-us/articles/7043724530842-Setting-up-c…
  3. support.atlassian.com/confluence-cloud/docs/create-edit-and-publis…
  4. appen.com/agentic-ai
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment