Human adjudication records review packet: an eight-item checklist for QA rulings before review

Short answer

A human adjudication records review packet is an eight-item checklist that describes a company's QA rulings well enough for SourceX to assess them, using only metadata the quality team already holds. The items: a cover note, the ruling task, inputs per ruling, outputs per ruling, the acceptance check, volume and retention, tool and rubric history, and sensitivity and rights. Most are prepared by the quality manager; the help desk administrator and counsel each fill in one. No ticket text or customer data goes in.

Visual briefFull-size diagram
Eight checklist cards fan in from both sides and settle into a folder marked no ticket text, each card ticked as it lands, from the ruling task to sensitivity and rights.
Illustrative checklist for a fictional support QA program. The folder describes rulings; it holds no ticket text, reviews or customer data.

The checklist is part of the guide to human adjudication records in an ordinary workflow, which follows two QA reviewers who disagreed about one escalation and the lead who settled it. That guide explains what a ruling record holds. This page lists what the company assembles to describe a whole archive of rulings, in the order a reviewer reads them.

The checklist, in reading order

Each item says what it shows and who holds the pen. Tick it when it can be written from what the team already knows.

  • ☐
    1. Cover note. Shows company size, the help desk and QA tools in use, months of history, and the named owner and contact. Prepared by the COO or head of support, with the introducer helping only on these metadata fields.
  • ☐
    2. Ruling task. Shows the decision a ruling makes, in one sentence. Prepared by the quality manager.
  • ☐
    3. Inputs per ruling. Shows the fields an adjudicator sees: work item ID, both original scores and comments, reviewer roles, the rubric version. Prepared by the quality manager with the QA tool admin.
  • ☐
    4. Outputs per ruling. Shows what a ruling leaves behind: final score, adjudicator role, written reason, any rubric change with its date. Prepared by the quality manager.
  • ☐
    5. Acceptance check. Shows the test a ruling must pass to be usable, fixed before anyone counts rulings. Prepared by the quality manager, reviewed by SourceX.
  • ☐
    6. Volume and retention. Shows double-reviewed items per month, the share that ended in disagreement, the share of rulings with a reason, and how long items and reviews are kept. Prepared by the help desk administrator.
  • ☐
    7. Tool and rubric history. Shows QA tool and help desk migration dates, and a dated list of rubric versions. Prepared by the quality manager.
  • ☐
    8. Sensitivity and rights. Shows customer identifiers in ticket text, employee names in reviews, and any client or vendor contract limits. Prepared by counsel or the CFO.

Items 2 to 5 define a ruling. Items 6 and 7 say how many exist and whether they can be read in context. Item 8 says whether they can be licensed.

Items 2 to 4: what a ruling is, what goes in, what comes out

Item 2, the ruling task. Narrow enough to check. For the main guide's escalation case: "Decide whether a Tier 1 escalation met the escalation policy in force, given the ticket history and both QA reviews." A task like "improve support quality" cannot be tested.

Item 3, inputs per ruling. Field names, not values:

InputTypical sourceWhy the reviewer needs it
Work item IDQA review recordJoins the ruling to the ticket it judged
Ticket historyHelp desk audit trailLets the reason be checked against events
First and second scores, with commentsQA toolShows the disagreement itself
Reviewer rolesQA tool user setupShows the adjudicator was a different, senior person
Rubric version in forcePolicy document historyShows which rule applied on that date

Zendesk's developer documentation describes ticket audits as a read-only history of all updates to a ticket, with events such as field changes and comments. Item 3 says whether the company's help desk keeps a history like that for the period the rulings cover.

Item 4, outputs per ruling. The final score, the adjudicator's role, the written reason, and any rubric change with its version number and date. The critical question for this item is whether the original scores survive. Zendesk's documentation notes that a dispute can be partially accepted, changing only some scores, and that scores cannot be changed after a rejection without a new dispute. Item 4 should state whether reports keep the before and after values, or only the final one.

Item 5: the acceptance check for one ruling

The acceptance check is a short list of conditions a ruling must meet to count. It is written before anyone looks at how many rulings meet it, otherwise it simply describes the archive. An Illustrative check for the escalation task, all conditions required:

  1. The written reason cites a specific policy or rubric clause.
  2. The reason refers to at least one event in the work item's history, such as a timestamped internal note.
  3. Both original scores are retained unchanged alongside the ruling.
  4. The final score is recorded once, by a person different from both reviewers.
  5. Any rubric change prompted by the ruling has a version number and a date.

In the main guide's case, the lead's reason ("Policy v3 requires documented steps, not customer-facing ones; internal notes count") cites a clause, points to the 10:12 internal note, and comes with a dated rubric change to version 4. It passes. A ruling that says only "agreed with B" fails conditions 1 and 2, however sensible it was.

Data vendors describe the same shape of work. Appen's writing on human-in-the-loop design lists multi-judgment tasks and adjudication steps as parts of a review workflow. The acceptance check is what lets a company's own rulings be compared with that kind of structured process.

Items 6 and 7: volume, retention and tool history

Item 6 is a handful of numbers, rough is fine: items reviewed twice per month, the share that ended in disagreement, the share of rulings with a filled-in reason, and how long tickets and reviews are each kept. If tickets are purged after 18 months but reviews are kept for five years, only 18 months of rulings can be checked against their tickets, and the item should say so.

Item 7 is two dated lists. First, every help desk or QA tool change, and whether review IDs and ticket IDs carried over. Second, every rubric or policy version with its effective date. Without the second list, a ruling made under version 3 and a later one made under version 4 look like contradictions rather than a change in policy.

Item 6 and 7 entries (Illustrative, fictional 240-person software company)Value
Items reviewed twice per monthAbout 400
Share ending in disagreementRoughly one in four
Rulings with a written reasonMost since dispute comments became mandatory
Ticket retentionSix years
Review retentionFour years, since the current QA tool was adopted
Rubric versionsSeven, each dated

Item 8, and the line partners do not cross

Item 8 lists sensitivities and limits, not conclusions: customer names and account details in ticket text, employee names in reviews and rulings, and any client contract (for an outsourced team) or vendor term that limits reuse of quality data. Counsel or the CFO prepares it. Authority to discuss a license usually sits with the COO or CEO.

The company must qualify on its own terms: 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Partners never assemble the packet and never receive reviews, exports or screenshots. Their part ends at the cover note's metadata. What the packet cannot settle: how many rulings pass the acceptance check, how names are de-identified, whether vendor or client terms allow use, and whether a buyer wants this task type. Those are decided in SourceX's review. To check fit before any of this, use the human adjudication records scorecard. Related record families sit in the data types hub, including human feedback and review and the call center QA scorecard; background is in human-in-the-loop data, RLHF data and the guide to assessing human review records.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client's quality manager could tick most of these eight items from memory, with original scores kept and rulings tied to tickets, register as a partner, check who qualifies, and read the terms before introducing the COO.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does the packet include example rulings?

No. It describes the class of rulings by field names, counts and dates. Ticket text, review comments and customer data stay inside the company until an agreement with SourceX is in place.

Who inside the company usually leads the packet?

The quality manager or head of support quality, because most items describe how reviews and rulings work. The help desk administrator, the support director and counsel each contribute one item.

What if some rulings have no written reason?

Count them and say so in item 6. A share of reasonless rulings is a known limit, not a failed packet. It tells SourceX which rulings are checkable and which are not.

Sources

  1. support.zendesk.com/hc/en-us/articles/9642557697306-Responding-t…
  2. developer.zendesk.com/api-reference/ticketing/tickets/ticket_audits
  3. appen.com/blog/human-in-the-loop
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment