The checklist is part of the guide to human adjudication records in an ordinary workflow, which follows two QA reviewers who disagreed about one escalation and the lead who settled it. That guide explains what a ruling record holds. This page lists what the company assembles to describe a whole archive of rulings, in the order a reviewer reads them.
The checklist, in reading order
Each item says what it shows and who holds the pen. Tick it when it can be written from what the team already knows.
- ☐1. Cover note. Shows company size, the help desk and QA tools in use, months of history, and the named owner and contact. Prepared by the COO or head of support, with the introducer helping only on these metadata fields.
- ☐2. Ruling task. Shows the decision a ruling makes, in one sentence. Prepared by the quality manager.
- ☐3. Inputs per ruling. Shows the fields an adjudicator sees: work item ID, both original scores and comments, reviewer roles, the rubric version. Prepared by the quality manager with the QA tool admin.
- ☐4. Outputs per ruling. Shows what a ruling leaves behind: final score, adjudicator role, written reason, any rubric change with its date. Prepared by the quality manager.
- ☐5. Acceptance check. Shows the test a ruling must pass to be usable, fixed before anyone counts rulings. Prepared by the quality manager, reviewed by SourceX.
- ☐6. Volume and retention. Shows double-reviewed items per month, the share that ended in disagreement, the share of rulings with a reason, and how long items and reviews are kept. Prepared by the help desk administrator.
- ☐7. Tool and rubric history. Shows QA tool and help desk migration dates, and a dated list of rubric versions. Prepared by the quality manager.
- ☐8. Sensitivity and rights. Shows customer identifiers in ticket text, employee names in reviews, and any client or vendor contract limits. Prepared by counsel or the CFO.
Items 2 to 5 define a ruling. Items 6 and 7 say how many exist and whether they can be read in context. Item 8 says whether they can be licensed.
Items 2 to 4: what a ruling is, what goes in, what comes out
Item 2, the ruling task. Narrow enough to check. For the main guide's escalation case: "Decide whether a Tier 1 escalation met the escalation policy in force, given the ticket history and both QA reviews." A task like "improve support quality" cannot be tested.
Item 3, inputs per ruling. Field names, not values:
| Input | Typical source | Why the reviewer needs it |
|---|---|---|
| Work item ID | QA review record | Joins the ruling to the ticket it judged |
| Ticket history | Help desk audit trail | Lets the reason be checked against events |
| First and second scores, with comments | QA tool | Shows the disagreement itself |
| Reviewer roles | QA tool user setup | Shows the adjudicator was a different, senior person |
| Rubric version in force | Policy document history | Shows which rule applied on that date |
Zendesk's developer documentation describes ticket audits as a read-only history of all updates to a ticket, with events such as field changes and comments. Item 3 says whether the company's help desk keeps a history like that for the period the rulings cover.
Item 4, outputs per ruling. The final score, the adjudicator's role, the written reason, and any rubric change with its version number and date. The critical question for this item is whether the original scores survive. Zendesk's documentation notes that a dispute can be partially accepted, changing only some scores, and that scores cannot be changed after a rejection without a new dispute. Item 4 should state whether reports keep the before and after values, or only the final one.
Item 5: the acceptance check for one ruling
The acceptance check is a short list of conditions a ruling must meet to count. It is written before anyone looks at how many rulings meet it, otherwise it simply describes the archive. An Illustrative check for the escalation task, all conditions required:
- The written reason cites a specific policy or rubric clause.
- The reason refers to at least one event in the work item's history, such as a timestamped internal note.
- Both original scores are retained unchanged alongside the ruling.
- The final score is recorded once, by a person different from both reviewers.
- Any rubric change prompted by the ruling has a version number and a date.
In the main guide's case, the lead's reason ("Policy v3 requires documented steps, not customer-facing ones; internal notes count") cites a clause, points to the 10:12 internal note, and comes with a dated rubric change to version 4. It passes. A ruling that says only "agreed with B" fails conditions 1 and 2, however sensible it was.
Data vendors describe the same shape of work. Appen's writing on human-in-the-loop design lists multi-judgment tasks and adjudication steps as parts of a review workflow. The acceptance check is what lets a company's own rulings be compared with that kind of structured process.
Items 6 and 7: volume, retention and tool history
Item 6 is a handful of numbers, rough is fine: items reviewed twice per month, the share that ended in disagreement, the share of rulings with a filled-in reason, and how long tickets and reviews are each kept. If tickets are purged after 18 months but reviews are kept for five years, only 18 months of rulings can be checked against their tickets, and the item should say so.
Item 7 is two dated lists. First, every help desk or QA tool change, and whether review IDs and ticket IDs carried over. Second, every rubric or policy version with its effective date. Without the second list, a ruling made under version 3 and a later one made under version 4 look like contradictions rather than a change in policy.
| Item 6 and 7 entries (Illustrative, fictional 240-person software company) | Value |
|---|---|
| Items reviewed twice per month | About 400 |
| Share ending in disagreement | Roughly one in four |
| Rulings with a written reason | Most since dispute comments became mandatory |
| Ticket retention | Six years |
| Review retention | Four years, since the current QA tool was adopted |
| Rubric versions | Seven, each dated |
Item 8, and the line partners do not cross
Item 8 lists sensitivities and limits, not conclusions: customer names and account details in ticket text, employee names in reviews and rulings, and any client contract (for an outsourced team) or vendor term that limits reuse of quality data. Counsel or the CFO prepares it. Authority to discuss a license usually sits with the COO or CEO.
The company must qualify on its own terms: 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.
Partners never assemble the packet and never receive reviews, exports or screenshots. Their part ends at the cover note's metadata. What the packet cannot settle: how many rulings pass the acceptance check, how names are de-identified, whether vendor or client terms allow use, and whether a buyer wants this task type. Those are decided in SourceX's review. To check fit before any of this, use the human adjudication records scorecard. Related record families sit in the data types hub, including human feedback and review and the call center QA scorecard; background is in human-in-the-loop data, RLHF data and the guide to assessing human review records.
Next step
Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
If a client's quality manager could tick most of these eight items from memory, with original scores kept and rulings tied to tickets, register as a partner, check who qualifies, and read the terms before introducing the COO.