This worked case is part of the quality scoring disagreements guide, which covers the inventory before a QA vendor change. Here, one hard case is followed from the call to the ruling.
The case: one over-limit credit, two opposite scores (Illustrative)
Everything below is Illustrative: a fictional 420-person home warranty company with an in-house contact center. It is not a real program, a quote or a completed deal.
- The call, Monday. A customer phones about a third failed repair on the same appliance. The agent offers a goodwill credit above the agent's normal limit, after a supervisor approves it in chat. The supervisor also adds a note on the customer account.
- Evaluation by Reviewer A, Wednesday. Reviewer A scores the policy-adherence item as an automatic fail, citing a 40-second snippet: the credit exceeded the agent's limit.
- Evaluation by Reviewer B, Wednesday. The same call is in that week's calibration set. Reviewer B scores the item as a pass, citing the same snippet: the exception was approved and noted.
- Calibration session, Friday. The two scores disagree on an item that drives the agent's overall result. The session flags it for the QA lead.
- Ruling. The QA lead checks the account and finds the supervisor's approval note. The ruling: an over-limit credit passes only when the approval is recorded on the customer account. Reviewer B's evaluation becomes the scoring index for this call.
- Correction. The agent's score is corrected and the agent is told why.
- Form change, the next week. The QA lead adds an "approved exception" answer to the policy-adherence item, with a dated form version, so the item stops producing splits.
Who decided what, and the record each decision left
| Step | Decision | Who made it | Record left behind | Where it usually lives |
|---|---|---|---|---|
| 1 | Grant an over-limit credit | Agent, approved by supervisor | Chat approval; account note | Contact center suite; CRM |
| 2 | Item fails | Reviewer A | Score, comment, snippet timestamps | QA platform |
| 3 | Item passes | Reviewer B | Score, comment, same snippet | QA platform |
| 4 | Escalate the split | Calibration session | Session record listing both scores | QA platform |
| 5 | Pass, with a stated rule | QA lead | Adjudication note; scoring index | QA platform comments or email |
| 6 | Correct the agent's score | QA lead | Corrected evaluation | QA platform |
| 7 | Change the form | QA lead with quality director | New form version with date | QA platform settings |
Calibration is a standard feature of QA platforms. Genesys, for example, documents calibration sessions as a way to keep scoring consistent across evaluators assessing the same interaction, with one evaluation marked as the scoring index and quality administrators comparing the variation between evaluators. Step 5 is exactly that index being chosen.
Why this sequence is useful evidence
Because it records judgment, not just activity. Most evaluations show one reviewer applying a form. This chain shows two competent people reading the same evidence in opposite ways, a third person resolving it with a rule, and the rulebook being changed so it does not happen again. That is the shape of an exception: a failure, the context that explains it, an authorized decision and a checked result. Records of that shape are why exception handling data interests AI developers, and why examples with an expected outcome matter in quality work.
It also depends on a record outside the QA platform. The ruling turned on an account note in the CRM. A history where the ruling says "approval verified" but the note no longer exists is weaker; the system map for this workflow shows which IDs keep that link intact.
Sizing the splits without opening a call
A quality team can describe how many such cases it holds from one agreement table, counted from calibration reports. The figures below are Illustrative arithmetic for one scoring item, not published data.
| Reviewer A / Reviewer B | B: pass | B: fail | Total |
|---|---|---|---|
| A: pass | 26 | 4 | 30 |
| A: fail | 2 | 8 | 10 |
| Total | 28 | 12 | 40 |
Raw agreement is 34 of 40, or 85%. That flatters the program, because reviewers who mostly mark "pass" agree often by chance. McHugh's review of the kappa statistic explains that Cohen introduced kappa because percent agreement ignores chance agreement. Here chance agreement is (30 × 28 + 10 × 12) / 1,600 = 60%, so kappa = (0.85 − 0.60) / (1 − 0.60) = 0.625, about 0.63.
For a licensing conversation the useful number is not kappa itself but the six off-diagonal calls. Each should have an adjudication note. "Our calibration history holds several hundred split decisions, each with a ruling" is a complete metadata answer.
Recognizing similar cases at a client
The pattern is easy to hear before it is ever seen. Listen for these signals when a client talks about quality:
- Agents or team leads complain that one item is "scored differently depending on who reviews you".
- The quality team mentions a weekly or monthly calibration meeting with a named lead.
- Someone recalls an item being reworded after an argument about a specific call.
- Supervisors can approve exceptions to policy limits, such as credits, refunds or fee waivers.
- Scores feed pay, bonuses or performance plans, which tends to make rulings careful and written.
Two or three of these together usually mean adjudicated splits exist. The partner then asks only for counts and years. Recordings involve customers and agents, so notice, consent and de-identification come before any discussion; the who-can-license page covers who decides.
The introduced company needs 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 90% of the data is in English. Buyer interest, the rights review result and any offer stay unknown until SourceX reviews the records. Partners sign the Channel Partner Agreement before submitting; the first valid introduction within the 90-day window is credited when the company submits a verified application. More on the record type is on the QA workflows page.
Next step
Refer a company. Earn up to $100,000. Default reward: 25% of Actual Supplier Payout, capped at $100,000 cumulatively per referred company. Eligibility and payment conditions apply, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
If a client's quality team can point to years of adjudicated splits like this one, register as a partner, check the company on who qualifies and read the program terms before you ask the quality director for an introduction.