AI agent escalation training data: a timeline that separates automated steps, the human handoff and the outcome

Short answer

The records that show whether an AI agent escalated correctly form one timeline with three lanes kept apart: what the automated system did and why it stopped, the handoff event with the context it passed on, and what the human did next, followed by the outcome after closure. A reviewer needs all of it, in order, with each action attributed to a machine or a person. Below is an Illustrative timeline, the fields that mark each lane, and how to confirm a company keeps them.

Visual briefFull-size diagram
A single timeline ribbon for one freight-damage chat: four blue automated segments, a handoff marker at minute 4, three green human segments, and an outcome flag at day 14 showing no reopen.
Illustrative case at a fictional distributor. The color change on the ribbon is the boundary a reviewer needs to see in the records.

This page is part of support escalation records, which covers escalation chains in general. This page focuses on one asset: the timeline itself.

The timeline (Illustrative)

Fictional case, invented for this page. A 450-person industrial supplies distributor runs an AI agent in its customer chat. A customer writes in about a pallet that arrived damaged and asks for a credit on 12 cartons.

TimeLaneEventWhere it is recorded
0:00AutomatedChat opens; agent matches the customer's email to the account and orderChat platform log
0:01AutomatedAgent looks up the shipment: delivered yesterday, receipt signed "damaged"Order lookup logged under the integration user
0:03AutomatedAgent asks for photos; customer uploads twoTranscript and attachments
0:04AutomatedAgent classifies the case as freight damage; the requested credit is above its limitAgent decision log with policy version
0:04HandoffTicket created by a trigger and routed to the claims group with a summary and the photosTicket event made by a business rule
0:19HumanClaims specialist checks the photos against the delivery receipt noteInternal note by a named agent
0:26HumanSpecialist approves the credit for 12 cartons and opens a carrier claimCredit memo in the ERP; carrier claim record
0:27HumanCustomer told the credit and its timing; ticket solvedPublic reply and status change
Day 14OutcomeNo reopen and no repeat contactTicket status and contact history
Day 30OutcomeCarrier accepts the claimCarrier claim record

The 15 minutes between handoff and pickup is a staffing question, not an escalation error. The escalation question is whether the agent stopped at the right point and passed on enough for the specialist to act without asking the customer again. Here it did both.

What each lane must contain

Automated lane. Every step the agent took, in order, with what it looked up and what it said. The step that matters most is the last one before the handoff: the reason the agent stopped. A policy limit, a low-confidence classification and a customer request for a person are different reasons, and a reviewer judges them differently. If the policy version in force that day is stored, the reviewer can check the agent against the rule it actually had.

Handoff. The event that moved the case, the group it went to, and the context passed with it: a summary, the attachments, the account and order. When the specialist's first action is to ask the customer something the agent already knew, the context was not good enough, and the record shows it.

Human lane. Each action authored by a named person or role: notes, approvals, replies. Linked records in other systems, like the credit memo and the carrier claim here, show the decision had consequences.

Outcome lane. What happened after the ticket closed: a reopen, a repeat contact within a set window, or a confirmed resolution. This lane is what lets a reviewer say the handoff was right rather than merely that it happened.

How the system marks the boundary

The timeline is only useful if automated and human actions can be told apart in the stored record, not just on screen. Helpdesks differ, so the question is always how this particular system marks it.

In Zendesk, ticket audits are a read-only history of every update to a ticket. Each audit holds one or more events, carries an author, and has a via object describing how the update was made. The via object's source distinguishes changes made by business rules such as triggers and automations, and names the rule. So a ticket created by a trigger in the handoff row above would show that rule, while the specialist's note would show a person as author.

Other platforms use different conventions: a dedicated integration user for the AI agent, a separate chat platform with its own log, or a flag on each message. Some companies lose the distinction when the chat transcript is pasted into the ticket as a single block. Ask how the separation survives, and where the agent's own decision log lives, because it is often in the chat vendor's system rather than the helpdesk.

Reading the timeline as an evaluator would

Teams that evaluate agents work from this same shape. Appen describes offering agent trajectories and failure annotations as data services, and triaging agent cases for expert review by flagged failure signals and severity. A company's own escalation history, with outcomes attached, is a human-made version of that labeled timeline: the specialist's action and the day-14 result act as the label.

An evaluator reading the case above asks four things:

  1. Did the agent try the safe steps it was allowed to take before stopping?
  2. Was the stop reason legitimate under the policy in force?
  3. Did the handoff carry enough context for the human to act at once?
  4. Did the outcome confirm the human's decision?

How trajectory records are defined more broadly is covered in agent trajectory data. Handoff transcripts on their own are covered in AI-to-human handoff data, and using past tickets to train support agents is covered in the historical tickets guide.

Confirming the records exist

Ask the head of support or customer operations, at metadata level only, and with the authorized sponsor's agreement. Never request transcripts, tickets or exports.

LaneQuestionAnswer that points to a usable timeline
AutomatedIs the agent's step log kept after the case is handed off?Yes, in the chat platform, linked by ticket ID
AutomatedIs the policy or prompt version recorded by date?Yes, changes are dated
HandoffIs the handoff an event with a reason, or just a reassignment?An event with a reason code or rule name
HumanAre human actions authored by named agents?Yes, by user, separate from integration users
OutcomeAre reopens and repeat contacts linked to the first case?Yes, within a set window
DepthHow many months or years of agent-assisted cases exist?Since the agent launched, with no gaps

Owners, authority and what stays open

The support operations manager usually knows where the lanes are stored; the chat platform and helpdesk administrators can describe the fields. Neither can authorize a licensing discussion. That is the CEO, who can also sign for the company; the COO or another executive responsible for customer records may sponsor the introduction. The company needs 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

What stays unknown until SourceX reviews it: whether the chat vendor's terms affect who controls the agent logs, how customer details and attachments are de-identified, SourceX's current data needs for agent-assisted escalation records, and any offer. The company's payout and the partner's reward are separate; SourceX funds the reward and never deducts it from what the company receives. For the wider context on agent deployment, see agentic AI in the enterprise.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client can keep its automated steps, handoffs, human actions and outcomes apart in the stored record, register as a partner, confirm the basics on who qualifies and read the program terms before you introduce the executive who owns those records.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does the company need an AI agent already for its records to count?

No. Escalations between tier 1 and tier 2 people follow the same pattern: a first responder, a point where they hand off, context passed on and a result. An AI agent in front makes the automated lane explicit, but human-only histories carry the same judgment.

What if automated and human actions are mixed together in one log?

The timeline is weaker but sometimes recoverable from system user names, integration accounts or rule names. Note that the company is unsure; SourceX checks this during its review.

Is the outcome lane really necessary?

Yes. Without it, a reviewer can see that a handoff happened but not whether it was needed. A reopen, a repeat contact or a confirmed resolution is what turns a log into a judged example.

Sources

  1. developer.zendesk.com/documentation/ticketing/reference-guides/via…
  2. developer.zendesk.com/api-reference/ticketing/tickets/ticket_audits
  3. appen.com/whitepapers/failure-mode-analysis-ai-agent-p…
  4. appen.com/agentic-ai
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment