Trajectory-level evaluation: when a refund is the right amount but issued without authority

Short answer

Trajectory-level evaluation grades the path to a result, not only the result: which steps were taken, in what order, and by whom. In an ordinary business workflow, a store refund of exactly the right amount still fails the check if the person who approved it was not authorized to. The records that support it show every step of a routine task with the person, the time and the rule that applied.

Visual briefFull-size diagram
A reference sequence of five refund steps sits above the actual sequence; the actual sequence skips the manager approval step, so the outcome check on the $312 refund passes while the trajectory check fails at step 3.
Illustrative store refund at a fictional home-improvement retailer. The money came out right; the path did not.

What does trajectory-level evaluation mean in an ordinary business workflow?

Trajectory-level evaluation checks whether the steps taken to reach a result were the right ones, in the right order, by someone allowed to take them. An outcome check asks only whether the end state is correct. Businesses already run both kinds of check, usually called reconciliation and audit.

AI developers use the same split. The research benchmark τ-bench grades an agent by comparing the database at the end of a conversation with an annotated goal state, an outcome check. Google Cloud's agent evaluation tools add trajectory metrics that compare the agent's sequence of tool calls with a reference sequence: exact match requires the same calls in the same order, in-order match allows extra calls as long as the required ones appear in order, and any-order match ignores order. Appen, a data services company, describes expert reviewers labeling each step of an agent's run as correct, suboptimal, incorrect or unnecessary, to find where the run broke down.

The difference matters most when a rule governs how something is done rather than what the result is. Approvals are the classic case.

A store refund, step by step (Illustrative)

The company is fictional: a 520-person home-improvement retailer with 30 stores. Its returns policy says a refund over $250 without a receipt requires a manager's approval, entered with the manager's own code, before the register will issue it. A customer returns a cordless tool bought five weeks ago without a receipt. The register finds the purchase on the customer's card: $312.

StepReference pathWhat actually happenedRecord left
1Look up the purchase by cardCashier looks up the purchase; $312 foundLookup event with cashier ID
2Inspect item, record conditionItem inspected, marked resellableReturn line with condition code
3Manager approves, own codeCashier enters a shared override code taped under the registerApproval event under a manager ID who was on break
4Refund to original card$312 refunded to the cardRefund transaction
5Restock and closeItem restockedInventory adjustment

Every outcome check passes: the refund matches the purchase, the item is back in stock, the customer leaves happy. A trajectory check fails at step 3, because the approval came from a code the manager did not enter. A grader would catch it if the record showed the cashier's ID on the approval; with a shared code, the log names the wrong person and the failure is invisible. That is the core lesson for introducers: trajectory-level evaluation is only as good as the authority records underneath it. The companion guide on action-space constraints covers how such permissions are set up in the first place.

Which records let someone grade the path, not just the result?

A trajectory can only be graded when the records hold both the steps that happened and the rule that said what should have happened. In the refund example, that means six kinds of record.

RecordExample systemWhat it lets a grader check
Written policy with effective datesPolicy library or store operations manualThe reference path at the time of the refund
Event log with user ID and timestamp per stepPoint-of-sale systemThe order of steps and who took each one
Approval events tied to an individual credentialPoint-of-sale systemWhether the approver was authorized and present
Refund transaction with amount and tenderPayment processor recordsThat the outcome matched the purchase
Inventory adjustmentInventory systemThat the item was handled as the policy required
Shift or timeclock recordsWorkforce systemWhether the named approver was on duty

The last row is the one most people overlook. It is what reveals that the manager whose code was used was on break, turning an invisible failure into a gradable one.

Where does a trajectory grade become uncertain?

Some cases cannot be graded cleanly, and a careful review marks them rather than forcing a verdict.

  • Policy changed mid-period. The approval threshold moved from $200 to $250 during the year, and the policy library keeps only the current version.
  • Shared credentials. As in the example, a shared code means the log names someone who did not act. The step looks compliant when it was not.
  • Steps done outside the system. A manager approves verbally and the cashier records nothing, so the log shows a gap rather than a violation.
  • Extra steps that are harmless. A cashier checks stock twice. In-order matching tolerates this; exact matching would wrongly fail it.

Knowing which of these apply is part of understanding the archive.

Does the company qualify, and what stays unknown?

Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The company also needs 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license the records, and an authorized contact.

Retail and payment records often involve a point-of-sale vendor and a payment processor, so the first rights question is which records the company holds itself and what its vendor contracts allow. Customer card data is sensitive and needs careful handling during any review. What stays unknown until SourceX reviews it: how many cases carry attributable approvals, whether shared-code periods can be identified, how customer and employee data will be handled, and whether any buyer wants this task type. More concepts sit in the data types hub.

Go deeper on trajectory-level evaluation

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the operations or finance lead confirms that approvals are logged under each person's own identity and rule changes are dated, register as a partner, check the company against who qualifies, and read the terms before you introduce the decision-maker.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

How is trajectory-level evaluation different from agent trajectory data?

Trajectory data is the recorded sequence of steps. Trajectory-level evaluation is the grading of that sequence against a reference path and the rules that applied. Companies whose records show both the steps and the rules support the second, not just the first.

Does a company need perfect process compliance for its records to be useful?

No. Cases where a step was skipped or done out of order are useful for evaluation, as long as the records show what actually happened and what should have happened.

Can I look at a few refund logs to judge whether the steps are recorded?

No. Ask whether approvals, overrides and refunds are logged with a user and time, and in which system. SourceX reviews records only under an agreement with the company.

Sources

  1. docs.cloud.google.com/vertex-ai/generative-ai/docs/models/evaluati…
  2. appen.com/agentic-ai-trajectory-analysis
  3. arxiv.org/abs/2406.12045
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment