What does trajectory-level evaluation mean in an ordinary business workflow?
Trajectory-level evaluation checks whether the steps taken to reach a result were the right ones, in the right order, by someone allowed to take them. An outcome check asks only whether the end state is correct. Businesses already run both kinds of check, usually called reconciliation and audit.
AI developers use the same split. The research benchmark τ-bench grades an agent by comparing the database at the end of a conversation with an annotated goal state, an outcome check. Google Cloud's agent evaluation tools add trajectory metrics that compare the agent's sequence of tool calls with a reference sequence: exact match requires the same calls in the same order, in-order match allows extra calls as long as the required ones appear in order, and any-order match ignores order. Appen, a data services company, describes expert reviewers labeling each step of an agent's run as correct, suboptimal, incorrect or unnecessary, to find where the run broke down.
The difference matters most when a rule governs how something is done rather than what the result is. Approvals are the classic case.
A store refund, step by step (Illustrative)
The company is fictional: a 520-person home-improvement retailer with 30 stores. Its returns policy says a refund over $250 without a receipt requires a manager's approval, entered with the manager's own code, before the register will issue it. A customer returns a cordless tool bought five weeks ago without a receipt. The register finds the purchase on the customer's card: $312.
| Step | Reference path | What actually happened | Record left |
|---|---|---|---|
| 1 | Look up the purchase by card | Cashier looks up the purchase; $312 found | Lookup event with cashier ID |
| 2 | Inspect item, record condition | Item inspected, marked resellable | Return line with condition code |
| 3 | Manager approves, own code | Cashier enters a shared override code taped under the register | Approval event under a manager ID who was on break |
| 4 | Refund to original card | $312 refunded to the card | Refund transaction |
| 5 | Restock and close | Item restocked | Inventory adjustment |
Every outcome check passes: the refund matches the purchase, the item is back in stock, the customer leaves happy. A trajectory check fails at step 3, because the approval came from a code the manager did not enter. A grader would catch it if the record showed the cashier's ID on the approval; with a shared code, the log names the wrong person and the failure is invisible. That is the core lesson for introducers: trajectory-level evaluation is only as good as the authority records underneath it. The companion guide on action-space constraints covers how such permissions are set up in the first place.
Which records let someone grade the path, not just the result?
A trajectory can only be graded when the records hold both the steps that happened and the rule that said what should have happened. In the refund example, that means six kinds of record.
| Record | Example system | What it lets a grader check |
|---|---|---|
| Written policy with effective dates | Policy library or store operations manual | The reference path at the time of the refund |
| Event log with user ID and timestamp per step | Point-of-sale system | The order of steps and who took each one |
| Approval events tied to an individual credential | Point-of-sale system | Whether the approver was authorized and present |
| Refund transaction with amount and tender | Payment processor records | That the outcome matched the purchase |
| Inventory adjustment | Inventory system | That the item was handled as the policy required |
| Shift or timeclock records | Workforce system | Whether the named approver was on duty |
The last row is the one most people overlook. It is what reveals that the manager whose code was used was on break, turning an invisible failure into a gradable one.
Where does a trajectory grade become uncertain?
Some cases cannot be graded cleanly, and a careful review marks them rather than forcing a verdict.
- Policy changed mid-period. The approval threshold moved from $200 to $250 during the year, and the policy library keeps only the current version.
- Shared credentials. As in the example, a shared code means the log names someone who did not act. The step looks compliant when it was not.
- Steps done outside the system. A manager approves verbally and the cashier records nothing, so the log shows a gap rather than a violation.
- Extra steps that are harmless. A cashier checks stock twice. In-order matching tolerates this; exact matching would wrongly fail it.
Knowing which of these apply is part of understanding the archive.
Does the company qualify, and what stays unknown?
Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The company also needs 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license the records, and an authorized contact.
Retail and payment records often involve a point-of-sale vendor and a payment processor, so the first rights question is which records the company holds itself and what its vendor contracts allow. Customer card data is sensitive and needs careful handling during any review. What stays unknown until SourceX reviews it: how many cases carry attributable approvals, whether shared-code periods can be identified, how customer and employee data will be handled, and whether any buyer wants this task type. More concepts sit in the data types hub.
Go deeper on trajectory-level evaluation
- Questions an introducer should ask about trajectory-level evaluation: a short question set for checking whether a company's step records could qualify.
- Record gaps that make trajectory-level evaluation hard to assess: a guide to the missing approvals and policy versions that weaken grading.
- Trajectory-level evaluation review packet: a checklist of what belongs in an evidence packet for SourceX review.
Next step
Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
If the operations or finance lead confirms that approvals are logged under each person's own identity and rule changes are dated, register as a partner, check the company against who qualifies, and read the terms before you introduce the decision-maker.