Trajectory-level evaluation review packet: nine metadata documents, in the order a reviewer reads them

Short answer

A trajectory-level evaluation review packet holds nine metadata-only documents in three groups. The task group: a task statement, the policy history with effective dates, and the reference trajectory. The steps group: a role and permission matrix, a log field dictionary and a credential practice history. The proof group: an outcome linkage note, a rights summary and a case scorecard with its acceptance check. Operations, loss prevention, IT, finance and legal each prepare part. None of it contains a transaction.

Visual briefFull-size diagram
A binder with three tabbed sections, Task, Steps and Proof, fills as nine labelled documents drop in order behind their tabs, ending with the acceptance check marked as the document written before outcomes are examined.
Illustrative packet for a no-receipt refund task. Every document describes records or policy; none contains transactions.

This packet is part of the guide to trajectory-level evaluation in an ordinary workflow, which follows one store refund that was the right amount but approved with a shared code. The packet is how a company shows that thousands of such refunds could be graded step by step. It is assembled by the company after an introduction, under SourceX's agreement with the company. Partners never prepare or handle it; they need to know it exists so their first questions point the right way.

How the packet is organized

A reviewer reads in the order a grader works. First, what the task is and what a correct run looks like. Second, whether the steps can be attributed and timed. Third, whether outcomes can be confirmed and the records can be licensed. Each group depends on the one before: there is no point mapping log fields for a task nobody has defined, and no point checking rights for records that cannot be attributed.

The structure mirrors how agent evaluations are built. Google Cloud's agent evaluation tools compare a run with a reference trajectory using exact, in-order or any-order matching; that needs documents 1 to 3. Research benchmarks such as TheAgentCompany split each task into checkpoints and give partial credit for the checkpoints passed; that needs each checkpoint visible in documents 4 to 6.

Packet contents, in order

OrderGroupDocumentWhat it showsWho prepares it
1TaskTask statement and rough volumeThe routine task in one paragraph and about how often it occurs a yearHead of store operations
2TaskPolicy history with effective datesEach version of the rule: thresholds, who may approve, required order of stepsStore operations with compliance
3TaskReference trajectoryThe correct sequence of steps, numbered, with what each step must recordStore operations
4StepsRole and permission matrixWhich roles may approve which refunds, and since whenLoss prevention with HR
5StepsLog field dictionaryFor each step, the fields the system records: user, role, terminal, timestamp, approval method, override reasonIT, point-of-sale administrator
6StepsCredential practice historyWhether shared codes, shared logins or assistant entries were used, and the dates they endedLoss prevention
7ProofOutcome linkage noteHow a reversal, chargeback or repeat refund links to the original transaction, and where it does notFinance
8ProofRights and contract summaryWho owns the logs; what point-of-sale vendor and processor terms say; who can authorize licensingLegal, signed off by the CEO
9ProofCase scorecard and acceptance checkFour described case types graded both ways, and the written check a correct run must passStore operations, drafted with SourceX

Documents 2 and 6 are the ones companies most often underestimate. Without effective dates, a refund graded against today's threshold may be graded wrong; without the credential history, nobody can say which years have trustworthy approver identities.

The reference trajectory (Illustrative)

For the running example, a no-receipt return refunded to the original card under the returns policy, document 3 would read:

  1. Look up the purchase by card; record the cashier ID and the amount found.
  2. Inspect the item; record a condition code.
  3. If the amount exceeds $250, a manager approves with their own credential; record approver ID, role and time.
  4. Refund the purchase amount less anything already refunded, to the original card.
  5. Restock or dispose of the item; record the inventory adjustment.

Each numbered step names what it must record, which is what makes document 5 checkable: every field in the dictionary should trace back to a step here.

The case scorecard: outcome check versus trajectory check (Illustrative)

Document 9 shows how the two kinds of grading differ on real case types. Four fictional refunds:

CaseWhat happenedOutcome checkTrajectory check
A$312 refund, manager approves with own code, then refundPassPass
B$312 refund, shared override code usedPassFail: approval not by an authorized person
C$312 refund issued first, manager approval entered ten minutes laterPassFail on order: approval must precede refund
DManager approves, but refund issued as $336 including tax already refunded on a prior partial returnFail: amount wrongFail at step 4

Cases B and C are the valuable ones, because an outcome-only grader would call them successes. Appen, which sells agent trajectory analysis, describes (checked 2026-10-18) reviewers labeling each step as correct, suboptimal, incorrect or unnecessary to find where a run breaks down; cases like B and C are what such labels are for.

Acceptance check, written before outcomes are examined. A case passes only if all five hold: the refund amount equals the purchase amount less anything already refunded; an approval event exists for any no-receipt refund over $250; the approver's role permits approval under the policy version in force that day; the approval timestamp precedes the refund timestamp; and the approver is not the cashier.

What the partner checks, and what stays with the company

Before introducing, a partner can confirm that documents 1, 4 and 8 could be written: is there a task with a rule about how it is done, do approvals carry individual identities, and who can authorize licensing? The trajectory-level evaluation scorecard turns those answers into a fit score, and the evidence gap analysis shows how to describe what is missing.

The partner never asks for receipts, register logs, approval reports or screenshots. Partners sign the Channel Partner Agreement before submitting introductions. The company needs 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

For how graded examples are used, see what model evaluation is and rubric-based evaluation. For what step records look like, see agent trajectory data and tool-use training data.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the operations lead could write the task statement and the loss prevention lead can date the credential history, register as a partner, check who qualifies, and read the terms before you introduce the CEO, who would sign document 8.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Why is the acceptance check written before outcomes are examined?

So the standard cannot be bent to fit the data. If the check is written after looking at cases, it tends to describe what happened rather than what should have happened.

Does the packet include any refunds?

No. It includes the policy, the reference path, the field list and four described case types. Real transactions are reviewed only later, under SourceX's agreement with the company and with card and employee data handled as agreed.

Can the partner help draft the reference trajectory?

No. The partner's role ends at a qualified introduction. The company's operations team writes the reference path, because only it knows how the process is supposed to run.

Sources

  1. docs.cloud.google.com/vertex-ai/generative-ai/docs/models/evaluati…
  2. arxiv.org/abs/2412.14161
  3. appen.com/agentic-ai-trajectory-analysis
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment