Trajectory-level evaluation evidence gaps: what step-by-step grading needs and what store records lack

Short answer

The gaps that break trajectory grading are the ones that hide who did what and when: missing permissions (approvals under shared codes or logins), missing context (an optional check field left blank), unclear versions (a rule changed with no date) and unproven outcomes (a reversal recorded elsewhere). Outcome evidence is usually strongest and approver evidence weakest. Each gap changes how much of a company's history can be graded, not whether the company is worth introducing.

Visual briefFull-size diagram
Five vertical columns compare the evidence step-level grading expects with what a typical retailer keeps: the refund amount is complete, while approver identity, condition check, rule version and final outcome each have a missing top segment drawn as a dashed orange outline.
Illustrative pattern, not measured data. Column heights show the typical shape described on this page.

This gap analysis belongs to the guide to trajectory-level evaluation in an ordinary workflow, which grades a single store transaction against its reference path. The examples below stay with a related task so the gaps are easy to compare: a return over $250 without proof of purchase needs a supervisor's personal sign-off before money goes back to the customer.

What step-level grading expects to find

Grading a trajectory means comparing what happened with a reference path and the rules in force. Google Cloud's agent evaluation tooling, for example, compares an agent's sequence of tool calls with a reference sequence and offers exact, in-order and any-order matching. Appen's trajectory analysis page (checked 2026-10-18) describes reviewers labeling each step of an agent's run as correct, suboptimal, incorrect or unnecessary. For a business archive to support either approach, each step must leave five kinds of evidence.

EvidenceQuestion it lets a grader answer
Approver identityDid an authorized person take the approval step, personally?
Step contextWas the required check done, and what did it find?
Rule versionWhich threshold and approver list applied on that date?
Sequence and timeDid approval come before the refund?
Final outcomeDid the refund stand, or was it reversed, disputed or repeated?

NIST's audit-content control for information systems (AU-3) covers much of the same ground from a security angle: the type of event, when and where, its source, its outcome and the identity of who was involved. The trajectory version adds one demand of its own, the rule version, because the reference path changes when the policy does.

Expected versus what retailers typically keep

The pattern below is a description of common shapes, not measured data.

EvidenceTypically presentUsually missingHow the gap shows up
Refund amount and tenderAlmost always: the payment record is completeRarely missingNo gap; this is why outcome-only checks look easy
Approver identityAn approval event existsThe real approver, when shared override codes, "manager on duty" logins or assistant entries are usedThe log names a supervisor who was not on shift
Step contextA return line with item and priceThe condition code, an optional field left blankThe grader cannot tell whether the item was inspected
Rule versionToday's threshold in the policy libraryWhen the threshold changed from $200 to $250Refunds between $200 and $250 from before the change are graded against the wrong threshold
Final outcomeThe refund transactionA later reversal held in the processor's portalA "correct" outcome that was not final

Sequence and time are left out of the table because point-of-sale timestamps are usually present; the trouble comes when the approval is entered into a different system with its own clock, which turns into a version of the context gap.

Four gaps, worked through (Illustrative)

Missing permissions. Most common and most damaging. A pooled sign-off code records an approval without a real approver, so money returned by a clerk acting alone looks identical to money a supervisor signed off. NIST's enhancement AU-3(1) names this problem in security terms: the individual identities of group account users. In grading terms, the failure case is indistinguishable from the success case, which removes exactly the examples a trajectory grader is for.

Missing context. The return line has a condition code field, but it is optional and blank on most rows. The step happened, or did not; nobody can say. The graded path then skips step 2, and only an any-order or in-order match that tolerates missing steps can use the case at all.

Unclear versions. The no-receipt threshold was $200 until last spring and $250 since. Without the change date, every $230 refund from last winter is graded against the wrong rule: one properly approved looks like an unnecessary approval, and one issued without approval looks compliant. Both are graded wrong.

Unproven outcomes. A card network later reverses a refund as a duplicate, but the reversal sits in the payment processor's portal with no link to the store transaction. The path may be perfect and the result still wrong.

Which gaps narrow the archive, and by how much

Not every gap costs the same. A rough way to think about it, before SourceX measures it in review:

  • Permissions gap with a known end date: the years after the fix remain fully gradable; earlier years support outcome checks only.
  • Permissions gap with no end: the approval step cannot be graded, which removes the main reason to grade trajectories in this task.
  • Context gap: cases with the field filled remain usable; the share matters.
  • Version gap: cases near a threshold become ambiguous; cases far from it are unaffected.
  • Outcome gap: steps can be graded, but the record cannot show whether a correct-looking path produced a result that held.

Stating the gap plainly in an introduction

Ask about shared codes early, plainly and without judgment. Many retailers know about them and fixed them at some date. Then record what the company told you, in its words:

Task: no-receipt refunds over $250, about 9,000 a year across stores, per the loss prevention lead. Present: approval events in the point-of-sale system since 2017; refunds and tender. Gap, per the company: shared manager codes in use until March 2021; condition codes optional. Unknown: whether threshold changes before 2022 were dated; whether processor reversals can be matched. Who decides: CFO, with the loss prevention lead as systems contact.

Illustrative content. Do not estimate gradable share, and do not promise a gap can be repaired. Before writing it, the trajectory-level evaluation scorecard helps decide whether to introduce at all; afterwards, the company prepares the trajectory-level evaluation review packet.

The company needs 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. On English share, the rule is: Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. For help judging it, see how to assess whether business documentation is primarily in English. For what step records look like, see agent trajectory data; for how graded examples are used, see what model evaluation is; for tracing evidence in diligence, MCP for due diligence Q&A.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the company can date when shared approvals ended, or never used them, register as a partner, check who qualifies, and read the terms before you send the introduction with its gaps named.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Which gap should I ask about first?

Shared approval codes or logins. It is the most common gap, it makes an unauthorized approval look authorized, and it often has a known end date that keeps later years usable.

If outcomes are incomplete, can the steps still be graded?

Yes, partly. The order and attribution of steps can be checked without the final outcome, but a reviewer cannot tell whether a correct-looking path produced a result that held. That narrows how the records can be used.

Will naming gaps hurt the company's chances?

Gaps are expected and SourceX measures them in its review. An introduction that states them plainly saves time for everyone; one that hides them tends to unravel at the first metadata call.

Sources

  1. docs.cloud.google.com/vertex-ai/generative-ai/docs/models/evaluati…
  2. appen.com/agentic-ai-trajectory-analysis
  3. nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r…
  4. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment