Tool-call provenance evidence gaps: a gap analysis of what reviewers expect and what audit logs keep

Short answer

Four gaps cause most of the trouble: the actor is not recorded, the rule version is lost, the input is overwritten, or the outcome is never linked. Two hidden gaps follow close behind: shared credentials that make the actor field meaningless, and retention limits that cut history short. Each one removes a different piece of evidence a reviewer expects. The honest introduction names which pieces the company says it has and which it does not know.

Visual briefFull-size diagram
Six horizontal bars compare the evidence a review expects with what a typical audit log keeps: the change itself and its timestamp are fully present, while the named actor, rule revision, input value at the time and linked outcome have missing segments drawn as dashed orange outlines.
Illustrative pattern, not measured data. Bar lengths show a typical shape described on this page, not survey results.

This gap analysis is part of the guide to tool-call provenance in a business workflow, which walks through one help desk status change from billing input to customer outcome. Here the question is the reverse: when a client's records fall short of that example, where do they usually fall short, and how much does it matter?

The evidence a review expects for each automated change

A reviewer judging an automated action wants six things recorded together. The list maps closely onto the audit-content control in NIST SP 800-53 (AU-3), which asks audit records to establish the type of event, when and where it occurred, its source, its outcome, and the identity of the individuals or subjects involved. Provenance for AI use adds one item security auditing does not stress: the version of the logic that ran.

#Expected evidenceWhat it answers
1The change itself, with old and new valueWhat happened to the record
2TimestampWhen, and in what order relative to other events
3Named actor: which rule, integration or personWho or what was responsible
4Rule or integration versionWhich logic ran at that moment
5Input value as it stood at the time, and who wrote itWhat the action relied on
6Linked outcome: reopen, refund, dispute, next invoiceWhether the action turned out right

Expected versus typical: the gap matrix

Companies vary, but the shape repeats. Items 1 and 2 are nearly always present, because every audit trail records a change and a time. The others thin out from there.

Expected evidenceWhat companies typically haveThe usual gapMetadata question to ask
Change with old and new valuePresent in most help desk and ERP audit trailsSome systems store only the new value"Does the history keep the before and after of each field?"
TimestampPresent almost everywhereClocks differ across systems, so cross-system order is fuzzy"Do your systems log in one time zone?"
Named actorOften present for people; mixed for automationEvery automated change shows as "system" or one service account"Can you tell a rule's change from a person's in the history?"
Rule versionLeast often presentOnly today's rule definition exists; past edits are gone"Are changes to workflow rules versioned or logged, and since when?"
Input at the timePresent if the field's own history is keptThe field shows today's value, not the value the rule read"If an input field changed later, can you still see its earlier value?"
Linked outcomePresent within one system, rare across systemsRefunds and disputes sit in billing or a payment portal with no shared key"Can a reopen or billing dispute be tied back to the ticket number?"

Platform detail matters at the version row. Zendesk's documentation, for example, says a trigger-made change records the trigger's ID and title, and adds a revision ID on Enterprise plans. Two companies on the same help desk can therefore sit on opposite sides of the version gap depending on their plan, which is why the question is "what does your history show?" rather than "which product do you use?"

Why each gap matters for a review

Actor missing. Without a named actor, an approved workflow and a manual override look identical. That distinction is the whole point of tool-call provenance.

Version missing. A trigger edited twice in a quarter ran three sets of logic. If only the latest survives, any change older than the last edit is attributed to logic that did not exist yet.

Input overwritten. The rule read a credit note ID at 14:02; finance corrected the field at 16:40. The ticket now shows the corrected value, and the action looks like it relied on information it never saw.

Outcome unlinked. The change is fully traced but nobody can say whether it was right. For training, a traced action with no result teaches sequence without judgment; for evaluation, it cannot be scored.

A company with one gap can still be worth introducing, because the gap narrows what can be licensed rather than erasing it. A company with all four has status histories, not provenance.

Retention and shared credentials: the two hidden gaps

These rarely come up unprompted, and both make a record look more complete than it is.

Retention. Audit trails are kept for a period that varies by platform, plan and company policy, and older records may be archived rather than deleted. Zendesk, for instance, usually archives tickets 120 days after they close; archived tickets drop out of views but stay reachable through search and the API. Ask how many years of history exist and whether older years are archived or purged, rather than assuming all of it survives.

Shared credentials. If three integrations and two staff members write through the same login, the actor field is technically filled in and practically useless. NIST's enhancement AU-3(1) names this directly as additional audit content: the individual identities of group account users. Ask whether each integration has its own API user and since when.

Describing the gap honestly in an introduction

Write down what the company told you, attributed to the company, and mark what nobody knows yet. An illustrative note:

Records: help desk audit history, about six years; billing system with credit notes. Present, per the support lead: field changes with before and after; trigger names on automated changes. Unknown: whether trigger revisions were captured before the 2023 plan upgrade. Gap, per the company: two integrations share one API user until 2024; refund disputes live in the payment portal with no ticket key. Who decides: COO, with the support operations lead as the systems contact.

This tells SourceX where to look without overstating anything. Do not estimate volumes you have not been given, and do not describe a gap as fixable; whether and how history can be reconstructed is part of the review. Before writing the note, the tool-call provenance scorecard helps decide whether an introduction makes sense at all, and the review packet shows what the company would assemble afterwards.

The company baseline still applies: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. For agent-connected systems, see MCP and data provenance; for dataset-level origin and rights, see what data provenance is; for the wider family of records, see tool-use training data. Governance expectations for training data in the EU are covered in AI Act Article 10.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the company's history names the actor on automated changes and at least one other gap is known rather than guessed, register as a partner, check who qualifies, and read the terms before you send the introduction note.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Is one missing element enough to rule a company out?

Rarely. One gap usually narrows what can be licensed, for example to the years after rule versioning began. A company missing the actor, the revision, the input and the outcome all at once holds status histories, not provenance.

Should I tell the company its logs have gaps?

Describe what the company told you, in its own terms, and let SourceX assess it. A partner is not auditing the system, and a gap is information for the review rather than a judgment.

Do archived tickets count as missing?

Not necessarily. On some platforms archived tickets drop out of views and reports but stay reachable another way. Ask whether old records are archived or deleted, which is a different answer.

Sources

  1. nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r…
  2. developer.zendesk.com/documentation/ticketing/reference-guides/via…
  3. developer.zendesk.com/documentation/ticketing/reference-guides/tic…
  4. support.zendesk.com/hc/en-us/articles/4408887617050-About-ticket…
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment