Verifiable rewards evidence gaps: what an AI-data review expects, what companies keep, and how to say what is missing

Short answer

Six kinds of evidence make a workflow checkable: the task, the inputs available at the time, the rule version in force, the approver's identity, the action log and the final outcome. Companies usually keep the task and the action well. The gaps cluster in rule versions, approver identity and final outcomes, and each one turns an exact yes-or-no check into a judgment call. The honest way to introduce such a company is to name the gap in one plain sentence rather than hide it or oversell it.

Visual briefFull-size diagram
Six paired bars comparing expected and typically available evidence for task, inputs, rule version, approver identity, action log and final outcome, with the missing part of each available bar outlined in orange.
Illustrative pattern, not survey data. The bar lengths show where gaps usually appear, not measured percentages.

This analysis is part of the reinforcement learning with verifiable rewards guide. The qualification scorecard decides whether to raise the topic at all; this page is for describing what you found.

What a verifiable-rewards review expects

A verifiable reward needs a check a program can run. The Tulu 3 paper defines RLVR as rewarding a model only when a verification function confirms its output is correct, and DeepSeek-R1 makes the same requirement concrete: answers must arrive in a specified format so a rule can verify them. For a business record, the equivalent is a complete set of evidence around one task, so a check can compare what was done with what should have been done.

EvidenceWhat it answersIllustrative example: inventory write-down
TaskWhat was asked, by whom, by whenClose item: review slow-moving stock
InputsWhat the person could see at the timeAging report as of the period end
Rule versionWhich policy applied on that dateReserve policy v4: 50% after 365 days without movement
Approver identityWho accepted it, separate from who did itNamed controller, timestamp
Action logWhat was actually doneReserve entry with amount and item list reference
Final outcomeWhether the result stoodNo reversal; later disposal matched the reserve

What companies typically have, and what is usually missing

Illustrative pattern drawn from how common systems are used, not survey data. Proportions vary widely between companies.

EvidenceWhat companies typically haveWhat is usually missingEffect on the check
TaskClose checklists, tickets, work ordersDue dates and owners in older yearsMinor: the task can often be reconstructed
InputsThe final report or documentThe version as it stood on the dayThe check may compare against later data
Rule versionThe current policyEarlier versions with effective datesCannot tell which rule applied
Approver identityAn approval flag or statusA named person distinct from the preparerCannot test segregation of duties
Action logThe posted entry or transactionEdits made before approvalMinor for outcome checks, larger for process checks
Final outcomeThe original recordThe link to later reversals, credits or disposalsCannot tell whether the result stood

Rule version, approver identity and final outcome matter most, because no amount of volume compensates for them. A verification function is only as good as the evidence it checks: an archive without the final outcome can confirm that a task looks complete, not that it was right.

How to describe each gap honestly in an introduction

Use one sentence per gap, stating what exists, what does not and who could confirm it. Avoid both minimizing ("minor data issues") and dramatizing ("records are incomplete").

GapHonest wordingAvoid
Rule version"Current policies are kept; earlier versions before 2022 were overwritten.""Policies are documented."
Approver identity"Approvals are recorded by status; names were added when the new ERP went live in 2021.""Full approval trail."
Final outcome"Reversals exist but are not linked to the original entry; matching would be by amount and date.""Outcomes are tracked."
Inputs"Supporting reports are stored as final versions only."Leaving it out.
Task"Close checklists exist from 2019; earlier tasks are implied by the entries themselves.""Every task is documented."

A plain description helps everyone. SourceX can judge whether the remaining evidence supports a verifiable task, a rubric-based review or neither, and the owner is not surprised later. Related reading: failure data for AI training covers why corrected errors matter, and ethically sourced AI training data covers consent and rights.

When a gap ends the conversation and when it does not

Gaps in evidence are a matter of degree. Gaps in eligibility are not. Stop if the company has fewer than 50 full-time employees at peak (contractors excluded), lacks a documented operating history or the rights to license, or has no authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Otherwise, describe and continue. A smaller archive with every link intact can be a stronger case than a large one with the outcome missing. Interest in verifiable rewards among vendors and model builders does not mean any particular archive qualifies; why AI companies pay for data sets out what is and is not known. What stays unknown until SourceX reviews it: rights, scope, de-identification of personal and counterparty details, and whether any buyer wants the record type. Partners describe gaps from conversation and never request ledgers or exports to prove them. If the owner proceeds, the review packet lists what the company prepares.

Disclose that you are a SourceX referral partner and may receive a reward, paid separately by SourceX and never deducted from the company's payout, and sign the Channel Partner Agreement before submitting.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

When the gaps are described plainly and the owner still wants to explore licensing, register as a partner, confirm the company on who qualifies and read the program terms.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Should I leave a company out if it has one of these gaps?

Usually not. A single gap often turns a checkable record into one that still supports review by a person or a rubric. Describe the gap and let SourceX's review decide; only gate failures such as size or rights end the conversation.

Can a company fix a gap before an introduction?

It can sometimes recover context, for example by locating old policy versions in a document system. It cannot create records after the fact that pretend to be contemporaneous, and it should not try.

Are these gaps the same in every department?

The categories repeat, but where they bite differs. Finance often keeps approvals well and loses rule versions; operations often keeps actions and loses final outcomes.

Sources

  1. arxiv.org/abs/2411.15124
  2. arxiv.org/abs/2501.12948
  3. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment