Counterfactual task variants evidence gaps: which side of the pair breaks, and how to say so

Short answer

Seven pieces of evidence make a counterfactual pair checkable: the rejected first version, the resubmission, a link between them, a coded reason, the rule in force on both dates, the approver's authority, and the outcome of each version. Companies usually keep the resubmission and the payment. The overwritten first version, the uncoded reason and the undated rule are the common gaps. One or two gaps narrow what can be licensed; they do not end the conversation if you describe them plainly.

Visual briefFull-size diagram
Seven pairs of vertical columns compare expected and available evidence per 10 fictional resubmitted invoices, grouped by which side of the pair they protect; the rejected first version, the coded reason and the dated threshold show tall outlined missing segments.
Illustrative estimate for a fictional building services contractor. Outlined segments are evidence the pair needs but the archive lacks.

This gap analysis sits under the guide to counterfactual task variants. That guide shows one invoice held and then paid. This page asks what happens when the records behind such a pair are incomplete, and how to say so.

What a pair review expects to find

A pair is two versions of one task that differ in one meaningful detail. The contrast-sets paper in Findings of EMNLP 2020 built such pairs deliberately, with small edits that change the correct answer, and found models scored significantly lower on them. A business pair can serve the same purpose only if a reviewer can prove the two versions are the same task, see the single difference and confirm what happened to each.

#EvidenceHalf of the pair it protectsWhat a reviewer proves with it
1First submission kept after rejectionFirst versionThat the rule fired at all
2Resubmission with timestampResubmissionThat the fix happened, and when
3Shared case number or revision linkBothThat A and B are the same task
4Coded rejection reasonBothWhich single detail differed
5Rule or threshold in force on both datesBothThat the rule did not change in between
6Approver identity and limit on the releaseResubmissionThat the release was valid
7Outcome of each version: held, paid, credited, disputedBothThat the pair ended differently for the stated reason

Expected versus usually held

EvidenceWhat companies typically keepWhat usually goes missingWhy
First submissionA hold recordThe rejected document itselfMany AP tools replace the image with the corrected one
ResubmissionAlways, as the paid invoiceRarely missingIt is the record of payment
LinkSame invoice numberA revision marker when the resubmission is keyed as newManual re-entry under a new internal ID
Coded reasonA hold flagThe reason as a fixed fieldFree-text notes such as "see email"
Dated ruleThe current thresholdWhen it changedPolicies updated without effective dates
Approver authorityThe approver's nameTheir limit on that dateLimit tables overwritten
OutcomesFinal paymentLater credits and disputes linked backCredits raised as separate documents

Two failure patterns deserve a second look. If the approval threshold moved from $10,000 to $15,000 between the two submissions, the pair confounds two changes; this is the problem the temporal context guide deals with. And if the resubmission corrected the amount as well as supplying the missing document, the pair has two differences and cannot isolate either.

A building services contractor, gap by gap (Illustrative)

A fictional 230-person building services contractor rejects and resubmits a steady stream of subcontractor invoices. Its AP manager was asked, for every 10 resubmitted invoices, roughly how many keep each item. These are estimates from a conversation, not counts from the system.

EvidenceExpected per 10Available per 10Missing per 10Half damaged
First submission kept1037First version
Resubmission10100None
Link between versions1082Both
Coded reason1046Both
Rule dated1055Both
Approver limit1073Resubmission
Outcome of each version1091Both

The pattern is clear. Payment and linkage are strong, so the company can show what was paid and tie most resubmissions to an invoice number. But in roughly seven of ten cases the rejected version is gone, which means most pairs survive only as a hold flag and a paid invoice. Those are still useful task-and-outcome records, as described in task and outcome pairs, but only about three in ten can serve as true counterfactual pairs. The AP manager added that the tool began keeping rejected images after a 2023 configuration change, so recent years are stronger.

Wording a gap for the introduction

Describe the gap, the years it affects and the reason, in a sentence each. Avoid both understatement and alarm.

  • Overwritten first version: "Rejected invoice images were replaced by the corrected version until mid-2023; since then both are kept."
  • Uncoded reason: "Hold reasons are coded for price and quantity mismatches; missing-approval holds are noted in free text."
  • Undated rule: "The approval threshold changed once, in 2022; the controller can confirm the date but it is not recorded in the AP tool."
  • Approver limit: "Approvers are recorded on every release; the limit table for years before 2022 was not kept."

A statement like this helps SourceX set the review scope at once. A claim that "we keep everything" is usually discovered to be untrue later, which costs everyone time.

When a gap stops the introduction

Record gaps narrow the review. These conditions stop the introduction until they change, and none of them depends on the records' quality.

ConditionWhy it stopsWhat to do
Fewer than 50 full-time employees at peak (contractors excluded)Fails eligibilityDo not introduce
A parent company, client or outsourced AP provider controls the recordsRights are not the company's to licenseFind who can authorize, then reassess
No documented operating historyFails eligibilityDo not introduce
No authorized contactNobody can open a licensing discussionIdentify the CEO or owner-CEO first

Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

What stays unknown until SourceX reviews it: how many pairs differ in exactly one detail, buyer interest in the process, the de-identification needed for subcontractor details and any offer. To screen the company first, use the counterfactual variants qualification scorecard; to see what the company would prepare later, read the review packet contents list. Process mining explains why event histories with rework steps are informative, the proprietary data moat guide covers why real exception records are hard to copy, and the wider catalog is under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the company passes the gates and you can name its pair gaps in a few plain sentences, register as a partner, check who qualifies and read the terms before you introduce the owner.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Is a pair with two changes useless?

Not useless, but it cannot show which change decided the outcome. A reviewer may still use it as an ordinary task-and-outcome record. Say how often resubmissions fix more than one thing, if the AP lead knows.

Can the company rebuild a missing first version from email?

Sometimes the email attachment survives, but whether it can be linked reliably is a question for SourceX's review. Do not ask the company to reconstruct anything before then.

Which gap matters most?

An overwritten first version, because without it there is no pair at all. The other gaps weaken a pair; that one removes it.

Sources

  1. aclanthology.org/2020.findings-emnlp.117
  2. openreview.net/pdf
  3. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment