Counterfactual task variants: the same invoice with and without its approval, and why the pair matters

Short answer

Counterfactual task variants are two or more versions of the same task that differ in one meaningful detail, so the effect of that detail can be seen. In an ordinary accounts payable workflow, the same $18,400 invoice submitted once without its approval attachment and once with it is a natural pair: the first was held, the second paid. Add a smaller invoice that was paid without an attachment, and the records show exactly which rule decided the outcome.

Visual briefFull-size diagram
A two-by-two grid crossing invoice amount, under or over $10,000, with approval attachment present or absent. Three cells end in Pay; only the over-$10,000, no-attachment cell ends in Hold, and the two over-$10,000 cells are outlined as the minimal pair.
Illustrative: one changed field flips the outcome only above the approval threshold. The under-threshold row is the control.

What does counterfactual task variants mean in an ordinary business workflow?

It means holding everything constant except one input and watching whether the outcome changes. In business records, that happens whenever a case is sent back for one missing or wrong item and then resubmitted, or whenever two near-identical cases are handled under different conditions.

The idea comes from AI research. In a 2020 ICLR paper, researchers at Carnegie Mellon had people revise documents minimally so that a different label applied, and found that models trained on original and revised examples together generally outperformed models trained on comparable amounts of original data. A paper in Findings of EMNLP 2020 proposed "contrast sets": small, meaningful edits to test examples that usually change the correct answer. Models scored significantly lower on them, by up to a quarter in some cases. Both papers point to the same lesson: a model that has only seen ordinary cases may never have learned which detail actually decides the outcome.

Researchers had to build their pairs by hand. Companies produce them without trying, every time a control rejects something and a person fixes exactly what was wrong.

The invoice pair, step by step (Illustrative)

A 420-person wholesale food distributor requires a signed purchase approval for any invoice over $10,000. A refrigeration contractor sends an $18,400 repair invoice. All values are fictional.

StepVariant A: first submissionVariant B: resubmission
Invoice$18,400, repair at the north warehouseSame invoice number, amount and vendor
Approval attachmentMissingSigned by the plant manager
Match to purchase orderPassesPasses
Threshold ruleOver $10,000 with no approval: holdOver $10,000 with approval: release
OutcomeHeld, routed to the plant managerPaid on day 34 of terms

Everything except the attachment is the same, so the attachment is what changed the outcome. Now add a control: a $4,200 invoice from the same contractor, also with no attachment, paid without a hold because it sits under the threshold. Together, the three records show that the attachment matters only above $10,000. That is the knowledge an AI model working in accounts payable needs, and it is exactly what a pile of routine paid invoices never reveals.

CaseAmountAttachmentOutcomeWhat it isolates
Variant A$18,400AbsentHoldThe rule firing
Variant B$18,400PresentPayThe rule satisfied
Control$4,200AbsentPayThe rule not applying below the threshold

Read across the rows and the threshold rule can be recovered from the records alone, without anyone having to write it down.

The fields that make a pair a pair

Two invoices are a counterfactual pair only if a reviewer can prove they are the same task. These fields carry that proof in the example.

FieldWhere it sitsWhy the pair depends on it
Invoice number, vendor and amountBoth submissionsShows Variant A and Variant B are the same invoice
Submission timestampsAP system or inbox logPuts the two versions in order
Attachment presence and typeDocument link on each versionThe one detail that changed
Hold reason codeHold record on Variant ANames the rule that fired, not just that something did
Approver identity on the attachmentSigned approval on Variant BShows the right person cleared it
Purchase order match resultMatching record on both versionsConfirms nothing else differed
Payment recordVariant B and the controlThe outcome on the other side of the change
Approval threshold in forceAP policy with effective dateExplains why the control paid without an attachment

If Variant B had been entered as a new invoice with a new internal number, and nothing linked it to Variant A, the company would hold two unrelated-looking records. The pair would exist in fact but not in the data.

What the example shows, and what it cannot

  • Real pairs rarely differ in exactly one thing. A resubmission often fixes the attachment and also corrects a date or a cost center. Only the field history shows what else moved.
  • The control matters as much as the pair. Without the $4,200 invoice, a reviewer could conclude that every invoice needs an attachment.
  • The rule may change. If the threshold was later raised, older and newer pairs follow different rules, which is why the policy's effective date belongs in the record.
  • Common pairs are not the whole story. Missing-attachment holds may be frequent while rarer variants, such as a wrong cost center or a duplicate, appear only a few times a year. Synthetic edits can fill gaps, but synthetic data rarely replaces real data for showing how a process actually responded.

Who owns these records, and what stays unknown

In a distributor like this one, the AP team and controller own invoice, hold and payment history, usually in an ERP or AP automation tool, and IT administers it. Supplier agreements that might restrict reuse sit with procurement or legal. Authority to discuss licensing usually rests with the CFO or CEO.

The introduced company needs 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license the records, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

What stays unknown until SourceX reviews the opportunity: SourceX's current data needs, the rights review outcome, the de-identification needed, the company's offer and any reward. Industry interest in counterfactual variants does not mean every AP archive qualifies. SourceX licenses approved business data directly from companies and manages the agreement, rights review, data preparation, delivery and payout; partners stay at metadata level and never handle invoices. Related record families: task and outcome pairs, accounts payable exception handling, golden trajectories, why task length matters for agents and the full list of data types.

Go deeper on counterfactual task variants

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client keeps rejected and resubmitted cases linked, with coded reasons and outcomes, register as a partner, confirm who qualifies, and read the program terms before introducing the decision-maker.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does the company have to create the variants?

No. The useful pairs usually exist already: an invoice rejected and resubmitted, a ticket escalated once and not the next time, a quote reissued after one term changed. Building controlled variants for AI evaluation is the buyer's work, not the company's or the partner's.

Are synthetic variants a substitute for real ones?

They can supplement real records, but a synthetic edit only tests the rule its author imagined. A real pair shows how the company's process actually responded to one change, including exceptions nobody wrote down.

Can I ask the client to pull a few example pairs?

No. Partners stay at metadata level: whether rejections and resubmissions are kept, roughly how many per year and in which system. SourceX reviews samples under its own agreement with the company.

Sources

  1. refer.sourcex.si/terms
  2. openreview.net/pdf
  3. aclanthology.org/2020.findings-emnlp.117

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment