# Counterfactual task variants in records | SourceX

Canonical: https://refer.sourcex.si/guides/concept-counterfactual-task-variants
Last updated: 2026-11-04

Counterfactual task variants are two or more versions of the same task that differ in one meaningful detail, so the effect of that detail can be seen. In an ordinary accounts payable workflow, the same $18,400 invoice submitted once without its approval attachment and once with it is a natural pair: the first was held, the second paid. Add a smaller invoice that was paid without an attachment, and the records show exactly which rule decided the outcome.

## What does counterfactual task variants mean in an ordinary business workflow?

It means holding everything constant except one input and watching whether the outcome changes. In business records, that happens whenever a case is sent back for one missing or wrong item and then resubmitted, or whenever two near-identical cases are handled under different conditions.

The idea comes from AI research. In a 2020 ICLR paper, researchers at Carnegie Mellon had people revise documents minimally so that a different label applied, and found that models trained on original and revised examples together generally outperformed models trained on comparable amounts of original data. A paper in Findings of EMNLP 2020 proposed "contrast sets": small, meaningful edits to test examples that usually change the correct answer. Models scored significantly lower on them, by up to a quarter in some cases. Both papers point to the same lesson: a model that has only seen ordinary cases may never have learned which detail actually decides the outcome.

Researchers had to build their pairs by hand. Companies produce them without trying, every time a control rejects something and a person fixes exactly what was wrong.

## The invoice pair, step by step (Illustrative)

A 420-person wholesale food distributor requires a signed purchase approval for any invoice over $10,000. A refrigeration contractor sends an $18,400 repair invoice. All values are fictional.

| Step | Variant A: first submission | Variant B: resubmission |
|---|---|---|
| Invoice | $18,400, repair at the north warehouse | Same invoice number, amount and vendor |
| Approval attachment | Missing | Signed by the plant manager |
| Match to purchase order | Passes | Passes |
| Threshold rule | Over $10,000 with no approval: hold | Over $10,000 with approval: release |
| Outcome | Held, routed to the plant manager | Paid on day 34 of terms |

Everything except the attachment is the same, so the attachment is what changed the outcome. Now add a control: a $4,200 invoice from the same contractor, also with no attachment, paid without a hold because it sits under the threshold. Together, the three records show that the attachment matters only above $10,000. That is the knowledge an AI model working in accounts payable needs, and it is exactly what a pile of routine paid invoices never reveals.

| Case | Amount | Attachment | Outcome | What it isolates |
|---|---|---|---|---|
| Variant A | $18,400 | Absent | Hold | The rule firing |
| Variant B | $18,400 | Present | Pay | The rule satisfied |
| Control | $4,200 | Absent | Pay | The rule not applying below the threshold |

Read across the rows and the threshold rule can be recovered from the records alone, without anyone having to write it down.

## The fields that make a pair a pair

Two invoices are a counterfactual pair only if a reviewer can prove they are the same task. These fields carry that proof in the example.

| Field | Where it sits | Why the pair depends on it |
|---|---|---|
| Invoice number, vendor and amount | Both submissions | Shows Variant A and Variant B are the same invoice |
| Submission timestamps | AP system or inbox log | Puts the two versions in order |
| Attachment presence and type | Document link on each version | The one detail that changed |
| Hold reason code | Hold record on Variant A | Names the rule that fired, not just that something did |
| Approver identity on the attachment | Signed approval on Variant B | Shows the right person cleared it |
| Purchase order match result | Matching record on both versions | Confirms nothing else differed |
| Payment record | Variant B and the control | The outcome on the other side of the change |
| Approval threshold in force | AP policy with effective date | Explains why the control paid without an attachment |

If Variant B had been entered as a new invoice with a new internal number, and nothing linked it to Variant A, the company would hold two unrelated-looking records. The pair would exist in fact but not in the data.

## What the example shows, and what it cannot

- **Real pairs rarely differ in exactly one thing.** A resubmission often fixes the attachment and also corrects a date or a cost center. Only the field history shows what else moved.
- **The control matters as much as the pair.** Without the $4,200 invoice, a reviewer could conclude that every invoice needs an attachment.
- **The rule may change.** If the threshold was later raised, older and newer pairs follow different rules, which is why the policy's effective date belongs in the record.
- **Common pairs are not the whole story.** Missing-attachment holds may be frequent while rarer variants, such as a wrong cost center or a duplicate, appear only a few times a year. Synthetic edits can fill gaps, but [synthetic data rarely replaces real data](/questions/will-synthetic-data-replace-real-data) for showing how a process actually responded.

## Who owns these records, and what stays unknown

In a distributor like this one, the AP team and controller own invoice, hold and payment history, usually in an ERP or AP automation tool, and IT administers it. Supplier agreements that might restrict reuse sit with procurement or legal. Authority to discuss licensing usually rests with the CFO or CEO.

The introduced company needs 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license the records, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

What stays unknown until SourceX reviews the opportunity: SourceX's current data needs, the rights review outcome, the de-identification needed, the company's offer and any reward. Industry interest in counterfactual variants does not mean every AP archive qualifies. SourceX licenses approved business data directly from companies and manages the agreement, rights review, data preparation, delivery and payout; partners stay at metadata level and never handle invoices. Related record families: [task and outcome pairs](/data-types/task-and-outcome-pairs), [accounts payable exception handling](/data-types/accounts-payable-exception-handling-process), [golden trajectories](/guides/concept-golden-trajectories), [why task length matters for agents](/guides/metr-ai-task-length-doubling) and the full list of [data types](/data-types).

## Go deeper on counterfactual task variants

- [Questions to ask a client about counterfactual task variants](/questions/does-counterfactual-task-variants-data-qualify): a short discovery script for checking whether a company keeps linked rejections and resubmissions.
- [Record gaps that make counterfactual task variants hard to assess](/questions/counterfactual-task-variants-evidence-gaps): a guide to the unlinked resubmissions and missing reason codes that break a pair.
- [Counterfactual task variants review packet](/resources/counterfactual-task-variants-review-packet): a list of the metadata a company prepares before SourceX reviews its exception history.

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client keeps rejected and resubmitted cases linked, with coded reasons and outcomes, [register as a partner](/register), confirm [who qualifies](/who-qualifies), and read the [program terms](/terms) before introducing the decision-maker.

## Sources

- https://refer.sourcex.si/terms
- https://openreview.net/pdf?id=Sklgs0NFvr
- https://aclanthology.org/2020.findings-emnlp.117/
