Counterfactual task variants review packet: the seven documents a company assembles for a pair review

Short answer

A counterfactual task variants review packet holds seven metadata-only documents, in this order: a cover sheet, the task specification, a pair inventory, a field comparison template, a register of thresholds with effective dates, an authority and outcome summary, and rights notes. Together they let a reviewer judge whether the company's rejected and resubmitted cases can serve as clean pairs. The introducer helps only with the cover sheet; the AP team, controller and legal prepare the rest once a review begins with SourceX.

Visual briefFull-size diagram
Seven labelled pages slide in from the right and clip into a ring binder in order: cover sheet, task specification, pair inventory, field comparison template, threshold register, authority and outcome summary, and rights notes, each marked with the role that prepares it.
Illustrative assembly order for a fictional accounts payable pair review. The pages describe records; none contains them.

The packet supports the guide to counterfactual task variants, where an invoice held for a missing approval and paid after resubmission forms the pair. Partners do not assemble it. It is described so you know what a company's records must be able to produce later.

Packet contents at a glance

OrderDocumentWhat it showsPrepared by
1Cover sheetHeadcount, AP system and years, migration dates, authorized contactIntroducer with the owner
2Task specificationThe decision rule the pairs test, written as an instructionController
3Pair inventoryCounts of rejected-then-resubmitted cases per year, by reason codeAP manager
4Field comparison templateWhich fields are compared to prove only one thing changedAP system administrator
5Threshold registerEach approval threshold with effective datesController
6Authority and outcome summaryApproval limits by role and year; counts of holds, releases, payments, creditsFinance
7Rights notesSupplier terms on reuse, personal or bank data present, consents neededLegal or procurement

Document notes, one at a time

Cover sheet. Peak full-time headcount (contractors excluded), the AP or ERP tool and the years it covers, any migration and its date, and the name of the CEO (in an owner-run company, usually the owner), who can authorize a licensing discussion. It names no suppliers.

Task specification. The rule written as an instruction a model or a new clerk could follow. For the fictional case: "Process this vendor invoice. If it exceeds $10,000 and has no signed purchase approval, place it on hold and route it to the budget owner. Otherwise, release it for payment once the purchase order matches."

Pair inventory. A small table: year, reason code, number of rejected cases, number resubmitted, and whether the first version was kept. It tells the reviewer how many candidate pairs exist and which reasons dominate, without opening one.

Field comparison template. The list of fields that must match for two versions to count as one task with one change: invoice number, vendor, amount, purchase order, cost center, service date, and the attachment field. The template is what lets a reviewer reject a pair where the resubmission also corrected the amount.

Threshold register. One row per threshold value, with the date it took effect and the date it ended. If the threshold moved from $10,000 to $15,000 in a given year, every pair straddling that date is flagged.

Authority and outcome summary. Approval limits by role for each year, plus counts of holds, releases, payments and later credits or disputes. Counts only.

Rights notes. Whether supplier agreements limit reuse of invoice data, whether bank details or personal data appear on the documents, and whether a parent company must consent.

The pair checklist and acceptance check (Illustrative)

Once a review is under way, the reviewer works through one candidate pair at a time. The checklist below is what each pair must eventually be able to produce from the company's systems.

  • ☐
    Both submissions with timestamps, the shared invoice number and the revision marker
  • ☐
    A field-by-field comparison showing exactly one material difference
  • ☐
    The threshold in force on both dates
  • ☐
    The hold and release events, with who acted and their approval limit
  • ☐
    Payment records, plus any credit or dispute that followed
  • ☐
    A control case under the threshold with the same missing item

Acceptance check. The pair passes only if all four hold: the two versions differ in the attachment and nothing else material; the threshold was the same on both dates; variant A was held and variant B released, each by someone with authority; and variant B was paid without a later credit. A pair where the resubmission also corrected the amount fails, because two things changed at once.

Here is how the check reads against three fictional candidates from a 180-person industrial parts distributor.

CandidateDifference between versionsThreshold same on both datesRelease authorizedPaid without creditResult
Pair 1, $18,400 repair invoiceAttachment onlyYesYesYesPass
Pair 2, $12,750 parts invoiceAttachment and amountYesYesYesFail: two changes
Pair 3, $16,200 service invoiceAttachment onlyNo, threshold changed between submissionsYesYesFail: rule moved

Pair 3 is a useful reminder of why the threshold register sits in the packet. The records looked perfect until the dates were set against the policy.

Why reviewers value pairs, and where that stops

Research shows why evaluators want pairs like this. The ICLR 2020 paper on counterfactually-augmented data had people revise documents minimally so a different label applied, and found that training on originals and revisions together generally helped. The contrast-sets paper found models scored significantly lower on small edits that change the answer. That evidence concerns evaluation in general. It says nothing about demand for a particular company's records, which only SourceX's review can establish.

Real pairs also carry something generated ones lack: they show how the company's process actually responded, including exceptions nobody wrote down. That is the argument in whether synthetic data will replace real data. Longer multi-step cases, where a hold leads to several follow-up actions, connect to why task length matters for agents.

Before you suggest a packet

Check three things in conversation first. The company needs 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license the records and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The AP lead should be able to say that rejected versions are kept, at least for recent years. And the owner should understand that nothing leaves the company before an agreement: no invoices, no exports, no sample pairs.

If those hold, the counterfactual variants qualification scorecard gives a quick read on fit, and the gap analysis for counterfactual variants shows how to describe what is missing. Accounts payable exception handling covers the wider record family, and other categories are under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the controller could produce a pair inventory and a threshold register from the company's own system, register as a partner, confirm who qualifies and read the terms before you introduce the decision-maker.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Who decides which pairs go in?

Nobody selects individual pairs at the packet stage. The pair inventory reports counts by reason code and year; choosing and checking actual pairs happens in SourceX's review under its agreement with the company.

Why does the packet need a control case?

Without a similar case below the threshold, a reviewer cannot tell whether the missing item matters always or only above a certain amount. The control describes that boundary.

Can the introducer help fill in the pair inventory?

No. The introducer helps with the cover sheet from what the owner says. Counts by reason code come from the AP team, from their own system.

Sources

  1. aclanthology.org/2020.findings-emnlp.117
  2. openreview.net/pdf
  3. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment