Does a company's counterfactual task variants data qualify? A scorecard for rejected and resubmitted cases

Short answer

Score the company on three gates and seven criteria. The gates are 50+ full-time employees at peak, rights to license the records, and a person who can authorize a licensing discussion. The criteria test whether rejected and resubmitted cases survive as pairs: history, kept rejections, coded reasons, linked versions, dated rules, kept outcomes and control cases. A strong fit keeps both versions under one case number with a coded reason and a payment record. Gather the answers in conversation, never from the records.

Visual briefFull-size diagram
A horizontal gauge from 0 to 14 with weak, borderline and strong zones; as three gates and seven criteria are ticked, the needle slides along the track and settles at 12 for a fictional 180-person industrial parts distributor, at the start of the strong zone.
Illustrative score for a fictional company. The scorecard sorts introductions; it does not predict acceptance or a reward.

This scorecard is part of the guide to counterfactual task variants, which explains why the same invoice held once and paid once is a natural pair. Here the narrower job is deciding whether a given company holds such pairs in a usable form, and how confident to be when you introduce it.

How the points add up

Each of the seven criteria scores 2 for a strong answer, 1 for borderline and 0 for weak, for a maximum of 14. Gates are pass or fail and sit outside the points.

TotalFitMeaning for the introduction
12 to 14StrongPairs likely exist in quantity, linked and coded; introduce with the basics confirmed
7 to 11BorderlinePairs exist but some break on linkage, reasons or dating; introduce and name the weak rows
0 to 6WeakThe opportunity is mostly theoretical; resolve the open questions first

Why pairs are worth scoring at all: researchers had to build them by hand. In the ICLR 2020 study on counterfactually-augmented data, people revised documents minimally so a different label applied, and models trained on originals and revisions together generally did better than models trained on comparable amounts of original data. The contrast-sets paper found models scored significantly lower on small, meaningful edits. A company that kept its rejections holds pairs of that kind as a by-product of its controls.

Gates first: size, rights, authority and English share

Check these before scoring anything. A failed gate ends the scorecard for now, however rich the records.

GatePassFail
Size50+ full-time employees at peak, contractors excludedUnder 50, or the headcount is mostly contractors
Rights to licenseThe company owns its AP, claims or request history, and supplier or client terms do not forbid reuseA parent, client or outsourced provider controls the records
Authorized contactA named CEO, or owner-CEO, who can open a licensing discussionNobody with that authority identified

English share applies only outside the U.S., Canada and the EU. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. For a company in those other regions, treat the 70% English rule as a fourth gate. Documented operating history is required too; it is scored below because its depth matters, but a company with none fails outright.

The seven criteria, strong to weak

CriterionStrong (2)Borderline (1)Weak (0)Question that settles it
Operating historyFive or more years in one AP or case systemHistory split by a migrationUnder two years, or rebuilt recordsWhich years does each system cover?
Rejected versions keptFirst submission stays in history after correctionKept for some document typesOverwritten by the corrected versionWhen an invoice is rejected and resubmitted, are both versions kept?
Reason codedFixed reason codes on every holdCodes for some holds, free text for others"See email" or nothingIs the reason for each rejection a fixed field?
Versions linkedSame case number, new revision markerLinked by a typed referenceResubmission gets an unrelated new IDCan the two versions be tied by a number?
Rule datedThresholds and policies kept with effective datesCurrent policy known, past dates reconstructableNobody knows when thresholds changedWhen did the approval threshold last change?
Outcome keptHold, release, payment and credit historyFinal payment onlyNo outcome stored with the caseIs each version's outcome kept: held, paid, credited?
Control casesSimilar cases below the threshold are in the same systemSome, in a different systemNone identifiableAre small invoices processed in the same tool?

The questions come in this order on purpose. If rejected versions are overwritten, the next four answers matter much less. For the general screen that sits behind these rows, see the company qualification checklist.

Three Illustrative companies, scored

All three companies are fictional, and the answers came from a conversation with each owner or finance lead.

Criterion180-person industrial parts distributor240-person property management firm90-person engineering consultancy
Operating history212
Rejected versions kept220
Reason coded210
Versions linked211
Rule dated111
Outcome kept221
Control cases110
Total12 of 14, strong9 of 14, borderline5 of 14, weak

The parts distributor keeps every rejected supplier invoice in its AP tool under the same invoice number with a revision marker and a coded hold reason. It drops a point on rule dating because the approval threshold changed in 2022 and only the controller remembers when, and another on controls because small invoices go through a corporate card process.

The property management firm keeps rejected maintenance invoices and their outcomes, but a 2021 move to a new accounting system split the history, and hold reasons are coded only for some vendors. Every row is fixable through review, so this is a borderline introduction with named open items.

The engineering consultancy overwrites rejected expense claims with the corrected version, and reasons live in email. Even with a solid outcome record, the pairs do not survive. It passes the gates, so an introduction is possible, but it should say plainly that resubmissions are not kept.

What the score leaves open

The score summarizes what a company says it holds. It cannot show how many pairs differ in exactly one material detail, because that needs a field-by-field comparison inside the company's systems after a discussion starts. It cannot show SourceX's current data needs for a given process, the de-identification required for supplier names and bank details, or what the company would be offered.

Two companion pages pick up where the scorecard stops. The evidence gaps page for counterfactual variants shows how to word the weak rows honestly, and the review packet contents list shows what the company prepares later. For related record families, see task and outcome pairs and accounts payable exception handling; the introducer's guide to operational data licensing covers the wider process, and the full list sits under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the company passes all three gates and scores in the strong or borderline range, register as a partner, confirm the details on who qualifies and read the program terms before you introduce its finance lead.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

What if the owner does not know whether rejected versions are kept?

Score it borderline and write unknown beside it. The AP manager or the administrator of the AP tool can usually answer in a sentence, and a guessed strong score helps nobody.

Do the pairs have to be invoices?

No. Any process where a case is sent back for one item and resubmitted can produce pairs: purchase requests, credit applications, claims, change orders, quotes. The scorecard works the same way for each.

Can a company with a weak score still be introduced?

If it passes the three gates, yes, as long as the introduction says what is weak. A failed gate is different: resolve it before anyone spends time.

Sources

  1. openreview.net/pdf
  2. aclanthology.org/2020.findings-emnlp.117
  3. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment