When a pilot needs a holdout group: a six-question feasibility tree that decides whether anyone should be held back, separates what a holdout may withhold from what it never can, and names the design to use when the answer is no

Short answer

A pilot needs a holdout only when four things hold at once: it asks whether an activity changed what people did, the outcome could plausibly happen without that activity, the activity can be withheld without withholding anything a CEO or partner is owed, and the held-back group can stay apart until the reading date. If any one fails, a holdout adds cost and no truth. Run a feasibility pilot, a delayed-start group or a dated before-and-after log instead, and say plainly which limits remain.

Visual briefFull-size diagram
A decision tree of six questions running down the left, about the decision at stake, a causal question, a background rate, fair withholding, separation, and visibility with size; each no branches right to a fallback: practice log, feasibility pilot, change outcome, delayed start, cluster or before-after, and no comparison; a yes to all six ends at a green box reading holdout, drawn before launch.
The tree is this page's method, not a statistical standard. Fallbacks are listed in the order the questions are asked.

"Holdout" also has an AI-evaluation meaning, a set of real cases kept out of training so a model can be tested on them. This page is about people and companies in a referral pilot.

A holdout answers one question: what would have happened anyway

Every referral network has background activity. Companies can apply to SourceX on their own, partners introduce companies through their ordinary practice, and some companies have already been in touch with SourceX, which makes them duplicates under section 3 of the terms. A pilot that counts introductions after launching a new activity cannot tell which of them the activity caused.

Clinical trial guidance has a name for the problem. ICH's guideline on choosing control groups (E10), paraphrased from its own text, says a control group exists to separate outcomes caused by the treatment from outcomes caused by other factors, and it calls the inability to control bias the major, well-recognized limitation of comparisons against external or historical controls. A referral pilot is not a clinical trial, but the logic transfers: without a concurrent comparison, "introductions rose after the campaign" may only mean "introductions rose".

TermMeaning on this page
ActivityThe pilot's optional extra: a campaign, a session, office hours, a feature, a new resource sent out
HoldoutA group drawn before launch that does not receive the activity until the reading date
OwedAnything a CEO or partner is entitled to whatever the pilot does; never withheld
Delayed-start groupA holdout that receives the activity on a fixed date after the reading
SpilloverHoldout members who hear about or receive the activity anyway

What a holdout may withhold, and what it never touches

This is the line that makes a holdout fair or unfair. A holdout withholds the pilot's extra effort. It never withholds information, choices or answers.

May be held back until the reading dateNever held back
A proactive outreach campaign from the platform teamSourceX's published pages, including who qualifies and the program terms
An invitation to a workshop or office hoursA full answer to any CEO or partner who asks
A newsletter feature or an emailed resourceA company's ability to apply on its own
A personal walkthrough of the programA CEO's request to be introduced, acted on as it would be without the pilot
A reminder campaign to partnersA partner's ordinary introductions, made in their usual practice
Early access to a new templateCorrection of anything said that turned out to be wrong

A CEO in the holdout who asks about SourceX gets a complete answer, and the row is marked "asked unprompted". It stays in the holdout for the analysis, because that question is exactly the background the holdout exists to count.

The holdout feasibility decision tree

Ask the questions in order. The first no ends the tree and names the fallback.

QuestionIf noIf yes
1. Which decision would the result change, and would a holdout's answer change it?Keep no holdout; run the activity as practice and log what happened and what it costGo to 2
2. Is the question causal (did the activity change what people did) rather than feasibility (can it run safely)?Run a feasibility pilot, which needs no comparison groupGo to 3
3. Could the outcome happen without the activity?Change the outcome: replies to the pilot's own message cannot happen in a holdout, so measure something that can, such as CEO-agreed introductionsGo to 4
4. Can the activity be withheld without withholding anything owed?Give everyone the activity, either in two dated waves or all at once with a before-and-after logGo to 5
5. Will the groups stay apart until the reading date?Hold back whole clusters (chapters, offices, fund vehicles); with too few clusters, use a dated before-and-after logGo to 6
6. Can you see the outcome equally in both groups, in numbers that could move the decision by the reading date?Do not hold anyone back; report the pilot descriptively or lengthen itDraw the holdout before launch, with a fixed end date

Question 6 hides two traps. The first is visibility. A platform team sees introductions its own partners make, but not companies that apply to SourceX directly, so a company-level holdout must be judged on introductions through the network, which both groups can produce and the team can see. The second is size. SourceX publishes no conversion rates, so there is no honest way to compute a sample size from one. Write a floor instead: the smallest number of outcomes, across both groups, below which you will not read the comparison, and the date on which you will look.

Five designs the tree can land on

DesignThe tree lands here whenWhat it can claimWhat it cannot claim
Practice logQuestion 1 says noWhat was done, what happened, what it costThat the activity caused any of it
Feasibility pilotQuestion 2 says noWhether the process runs safely and as writtenAny effect size
Delayed startQuestion 4 says no, or as the ending of any holdoutA fair comparison while the second wave waits, if groups stay apartAnything about people who could not be kept apart
Cluster holdoutQuestion 5 says no and there are enough clustersA comparison between drawn groups of chapters or officesEffects on individuals inside a cluster
Individual holdoutAll six say yesThe cleanest comparison this kind of pilot allowsEffects beyond the population it was drawn from

When the tree ends at a feasibility pilot because only a few companies pass the published fit facts, the small-network feasibility test sets out what to count instead.

Measuring both groups the same way

  • Primary: CEO-agreed introductions submitted through the network's partners, per member or company drawn, read at a fixed date. The CEO's written yes comes first, since SourceX's screening guidance treats an introduction the decision maker never heard about as a poor fit.
  • Progress: verified applications and the stage words partners copy from their dashboards. A credited introduction needs the company's own verified application within the 90-day window, so the reading date sits at least 90 days after the last introduction counted.
  • Duplicates: companies SourceX already knew, reported as their own row in both groups.
  • Spillover: holdout members who received or heard about the activity anyway, counted and published.

Trust signals need separate handling. Stop requests, complaints and notes that an approach felt like pressure can only arise where someone was contacted, so the holdout cannot benchmark them. Give them an absolute threshold written in advance, such as one complaint about pressure pausing the activity, rather than a comparison.

Stop rules belong in the plan before launch. A holdout CEO refused an answer is a breach that pauses the pilot. Spillover above a share you write down in advance turns the comparison into a descriptive report. A harm threshold crossed in the active group stops the activity for everyone. On the end date, the holdout receives the activity if it proved neutral or helpful; if it proved harmful, nobody does.

Three pilots through the tree (Illustrative)

Illustrative. Generic networks, invented settings, no results implied.

PilotWhere it leaves the treeDesign chosenWhy
A seed fund with nine companies that pass the known fit facts asks whether its first referral protocol worksQuestion 2: a feasibility questionFeasibility pilot, no holdoutNine companies can show whether the process holds; they cannot carry a comparison
A growth fund's platform team plans a data-licensing feature in its newsletter to 120 portfolio foundersQuestion 5: founders share one community channel, and there are only three fund vehicles to cluster byDated before-and-after log, limits statedA feature reposted in a shared channel reaches everyone, and three clusters are too few to draw
An angel network of 400 members in eight city chapters tests monthly office hours with its platform teamPasses all sixCluster holdout: four chapters drawn by lot start now, four after the reading dateChapters rarely mix, members' introductions are visible on their own dashboards, and the floor is written down before launch

In the second pilot, the log compares the quarter before the feature with the quarter after, names everything else that changed in between and calls the result descriptive. In the third, the four delayed chapters get office hours on a fixed date whatever the result shows, unless a harm threshold was crossed.

Two published designs show the pieces in use: the MSP pilot that protects client trust holds back a few accounts to read service scores against them, and the workshop test uses a later group that attends after the counting window, which is the delayed-start ending in practice.

Where a holdout misleads even when the tree says yes

  • Spillover you did not see. Members talk outside every channel you know about; a small measured spillover is a floor, not a ceiling.
  • Unequal attention. A team that knows which group is which may answer holdout questions more slowly; the never-withhold list applies to tone as well as content.
  • Telling people they were held back. Announcing the draw can make holdout members act differently; describe the pilot without naming groups until the reading.
  • Moving programs. SourceX checks relevance to current data needs in review, and programs can change mid-pilot; a dated change affects both groups and is reported.
  • Credit noise. One partner is credited per company and SourceX decides overlapping claims from dated records, so introductions counted by the pilot may later be credited elsewhere; count introductions, not credit.
  • A clean result that does not travel. A holdout drawn from one network describes that network; screening changes and similar designs show how a fixed denominator keeps the reading honest within it.

Method note. The tree applies SourceX's published terms, who-qualifies and how-it-works pages, checked 2026-10-28, and the control-group reasoning of ICH E10. It sets no sample sizes, because no published conversion rate exists to set them from, and it leaves out demand, record quality and payouts.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

Whatever design the tree picks, every introduction inside it starts the same way: register, make sure the person submitting has signed the Channel Partner Agreement, check the company against who qualifies and the program terms, and browse related designs in the guides hub.

Designing a fair pilot is referral work. Evaluating SourceX as a company to invest in is a separate decision, made from the materials SourceX shares on request through its investor overview.

This article is for information only and is not an offer to sell or a solicitation of an offer to buy securities. Any offering by SourceX is made only to verified accredited investors through its offering documents.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Is it fair to the companies held back?

Only when the holdout withholds an optional extra and nothing a CEO is owed. Held-back companies keep every published page, can apply on their own, and get a full answer if their CEO asks; the pilot simply does not start the extra activity with them before the reading date.

Can last year's numbers stand in for a holdout?

They can be reported, not relied on. Comparisons with an earlier period cannot rule out everything else that changed, including SourceX's own programs, which set their own criteria and can move. Label a before-and-after count as descriptive and state what else changed in the meantime.

What if a held-back company is introduced by another partner?

Let it happen and record it. A holdout never blocks anyone's ordinary introductions, SourceX credits a single introduction per company from dated records, and an introduction from outside the pilot is part of the background the holdout exists to measure.

Sources

  1. refer.sourcex.si/terms
  2. refer.sourcex.si/how-it-works
  3. refer.sourcex.si/who-qualifies
  4. database.ich.org/sites/default/files/E10_Guideline.pdf
  5. sourcex.si/investors

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment