"Holdout" also has an AI-evaluation meaning, a set of real cases kept out of training so a model can be tested on them. This page is about people and companies in a referral pilot.
A holdout answers one question: what would have happened anyway
Every referral network has background activity. Companies can apply to SourceX on their own, partners introduce companies through their ordinary practice, and some companies have already been in touch with SourceX, which makes them duplicates under section 3 of the terms. A pilot that counts introductions after launching a new activity cannot tell which of them the activity caused.
Clinical trial guidance has a name for the problem. ICH's guideline on choosing control groups (E10), paraphrased from its own text, says a control group exists to separate outcomes caused by the treatment from outcomes caused by other factors, and it calls the inability to control bias the major, well-recognized limitation of comparisons against external or historical controls. A referral pilot is not a clinical trial, but the logic transfers: without a concurrent comparison, "introductions rose after the campaign" may only mean "introductions rose".
| Term | Meaning on this page |
|---|---|
| Activity | The pilot's optional extra: a campaign, a session, office hours, a feature, a new resource sent out |
| Holdout | A group drawn before launch that does not receive the activity until the reading date |
| Owed | Anything a CEO or partner is entitled to whatever the pilot does; never withheld |
| Delayed-start group | A holdout that receives the activity on a fixed date after the reading |
| Spillover | Holdout members who hear about or receive the activity anyway |
What a holdout may withhold, and what it never touches
This is the line that makes a holdout fair or unfair. A holdout withholds the pilot's extra effort. It never withholds information, choices or answers.
| May be held back until the reading date | Never held back |
|---|---|
| A proactive outreach campaign from the platform team | SourceX's published pages, including who qualifies and the program terms |
| An invitation to a workshop or office hours | A full answer to any CEO or partner who asks |
| A newsletter feature or an emailed resource | A company's ability to apply on its own |
| A personal walkthrough of the program | A CEO's request to be introduced, acted on as it would be without the pilot |
| A reminder campaign to partners | A partner's ordinary introductions, made in their usual practice |
| Early access to a new template | Correction of anything said that turned out to be wrong |
A CEO in the holdout who asks about SourceX gets a complete answer, and the row is marked "asked unprompted". It stays in the holdout for the analysis, because that question is exactly the background the holdout exists to count.
The holdout feasibility decision tree
Ask the questions in order. The first no ends the tree and names the fallback.
| Question | If no | If yes |
|---|---|---|
| 1. Which decision would the result change, and would a holdout's answer change it? | Keep no holdout; run the activity as practice and log what happened and what it cost | Go to 2 |
| 2. Is the question causal (did the activity change what people did) rather than feasibility (can it run safely)? | Run a feasibility pilot, which needs no comparison group | Go to 3 |
| 3. Could the outcome happen without the activity? | Change the outcome: replies to the pilot's own message cannot happen in a holdout, so measure something that can, such as CEO-agreed introductions | Go to 4 |
| 4. Can the activity be withheld without withholding anything owed? | Give everyone the activity, either in two dated waves or all at once with a before-and-after log | Go to 5 |
| 5. Will the groups stay apart until the reading date? | Hold back whole clusters (chapters, offices, fund vehicles); with too few clusters, use a dated before-and-after log | Go to 6 |
| 6. Can you see the outcome equally in both groups, in numbers that could move the decision by the reading date? | Do not hold anyone back; report the pilot descriptively or lengthen it | Draw the holdout before launch, with a fixed end date |
Question 6 hides two traps. The first is visibility. A platform team sees introductions its own partners make, but not companies that apply to SourceX directly, so a company-level holdout must be judged on introductions through the network, which both groups can produce and the team can see. The second is size. SourceX publishes no conversion rates, so there is no honest way to compute a sample size from one. Write a floor instead: the smallest number of outcomes, across both groups, below which you will not read the comparison, and the date on which you will look.
Five designs the tree can land on
| Design | The tree lands here when | What it can claim | What it cannot claim |
|---|---|---|---|
| Practice log | Question 1 says no | What was done, what happened, what it cost | That the activity caused any of it |
| Feasibility pilot | Question 2 says no | Whether the process runs safely and as written | Any effect size |
| Delayed start | Question 4 says no, or as the ending of any holdout | A fair comparison while the second wave waits, if groups stay apart | Anything about people who could not be kept apart |
| Cluster holdout | Question 5 says no and there are enough clusters | A comparison between drawn groups of chapters or offices | Effects on individuals inside a cluster |
| Individual holdout | All six say yes | The cleanest comparison this kind of pilot allows | Effects beyond the population it was drawn from |
When the tree ends at a feasibility pilot because only a few companies pass the published fit facts, the small-network feasibility test sets out what to count instead.
Measuring both groups the same way
- Primary: CEO-agreed introductions submitted through the network's partners, per member or company drawn, read at a fixed date. The CEO's written yes comes first, since SourceX's screening guidance treats an introduction the decision maker never heard about as a poor fit.
- Progress: verified applications and the stage words partners copy from their dashboards. A credited introduction needs the company's own verified application within the 90-day window, so the reading date sits at least 90 days after the last introduction counted.
- Duplicates: companies SourceX already knew, reported as their own row in both groups.
- Spillover: holdout members who received or heard about the activity anyway, counted and published.
Trust signals need separate handling. Stop requests, complaints and notes that an approach felt like pressure can only arise where someone was contacted, so the holdout cannot benchmark them. Give them an absolute threshold written in advance, such as one complaint about pressure pausing the activity, rather than a comparison.
Stop rules belong in the plan before launch. A holdout CEO refused an answer is a breach that pauses the pilot. Spillover above a share you write down in advance turns the comparison into a descriptive report. A harm threshold crossed in the active group stops the activity for everyone. On the end date, the holdout receives the activity if it proved neutral or helpful; if it proved harmful, nobody does.
Three pilots through the tree (Illustrative)
Illustrative. Generic networks, invented settings, no results implied.
| Pilot | Where it leaves the tree | Design chosen | Why |
|---|---|---|---|
| A seed fund with nine companies that pass the known fit facts asks whether its first referral protocol works | Question 2: a feasibility question | Feasibility pilot, no holdout | Nine companies can show whether the process holds; they cannot carry a comparison |
| A growth fund's platform team plans a data-licensing feature in its newsletter to 120 portfolio founders | Question 5: founders share one community channel, and there are only three fund vehicles to cluster by | Dated before-and-after log, limits stated | A feature reposted in a shared channel reaches everyone, and three clusters are too few to draw |
| An angel network of 400 members in eight city chapters tests monthly office hours with its platform team | Passes all six | Cluster holdout: four chapters drawn by lot start now, four after the reading date | Chapters rarely mix, members' introductions are visible on their own dashboards, and the floor is written down before launch |
In the second pilot, the log compares the quarter before the feature with the quarter after, names everything else that changed in between and calls the result descriptive. In the third, the four delayed chapters get office hours on a fixed date whatever the result shows, unless a harm threshold was crossed.
Two published designs show the pieces in use: the MSP pilot that protects client trust holds back a few accounts to read service scores against them, and the workshop test uses a later group that attends after the counting window, which is the delayed-start ending in practice.
Where a holdout misleads even when the tree says yes
- Spillover you did not see. Members talk outside every channel you know about; a small measured spillover is a floor, not a ceiling.
- Unequal attention. A team that knows which group is which may answer holdout questions more slowly; the never-withhold list applies to tone as well as content.
- Telling people they were held back. Announcing the draw can make holdout members act differently; describe the pilot without naming groups until the reading.
- Moving programs. SourceX checks relevance to current data needs in review, and programs can change mid-pilot; a dated change affects both groups and is reported.
- Credit noise. One partner is credited per company and SourceX decides overlapping claims from dated records, so introductions counted by the pilot may later be credited elsewhere; count introductions, not credit.
- A clean result that does not travel. A holdout drawn from one network describes that network; screening changes and similar designs show how a fixed denominator keeps the reading honest within it.
Method note. The tree applies SourceX's published terms, who-qualifies and how-it-works pages, checked 2026-10-28, and the control-group reasoning of ICH E10. It sets no sample sizes, because no published conversion rate exists to set them from, and it leaves out demand, record quality and payouts.
Next step
Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
Whatever design the tree picks, every introduction inside it starts the same way: register, make sure the person submitting has signed the Channel Partner Agreement, check the company against who qualifies and the program terms, and browse related designs in the guides hub.
Designing a fair pilot is referral work. Evaluating SourceX as a company to invest in is a separate decision, made from the materials SourceX shares on request through its investor overview.
This article is for information only and is not an offer to sell or a solicitation of an offer to buy securities. Any offering by SourceX is made only to verified accredited investors through its offering documents.