# Referral experiment failure criteria | SourceX

Canonical: https://refer.sourcex.si/guides/a-referral-experiment-with-clear-failure-criteria
Last updated: 2026-11-04

Failure criteria are the results, written down before a test starts, that would make you stop it or reject the change. A referral test needs four kinds: trust failures (a CEO pressed, or introduced without knowing), quality failures (more submissions but no more introductions reaching a verified application), waste failures (the extra submissions are duplicates or miss the published fit facts) and design failures (the test cannot be read). Each gets a count, a check date and a consequence, fixed before the first send. This page reports no result.

The worked case is the cheapest lever a partner growth manager has: a monthly reminder to registered partners. It is also the lever most likely to raise activity without raising anything that matters, which is why it is a good place to practice writing failure criteria. Other designs in this cluster test what partners are told before they start, such as [the enablement kit before a rollout](/guides/the-partner-enablement-test-before-a-large-network-rollout) and [where the reward sits in a recruitment invitation](/guides/the-experiment-that-separates-reward-interest-from-company-fit); this one tests a nudge sent after registration.

## Why a reminder test needs failure criteria more than most

A reminder works on partners, but its effects land on CEOs. A partner nudged into action may write to a CEO they barely know, submit a company they never checked, or submit one SourceX already knows. None of that helps anyone. Section 5 of the terms says, "A lead, meeting or signed agreement alone does not earn a payable reward," and under section 3 a company that already has "any earlier lead, account or email correspondence" with SourceX is a duplicate and is not credited to a later partner.

Microsoft's experimentation team names the kind of measure that catches this. Its during-experiment guidance lists guardrail metrics, which "Measure aspects of the product that we don't want to degrade but won't necessarily improve." In a referral program the guardrails are plain: the CEO knew about and wanted the introduction, nobody felt pressed, and partners still want to hear from you. A reminder can lift every activity count while quietly wearing those down.

| Term | How this page uses it |
|---|---|
| Allocated partner | A registered partner assigned to an arm before the first reminder, whether or not they ever open one |
| Qualified introduction | An introduction the CEO agreed to in writing that reaches a verified application inside the 90-day window, with no published fit fact failing |
| Submission | Any introduction a partner sends, qualified or not |
| Failure criterion | A count, fixed in advance, that triggers a named consequence when reached |
| Read date | Day 120 after the third reminder, so every introduction prompted by it has run its 90-day window |

## The reminder test, written down before anything is sent

| Element | Setting (Illustrative) |
|---|---|
| Sampling frame | Every partner in your network who registered at least 60 days ago, has signed the Channel Partner Agreement and has not opted out of program email |
| Consent | A notice in the welcome email: dates and dashboard stages of introductions may be counted to improve the program; any partner may ask to be left out |
| Allocation | A list drawn before the first send, in blocks by office, so partners who share an office and its CEOs land in the same arm |
| Arm A | Current practice: no reminder |
| Arm B | One reminder on the first business day of each month, three months, fixed text |
| Primary count | Qualified introductions per allocated partner, arm B against arm A |
| Beside it | Submissions per allocated partner; duplicates; submissions with a failed fit fact |
| Negative feedback | Partners who opt out of reminders or deregister; any CEO complaint reaching the partner, the network or SourceX |
| Data held | Counts by dashboard stage, which partners read from their own dashboards; a code of the partner's choosing per company, never a company name, file or figure |

The reminder itself is short and truthful. It names the published fit facts and links [who qualifies](/who-qualifies), says the company's CEO decides, invites the partner to think of one company whose CEO they know, and offers a one-click way to stop reminders. It never carries a deadline, a payout figure, a list of companies or the words "your contacts qualify".

## The four failure types and their tripwires

This is the page's asset: each failure type with what is counted, the threshold, when it is checked and what happens. The thresholds are study settings; change them if you like, but only before the first send.

| Failure type | What is counted | Threshold (Illustrative) | Checked | Consequence |
|---|---|---|---|---|
| Trust | Introductions submitted without the CEO's written agreement; complaints of pressure or repeated contact; any message to a CEO after a no | One case in arm B | Weekly | Stop reminders at once; review the case; any restart needs new text and a new plan |
| Trust | Arm B partners who opt out of reminders or deregister | More than one in five by the end of month two | Monthly | Stop reminders; partners are telling you the message is unwelcome |
| Design | Arm A partners who received a reminder; arm sizes off the planned split; partners with no outcome data at read | Any contamination, a split off by more than a tenth, or more than a tenth missing | At each check and at read | No verdict; report the counts; fix and rerun |
| Waste | Duplicates plus submissions with a failed fit fact, in arm B | More of these than qualified introductions in arm B | At read | Revise the reminder's fit wording before any rerun |
| Quality | Qualified introductions per allocated partner | Arm B not ahead of arm A by the margin fixed in the plan | At read | Keep current practice |

Two rules sit behind the table. Trust is checked first and continuously, because waiting for the read date to discover pressure would mean spending more CEOs' goodwill to confirm it. And the design check comes before the quality reading. Microsoft's guidance calls arms that drift from the planned split a sample ratio mismatch, and says: "SRMs typically invalidate the A/B test and make any results and metric movements untrustworthy." A referral test whose arms drift apart has the same problem, whatever the counts say.

The margin for quality has to be set with small numbers in mind. A few qualified introductions either way can come from chance alone; [the five-replies arithmetic](/guides/how-to-avoid-declaring-a-winner-from-five-replies) shows how lopsided a small split must be before it means anything.

## The predefined continue-stop decision rules

| Reading | Verdict | What changes | What it never justifies |
|---|---|---|---|
| A trust tripwire is hit at any point | Stop now | Reminders end; the case count is published | Resuming because submissions were up |
| A design failure | No verdict | The fault is fixed and the test rerun | Reading the rates anyway |
| Submissions up, qualified introductions flat | Quality failure | Current practice stays | Counting submissions as success |
| Qualified introductions up, waste threshold crossed | Revise | Fit wording rewritten, test rerun | Adopting the reminder as written |
| Qualified introductions up by the margin, no tripwire crossed | Continue | Monthly reminder adopted, rechecked with the next cohort | Sending reminders more often without a new test |
| No difference on any count | Keep current practice | Nothing; the quieter option wins a tie | Rerunning until something shows |

A stop is a result, not an embarrassment. CONSORT's item on interim analyses, paraphrased from its own page, asks a trial report to say when interim looks happened, what prompted them, how many there were, whether they were planned or improvised, and whether stopping rules were fixed in advance. Publish your checks the same way: every weekly trust check, including the ones that found nothing, and the date and count of any check that fired.

## Every partner and every introduction keeps a row

Every allocated partner stays in the denominator: those who never opened a reminder, those who deregistered and those who made no introduction at all. Every introduction stays too, in exactly one row.

| Row in the report | Arm A | Arm B |
|---|---|---|
| Partners allocated | Pending | Pending |
| Submissions | Pending | Pending |
| CEO declined or never replied | Pending | Pending |
| Duplicates under section 3 | Pending | Pending |
| Failed a published fit fact | Pending | Pending |
| Declined in SourceX's review | Pending | Pending |
| Still open at the read date | Pending | Pending |
| Qualified introductions | Pending | Pending |
| Trust cases and partner opt-outs | Pending | Pending |

An introduction still open at the read date is counted as open, not as a loss. Only the arm comparison supports a causal reading about the reminder, because allocation was random. Anything else, such as why CEOs declined or which offices did better, is description. If the reminder works partly by thinning out weak submissions, the per-partner count still shows it; [the screening cohort design](/guides/when-better-screening-lowers-applications-but-improves-outcomes) explains why the bottom of the fraction has to stay fixed.

## What every reminder must keep true

The reminder can only point partners at companies that might fit, so it has to describe fit correctly. A company needs 50 or more full-time employees at peak, contractors left out; records it created over years of operation; clear rights to license those records; and a CEO who can authorize the discussion and has said yes to it. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

The partner describes, at most, what kind of records exist and roughly how many years they cover. They never request or forward files, samples or logins; SourceX's process page says partners do not need to export, upload or describe confidential records. Whether the records match a current program, whether customer contracts limit their use and what a license would cover stay unknown until SourceX's own review. A reminder that says otherwise fails the trust check on its first send.

## Reading the result with its uncertainty

Even a clean run leaves doubt. A three-month reminder may pull introductions forward rather than add new ones; the day-120 read catches some of that, not all. Partners in different arms talk to each other, which narrows any gap. Dashboard stages can lag, so the report states the date each stage was read. SourceX's criteria follow each program and can move mid-test, and a qualified introduction can still be declined in review for reasons no reminder touches. Microsoft's guidance makes a related point about patience: where movements are not highly certain, it recommends running a test to its pre-determined end rather than calling it early. Hold to the read date unless a tripwire fires.

If the verdict is continue, keep the failure criteria running in ordinary operation; the partner check-in agenda in [the partner check-in guide](/guides/the-partner-check-in-that-produces-better-introductions) is a natural place to read them each quarter. If you cannot reach a sample big enough to read, the alternatives in [when a pilot needs a holdout group](/guides/when-a-pilot-needs-a-holdout-group) cover designs that ask less of a small network.

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

Write your four tripwires before the first reminder goes out; partners who act on one can [register](/register), sign the Channel Partner Agreement, check each company against [who qualifies](/who-qualifies) and read the [program terms](/terms) before any CEO hears about it.

## Sources

- https://refer.sourcex.si/terms
- https://refer.sourcex.si/who-qualifies
- https://refer.sourcex.si/how-it-works
- https://www.microsoft.com/en-us/research/articles/patterns-of-trustworthy-experimentation-during-experiment-stage/
- https://www.consort-spirit.org/item16b-interimanalyses
