A referral experiment with clear failure criteria: a partner-reminder test that names four ways it can fail, fixes the count and the consequence for each before the first reminder goes out, and treats a stop as a finished result

Short answer

Failure criteria are the results, written down before a test starts, that would make you stop it or reject the change. A referral test needs four kinds: trust failures (a CEO pressed, or introduced without knowing), quality failures (more submissions but no more introductions reaching a verified application), waste failures (the extra submissions are duplicates or miss the published fit facts) and design failures (the test cannot be read). Each gets a count, a check date and a consequence, fixed before the first send. This page reports no result.

Visual briefFull-size diagram
A pipeline running left to right from a reminder test through four checkpoints, trust, design, waste and quality; each checkpoint has an orange exit arrow below it labeled stop now, no verdict, revise or keep current practice, and only the path that clears all four reaches a green box reading continue.
Illustrative layout of the continue-stop rules on this page. The reminder test has not run; no results exist.

The worked case is the cheapest lever a partner growth manager has: a monthly reminder to registered partners. It is also the lever most likely to raise activity without raising anything that matters, which is why it is a good place to practice writing failure criteria. Other designs in this cluster test what partners are told before they start, such as the enablement kit before a rollout and where the reward sits in a recruitment invitation; this one tests a nudge sent after registration.

Why a reminder test needs failure criteria more than most

A reminder works on partners, but its effects land on CEOs. A partner nudged into action may write to a CEO they barely know, submit a company they never checked, or submit one SourceX already knows. None of that helps anyone. Section 5 of the terms says, "A lead, meeting or signed agreement alone does not earn a payable reward," and under section 3 a company that already has "any earlier lead, account or email correspondence" with SourceX is a duplicate and is not credited to a later partner.

Microsoft's experimentation team names the kind of measure that catches this. Its during-experiment guidance lists guardrail metrics, which "Measure aspects of the product that we don't want to degrade but won't necessarily improve." In a referral program the guardrails are plain: the CEO knew about and wanted the introduction, nobody felt pressed, and partners still want to hear from you. A reminder can lift every activity count while quietly wearing those down.

TermHow this page uses it
Allocated partnerA registered partner assigned to an arm before the first reminder, whether or not they ever open one
Qualified introductionAn introduction the CEO agreed to in writing that reaches a verified application inside the 90-day window, with no published fit fact failing
SubmissionAny introduction a partner sends, qualified or not
Failure criterionA count, fixed in advance, that triggers a named consequence when reached
Read dateDay 120 after the third reminder, so every introduction prompted by it has run its 90-day window

The reminder test, written down before anything is sent

ElementSetting (Illustrative)
Sampling frameEvery partner in your network who registered at least 60 days ago, has signed the Channel Partner Agreement and has not opted out of program email
ConsentA notice in the welcome email: dates and dashboard stages of introductions may be counted to improve the program; any partner may ask to be left out
AllocationA list drawn before the first send, in blocks by office, so partners who share an office and its CEOs land in the same arm
Arm ACurrent practice: no reminder
Arm BOne reminder on the first business day of each month, three months, fixed text
Primary countQualified introductions per allocated partner, arm B against arm A
Beside itSubmissions per allocated partner; duplicates; submissions with a failed fit fact
Negative feedbackPartners who opt out of reminders or deregister; any CEO complaint reaching the partner, the network or SourceX
Data heldCounts by dashboard stage, which partners read from their own dashboards; a code of the partner's choosing per company, never a company name, file or figure

The reminder itself is short and truthful. It names the published fit facts and links who qualifies, says the company's CEO decides, invites the partner to think of one company whose CEO they know, and offers a one-click way to stop reminders. It never carries a deadline, a payout figure, a list of companies or the words "your contacts qualify".

The four failure types and their tripwires

This is the page's asset: each failure type with what is counted, the threshold, when it is checked and what happens. The thresholds are study settings; change them if you like, but only before the first send.

Failure typeWhat is countedThreshold (Illustrative)CheckedConsequence
TrustIntroductions submitted without the CEO's written agreement; complaints of pressure or repeated contact; any message to a CEO after a noOne case in arm BWeeklyStop reminders at once; review the case; any restart needs new text and a new plan
TrustArm B partners who opt out of reminders or deregisterMore than one in five by the end of month twoMonthlyStop reminders; partners are telling you the message is unwelcome
DesignArm A partners who received a reminder; arm sizes off the planned split; partners with no outcome data at readAny contamination, a split off by more than a tenth, or more than a tenth missingAt each check and at readNo verdict; report the counts; fix and rerun
WasteDuplicates plus submissions with a failed fit fact, in arm BMore of these than qualified introductions in arm BAt readRevise the reminder's fit wording before any rerun
QualityQualified introductions per allocated partnerArm B not ahead of arm A by the margin fixed in the planAt readKeep current practice

Two rules sit behind the table. Trust is checked first and continuously, because waiting for the read date to discover pressure would mean spending more CEOs' goodwill to confirm it. And the design check comes before the quality reading. Microsoft's guidance calls arms that drift from the planned split a sample ratio mismatch, and says: "SRMs typically invalidate the A/B test and make any results and metric movements untrustworthy." A referral test whose arms drift apart has the same problem, whatever the counts say.

The margin for quality has to be set with small numbers in mind. A few qualified introductions either way can come from chance alone; the five-replies arithmetic shows how lopsided a small split must be before it means anything.

The predefined continue-stop decision rules

ReadingVerdictWhat changesWhat it never justifies
A trust tripwire is hit at any pointStop nowReminders end; the case count is publishedResuming because submissions were up
A design failureNo verdictThe fault is fixed and the test rerunReading the rates anyway
Submissions up, qualified introductions flatQuality failureCurrent practice staysCounting submissions as success
Qualified introductions up, waste threshold crossedReviseFit wording rewritten, test rerunAdopting the reminder as written
Qualified introductions up by the margin, no tripwire crossedContinueMonthly reminder adopted, rechecked with the next cohortSending reminders more often without a new test
No difference on any countKeep current practiceNothing; the quieter option wins a tieRerunning until something shows

A stop is a result, not an embarrassment. CONSORT's item on interim analyses, paraphrased from its own page, asks a trial report to say when interim looks happened, what prompted them, how many there were, whether they were planned or improvised, and whether stopping rules were fixed in advance. Publish your checks the same way: every weekly trust check, including the ones that found nothing, and the date and count of any check that fired.

Every partner and every introduction keeps a row

Every allocated partner stays in the denominator: those who never opened a reminder, those who deregistered and those who made no introduction at all. Every introduction stays too, in exactly one row.

Row in the reportArm AArm B
Partners allocatedPendingPending
SubmissionsPendingPending
CEO declined or never repliedPendingPending
Duplicates under section 3PendingPending
Failed a published fit factPendingPending
Declined in SourceX's reviewPendingPending
Still open at the read datePendingPending
Qualified introductionsPendingPending
Trust cases and partner opt-outsPendingPending

An introduction still open at the read date is counted as open, not as a loss. Only the arm comparison supports a causal reading about the reminder, because allocation was random. Anything else, such as why CEOs declined or which offices did better, is description. If the reminder works partly by thinning out weak submissions, the per-partner count still shows it; the screening cohort design explains why the bottom of the fraction has to stay fixed.

What every reminder must keep true

The reminder can only point partners at companies that might fit, so it has to describe fit correctly. A company needs 50 or more full-time employees at peak, contractors left out; records it created over years of operation; clear rights to license those records; and a CEO who can authorize the discussion and has said yes to it. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

The partner describes, at most, what kind of records exist and roughly how many years they cover. They never request or forward files, samples or logins; SourceX's process page says partners do not need to export, upload or describe confidential records. Whether the records match a current program, whether customer contracts limit their use and what a license would cover stay unknown until SourceX's own review. A reminder that says otherwise fails the trust check on its first send.

Reading the result with its uncertainty

Even a clean run leaves doubt. A three-month reminder may pull introductions forward rather than add new ones; the day-120 read catches some of that, not all. Partners in different arms talk to each other, which narrows any gap. Dashboard stages can lag, so the report states the date each stage was read. SourceX's criteria follow each program and can move mid-test, and a qualified introduction can still be declined in review for reasons no reminder touches. Microsoft's guidance makes a related point about patience: where movements are not highly certain, it recommends running a test to its pre-determined end rather than calling it early. Hold to the read date unless a tripwire fires.

If the verdict is continue, keep the failure criteria running in ordinary operation; the partner check-in agenda in the partner check-in guide is a natural place to read them each quarter. If you cannot reach a sample big enough to read, the alternatives in when a pilot needs a holdout group cover designs that ask less of a small network.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

Write your four tripwires before the first reminder goes out; partners who act on one can register, sign the Channel Partner Agreement, check each company against who qualifies and read the program terms before any CEO hears about it.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Is a stopped experiment a failed experiment?

No. A stop triggered by a rule you wrote in advance is a finished result: it tells you the change cost more than it returned, on the measure you chose before you could see the data. Publish it with the same care as a success.

Why is one complaint enough to stop the reminders?

Because the reminder exists to produce introductions a CEO wanted, and a complaint about pressure shows the opposite happened at least once. The threshold is a study setting; a growth manager can set it differently, but only before the first send.

Can we judge the reminder on submissions if verified applications take too long?

Not as the deciding count. Credit under section 2 of the terms needs a verified application, and section 5 says a lead or meeting alone earns no payable reward. Submissions are reported beside the deciding count, never in its place.

Sources

  1. refer.sourcex.si/terms
  2. refer.sourcex.si/who-qualifies
  3. refer.sourcex.si/how-it-works
  4. microsoft.com/en-us/research/articles/patterns-of-trustwor…
  5. consort-spirit.org/item16b-interimanalyses

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment