Can your churn test detect anything?
Most retention tests are too small to tell a real effect from luck, and nobody checks before they start. Enter four numbers and find out how many accounts yours needs.
Why the numbers get so large
Churn is a rare event measured on a limited number of accounts. Two groups of 100 identical accounts will often differ by several points through chance alone. A test has to be big enough that the effect you care about stands clear of that.
Two things shrink the number. A bigger effect is easier to see. And a higher starting churn rate gives the test more events to work with, which is why testing on an at-risk segment needs a fraction of the accounts that testing on the whole base does.
If your test is too small
Narrow the group. Test where churn is concentrated, not across everyone.
Measure something earlier. A leading sign that moves faster than cancellation, agreed and written down before the test starts.
Run longer, or pool two rounds. New accounts keep arriving.
Or run it anyway and call it what it is: a direction with a range, not a verdict. That is still better than a before-and-after chart.
The math
A two-sided test of two proportions with equal group sizes, at 95% confidence (α = 0.05) and 80% power, using Cohen's effect size h = 2·asin(√p₁) − 2·asin(√p₂). Accounts per group = 2 · (z0.975 + z0.80)² / h², rounded up. This is the same calculation as NormalIndPower in Python's statsmodels.
It assumes accounts are assigned at random and each account's outcome is independent. It does not adjust for accounts that leave the test early or for checking results before the end, both of which need more accounts, not fewer. Nothing you type here is sent anywhere. Privacy.
I show clients this number before they pay.
The Retention Diagnostic finds the group leaving fastest, sizes the test with this math, and reads the result against a comparison group. If your group is too small for a verdict, you hear that before you sign.
Don't know your group's churn rate yet? Curve Split draws it from a subscriptions export.