A/B testing statistical significance tells you the probability that the observed difference between your control and variant is real and not due to random chance — but significance alone does not tell you whether to ship, because a statistically significant result can still be too small to matter or too expensive to maintain.
Statistical significance is the most misunderstood concept in product analytics. Teams ship variants because they hit p<0.05 without understanding what that means. They stop tests early when they see a winning result. They celebrate 0.1% conversion lifts as victories without calculating whether the engineering maintenance cost is worth it. This guide gives product managers a practical framework for making confident, statistically sound ship decisions.
What Statistical Significance Actually Means
When you run an A/B test with p=0.05, it means: if there were no real difference between control and variant, you would see a result this extreme or more extreme by chance 5% of the time.
This is not the same as: "There is a 95% probability that the variant is better."
H3: Type I and Type II Errors
- Type I error (false positive): You conclude the variant is better when it isn't. Probability = alpha (typically 0.05). Result: you ship something that doesn't help.
- Type II error (false negative): You conclude there's no difference when there is one. Probability = beta (typically 0.20). Result: you don't ship something that would have helped.
Most teams worry about false positives. The more common problem is false negatives from underpowered tests.
The Four Requirements for a Valid A/B Test
H3: Requirement 1 — Minimum Sample Size
Before running a test, calculate the required sample size based on:
- Baseline conversion rate: What is the current metric value?
- Minimum detectable effect (MDE): What is the smallest improvement worth detecting?
- Statistical power: 80% is standard (meaning you'll miss real effects 20% of the time)
- Significance level: 0.05 is standard (5% false positive rate)
Rule of thumb: Smaller effects require larger samples. A 1% lift on a 10% baseline requires ~100,000 users per variant. A 10% lift on a 10% baseline requires ~10,000 users per variant.
According to Lenny Rachitsky's writing on experimentation, the most common A/B testing mistake at startups is running underpowered tests — the sample size is too small to detect the effect size the team actually cares about, so they get inconclusive results and make subjective decisions anyway.
H3: Requirement 2 — Pre-Registration
Define your hypothesis, primary metric, and success threshold before looking at any data. Changing the primary metric after seeing partial results is p-hacking — it inflates your false positive rate dramatically.
H3: Requirement 3 — Full Test Duration
Run the test for the full pre-calculated duration. Peeking at results daily and stopping early when you see significance inflates the false positive rate. The "always valid" sequential testing methods (like the mSPRT) are designed for continuous monitoring — standard t-tests are not.
Minimum test duration: 1-2 full business cycles (usually 1-2 weeks minimum) to account for day-of-week effects.
H3: Requirement 4 — Metric Guardrails
For every primary metric you're trying to improve, monitor at least one guardrail metric that you must not degrade:
- Improving conversion rate while tracking support ticket volume (did we confuse users?)
- Improving session length while tracking satisfaction score (did we add friction?)
- Improving feature adoption while tracking other feature adoption (did we cannibalize?)
According to Shreyas Doshi on Lenny's Podcast, the single most valuable discipline in A/B testing is defining guardrail metrics before starting the test — teams that don't monitor guardrails frequently ship variants that improve one metric while silently degrading another.
When to Ship Despite Non-Significance
Statistical significance is not the only criterion for shipping. Ship if:
H3: Directional Positive with Low Cost
If the result trends positive (even if not significant) and the change is simple copy or color, ship. The expected value of a directional positive exceeds the cost of a borderline call.
H3: Engineering Maintenance Cost Justifies It
A statistically significant 0.05% conversion lift on 1 million monthly visitors = 500 more conversions/month. If the variant requires ongoing engineering maintenance, is that worth 500 conversions? Often no.
When Not to Ship Despite Significance
- Effect size is trivially small: Significance + tiny effect = true but meaningless
- Guardrail metric degraded: A significant conversion improvement that degrades NPS is a bad trade
- Test population isn't representative: Test ran during a holiday period, product launch, or other anomalous period
According to Annie Pearl on Lenny's Podcast discussing experimentation culture, the teams that make the best A/B test decisions are the ones that treat significance as a necessary but not sufficient condition for shipping — they also evaluate effect size, guardrail impact, and implementation cost before making the ship call.
FAQ
Q: What is statistical significance in A/B testing? A: The probability that the observed difference between control and variant is not due to random chance. At p=0.05, if there were no real difference, you would observe a result this extreme 5% of the time by chance.
Q: How do you calculate sample size for an A/B test? A: Use a sample size calculator with your baseline conversion rate, minimum detectable effect, statistical power of 80%, and significance level of 0.05. Smaller effects and lower baselines require larger samples.
Q: Why should you not stop an A/B test early? A: Early stopping when you observe significance inflates the false positive rate. A test designed for 1000 users per variant stopped at 200 because results look good will produce unreliable conclusions.
Q: What is a guardrail metric in A/B testing? A: A metric you monitor alongside your primary metric to ensure the variant doesn't improve one dimension while degrading another. For example, monitoring support volume when testing a conversion optimization.
Q: When should you ship a variant despite non-significance? A: When the result trends directionally positive, the change is low-cost to maintain, and the effect size is practically meaningful even if the sample is too small for statistical confidence.
HowTo: Run a Statistically Valid A/B Test
- Define your hypothesis, primary metric, minimum detectable effect, and guardrail metrics before looking at any data or building any variant
- Calculate the required sample size using your baseline conversion rate, minimum detectable effect, 80 percent power, and 0.05 significance level
- Run the test for the full calculated duration including at least one full business cycle — do not stop early even if early results look significant
- Monitor guardrail metrics throughout the test to catch variants that improve the primary metric while degrading another dimension
- Evaluate the result on three criteria: statistical significance, practical effect size, and guardrail metric impact
- Make the ship decision based on all three criteria — a significant but tiny effect or a significant effect with guardrail degradation may still be the wrong call to ship