Sample size is the minimum number of visitors (sessions or unique users) that each variant in your A/B test needs to receive before you can draw statistically reliable conclusions. Running a test with too few visitors produces results that are largely noise — you might see a 20% lift on 300 sessions that disappears entirely when 3,000 sessions are observed. Getting sample size right is a prerequisite for trustworthy experimentation.
The required sample size per variant depends on three inputs:
n = (z² × p(1 − p)) / e²
Where:
- z = z-score for your confidence level (1.96 for 95% confidence)
- p = baseline conversion rate (your current control rate)
- e = minimum detectable effect (MDE) — the smallest lift you care about detecting
Example: If your current conversion rate is 3% and you want to detect a 15% relative improvement (from 3% to 3.45%), with 95% confidence, you need approximately 15,000 sessions per variant. Most A/B testing tools include a sample size calculator — input your baseline rate and MDE and it does the math.
Why Sample Size Matters for Ecommerce
An underpowered test is worse than no test at all because it gives you false confidence in noisy data. D2C brands with lower-traffic stores — say, 20,000 monthly sessions — face a real constraint: to reach the sample sizes needed for reliable A/B test results, tests may need to run for 3–6 weeks. That's often longer than the festive season itself. Understanding sample size requirements helps you prioritize which tests to run (high-traffic pages first), set realistic test durations, and avoid the "peeking problem" of calling tests early. It also helps you size up realistically: if you need 50,000 sessions per variant to detect a 10% lift and you only get 5,000 sessions a week, that test will take months.
Real-World Example
A D2C kitchenware brand on Shopify wanted to test a new checkout page design. Their checkout page received about 2,000 sessions per week with a 58% completion rate. Using a sample size calculator, they found they'd need 12,000 sessions per variant to detect a 5% relative improvement at 95% confidence — a 6-week test at their current traffic. Instead of testing on checkout, they moved the test to their product listing page (12,000 sessions/week) and ran the same test in 2 weeks. The learning: design your test program around your traffic reality, not your ideal.
How to Improve / Optimize Sample Size
- Increase traffic to tested pages: More sessions per week means you reach required sample sizes faster. Paid traffic, email campaigns, or push notifications to tested pages all help.
- Increase your minimum detectable effect: If you're willing to detect only large improvements (20%+ lifts), you need far smaller sample sizes. This means prioritizing high-impact hypotheses.
- Test on higher-conversion pages: A checkout page with 60% completion rate needs fewer sessions to detect the same relative change as a product page with 2% conversion rate.
- Reduce the number of variants: Every additional variant splits traffic further and increases the sample size needed for each. Stick to two variants (A vs. B) when traffic is limited.
- Use sequential testing methods: Bayesian testing approaches can sometimes require fewer sessions for equivalent confidence compared to traditional frequentist methods.
Sample Size in A/B Testing
Sample size planning is the first step of any A/B test — not an afterthought. Before you launch a test, calculate the required sample size, estimate how long it will take at current traffic levels, and decide whether the test is feasible. A test that will take 6 months to reach significance on a page you redesign every quarter is a test you shouldn't run.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.