Put this into practice
Run A/B tests and personalize your store without code. 14-day free trial, no credit card.
Start free trial →Run A/B tests and personalize your store without code. 14-day free trial, no credit card.
Start free trial →A z-test is a statistical hypothesis test used to compare two proportions — in A/B testing, most commonly the conversion rates of a control group and a variant group. When sample sizes are large (generally 30+ per group, which is almost always true in ecommerce), the z-test for proportions is the standard method for determining whether the observed difference in conversion rates between two groups is statistically significant or within the range of random variation.
z = (p₁ − p₂) / √[p̂(1 − p̂)(1/n₁ + 1/n₂)]
Where:
The z-score is then looked up in a standard normal distribution table (or converted using software) to derive the p-value. For a two-tailed test at 95% confidence, the critical z-value is ±1.96.
The z-test is the engine behind the "confidence" percentage you see in most A/B testing dashboards. When your tool says "95% confident the variant is better," it has computed a z-statistic and found that it exceeds the critical threshold. Understanding this means you can interpret results more accurately — and catch situations where a tool might be using a different variant of the test (e.g., one-tailed vs. two-tailed) than you expect, which changes the effective confidence level.
Plum Goodness ran a product page test comparing their original CTA ("Add to Cart") against a variant ("Add to Bag — Ships in 24 hrs"). Control: 18,400 visitors, 756 add-to-carts (4.11% CR). Variant: 18,600 visitors, 874 add-to-carts (4.70% CR). Pooled proportion: (756 + 874) / (18,400 + 18,600) = 4.41%. z = (0.0470 − 0.0411) / √[0.0441 × 0.9559 × (1/18,400 + 1/18,600)] = 4.82, p < 0.0001. The result was highly significant — the urgency-framed CTA clearly outperformed the generic one.
Most A/B testing platforms use z-tests or chi-square tests interchangeably for conversion rate significance.
The z-test is what most A/B testing platforms compute when you test a binary metric like conversion rate. It's fast, mathematically elegant, and well-suited to the high-volume, binary-outcome nature of ecommerce experimentation. The z-score and its associated p-value are the direct inputs to your significance threshold decision.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.