Run A/B tests and personalize your store without code. 14-day free trial, no credit card.
Start free trial →In-depth guides and case studies where this concept is put to work.
A t-test is a statistical test that compares the means of two groups to determine whether the difference between them is statistically significant. In A/B testing, t-tests are most appropriate for continuous metrics — average order value, revenue per visitor, time on page, or items per cart — where you are comparing averages rather than proportions. The t-test accounts for the variability within each group (the spread of individual order values) when calculating whether the mean difference is meaningful or just noise.
For an independent samples t-test (the type used in A/B testing):
t = (Mean₁ − Mean₂) / √[(s₁²/n₁) + (s₂²/n₂)]
Where:
The t-statistic is converted to a p-value using a t-distribution with degrees of freedom ≈ n₁ + n₂ − 2. If p < 0.05, the difference in means is statistically significant at the 95% confidence level.
Conversion rate tests use chi-square or z-tests (binary outcomes). But when your experiment's success metric is average order value, cart size, or revenue per visitor, those tests don't apply — you need a t-test. AOV data in ecommerce is notoriously noisy: most orders cluster around a typical value, but a small number of high-value orders skew the mean. The t-test handles this by incorporating variance directly into the significance calculation, meaning it's less likely to be fooled by a few outlier orders into declaring a spurious winner.
Sugar Cosmetics ran a bundle recommendation experiment where the variant showed "Frequently Bought Together" sets on the cart page. Their primary metric was AOV. Control AOV: ₹1,140 (standard deviation ₹680, n = 8,200). Variant AOV: ₹1,290 (SD ₹710, n = 8,400). Running a two-sample t-test gave t = 8.7, p < 0.001 — a highly significant result. The high SD (large spread in order values) was expected for a cosmetics brand with a wide price range; the t-test correctly accounted for this variance and confirmed the result was genuine.
Advanced testing platforms allow you to specify revenue or AOV as an experiment metric and will apply a t-test automatically. In Python, scipy.stats.ttest_ind runs an independent samples t-test; in R, t.test() does the same. Understanding which test your platform uses for which metric type helps you interpret results accurately.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.