Run A/B tests and personalize your store without code. 14-day free trial, no credit card.
Start free trial →In-depth guides and case studies where this concept is put to work.
Sample Ratio Mismatch (SRM) is a data quality problem in A/B testing that occurs when the actual ratio of users assigned to each variant differs significantly from the intended ratio. If you set up an A/B test with a 50/50 traffic split and end up with 52,000 users in control vs. 44,000 in the variant, that 8,000-user discrepancy is a sample ratio mismatch. SRM invalidates the test results because something in the assignment or tracking pipeline is behaving differently for different groups — meaning the two groups are not truly comparable, and any observed conversion difference may be caused by the mismatch, not the variant itself.
Detection: Run a chi-squared test on the observed vs. expected counts. If p < 0.01, SRM is present and the test should not be trusted.
SRM is one of the most common and most overlooked sources of invalid A/B test results. When it is present, your test is telling you something about your tracking infrastructure, not your variant. If you ship a "winning" variant based on SRM-affected data, you may ship a change that actually harms conversion — the lift you observed was an artifact of the mismatch.
For Indian D2C brands relying on A/B test data to make product page, pricing, and UX decisions, SRM represents a real risk. Common causes include bot traffic being blocked differently for control vs. variant, variant pages loading slower (causing early exits), the variant JavaScript failing on certain browsers, or CDN caching serving some users stale content. Any of these create biased samples that look like legitimate results.
At scale, SRM can cost significant revenue. A brand doing ₹5 crore/month that ships a "losing" variant as a "winner" due to SRM could see a 5–10% conversion drop — ₹25–50 lakh/month — before someone investigates.
A D2C electronics accessories brand (similar to Boat) runs an A/B test on their checkout page, testing a simplified one-page checkout against the control multi-step flow. After 7 days, the simplified checkout shows a 9% lift in checkout completion rate — statistically significant. Before shipping, their analytics team runs an SRM check: control has 18,200 users; variant has 14,700 users — a 19% shortfall against the expected 50/50 split. Investigation reveals the simplified checkout page is failing to load on older Android WebView versions, which affects ~15% of their traffic. These users get a blank page and bounce — they are logged as variant users with no conversion, massively deflating the variant's true conversion rate. The actual checkout improvement, once the WebView bug is fixed, may be much larger or smaller than 9%. The test is invalidated.
SRM detection is considered a mandatory quality check in mature experimentation programmes. Companies like Microsoft, LinkedIn, and Booking.com have published engineering blogs describing SRM as one of the top three causes of invalid A/B test results. For ecommerce teams, implementing automated SRM detection — running a chi-squared test on sample counts every 24 hours and alerting if p < 0.01 — is a high-ROI addition to the testing workflow.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.