Concepts referenced in this article, defined.
Run rigorous A/B tests and personalize every visit on Shopify or any storefront โ no engineers required.
Sample ratio mismatch (SRM) occurs when the actual proportion of users assigned to each variant in your A/B test differs meaningfully from what you intended. If you set up a 50/50 split but end up with 55% in the control and 45% in the variant, you have an SRM โ and your test results are invalid. SRM is one of the most under-diagnosed problems in A/B testing, and it causes confident-looking results to be completely untrustworthy.
This problem is more common than most teams realize. A 2019 study by Microsoft found SRM in a significant percentage of their internal experiments. For Indian D2C brands running tests on Shopify, the risk is real โ especially when using third-party analytics, redirect-based testing, or running during high-bot-traffic periods (like festive sale events).
Imagine you're testing two versions of your product page on Nykaa or a similar platform. You set up a 50/50 split. After two weeks, you check results:
Variant B looks like it's winning โ a 27% relative lift! But wait: you intended 50/50, but got 55%/45%. That's a 15,000 visitor discrepancy. Before celebrating, you have an SRM. The variant might be getting a different type of visitor, not just more conversions.
The danger: If Variant B received fewer visitors because it was shown to a subset with higher purchase intent (e.g., visitors coming from a specific UTM source, or returning customers), then the CVR difference isn't caused by the variant โ it's caused by who saw it.
Method 1: Visual inspection Compare your expected vs. actual visitor counts. If you set a 50/50 split and one variant has more than 52-53% of visitors, investigate. This rough check catches major SRMs.
Method 2: Chi-square test The formal statistical test for SRM. Calculate:
Most A/B testing platforms (including CustomFit.ai) run this check automatically and flag SRM before you even look at conversion data.
Interpretation: A p-value below 0.01 from the chi-square test indicates statistically significant SRM. At this threshold, the probability of seeing this visitor imbalance by chance is less than 1%.
Method 3: Check traffic by segment Break down visitors by device, browser, traffic source, and geography. If one variant has a very different composition (e.g., 60% mobile in control vs. 40% mobile in variant), you have a segment-level SRM even if total counts look balanced.
Understanding why SRM happens helps you prevent it:
1. Bot and crawler traffic Bots don't convert, but they inflate visitor counts unevenly. If your testing platform serves variant pages differently to bots vs. humans, you'll get SRM. This is especially prevalent during Diwali or end-of-season sale periods when bot traffic spikes alongside real traffic.
2. Redirect-based testing implementation
When you test using URL redirects (e.g., Control stays on /product/X, Variant redirects to /product/X?v=b), browsers handle redirects differently. Some users get bounced before the redirect completes, others cache the redirect. This creates uneven variant counts.
3. Randomization bugs The most painful cause. If your randomization code has a bug โ for example, if it uses the wrong hash function or re-randomizes users on each visit โ users can get assigned to different variants across sessions, causing double-counting or systematic skew.
4. Caching CDN or browser caching is a major SRM cause for ecommerce sites. If your CDN caches the control version and serves it to some users regardless of their variant assignment, those users will be counted in analytics for the variant they were assigned but see the control experience. This is particularly tricky for Indian brands using Cloudflare or AWS CloudFront.
5. Third-party script interference Tag managers, analytics scripts, and ad pixels can load inconsistently. If a script that fires the variant tracking event fails to load for some users, those users won't be counted โ even though they saw the variant.
6. Unequal traffic routing If you're using a multi-CDN setup or have regional traffic routing, different variants might be preferentially served in different geographies. For an Indian D2C brand, this could mean Variant A is shown primarily to Mumbai visitors and Variant B to Delhi visitors โ with very different purchase behaviors.
7. Experiment start/stop timing If you start a test at 2 PM and one variant's assignment starts at 2 PM while the other's starts at 2:15 PM, you've already created an SRM for the first 15 minutes of traffic.
When you act on SRM-contaminated results, you're making product decisions based on flawed data. Real consequences:
False wins: A variant appears to convert better only because it received higher-intent visitors. You ship the change, and conversion rate returns to baseline โ or drops below it.
False losses: A genuinely better variant loses the test because it received lower-intent visitors. You discard an idea that would have worked.
Misattribution: You attribute a CVR change to your test variable when the real cause was a visitor mix change โ perhaps a traffic campaign that started mid-test.
For a brand like Kapiva doing โน1โ2 crore in monthly revenue, a false win that "proves" a variant is 10% better when it isn't could lead to a significant and sustained revenue loss when rolled out.
Step 1: Stop interpreting the current test results. Do not try to correct for SRM mathematically โ the data is compromised.
Step 2: Diagnose the root cause. Check each of the causes above systematically. Start with your implementation (caching, redirects, randomization code) before blaming traffic sources.
Step 3: Fix the issue. For caching: configure your CDN to vary responses based on your variant cookie. For bots: add bot filtering before variant assignment. For bugs: fix your randomization code and have a second engineer review it.
Step 4: Restart the test from zero. Don't resume โ historical data from the SRM period cannot be used. Clear all variant assignments and start fresh.
Step 5: Build SRM detection into your pre-launch checklist. Monitor SRM daily for the first 48 hours of any new test. Catching it early limits wasted time.
CustomFit.ai is a Shopify-native testing platform, which means:
This is one reason the platform can report reliable results like Bellavita's 11% CVR improvement โ because the tests that produced those numbers were SRM-free.
Check SRM within 24โ48 hours of launch โ the earlier you catch it, the less time you waste on a contaminated test.
Use server-side randomization whenever possible โ client-side JavaScript testing is more prone to SRM from ad blockers, slow connections, and script failures.
Filter bots before variant assignment โ never include bot traffic in your visitor counts or variant assignment logic.
Test your randomization logic with a simulated 50/50 coin flip โ before running real experiments, verify your assignment code produces roughly equal splits across 10,000 simulated users.
Audit CDN caching settings before launching a test โ confirm that your CDN isn't caching the control page and serving it to variant users.
Monitor SRM separately for each browser and device โ an overall balanced split can mask device-level SRM.
Document your test implementation for peer review โ have a second team member check the implementation before launch. SRM bugs are often obvious in retrospect.
Related reading: Statistical Significance | Sample Size in A/B Testing | How Long to Run an A/B Test | A/B Testing Metrics | A/B Testing Pillar Guide