What is sample ratio mismatch in A/B testing?

Sample ratio mismatch (SRM) occurs when the actual ratio of users assigned to test variants differs significantly from the intended ratio — for example, getting 55%/45% when you expected 50%/50%.

Why does sample ratio mismatch matter?

SRM indicates a systematic bias in how users were assigned to variants. This bias contaminates your results — a 'winning' variant may have won because it got different users, not because it was better.

How do I detect sample ratio mismatch?

Run a chi-square test on your observed vs. expected visitor counts per variant. Most testing platforms include SRM detection automatically. A p-value below 0.01 indicates SRM.

What causes sample ratio mismatch?

Common causes include bot traffic, browser redirects, implementation bugs in the randomization code, caching issues, and uneven traffic sources hitting different variants.

What should I do if I detect SRM?

Stop the test, investigate the root cause, fix the issue, and restart. Do not try to interpret results from an SRM-affected test — they are not valid.

What Is Sample Ratio Mismatch (SRM) in A/B Testing?

Sample ratio mismatch (SRM) occurs when the actual proportion of users assigned to each variant in your A/B test differs meaningfully from what you intended. If you set up a 50/50 split but end up with 55% in the control and 45% in the variant, you have an SRM — and your test results are invalid. SRM is one of the most under-diagnosed problems in A/B testing, and it causes confident-looking results to be completely untrustworthy.

This problem is more common than most teams realize. A 2019 study by Microsoft found SRM in a significant percentage of their internal experiments. For Indian D2C brands running tests on Shopify, the risk is real — especially when using third-party analytics, redirect-based testing, or running during high-bot-traffic periods (like festive sale events).

What SRM Looks Like in Practice

Imagine you're testing two versions of your product page on Nykaa or a similar platform. You set up a 50/50 split. After two weeks, you check results:

Control (Version A): 8,200 visitors, 4.1% CVR
Variant (Version B): 6,800 visitors, 5.2% CVR

Variant B looks like it's winning — a 27% relative lift! But wait: you intended 50/50, but got 55%/45%. That's a 15,000 visitor discrepancy. Before celebrating, you have an SRM. The variant might be getting a different type of visitor, not just more conversions.

The danger: If Variant B received fewer visitors because it was shown to a subset with higher purchase intent (e.g., visitors coming from a specific UTM source, or returning customers), then the CVR difference isn't caused by the variant — it's caused by who saw it.

How to Detect SRM

Method 1: Visual inspection Compare your expected vs. actual visitor counts. If you set a 50/50 split and one variant has more than 52-53% of visitors, investigate. This rough check catches major SRMs.

Method 2: Chi-square test The formal statistical test for SRM. Calculate:

Expected count per variant: (Total visitors) / (Number of variants)
Observed count: Actual visitors per variant
Chi-square statistic from these values

Most A/B testing platforms (including CustomFit.ai) run this check automatically and flag SRM before you even look at conversion data.

Interpretation: A p-value below 0.01 from the chi-square test indicates statistically significant SRM. At this threshold, the probability of seeing this visitor imbalance by chance is less than 1%.

Method 3: Check traffic by segment Break down visitors by device, browser, traffic source, and geography. If one variant has a very different composition (e.g., 60% mobile in control vs. 40% mobile in variant), you have a segment-level SRM even if total counts look balanced.

Common Causes of SRM

Understanding why SRM happens helps you prevent it:

1. Bot and crawler traffic Bots don't convert, but they inflate visitor counts unevenly. If your testing platform serves variant pages differently to bots vs. humans, you'll get SRM. This is especially prevalent during Diwali or end-of-season sale periods when bot traffic spikes alongside real traffic.

2. Redirect-based testing implementation When you test using URL redirects (e.g., Control stays on /product/X, Variant redirects to /product/X?v=b), browsers handle redirects differently. Some users get bounced before the redirect completes, others cache the redirect. This creates uneven variant counts.

3. Randomization bugs The most painful cause. If your randomization code has a bug — for example, if it uses the wrong hash function or re-randomizes users on each visit — users can get assigned to different variants across sessions, causing double-counting or systematic skew.

4. Caching CDN or browser caching is a major SRM cause for ecommerce sites. If your CDN caches the control version and serves it to some users regardless of their variant assignment, those users will be counted in analytics for the variant they were assigned but see the control experience. This is particularly tricky for Indian brands using Cloudflare or AWS CloudFront.

5. Third-party script interference Tag managers, analytics scripts, and ad pixels can load inconsistently. If a script that fires the variant tracking event fails to load for some users, those users won't be counted — even though they saw the variant.

6. Unequal traffic routing If you're using a multi-CDN setup or have regional traffic routing, different variants might be preferentially served in different geographies. For an Indian D2C brand, this could mean Variant A is shown primarily to Mumbai visitors and Variant B to Delhi visitors — with very different purchase behaviors.

7. Experiment start/stop timing If you start a test at 2 PM and one variant's assignment starts at 2 PM while the other's starts at 2:15 PM, you've already created an SRM for the first 15 minutes of traffic.

The Business Impact of Ignoring SRM

When you act on SRM-contaminated results, you're making product decisions based on flawed data. Real consequences:

False wins: A variant appears to convert better only because it received higher-intent visitors. You ship the change, and conversion rate returns to baseline — or drops below it.

False losses: A genuinely better variant loses the test because it received lower-intent visitors. You discard an idea that would have worked.

Misattribution: You attribute a CVR change to your test variable when the real cause was a visitor mix change — perhaps a traffic campaign that started mid-test.

For a brand like Kapiva doing ₹1–2 crore in monthly revenue, a false win that "proves" a variant is 10% better when it isn't could lead to a significant and sustained revenue loss when rolled out.

How to Fix SRM

Step 1: Stop interpreting the current test results. Do not try to correct for SRM mathematically — the data is compromised.

Step 2: Diagnose the root cause. Check each of the causes above systematically. Start with your implementation (caching, redirects, randomization code) before blaming traffic sources.

Step 3: Fix the issue. For caching: configure your CDN to vary responses based on your variant cookie. For bots: add bot filtering before variant assignment. For bugs: fix your randomization code and have a second engineer review it.

Step 4: Restart the test from zero. Don't resume — historical data from the SRM period cannot be used. Clear all variant assignments and start fresh.

Step 5: Build SRM detection into your pre-launch checklist. Monitor SRM daily for the first 48 hours of any new test. Catching it early limits wasted time.

How CustomFit.ai Prevents SRM

CustomFit.ai is a Shopify-native testing platform, which means:

Variant assignment happens server-side before the page renders — eliminating redirect-based SRM
Bot traffic is filtered based on Shopify's own bot detection before variant assignment
SRM is checked automatically after 100 visitors per variant, and you're alerted before results become meaningful
Caching is handled at the Shopify level with proper cache-key variation per experiment

This is one reason the platform can report reliable results like Bellavita's 11% CVR improvement — because the tests that produced those numbers were SRM-free.

Tips / Best Practices

Check SRM within 24–48 hours of launch — the earlier you catch it, the less time you waste on a contaminated test.
Use server-side randomization whenever possible — client-side JavaScript testing is more prone to SRM from ad blockers, slow connections, and script failures.
Filter bots before variant assignment — never include bot traffic in your visitor counts or variant assignment logic.
Test your randomization logic with a simulated 50/50 coin flip — before running real experiments, verify your assignment code produces roughly equal splits across 10,000 simulated users.
Audit CDN caching settings before launching a test — confirm that your CDN isn't caching the control page and serving it to variant users.
Monitor SRM separately for each browser and device — an overall balanced split can mask device-level SRM.
Document your test implementation for peer review — have a second team member check the implementation before launch. SRM bugs are often obvious in retrospect.

Key Takeaways

SRM occurs when actual visitor ratios per variant differ from your intended split — it invalidates test results completely
Detect SRM with a chi-square test on expected vs. observed visitor counts, or use a platform that does this automatically
Common causes include bots, redirect-based testing, caching, randomization bugs, and third-party script failures
Never try to interpret results from an SRM-affected test — stop, diagnose, fix, and restart
Indian D2C brands are particularly exposed to SRM during high-traffic festive sale events when bot traffic spikes
Server-side testing tools like CustomFit.ai prevent most SRM causes by design