Concepts referenced in this article, defined.
Run rigorous A/B tests and personalize every visit on Shopify or any storefront โ no engineers required.
QA (quality assurance) for A/B tests means systematically verifying your test setup before exposing real visitors to it. A broken A/B test โ one where variants don't render, tracking doesn't fire, or traffic splits malfunction โ wastes visitor traffic, corrupts your data, and can actively damage user experience. A 45-minute QA process before launch prevents these failures. This checklist covers everything to verify before going live.
Unlike a broken page (which is immediately visible), a broken A/B test can run for days or weeks before anyone notices. Common failure modes that QA catches:
Variant renders incorrectly on mobile. Your desktop QA looked fine, but on an Android phone the variant's headline wraps to three lines and covers the product image. You run the test for two weeks, collecting data on a broken experience.
Conversion event fires on the wrong trigger. Your test is supposed to measure purchase completions, but you accidentally configured the conversion goal to fire on "add to cart." Your results show the variant converts 40% better โ but that's add-to-cart rate, not purchase rate. You roll out the change based on false data.
Both control and variant show to the same visitor. A JavaScript conflict between your testing tool and another script causes the variant to flash and revert to control on every page load. Visitors see both versions. Your results are meaningless.
Anti-flicker snippet placed incorrectly. Every visitor sees a 300ms flash of the control version before the variant loads โ contaminating the variant experience, especially for above-the-fold changes.
Traffic split is not 50/50. A configuration error routes 90% of visitors to the control and 10% to the variant. You'll need 5x more traffic to reach significance, and you won't realize the problem until you look at your analytics weeks later.
Each of these failures is preventable with a structured pre-launch QA process.
1.1 Render check โ desktop (Chrome) Force yourself into the variant using your testing tool's preview or QA URL. Verify:
1.2 Render check โ mobile (iOS and Android) This is the most commonly skipped and most important step. Use real devices if possible; use Chrome DevTools device simulator as a minimum:
1.3 Render check โ across browsers At minimum: Chrome (latest), Safari (for iPhone users), Firefox, Samsung Internet (widely used on budget Android phones in India).
1.4 Cross-page consistency check (for multi-page tests) If your test changes elements on multiple pages, verify every page in the test scope shows the correct variant โ not just the first page.
2.1 Conversion goal verification Navigate through the exact path required to trigger your conversion goal:
2.2 No duplicate conversions Refresh your order confirmation page. Does the conversion event fire again? Duplicate conversions inflate your results โ a single order counted twice shows twice the conversion rate for a variant. Implement deduplication logic if your tool doesn't handle this automatically.
2.3 Impression tracking Verify that variant impressions are being recorded. Your testing tool should show a count of visitors in each variant within minutes of starting the QA session.
2.4 Analytics integration If you've connected your testing tool to Google Analytics or another analytics platform, verify that variant data is flowing through correctly. Check for variant dimension data in your analytics platform's real-time view.
3.1 Split verification โ manual Open your site in 10 incognito windows in rapid succession. Record which variant each window shows. At 50/50 split, you'd expect roughly 5 control and 5 variant โ though random variation means this might be 4/6 or 7/3 in a small sample. If all 10 windows show the same variant, your split is broken.
3.2 Split verification โ testing tool dashboard Check your testing tool's real-time visitor dashboard. After 50+ visitors, verify the traffic split is approaching your configured ratio.
3.3 Mutual exclusivity Verify that a single visitor always sees the same variant โ they don't see control on one visit and variant on the next. Open the site in the same browser (not incognito) multiple times. You should see the same variant each time.
4.1 Anti-flicker verification On a throttled connection (Chrome DevTools โ Network โ "Slow 3G"):
4.2 JavaScript conflict check Open Chrome DevTools โ Console. Load the variant page. Check for any JavaScript errors. Errors can indicate conflicts between your testing tool and other scripts (chat widgets, analytics tags, review app scripts).
4.3 Page speed impact Run the test URL through Google PageSpeed Insights or GTmetrix with the test active. Compare to baseline (without the test running). If your testing script adds more than 200ms to page load time, investigate the script loading sequence.
4.4 Form functionality If your test variant modifies any form (search, newsletter sign-up, customer login, checkout address fields), test all form submissions. Testing tool modifications can sometimes break form event listeners.
5.1 Discount and promotion compatibility If your store has active promotions, coupon codes, or cart-level discounts, verify they work correctly with both the control and variant active. Testing tool modifications occasionally interfere with JavaScript-based promotion applications.
5.2 Cart and checkout flow Walk through the complete purchase flow with the test active:
5.3 Responsive behavior with dynamic content If your store shows dynamic content (personalized recommendations, recently viewed, low stock warnings), verify this content loads correctly in both control and variant conditions.
Before launching any test, document your QA findings. A simple template:
Test name: [Name]
Test ID: [ID from testing tool]
Launch date: [Date]
QA completed by: [Name]
QA date: [Date]
Desktop render: PASS / FAIL (notes)
Mobile render (iPhone): PASS / FAIL (notes)
Mobile render (Android): PASS / FAIL (notes)
Conversion tracking: PASS / FAIL (notes)
Traffic split: PASS / FAIL (notes)
Anti-flicker: PASS / FAIL (notes)
JavaScript errors: NONE / [list errors]
Page speed impact: [baseline vs test, in ms]
Checkout flow: PASS / FAIL (notes)
This 5-minute document creates accountability, a record of your QA process, and a reference when troubleshooting issues after launch.
QA doesn't end at launch. In the first 24 hours after launching a test:
Check visitor counts are accumulating in both variants. If one variant shows zero visitors after 2 hours, something is broken.
Verify no customer complaints or support tickets related to the test pages. Monitor your WhatsApp support or email inbox for unusual volume.
Check conversion tracking is functioning. Verify that orders are being attributed to variants in your testing tool's dashboard.
Look for JavaScript console errors that might indicate conflicts that only emerge in production (with real CDN environments, different cache states, etc.).