A/B testing works by splitting traffic between two versions of a page, measuring which performs better on a conversion metric, and declaring a winner at statistical significance.
Concepts referenced in this article, defined.
Run rigorous A/B tests and personalize every visit on Shopify or any storefront โ no engineers required.
A/B testing works by randomly splitting your website visitors into two groups โ one sees the original version of a page (the control), the other sees a changed version (the variant) โ then measuring which group converts better on a specific metric. The assignment happens instantly when a visitor lands on the page, is locked via a cookie so the experience stays consistent, and results accumulate until the difference between the two versions is statistically significant. When significance is reached and the test period is complete, you ship the winner and move to the next test.
If you're building a D2C brand and want to understand exactly what's happening inside an A/B test โ technically, not just conceptually โ this guide walks through every step.
Every A/B test โ whether you're running it on a Shopify product page or a landing page for a fashion brand โ follows the same five-step sequence. Understanding each step helps you run tests correctly and avoid the errors that invalidate results.
When a visitor hits your page, the A/B testing system needs to decide instantly: control or variant? This decision uses a deterministic hashing algorithm.
Here's how it works: the system takes a unique identifier โ usually a cookie ID generated on first visit, or a logged-in user ID โ and runs it through a hash function. The output is a number. That number maps to a bucket (say, 0โ49 = control, 50โ99 = variant for a 50/50 split). The same input always produces the same output, which means the same visitor always lands in the same bucket.
This is important for two reasons:
For a 50/50 split, both groups are as equal as possible. You can also run unequal splits (80/20) if you want to limit exposure to a riskier variant.
Once the assignment is made, it's stored in a first-party cookie in the visitor's browser. This cookie typically has a lifespan of 30โ90 days, long enough to cover the test duration.
Every time that visitor returns to the page, the testing system reads the cookie and serves the same experience. No flickering. No switching between versions. This consistency matters because:
Client-side tests inject the variant via JavaScript after the page loads, which can sometimes cause a brief flash of original content (FOOC) before the variant appears. Server-side tests serve the correct version from the origin server, with no flash โ we'll cover this distinction more below.
With visitors consistently bucketed into control and variant, the testing tool begins collecting data. For each group, it tracks:
The data flows into the testing platform in near real-time. Each conversion event is tagged with the variant that visitor belongs to, creating two parallel streams of data that will be compared at analysis time.
One nuance: you want to track unique visitors, not sessions. If a visitor visits three times before converting, that's one convert in one bucket โ not three sessions contributing to the numerator.
This is where most marketers get lost. Here's the plain-language version.
When you observe that the variant has a 4.2% conversion rate and the control has a 3.8% conversion rate, you have a 10.5% relative lift. But is that difference real, or could it be random noise?
P-value tells you the probability that the observed difference happened by chance, assuming there was no real difference (the null hypothesis). A p-value of 0.05 means there's a 5% probability the result is random noise. When p < 0.05, you've reached 95% statistical significance โ the industry standard.
Confidence interval shows the range of likely true values for the lift. A 95% confidence interval of [+2% to +18%] means you're 95% confident the true lift is somewhere in that range. Wider intervals mean more uncertainty; narrow intervals mean you have enough data to be precise.
Sample size is the other side of this equation. The smaller the true effect you're trying to detect, the larger the sample you need. A tool like a sample size calculator will ask you for:
Running this calculation before you start is non-negotiable. It tells you how many visitors each variant needs before results are valid.
A test is ready to call when both conditions are true:
Never call a winner based on significance alone. A test that reaches 95% significance on day 3 with 400 visitors is almost certainly a false positive โ you need both the statistical threshold and the time threshold.
When both are met:
There are two fundamentally different ways to deliver A/B test variants, and each has meaningful trade-offs for D2C brands.
Client-side testing works via a JavaScript snippet that loads in the browser after the page HTML is delivered. The snippet reads the visitor's bucket assignment and modifies the DOM (Document Object Model) to render the variant โ changing headline text, button colors, image swaps, etc. Tools like Google Optimize (deprecated), VWO, and many others use this approach.
The downside: because the modification happens after page load, there's a window where the visitor sees the original before the JavaScript executes. This "flicker" is usually milliseconds, but it's a real issue on slower connections and can contaminate behavioral data if users see both versions.
Server-side testing makes the branching decision at the server or CDN level, before any HTML is delivered to the browser. The visitor gets the correct version immediately, with zero flicker. This approach also handles complex experiments โ different pricing, different product recommendations, different page layouts โ that would be difficult to implement purely in JavaScript.
Server-side testing is harder to set up but produces cleaner data. For D2C brands doing serious CRO, it's worth the investment.
Traditional A/B testing is binary and static: 50% to control, 50% to variant, hold it until significance. AI-powered testing introduces dynamic traffic allocation that improves both speed and accuracy.
Multi-armed bandit algorithms treat each variant as an "arm" of a slot machine. Early in the test, the algorithm explores โ spreading traffic relatively evenly to gather data. As data accumulates, it begins exploiting โ shifting more traffic toward the variant that's currently winning. By the end of the test, the majority of traffic may be going to the better-performing version.
The benefit: you capture the lift sooner, and you lose less revenue to the underperforming variant during the test. The trade-off: pure bandit results are harder to interpret with classical statistics, so some platforms use a hybrid approach.
Predictive intent scoring goes further. Instead of assigning all visitors to the same test, the system segments by predicted intent โ high-intent purchasers, browsers, price-sensitive shoppers โ and learns which variant performs best for each segment. A variant that lifts conversions for high-intent shoppers might have no effect on casual browsers, and showing the result as a single aggregate number would mask both findings.
Automatic winner detection uses sequential testing methods that continuously monitor significance and flag when results are conclusive, rather than waiting for a fixed endpoint.
For Indian D2C brands on Shopify, the testing workflow is concrete. Here's how it runs on CustomFit.ai:
No developer tickets. No waiting for sprint cycles. Tests that used to take weeks to set up now launch in an afternoon.
Can I test more than two versions at once? Yes โ testing three or more variants simultaneously is called A/B/n testing or multivariate testing. It requires proportionally more traffic (if you have three variants at 33% each, each group is smaller) and takes longer to reach significance. Keep variants to 2โ3 unless you have very high traffic.
What if my traffic is seasonal? Always run tests across at least one full weekly cycle to account for weekday/weekend behavior differences. For seasonal categories (festive wear, wedding lehengas), avoid launching tests during major sale periods โ the behavioral data from sale traffic doesn't represent normal shoppers.
Does A/B testing work on mobile apps? Yes, with server-side testing or mobile SDKs. The mechanics are identical: users are assigned to buckets via a unique identifier, variants are served consistently, and conversions are tracked. Mobile introduces session complexity (apps don't use cookies the same way), so use a user ID as the assignment key.
What's the difference between A/B testing and multivariate testing? A/B testing changes one element and compares two complete versions. Multivariate testing (MVT) tests multiple elements simultaneously โ headline + image + button โ to find the best combination. MVT requires significantly more traffic and is generally more useful for optimization teams with established testing programs.
Understanding what A/B testing is is the foundation. Knowing how to run A/B tests correctly is what turns that knowledge into revenue.
1,000+ D2C brands use CustomFit.ai to run A/B tests โ without code, without developer tickets. 14-day free trial ยท No credit card required.