What is a multi-armed bandit in A/B testing?

A multi-armed bandit (MAB) is an adaptive testing algorithm that dynamically shifts more traffic to better-performing variants during the test — unlike traditional A/B testing where traffic stays 50/50 throughout. The name comes from a metaphor of choosing between multiple slot machines to maximize winnings.

When should I use multi-armed bandit instead of A/B testing?

Use multi-armed bandit when you have low traffic, want to minimize revenue lost to underperforming variants, or are testing many variants simultaneously. Use traditional A/B testing when you need statistically rigorous learning, are testing hypotheses, or have high enough traffic for clean experiments.

Does multi-armed bandit give less accurate results than A/B testing?

Yes. Because MAB shifts traffic mid-test, it produces less statistically clean results and is more prone to Type I errors (false positives). The tradeoff is lower opportunity cost during the test — MAB earns more revenue while learning, but learns less precisely.

Can I run multi-armed bandit tests on Shopify?

Yes. CustomFit.ai, Dynamic Yield, and Optimizely all support MAB testing on Shopify stores. CustomFit.ai's no-code interface lets you configure MAB experiments without developer involvement.

What is the main disadvantage of multi-armed bandit testing?

The main disadvantage is reduced statistical rigor. MAB can declare a 'winner' based on early noise, especially in low-traffic situations. It is also harder to run on low-traffic sites where the algorithm needs sufficient data to optimize meaningfully.

Multi-Armed Bandit vs A/B Testing

Multi-armed bandit (MAB) testing reallocates traffic to winning variants while the experiment is still running. Traditional A/B testing keeps a fixed split until the test reaches statistical significance. Both methods test which version of a page, offer, or element performs better, but they trade off statistical certainty against opportunity cost in different ways. Knowing when to use each one matters for any ecommerce brand running an optimization program.

How traditional A/B testing works

Traditional A/B testing follows a fixed protocol:

Define a hypothesis
Split traffic 50/50 between control and variant
Wait until the predetermined sample size is reached
Analyze results at 95% confidence
Deploy the winner

The traffic split stays fixed at 50/50 regardless of early results. If Variant B is clearly winning after day 3, you still send 50% of traffic to the losing Variant A until the test concludes. That is the cost of A/B testing: you give up short-term revenue for statistical certainty.

Advantages of A/B testing:

Statistically rigorous, with a low false positive rate when run correctly
Results are interpretable, so you know exactly why one variant won
Easy to combine with hypothesis-driven learning programs
Supported by all major testing platforms

Disadvantages:

Traffic to the losing variant wastes potential revenue
Requires a sufficient sample size, which can take weeks for low-traffic sites
Only tests two variants efficiently (multi-variant A/B tests require much larger samples)

How multi-armed bandit testing works

Multi-armed bandit algorithms start with equal traffic allocation, then keep shifting more traffic toward better-performing variants as data comes in. Common algorithms include:

Epsilon-greedy: Allocates ε% of traffic randomly (exploration) and (1-ε)% to the current best performer (exploitation). It is simple, but it can get stuck on early leaders.

Thompson Sampling: Uses Bayesian probability to estimate each variant's true performance and samples traffic proportionally. It is more sophisticated and is a good default for ecommerce.

Upper Confidence Bound (UCB): Favors variants with high uncertainty so every variant gets adequate exploration. It works well for multi-variant experiments.

Here is how that plays out over a two-week test:

Start: 50% Control / 50% Variant B
Day 3 (Variant B showing +8% CVR): 35% Control / 65% Variant B
Day 7 (Variant B showing +12% CVR): 20% Control / 80% Variant B
Day 14 (confident in winner): 5% Control / 95% Variant B

The algorithm keeps adjusting as it goes, sending less traffic to the underperformers.

Advantages of MAB:

Lower opportunity cost, since more traffic reaches the winning variant sooner
Handles multiple variants at once
Self-corrects for seasonality, because the algorithm responds to changing performance
Better suited to low-traffic sites where traditional A/B tests take too long

Disadvantages:

Lower statistical rigor, with higher false positive rates
Harder to extract clean learnings from (why did it win?)
Sensitive to early noise, so it can prematurely allocate to a false leader
Requires more capable platform support

Side-by-side comparison

Factor	A/B Testing	Multi-Armed Bandit
Traffic allocation	Fixed 50/50	Dynamic, shifts toward winner
Statistical rigor	High	Lower
Revenue during test	Lower (traffic to loser)	Higher (optimizes in real-time)
Learning quality	High	Lower
Traffic requirement	High (1,000+ per variant)	Lower
Best for	Hypothesis testing	Revenue optimization
Multiple variants	Requires large sample	Handles well
Result stability	Stable	Can fluctuate

When to use A/B testing

Use traditional A/B testing when:

You want to learn, not just optimize. If your goal is to understand why a change works, and to build a model of how your customers behave, A/B testing gives you cleaner, more interpretable results.

You have sufficient traffic. Sites with 10,000+ monthly visitors per page can reach statistical significance in a reasonable time, so the opportunity cost of the fixed split is acceptable.

You are testing structural changes. Major page redesigns, new checkout flows, or pricing page restructures are worth the statistical rigor of A/B testing. Getting a structural change wrong has long-term consequences.

You are running a regulated experiment. If your test results will inform product decisions, go to investors, or change pricing, you need A/B testing's statistical standards.

You are building a testing program from scratch. Hypothesis-driven A/B testing builds team learning. MAB optimizes without giving the team the same development.

When to use multi-armed bandit

Use MAB when:

You have low traffic. Sites with under 5,000 monthly visitors per page cannot reach A/B testing significance quickly. MAB pulls value from limited data while the experiment runs.

You are testing many variants. Testing 5+ variants with A/B testing requires impractically large samples. MAB handles multi-variant experiments without that overhead.

You want to minimize revenue loss. During high-stakes periods such as the festive season or product launches, you may not want to send 50% of traffic to an underperforming variant for 3 to 4 weeks.

You are optimizing for short-term conversion. If your goal is pure revenue optimization rather than learning, MAB earns more revenue during the test window.

Your traffic patterns shift seasonally. MAB adapts to changing performance in real time. A static A/B test started in October may be optimizing on Diwali traffic that doesn't represent November buyer behavior.

Hybrid approaches

Some testing programs use both methods:

Explore-then-exploit: Run a standard A/B test for 2 to 3 weeks to find a statistically significant winner, then use MAB to keep optimizing within the proven winner space.

Bandit for initial screening, A/B for validation: Use MAB to spot promising variants from a large set, then run a standard A/B test on the top 2 to 3 performers to validate them rigorously.

Time-based switching: Default to A/B testing during stable traffic periods, and switch to MAB during high-stakes, short-window events such as Black Friday or Diwali, when opportunity cost is highest.

Practical example: Kapiva homepage hero

Say Kapiva wants to test 4 different homepage hero messages for their Apple Cider Vinegar product.

With A/B testing: Testing 4 variants at 95% confidence would require 2,500 visitors per variant, or 10,000 total visitors. At Kapiva's traffic levels, this would take 4 to 6 weeks.

With MAB (Thompson Sampling): The algorithm starts at 25% each and begins shifting within 3 to 4 days toward the top performer. By day 10, the winning variant is receiving 60 to 70% of traffic, which limits how much revenue the poorer performers lose.

Conclusion for this scenario: MAB fits here because there are 4 options to test, the traffic level is medium, and Kapiva wants to limit exposure to the underperforming messages.

Tips and best practices

Don't let MAB run indefinitely. Set a maximum runtime, typically 30 days, even for MAB tests. Algorithms can get stuck in local optima and stop improving.

Use A/B testing for learning and MAB for optimizing. The two tools complement each other. Build your understanding of what works through A/B testing, then let MAB optimize within that understanding.

Validate MAB winners with a short A/B test. If MAB strongly favors one variant, run a clean A/B test against it to confirm the result before making permanent changes.

Track statistical significance even in MAB. Most MAB platforms report a confidence score. Do not treat a 60% confidence winner as a conclusive result.

Check that your platform supports MAB properly. Basic A/B tools that approximate MAB with manual traffic reallocation are not true MAB. They introduce human bias and get no benefit from algorithmic optimization.

Key takeaways

A/B testing gives you higher statistical rigor at the cost of more traffic wasted on underperforming variants
Multi-armed bandit reduces opportunity cost but produces less precise results
Use A/B testing when you want to learn and have sufficient traffic; use MAB when traffic is limited or you are testing many variants
Hybrid approaches (MAB for screening, A/B for validation) combine the benefits of both methods
MAB pays off during high-stakes, short-window events where the opportunity cost of traditional A/B testing is highest
Validate MAB winners with a follow-up A/B test for important structural decisions

Related reading: