The most important decision you make before launching an A/B test is choosing the right primary metric. Conversion rate is the most common, but it's often the wrong choice β a variant that increases CVR while dropping average order value can hurt your business. This guide covers which metrics to track, how to set up a measurement hierarchy, and how Indian D2C brands can avoid the vanity metric trap.
Before you touch your test setup, you need a measurement plan. Without one, you'll end up with "winning" tests that don't move revenue β a problem endemic to ecommerce brands that haven't formalized their CRO process.
The Metric Hierarchy: Primary, Secondary, and Guardrail
Every A/B test needs three types of metrics defined upfront:
Primary metric: The single KPI that determines the winner. You commit to this before launch. It's the metric you optimize for. If you're testing a product page, this might be add-to-cart rate. If you're testing checkout flow, it's order completion rate.
Secondary metrics: Supporting KPIs that provide context. If your primary metric is add-to-cart rate, secondary metrics might include product page scroll depth, time on page, or wishlist adds. These help you understand why the primary metric moved.
Guardrail metrics: KPIs you monitor to ensure you're not winning in one area while breaking another. For a Nykaa-type beauty brand, a guardrail metric might be return rate β if your test increases CVR but customers are returning products at higher rates, the variant isn't truly better.
This hierarchy prevents the most dangerous A/B testing mistake: declaring a winner on a metric that doesn't actually reflect business health.
Core Ecommerce A/B Testing Metrics
Conversion Rate (CVR)
The percentage of visitors who complete your goal action. For most ecommerce tests, this is a purchase. CVR is the most tracked metric but can be misleading in isolation.
Formula: (Conversions / Unique Visitors) Γ 100
When to use as primary: When testing changes that should directly affect purchase decisions β CTAs, product descriptions, trust signals, pricing display.
When to be careful: CVR can increase while AOV drops, leaving revenue flat. Always check RPV alongside CVR.
Revenue Per Visitor (RPV)
The average revenue generated per unique visitor. This is arguably the most honest single metric for ecommerce tests.
Formula: Total Revenue / Unique Visitors
Why it's better than CVR: A variant that converts more people at βΉ800 AOV vs. your control converting fewer at βΉ1,400 AOV could be a revenue-negative "win" on CVR. RPV catches this.
Brands like Bellavita track RPV as their north star metric precisely because their product mix varies β a high CVR on low-margin items would look like a win but isn't.
Add-to-Cart Rate (ATC)
The percentage of product page visitors who add an item to their cart. ATC is an excellent proxy metric for product page tests because it measures purchase intent before the checkout friction point.
When to use: When testing product images, descriptions, pricing display, social proof placement, or CTA copy on PDPs (product detail pages).
Watch for: A high ATC with low checkout completion suggests cart/checkout friction, not a problem with your product page test.
Checkout Completion Rate
The percentage of users who begin checkout and complete a purchase. This is your primary metric for any checkout flow test.
Formula: (Orders / Checkout Initiations) Γ 100
Indian D2C nuance: Track this separately for COD (cash on delivery) and prepaid orders. COD completion rates are typically lower (customers cancel upon delivery). A test that shifts your mix toward prepaid may look like a checkout CVR win even if total completions are flat.
Average Order Value (AOV)
The average revenue per completed order. Essential as a secondary or guardrail metric for any pricing, upsell, or bundle test.
Formula: Total Revenue / Number of Orders
Bounce Rate
The percentage of single-page sessions. Useful as a secondary metric for landing page tests β a winning variant that also reduces bounce rate suggests stronger engagement beyond the conversion action.
Time to Purchase
How long it takes a visitor to complete a purchase from their first session. Relevant for tests that affect the research-to-buy journey β useful for high-consideration products like supplements (Kapiva, Himalaya) or electronics (Boat, Noise).
Micro-Conversion Metrics
Micro-conversions are intermediate steps that predict purchase intent. They're particularly useful for low-traffic sites where macro-conversion sample sizes take too long to reach.
Email sign-up rate: Measures whether your lead capture is working. Test this on landing pages and exit popups.
Wishlist add rate: A strong signal of purchase intent for deferred purchases β high in Indian ecommerce where customers research on mobile and buy later on desktop.
Video play rate: For brands using product demonstration videos (Boat, Sugar Cosmetics), video play rate predicts engagement quality.
Review read rate: How many visitors interact with reviews. Higher interaction with social proof correlates with higher conversion on high-consideration purchases.
Scroll depth: How far visitors scroll on long-form product pages. A key input for above-the-fold and content hierarchy tests.
Vanity Metrics to Avoid as Primary Metrics
These metrics feel meaningful but rarely drive decisions:
Total page views: More page views doesn't mean more revenue. A confusing page might get more views as users navigate back and forth.
Click-through rate (CTR) in isolation: If you test a button color, CTR tells you what got clicked β but if those clicks don't convert, the test didn't help.
Social shares: Not a purchasing signal for most D2C brands.
Time on page (in isolation): Could mean the content is engaging, or it could mean users are confused.
Use these as secondary metrics to understand behavior, never as primary decision metrics.
Setting Up Your Measurement Plan
Before every test, document:
- Primary metric + current baseline value (e.g., "Add-to-cart rate: 6.3%")
- Minimum detectable effect (e.g., "We need at least a 10% relative lift β to 6.93%")
- Secondary metrics (e.g., "RPV, checkout completion rate")
- Guardrail metrics (e.g., "Return rate, AOV β must not drop more than 5%")
- Sample size required per variant (calculated)
- Test start and end date
This document becomes your test's contract. You evaluate results only against what you defined here, not against every metric that moved.
Segmenting Your Metrics
Raw metrics lie. Always segment results to understand the full picture:
By device: Mobile vs. desktop behavior differs dramatically in Indian ecommerce. A test that wins on mobile might harm desktop conversions. CustomFit.ai lets you segment test results by device automatically.
By traffic source: Organic search visitors behave differently from social media traffic. An Instagram-driven brand like Plum or mCaffeine will see very different CVR patterns from paid vs. organic visitors.
By new vs. returning visitors: Returning visitors have higher intent and often convert at 2β3Γ the rate of new visitors. A test that wins overall might be driven entirely by returning visitors and could harm new visitor experience.
By COD vs. prepaid: As noted, this split is critical for Indian ecommerce interpretation.
How CustomFit.ai Tracks A/B Testing Metrics
CustomFit.ai is built for Shopify stores and integrates directly with your order data β so revenue metrics like RPV and AOV are calculated from actual transaction data, not proxy events. This means:
- No manual revenue tracking setup
- CVR, RPV, and AOV update in real time for each variant
- Results automatically segmented by device and traffic source
- Guardrail metric alerts if key secondary metrics drop below threshold
Brands using CustomFit.ai see an average 11% CVR improvement because they're optimizing for the right metrics from day one.
Tips / Best Practices
-
Define your primary metric before you build the test β not after you see the data.
-
Use RPV instead of CVR whenever your AOV varies significantly β common for brands with wide product ranges like Boat or Sugar.
-
Track COD and prepaid conversion separately β they have different economics and different behavioral profiles.
-
Set guardrail thresholds before launch β decide what "unacceptable drop" means for each guardrail metric. For example: "Test fails if return rate increases by more than 2 percentage points."
-
Never add new metrics mid-test β if a metric wasn't in your measurement plan, you can't use it to declare a winner without inflating false positive risk.
-
Segment results before acting β a variant that wins overall but loses on mobile (which is 70%+ of Indian ecommerce traffic) isn't a winner you should roll out.
-
Calculate business impact in βΉ, not percentages β a 0.5% CVR lift on βΉ50L monthly revenue is βΉ25,000/month. Make the business case explicit.
Key Takeaways
- Every A/B test needs a primary metric, secondary metrics, and guardrail metrics defined before launch
- Revenue per visitor (RPV) is more honest than conversion rate alone for most ecommerce tests
- Indian D2C brands must track COD vs. prepaid separately β they have different conversion patterns and economics
- Micro-conversions (ATC rate, wishlist adds, scroll depth) are essential metrics for low-traffic sites
- Vanity metrics like total pageviews and CTR in isolation are secondary metrics at best, never primary decision metrics
- Always segment results by device, traffic source, and visitor type before declaring a winner
Related reading: Statistical Significance Explained | Conversion Rate Definition | How Long to Run an A/B Test | A/B Testing Headlines | A/B Testing Pillar Guide
ab testing
ab testing
ab testing