From the conversion glossary
Concepts referenced in this article, defined.
A/B Testing Push Notifications
Start lifting conversions today.
Run rigorous A/B tests and personalize every visit on Shopify or any storefront โ no engineers required.
Concepts referenced in this article, defined.
Run rigorous A/B tests and personalize every visit on Shopify or any storefront โ no engineers required.
A/B testing push notifications means delivering two versions of a push message to randomly split segments of your subscriber base and measuring which drives more clicks, sessions, and purchases. Push notifications are the most time-sensitive marketing channel for D2C ecommerce โ and the one brands get wrong most often by guessing instead of testing. Brands that test systematically report CTR improvements of 20โ40% and measurable lifts in daily active users within 60 days.
Push notifications compete for attention on locked screens alongside messages from family, social apps, and every other brand the customer has installed. The difference between a notification that gets clicked and one that gets dismissed โ or leads to an unsubscribe โ often comes down to a single word in the title.
Indian D2C brands face a particularly crowded push environment. Consumers who shop across Nykaa, Mamaearth, Boat, mCaffeine, and Plum receive dozens of push notifications daily. Without testing, you are publishing based on intuition in a channel where intuition has a poor track record.
What makes push notifications uniquely testable:
Step 1: Choose your test variable
Pick one element to test. The most impactful variables, in order:
Step 2: Write your hypothesis
Example: "I believe leading with a โน discount in the title ('Get โน300 off today') will increase CTR vs. a benefit headline ('Your skin routine just got better') because Indian shoppers are price-motivated and discount-first framing triggers immediate action."
Step 3: Split your audience randomly
Most push platforms (OneSignal, CleverTap, WebEngage, MoEngage) allow you to split subscribers randomly for A/B tests. Use a 50/50 split for push notification tests โ unlike email where you hold a winner group, push notifications are typically sent to the full audience at once.
Step 4: Send simultaneously
Both variants must go out at exactly the same time. A 30-minute gap between variants introduces send-time bias that will contaminate your results.
Step 5: Measure the right metrics
Primary: Click-through rate (CTR) โ percentage of recipients who tap the notification Secondary: Conversion rate โ percentage of clickers who complete a purchase Tertiary: Opt-out rate โ if a variant drives more unsubscribes, that "win" on CTR may be net negative
Step 6: Document and apply learnings
A test without documentation is a lost learning. Record every experiment in a running log and use winners as your new control for future tests.
The title is the first โ and often only โ line a user reads before deciding to tap or dismiss. Test these approaches:
Body copy expands the title promise. Test:
Time of day and day of week are among the highest-variance push notification variables:
Test at least two send times with matched content before drawing conclusions.
Push notifications with images (product photos, offer banners) consistently outperform text-only in ecommerce categories. Test:
On Android, you can add up to 3 action buttons. Test:
Use a chi-square test on your click counts. Most push platforms calculate this automatically, but understanding the inputs helps you design better tests:
| Subscribers per variant | Minimum detectable effect |
|---|---|
| 500 | ~8 percentage points |
| 1,000 | ~5 percentage points |
| 5,000 | ~2 percentage points |
| 10,000+ | ~1 percentage point |
Target 95% confidence (p < 0.05) before declaring a winner. For most ecommerce push programs, this means a minimum of 1,000 subscribers per variant.
Testing too many variables at once. Changing the title, image, and send time simultaneously makes it impossible to know what moved the metric. Test one variable per experiment.
Ignoring the opt-out signal. A notification variant that drives 10% more clicks but 5% more unsubscribes is probably net negative. Track both.
Sending to everyone before testing. Many brands send campaigns to their full list and then wish they had tested. Reserve 20% of your list for an ongoing test cadence before scaling winning variants.
Stopping at CTR. Push notifications are a gateway, not a destination. Track session depth, add-to-cart rate, and purchase rate for every campaign โ not just click counts.
Ignoring notification fatigue. Sending more than 1โ2 pushes per day accelerates opt-outs. Test frequency as a separate variable rather than assuming "more is better."
Match notification content to user segment. Buyers who purchased last week respond differently than subscribers who have never purchased. Segment first, then test within segments.
Test festive messaging early. Diwali, Holi, Valentine's Day, and Independence Day push notifications have a short window. Run your festive format tests 2โ3 weeks before the event so you deploy proven variants during the peak window.
Personalize with purchase history. "[Name], you're almost out of your [product]" outperforms generic restock reminders. Most push platforms support this variable substitution.
Create a "push playbook". After 10โ15 tests, you will have a clear picture of what works for your brand. Document winning formulas (e.g., "[Discount] + [product name] + urgency cue") and train your content team to write to them.
Connect push to the full A/B testing program. Learnings from push notification tests often inform email subject line and SMS copy decisions. Share data across channels.
Related reading: