Push Notification A/B Test Schema: Fields and Variant Splits

A push notification A/B test campaign schema should store the campaign ID, hypothesis, eligible audience, message variants, traffic weight for each variant, schedule, primary metric, guardrail metrics, attribution window, and status. Define variants as an array with stable IDs and complete notification payloads. For a two-variant test, start with a 50/50 split unless you have a documented reason to limit exposure, and keep each subscriber assigned to one variant for the whole campaign.

Which fields belong in a push notification A/B test campaign schema?

Keep campaign settings separate from the content and allocation of each variant. This makes the experiment easier to validate before launch and prevents reporting code from confusing a message label with a campaign identifier.

Field	What it records	Example
`campaign_id`	Stable identifier used across delivery and reporting	`push_cart_2026_09`
`hypothesis`	The single expected behavior change	A specific title will increase notification opens
`audience`	Eligibility rules and exclusions	Cart abandoners who can receive push
`assignment_unit`	The ID used to keep assignment consistent	`subscriber_id`
`variants`	Message payload and traffic weight for every version	Control and variant B
`schedule`	Start time, end time, and time zone	ISO 8601 timestamps with a named time zone
`primary_metric`	The metric used to choose the winner	Notification click-through rate
`guardrail_metrics`	Measures that can disqualify an apparent win	Opt-out rate and conversion rate
`attribution_window`	How long an event counts after delivery	A window chosen before launch
`status`	Campaign lifecycle state	`draft`, `scheduled`, `running`, or `complete`

A compact, platform-neutral representation looks like this:

{
  "campaign_id": "push_cart_2026_09",
  "hypothesis": "A specific product reminder will increase notification CTR",
  "audience": {
    "segment": "cart_abandoners",
    "exclusions": ["push_opt_out"]
  },
  "assignment_unit": "subscriber_id",
  "variants": [
    {
      "variant_id": "control",
      "traffic_weight": 50,
      "payload": {
        "title": "Your cart is waiting",
        "body": "Return to finish your order"
      }
    },
    {
      "variant_id": "title_specific",
      "traffic_weight": 50,
      "payload": {
        "title": "Still deciding? Review your cart",
        "body": "Return to finish your order"
      }
    }
  ],
  "primary_metric": "notification_ctr",
  "guardrail_metrics": ["push_opt_out_rate", "conversion_rate"],
  "status": "draft"
}

The variant weights must add up to 100% of the experiment audience. Assign on a stable subscriber or device identifier, not separately on every send; otherwise one person can receive both versions and contaminate the comparison. If the campaign includes a no-message holdout, model that group explicitly rather than treating it as another notification payload.

Why Push Notification A/B Testing Matters for D2C

Push notifications compete for attention on locked screens alongside messages from family, social apps, and every other brand the customer has installed. The difference between a notification that gets clicked and one that gets dismissed — or leads to an unsubscribe — often comes down to a single word in the title.

Indian D2C brands face a particularly crowded push environment. Consumers who shop across Nykaa, Mamaearth, Boat, mCaffeine, and Plum receive dozens of push notifications daily. Without testing, you are publishing based on intuition in a channel where intuition has a poor track record.

What makes push notifications uniquely testable:

High volume means fast significance (if your subscriber base is large enough)
Results are available within hours, not days
Even small CTR improvements compound: a 3% CTR lift on 100,000 daily pushes equals 3,000 additional store visits per day
Testing is free — it requires only your existing subscriber base and a basic push platform

How to A/B Test Push Notifications: Step by Step

Steps ab process

Step 1: Choose your test variable

Pick one element to test. The most impactful variables, in order:

Title — the first line most users see on a lock screen
Body copy — the supporting message beneath the title
Send time — morning vs. evening, weekday vs. weekend
Image or icon — product photo vs. brand logo vs. offer badge
CTA (action button text) — "Shop Now" vs. "Claim Offer" vs. "See Deal"

Step 2: Write your hypothesis

Example: "I believe leading with a ₹ discount in the title ('Get ₹300 off today') will increase CTR vs. a benefit headline ('Your skin routine just got better') because Indian shoppers are price-motivated and discount-first framing triggers immediate action."

Step 3: Split your audience randomly

Most push platforms (OneSignal, CleverTap, WebEngage, MoEngage) allow you to split subscribers randomly for A/B tests. Use a 50/50 split for push notification tests — unlike email where you hold a winner group, push notifications are typically sent to the full audience at once.

Step 4: Send simultaneously

Both variants must go out at exactly the same time. A 30-minute gap between variants introduces send-time bias that will contaminate your results.

Step 5: Measure the right metrics

Primary: Click-through rate (CTR) — percentage of recipients who tap the notification Secondary: Conversion rate — percentage of clickers who complete a purchase Tertiary: Opt-out rate — if a variant drives more unsubscribes, that "win" on CTR may be net negative

Step 6: Document and apply learnings

A test without documentation is a lost learning. Record every experiment in a running log and use winners as your new control for future tests.

What to Test: High-Impact Push Notification Variables

Variables what test

Title Copy

The title is the first — and often only — line a user reads before deciding to tap or dismiss. Test these approaches:

Discount-first vs. benefit-first: "₹400 off Kapiva Apple Cider Vinegar" vs. "Support your gut health naturally"
Urgency vs. curiosity: "Last 6 hours of sale" vs. "Something big is ending tonight"
Social proof vs. offer: "5,000 orders this week — here's why" vs. "Weekend flash sale: 30% off"
Question vs. statement: "Did your skin care run out?" vs. "Restock your skin care — 20% off today"

Body Copy

Body copy expands the title promise. Test:

Specific details ("Free COD on orders over ₹499") vs. vague benefits ("Shop with confidence")
Short one-liners vs. two-sentence elaborations
Including product names vs. category-level messaging

Send Time

Time of day and day of week are among the highest-variance push notification variables:

7–9 AM: Strong for wellness (supplements, skincare routines)
12–1 PM: Lunch break browsing — good for fashion, accessories
7–9 PM: Post-dinner prime time — highest overall CTR for most categories
Saturday morning: Highest purchase intent day for Indian D2C

Test at least two send times with matched content before drawing conclusions.

Rich Media (Images)

Push notifications with images (product photos, offer banners) consistently outperform text-only in ecommerce categories. Test:

Product on white background vs. lifestyle image
Discount badge overlay vs. clean product shot
Animated GIF vs. static image (supported on Android)

Action Buttons

On Android, you can add up to 3 action buttons. Test:

Single CTA vs. two-button choice (e.g., "Shop Now" + "Remind Me Later")
"Buy Now" vs. "See Deal" vs. "Claim Offer"
Button with urgency cue vs. button without

Statistical Significance for Push Tests

Use a chi-square test on your click counts. Most push platforms calculate this automatically, but understanding the inputs helps you design better tests:

Subscribers per variant	Minimum detectable effect
500	~8 percentage points
1,000	~5 percentage points
5,000	~2 percentage points
10,000+	~1 percentage point

Target 95% confidence (p < 0.05) before declaring a winner. For most ecommerce push programs, this means a minimum of 1,000 subscribers per variant.

Common Push Notification Testing Mistakes

Testing too many variables at once. Changing the title, image, and send time simultaneously makes it impossible to know what moved the metric. Test one variable per experiment.

Ignoring the opt-out signal. A notification variant that drives 10% more clicks but 5% more unsubscribes is probably net negative. Track both.

Sending to everyone before testing. Many brands send campaigns to their full list and then wish they had tested. Reserve 20% of your list for an ongoing test cadence before scaling winning variants.

Stopping at CTR. Push notifications are a gateway, not a destination. Track session depth, add-to-cart rate, and purchase rate for every campaign — not just click counts.

Ignoring notification fatigue. Sending more than 1–2 pushes per day accelerates opt-outs. Test frequency as a separate variable rather than assuming "more is better."

Tips and Best Practices

Match notification content to user segment. Buyers who purchased last week respond differently than subscribers who have never purchased. Segment first, then test within segments.

Test festive messaging early. Diwali, Holi, Valentine's Day, and Independence Day push notifications have a short window. Run your festive format tests 2–3 weeks before the event so you deploy proven variants during the peak window.

Personalize with purchase history. "[Name], you're almost out of your [product]" outperforms generic restock reminders. Most push platforms support this variable substitution.

Create a "push playbook". After 10–15 tests, you will have a clear picture of what works for your brand. Document winning formulas (e.g., "[Discount] + [product name] + urgency cue") and train your content team to write to them.

Connect push to the full A/B testing program. Learnings from push notification tests often inform email subject line and SMS copy decisions. Share data across channels.

Key Takeaways

A/B testing push notifications requires at least 1,000 subscribers per variant and simultaneous send times
Test one variable at a time: title, body, image, send time, or CTA
Track CTR, conversion rate, AND opt-out rate — CTR alone can be misleading
Rich media (images) consistently outperform text-only for ecommerce categories
Document every experiment to build a brand-specific push notification playbook
Festive season and send-time tests offer the largest absolute CTR gains for Indian D2C brands

Related reading: