A/B testing without documentation is just an activity. With documentation, it becomes a program. You capture why you ran each test, what you expected, what actually happened, and what you learned. That record compounds over time. This free template covers everything your team needs to document before, during, and after every test. Copy it and use it.
Most D2C teams run A/B tests without proper documentation. The result: the same ideas get re-tested years later because no one remembers the results, new team members can't explain why the site looks the way it does, and previous learnings go to waste because they were never written down.
Teams with good documentation work differently. They know exactly which elements have been tested, what the results were, and which hypotheses remain. New team members onboard faster. Investors can see a rigorous experimentation process. The testing program becomes a strategic asset rather than a scattered collection of experiments.
The complete A/B testing documentation template
Below is the full template. Copy this into Notion, Google Docs, or Airtable for each test.
1. Test overview
Test Name: [Descriptive name, e.g., "Homepage Hero, Value Prop vs Social Proof Focus"]
Test ID: [ID from testing tool]
Created by: [Name]
Creation date: [Date]
Status: Planning / Running / Completed / Abandoned
2. Hypothesis statement
The most important part of any test document. A good hypothesis has three components: observation, proposed change, and expected outcome.
Observation:
[What did you see in your data, session recordings, or customer feedback
that suggests this change is worth testing?]
Example: "Heatmap data shows 78% of mobile visitors never scroll past the
hero section on the homepage. The current hero focuses on product imagery
with no explicit value proposition. Customer interviews indicate that
new visitors don't understand what differentiates us from competitors."
Proposed Change:
[What specifically will you change in the variant?]
Example: "Replace the current product-image-focused hero with a headline
stating our core value proposition ('India's fastest-delivered skincare,
with a 30-day returns guarantee'), with one product image and a prominent CTA."
Expected Outcome:
[What metric do you expect to improve, by approximately how much, and why?]
Example: "We expect the variant to improve homepage-to-product-page
click-through rate by 15 to 20%, because communicating the value proposition
clearly reduces the information gap that causes visitors to bounce without
engaging further."
3. Test setup
Test Type: A/B / Multivariate / Personalization
Pages in scope:
- [List every URL or page template included in the test]
Traffic allocation:
- Control: [X]%
- Variant A: [Y]%
- Variant B (if multivariate): [Z]%
Target audience:
- All visitors / New visitors only / Returning visitors only /
Mobile only / [Custom segment definition]
Testing tool: [CustomFit.ai / VWO / etc.]
Test ID in tool: [ID]
Launch date: [Date]
Planned end date: [Date]
Maximum duration: [e.g., 30 days]
Sample size target: [Required visitors per variant for significance]
Statistical significance target: [95% / 90% confidence]
4. What changes in the variant
Variant description:
[Clear written description of every change made in each variant.
Be specific enough that someone could recreate the variant from the description alone.]
Variant A changes:
1. [Change 1, element, change made, reason]
2. [Change 2]
3. [Change 3]
What STAYS THE SAME:
[List elements that are intentionally kept identical between control and variant.
This is important for isolating the effect of the changes.]
5. Metrics
Primary metric (decision metric):
[One metric that determines whether the variant wins or loses]
Example: "Purchase completion rate (orders / sessions)"
Secondary metrics (diagnostic):
[2 to 4 additional metrics you'll track to understand how the primary
metric changed and why]
Example:
- Homepage to product page click-through rate
- Add-to-cart rate
- Revenue per session
Guardrail metrics:
[Metrics you're monitoring to ensure the variant doesn't cause harm
in other areas]
Example:
- Return rate (we don't want a misleading variant to increase returns)
- Customer support contact rate (we don't want confused customers)
- Page load time (we don't want the variant to slow the page significantly)
Baseline (pre-test) values:
- Primary metric baseline: [X%] (measured over [date range])
- Secondary metric 1 baseline: [Y%]
- Secondary metric 2 baseline: [Z%]
6. QA checklist
[ ] Desktop render verified (Chrome)
[ ] Mobile render verified (iPhone)
[ ] Mobile render verified (Android)
[ ] Conversion tracking fires correctly
[ ] Traffic split verified (both variants receiving visitors)
[ ] Anti-flicker functioning (no visible flicker on throttled connection)
[ ] No JavaScript console errors
[ ] Checkout flow complete walkthrough done
[ ] Form submissions functional
[ ] Page speed impact checked (baseline vs. test)
QA completed by: [Name]
QA date: [Date]
QA notes: [Any issues found and resolved]
7. Results
(Complete after test ends)
Test end date: [Date]
Reason for ending: [Reached significance / Reached max duration / Abandoned]
Final sample sizes:
- Control: [N visitors]
- Variant A: [N visitors]
- Variant B (if applicable): [N visitors]
Primary metric results:
- Control: [X%]
- Variant A: [Y%]
- Absolute difference: [+/-Z percentage points]
- Relative difference: [+/-W%]
- Statistical significance: [X% confidence]
- Result: Control wins / Variant A wins / Inconclusive
Secondary metrics:
- [Metric 1]: Control [X%] vs Variant [Y%], [direction]
- [Metric 2]: Control [X%] vs Variant [Y%], [direction]
Revenue impact estimate:
- Estimated incremental revenue from winner (if implemented):
[CVR improvement] Γ [monthly sessions] Γ [average order value] = [βΉX/month]
8. Segment analysis
Device type results:
- Mobile: Control [X%] vs Variant [Y%], [Win/Loss/Neutral]
- Desktop: Control [X%] vs Variant [Y%], [Win/Loss/Neutral]
Traffic source results:
- Paid traffic: Control [X%] vs Variant [Y%]
- Organic: Control [X%] vs Variant [Y%]
- Direct: Control [X%] vs Variant [Y%]
New vs returning visitor results:
- New visitors: Control [X%] vs Variant [Y%]
- Returning visitors: Control [X%] vs Variant [Y%]
Key segment finding:
[1 to 2 sentences summarizing the most important segment insight]
9. Decision and implementation
Decision: Implement winner / Stay on control / Run follow-up test / Needs more data
If implementing:
- Implementation date: [Date]
- Implementation method: [CustomFit.ai rollout / Developer implementation]
- Implemented by: [Name]
If not implementing:
- Reason: [Why the winner wasn't implemented despite statistical significance]
10. Learnings and next steps
What we learned:
[2 to 4 bullet points of genuine insight from this test, applicable beyond just this specific test]
Example:
β’ Adding specific delivery promise ("arrives by [date]") outperforms
generic "fast shipping" messaging for mobile visitors, suggests
specificity matters more for mobile where context is limited.
β’ Trust signals work differently for paid vs organic traffic, paid
traffic needs more reassurance before converting.
What we'd do differently:
[Honest reflection on test design, setup, or analysis that could be improved]
Follow-up tests generated:
1. [Hypothesis for test inspired by this result]
2. [Hypothesis]
Test ID linked to this result: [If a follow-up test has already been created]
Where to store your A/B test documentation
Notion works well for teams that want rich documentation with linked databases. Create a master "Experiments" database where each record is one test, with all 10 sections as properties or sub-pages.
Airtable suits teams that want spreadsheet-style filtering and reporting alongside rich documentation. The grid view lets you filter tests by status, page, primary metric, or result.
Google Sheets is the simplest starting point. A single spreadsheet with one row per test and key columns covers the essentials. It handles small teams well, though rich text fields are awkward.
Confluence is common in larger organizations with existing Atlassian tooling, and integrates well with Jira for test backlog management.
The right tool is whichever one your team will actually use consistently. Start with Google Sheets if you're unsure. You can migrate to a richer tool as your documentation practice matures.
Tips and best practices
Write the hypothesis before setting up the test. The temptation is to configure the test in your testing tool first and write documentation later. Reverse that order. Writing the hypothesis forces clarity about what you're testing and why, and often reveals test design problems before you've spent time on a poorly-specified experiment.
Document null results as thoroughly as winners. A test that shows no significant difference between control and variant is still valuable. It tells you the change didn't matter, which is itself a finding. "We tested X, it didn't work, here's what we learned" is exactly the institutional knowledge that prevents repeated effort.
Review test documentation in retrospectives. Monthly or quarterly, go through recent completed tests as a team. Look for patterns: do tests on mobile consistently outperform desktop? Do tests on paid traffic pages behave differently from organic pages? These patterns generate new hypotheses worth testing.
Link related tests. When test B was inspired by a learning from test A, link them in your documentation. This creates a searchable chain of reasoning that helps new team members understand the experimentation history.
Key takeaways
- A/B test documentation turns a series of experiments into a compounding learning program.
- The hypothesis statement (observation, proposed change, expected outcome) is the most important thing to write before any test.
- Document all 10 sections: hypothesis, setup, variant changes, metrics, QA, results, segments, decision, and learnings.
- Document null results and losing tests as thoroughly as winning ones. The learning is equally valuable.
- Use CustomFit.ai's dashboard for test management and pair it with Notion or Airtable for documentation your whole team can access and build on.
ab testing
ab testing
ab testing