A/B Test Planning Template

Hypothesis Format Template

Hypothesis Checklist

• Insight-Driven: Is your hypothesis based on data (analytics, heatmaps, user research) rather than opinion?
• Specific Change: Can a developer or designer implement exactly what you're proposing without ambiguity?
• Measurable Outcome: Have you defined the primary metric and the minimum lift you're testing for?
• Business Justification: If this test wins, will it meaningfully impact revenue or key business goals?

Sample Size Calculation

• Baseline Conversion Rate: What's your current conversion rate for the metric you're testing? Pull from the last 30 days.
• Minimum Detectable Effect: What's the smallest lift that would be worth implementing? Usually 10-20% relative improvement.
• Statistical Power: Use 80% power (standard) unless you need higher confidence. Higher power requires more traffic.
• Significance Level: Use 95% confidence (α = 0.05) as standard. Don't peek at results early or you'll inflate false positives.

Test Duration Best Practices

• Full Week Cycles: Run tests in complete 7-day cycles to account for weekday vs weekend behavior differences.
• Minimum 2 Weeks: Even with enough traffic, run for at least 2 weeks to capture behavioral variations and smooth out anomalies.
• Avoid External Events: Don't run tests during major holidays, sales events, or product launches that could skew results.
• Traffic Allocation: Use 50/50 splits for most tests. Only use uneven splits if you need to minimize risk exposure.

Tracking Setup Checklist

• Primary Metric Tracking: Confirm your conversion goal fires correctly in both variations. Test with browser dev tools.
• Secondary Metrics: Track 2-3 secondary metrics to understand side effects (bounce rate, time on page, scroll depth).
• Variation Assignment: Ensure users are consistently bucketed into the same variation across sessions (use cookies or user IDs).
• Guardrail Metrics: Monitor revenue per visitor and cart abandonment to catch unexpected negative impacts early.

Pre-Launch QA Checklist

• Visual QA: Test both variations on desktop, mobile, and tablet. Check all major browsers (Chrome, Safari, Firefox).
• Tracking Verification: Complete the conversion flow in both variations and confirm events fire in your analytics tool.
• Flicker Testing: Check for visual "flicker" when the test loads. Use server-side testing or anti-flicker snippets if needed.
• Performance Impact: Confirm the test doesn't slow page load. Run Lighthouse tests on both variations.

Statistical Analysis Checklist

• Reach Sample Size: Don't end the test early, even if results look promising. Wait until you hit your calculated sample size.
• Statistical Significance: Use a proper significance calculator (like our T-Test Calculator) rather than relying on platform calculations alone.
• Practical Significance: Even if statistically significant, is the lift large enough to matter to your business?
• Secondary Metrics Review: Check if the winning variation had negative impacts on other important metrics.

Segmentation Analysis

• Device Segmentation: Did the treatment perform differently on mobile vs desktop? Common for design changes.
• Traffic Source: Compare results by traffic source (paid, organic, direct, email). Different audiences may respond differently.
• New vs Returning: New visitors often respond differently than returning visitors to messaging changes.
• Geographic Differences: For international sites, check if results vary by region or language.

Decision Matrix

What to Document

• Test Details: Original hypothesis, variations tested, sample size, and duration.
• Results Summary: Primary metric lift, statistical significance, and confidence intervals.
• Key Learnings: Why did it win or lose? What does this tell you about your users?
• Next Steps: Follow-up tests to run, other pages to apply learnings to.