A/B Test
An experiment that compares two versions of a webpage, email, or ad to see which performs better.
An A/B test randomly splits traffic between two (or more) variants of an experience and measures which produces better outcomes on a defined success metric. Done right, A/B testing is the gold standard for making data-driven changes; done wrong, it's worse than guessing because it gives false confidence to bad decisions.
Key rules: test one change at a time (or use multivariate testing with proper math), run until statistical significance, don't peek at results early, and report based on the metric you committed to before the test started. Tests with insufficient sample size, premature endings, or shifting success metrics are essentially storytelling, not science.
Test variant A (current CTA: 'Sign Up') vs variant B (new CTA: 'Start Free Trial'). After 2 weeks and 15,000 visitors per variant, B converts 18% better with statistical significance. Roll out B as the new control.
Frequently asked questions
How do I know if a test is statistically significant?
Use a p-value calculator or trust your A/B testing platform. Industry standard is 95% confidence (p < 0.05) before declaring a winner. Many platforms peek early and declare significance too soon — don't trust their alerts blindly.
Related terms
Need help applying A/B Test to your business?
Book a free 30-minute strategy call. I'll show you how A/B Test fits into a real growth strategy for your business.
Book a free strategy call