A/B Significance Engine
Determine winning variations with 95% confidence
Variation A
Variation B
Variation B outperformed Variation A by 42.9%.
Results are not significant yet.
There is currently a 2.1% probability that Variation B is better. We recommend continuing the test until reaching 95% confidence.
What is Confidence?
Statistical significance (Confidence) measures the likelihood that the difference between variations is not due to random chance. In A/B testing, a 95% confidence level is the industry standard for making business decisions.
When to stop testing?
You should stop a test only after achieving a significant result and reaching your pre-determined sample size. Stopping early (peeking) can lead to false positives. Ensure your variations have run for at least one full business cycle.
What is A/B Testing?
A/B testing (split testing) is a method of comparing two versions of a webpage, app, or other product experience to determine which one performs better. You split your audience into two groups - one sees version A (control) and the other sees version B (variant).
Version A
Original/Control
Version B
Variant/Test
Winner
Statistically Significant
Key Metrics Explained
Statistical Significance
Confidence that results are not due to chance. Aim for 95%+
Conversion Rate
Percentage of users who complete desired action
Lift
Percentage improvement of variant over control
Sample Size
Number of visitors needed for reliable results
P-Value
Probability results are due to chance (lower is better)
Confidence Level
1 - p-value, typically 95% or higher
Best Practices
Test One Thing
Change only one element at a time
Run Long Enough
Collect enough data before concluding
Use Real Traffic
Test with actual target audience
Avoid Peeking
Don't stop test early if results look good