Before the test
Define the decision in advance.
Choose the primary conversion, confidence level and analysis point before seeing results. Write down what improvement would matter enough to act on.
A/B Test Calculator · Evidence before rollout
An uplift is a number. A decision needs context.
Compare two conversion rates and the uncertainty around their difference. Use your completed experiment to understand what the evidence supports before rolling out a change.
Compare your variants
A/B Test Calculator
Bring the counts, not just the percentages.
Enter independent, randomly assigned visitors and how many converted in each group. Count each person once. Your figures are calculated in your browser.
02 / Read the uncertainty
Your test comparison
Enter your counts or load an illustrative example.
Method and assumptions
Individual conversion-rate intervals use Wilson scores. The difference interval uses the two-sided Newcombe hybrid-score method without continuity correction, for B minus A. Relative uplift is an observed estimate, not a probability that B wins.
These intervals assume independent binary outcomes and a fixed analysis point chosen before the experiment. Repeated checking and stopping when results look favourable, testing many variants or metrics, overlapping visitors and biased assignment are not accounted for.
Fewer than five converting or non-converting visitors in either arm triggers a sparse-data warning. This is a cautious display rule, not a sufficient sample-size or power test. We do not validate traffic allocation, randomisation, duration or practical business value.
A 95% confidence method covers the true value in about 95% of repeated samples under its assumptions; it is not a 95% probability that this observed uplift is correct. An interval containing zero does not prove equivalence.
Plain-text summary
Put it to work
A useful result starts with a fair comparison.
The calculator can work with your counts. It cannot repair a biased experiment or decide whether an improvement is valuable to your business.
When counting
Keep the unit consistent.
Use visitors and converting visitors from the same experiment window. Repeat purchases or multiple events from one person are not extra converting visitors.
When reading
Look beyond the relative uplift.
Read the absolute difference and its interval alongside the uplift. Check experiment quality and practical value before making the rollout decision.
Illustrative example
30% uplift. Here is the rest of the story.
Illustrative completed experiment: 10,000 visitors per variant, with 200 converting in A and 260 in B. The confidence level was set to 95% in advance.
A · Control
2.0%
200 converting visitors / 10,000 visitors
B · Variant
2.6%
260 converting visitors / 10,000 visitors
Observed difference
+0.6 percentage points
That is a 30% relative uplift from A’s 2% baseline. It is an observed estimate, not a 30% probability of success.
95% difference interval
+0.185 to +1.018 points
The interval for B minus A sits above zero. Under the stated assumptions, this supports a higher conversion rate for B; the exact size remains uncertain.
The rollout question
Is the difference worth acting on?
Consider lead quality, margin, implementation effort and other important outcomes. Statistical evidence alone does not answer the commercial question.
This example uses the calculator’s Newcombe difference interval. It is illustrative evidence, not a result achieved by ZCMarketing or a promise for your experiment.
Before you move on
Three checks before declaring a winner.
Respect the planned stopping point.
Repeatedly checking and stopping when a favourable interval appears is not accounted for by this calculator. Use the analysis plan you set before the experiment.
Review the test conditions.
Overlapping visitors, biased assignment, tracking changes and multiple comparisons can undermine interpretation. The tool does not validate these conditions.
Treat an unclear result honestly.
An interval containing zero means a difference is not established by this method. It does not prove the variants are equivalent.
Your questions
Know what the result means.
Does 95% confidence mean B has a 95% chance of winning?
No. Under the method’s assumptions, intervals from repeated samples cover the true value about 95% of the time. This is not a probability that B wins or that the observed uplift is correct.
Can I use total orders instead of converting visitors?
No. The model expects a binary outcome for each independent visitor: converted or did not convert. Count each converting visitor once, even if they ordered or submitted a form several times.
Will it tell me how long to run a test?
No. It does not calculate sample size, power or duration. Sparse outcomes trigger a caution, but the absence of that warning does not establish an adequate experiment.
Build the test around a meaningful decision.
Tell us what you want visitors to do and where they hesitate. We can help shape a focused website improvement, with a clear measurement question behind it.
Send an enquiry