Free Tool
Email A/B Test Calculator
A test where B beat A by 0.4% on 200 subscribers proved nothing. This calculator runs the significance check — and rates a single send while it is here.
Runs entirely in your browser. Nothing you type leaves your device, and nothing is stored or sent anywhere. The output is a planning aid built from the figures you enter — check the results before relying on them.
What this tool does
Enter both versions’ sends and opens (or clicks) and the calculator returns each rate and the two-proportion z-test: the z-score, the p-value and whether the difference is significant at 95% confidence. A difference that is not significant is a difference you have not seen — run it longer or with more subscribers, and do not act on it.
The single-send mode rates one email on its own: opens, clicks, click-to-open rate and the unsubscribes, with the same honest framing — one send is a data point, and a list that improves needs the trend, not the single number.
Try it
Waiting. Enter both versions, or switch to the single-send mode, then press Calculate.
How it works
The test is a two-proportion z-test. The pooled rate is the total opens over the total sends; the standard error is the square root of the pooled rate times one minus the pooled rate, times one over each sample size. The z-score is the rate difference divided by that error, and the p-value is the two-tailed probability under the normal curve — approximated with the Abramowitz and Stegun formula, accurate to about 1.5 × 10⁻⁷.
The single-send mode is division: opens over sends, clicks over sends, clicks over opens for the click-to-open rate, unsubscribes over sends. All four are percentages of the same send, so they can be read together.
Assumptions and limitations
Significance is a threshold, not a verdict on the idea.
- The z-test assumes the two samples are independent and each subscriber sees only one version. If the same subscriber got both versions, the test does not apply.
- A 95% threshold means a 5% false-positive rate: one test in twenty will call a chance difference significant. If you run many tests, some of the winners are noise — that is the price of testing often.
- The test says whether the difference is real, not whether it matters. A 0.2-point difference can be significant on a huge list and still not worth acting on.
- One send is one data point. A single send’s open rate tells you about that send; the trend across a dozen tells you about the list.
Worked example
Example (illustrative inputs, not a real campaign). Two subject lines, 1,000 sends each:
| Item | Figure |
|---|---|
| Version A | 210 opens — 21.0% |
| Version B | 260 opens — 26.0% |
| Difference | +5.0 percentage points |
| Significance | p ≈ 0.0083 — significant at 95% |
| Single send | 800 opens on 2,000 sends — 40.0% open, 2.0% click, 5.0% click-to-open |
Frequently asked questions
How many subscribers does a test need?
Enough that the difference you care about is bigger than the noise. A 5-point difference shows on 1,000 sends per version; a 1-point difference needs roughly 10,000. Small lists should test big changes, not small ones.
Can I stop a test early if one version is winning?
Only if the test has already reached significance — stopping at a random peak is how small differences get called winners. Decide the sample size before you start and let it run.
What do I do with a non-significant result?
Nothing. A non-significant difference is a difference you have not seen. Keep the current version, change something bigger, and test again — do not switch to the version that was ahead by chance.
What is click-to-open rate, and why does it matter?
Clicks divided by opens: of the people who opened, how many clicked. It separates the subject line from the content — a high open rate with a low click-to-open rate is a content problem, not a subject problem.