A/B testing

What an A/B test compares

Split your visitors at random, show one group the current version and the other a change, then compare. That definition is settled and it is not the hard part.

The definition is not where tests fail.

That is what a test is for. Every line in that box is a decision somebody made. The one above is the kind nobody in the room can call in advance. Taking the one-time option away might lift subscriptions, or it might cost you the people who wanted one box.

Tests fail in execution. The split does not come out the way it was configured. The tool measures something other than what you think. The result gets read before the test finished. Or the thing that moved was the novelty, not the change. Those failures do not announce themselves. A broken test produces a number that looks exactly like a working test’s number.

The vendors selling testing tools answer "what is an A/B test". Far fewer answer "how do I run one without fooling myself".

That second question is the harder one, and it is where most of this goes. Whether to test at all, how many of your ideas will win and what the programme costs is a different one, answered in our guide to conversion rate optimization.

What you split on decides what you can measure

Before anything else comes the unit of randomisation. You can split by session, by browser cookie or by logged-in user, and the choice changes what the result can mean.

The split unit and the metric denominator have to match.

If you split by session, a returning shopper can see version A on Monday and version B on Tuesday. Their behaviour is then a mixture of both, and the comparison is diluted. If you split by cookie, the same person on a phone and a laptop counts as two people. If you split by logged-in user, you only cover the ones who log in.

That rule is the same one behind what a conversion rate is: before two people can compare a rate they have to agree on what it divides. The consequence is the same here, because two teams using the same word can be measuring different populations.

This is the part of the setup that cannot be fixed afterwards. Sample size can be extended, a metric can be recalculated, but a test that split on the wrong unit has to be re-run.

Two checks before the result

Splitting on the right unit is necessary and it is not enough. A test result answers the question you asked only if the test itself worked. There are two cheap ways to find out. Use both checks before reading a result. What remains unanswered is how often these checks fail in small ecommerce programmes, and what each one cannot catch.

First: did the split come out the size you asked for.

Microsoft Research, read 2 September 2026How often the split itself is broken in serious programmes

Microsoft states it as a rule: every A/B test must pass the sample ratio mismatch check before it is analysed for its effects. A store without that gate is not running fewer broken tests, it is finding out about fewer of them.

The check, stated as a gate by the people who run it

1. 1Microsoft does not treat this as advice. No result is revealed until the split passes the check.
2. 2The threshold is published too: a p-value under 0.0005, set deliberately low so the gate does not cry wolf.

The check is arithmetic you can do yourself. Take the number of visitors in each group, compare against the split you configured, and ask how likely that difference is by chance.

When the check fires

When the check fires, the fix is not to rebalance the numbers. Microsoft’s account of an MSN experiment shows why.

Microsoft Research, read 2 September 2026What a broken split can do to a result

Microsoft draws the general lesson themselves: missing users are rarely just some users, they are often the ones most affected by what was being tested. Microsoft records that this test had enough power to detect very small changes, and the direction was still wrong. Their instruction is precise: do not trust the results of a test with a mismatch until you diagnose the root cause. In this case the diagnosis found the cause and the corrected result reversed.

Second check: run an A/A test

Second: run an A/A test. Show both groups the same thing and see what your tool reports. Since nothing differs, any significant result it produces is a false positive. An A/A test catches a broken split or a metric wired to the wrong event. It does not measure your tool’s error rate. As the calculator above shows, twenty runs at a 95 percent threshold are expected to produce about one significant result on their own.

Four ways a valid test still misleads you

A test can pass both of those checks and still report something untrue. These failure modes come from LinkedIn’s published research on the automatic detection it built for its own experimentation platform. That is evidence they are worth catching, not evidence of how often they happen. That research does not publish an incidence rate.

Stopping early is the one with a paper written about it.

Watching a test and stopping when it looks significant is not impatience, it is a different statistical procedure from the one the p-value assumes. Johari, Koomen, Pekelis and Walsh put it directly: frequentist p-values and confidence intervals are "wholly unreliable if users endogenously choose samples sizes by continuously monitoring their tests". Their answer was always-valid p-values, built for exactly this, and the paper reports the method implemented in a large-scale commercial A/B testing platform.

So the choice is a real one, not a scolding. Either fix the horizon before launch and do not look, or use a tool whose statistics are designed to be looked at. What you cannot do is use fixed-horizon statistics and peek anyway.

Contentsquare read its 2026 digital experience panel and found conversion rate down 5.1 percent year over year while frustration signals fell 4.3 percent. Measured frustration signals fell while the conversion rate fell too.

Borrowing somebody else's benchmark instead has the same problem one level up. Ecommerce conversion benchmarks mislead because traffic source, business model and calculation method all differ between stores.

Fix the metric you will judge on before launch, not after you have seen which way it went.

None of these is caught by significance. A test can be perfectly powered, correctly split and stopped on schedule, and still be measuring novelty. Which is why some things are better fixed than tested.

How to run an A/B test in order

Everything above collapses into a sequence, and the order decides whether the result means anything. Four steps happen before a single visitor sees the change. The other two can wait, because nothing in the first four can be reconstructed once the data has arrived.

Six steps, and four of them happen before launch.

1. Write down what would disprove the idea

   Not what would confirm it. If no result would change your mind, you are not running a test, you are running a demonstration.

2. Pick the split unit and the metric together

   They have to be the same thing. This is the decision that cannot be fixed later.

3. Compute the sample and the stop date

   Before launch, in writing. The arithmetic and why underpowered tests produce false winners are in the CRO method.

4. Run an A/A test if you have never sanity-checked the tool

   Once per tool, not once per test. It catches a broken split or a miswired metric, which is a smaller claim than validating the statistics.

5. Check the sample ratio before you look at the outcome

   The split first, the result second. Microsoft’s rule is not to trust the result of a test with a mismatch until you have diagnosed the cause, which is a diagnosis, not a discard.

6. Read the result at the horizon you set

   Or use always-valid statistics, and know which one your tool gives you.

The first step is the cheapest of the four and the easiest to leave out, because nothing downstream refuses to run without it. A hypothesis that cannot be wrong will always be confirmed by something.

Step three is the one that decides whether the rest is worth doing at all, and it is arithmetic, not judgment. The CRO method works through what a given baseline and effect size demand, and what happens to your results when you launch below that number anyway.

What A/B testing costs

That sequence needs somewhere to run, and one major free option ended on a known date. Know this before you price the work, because it retired Google’s free first-party testing tool.

Google Optimize was retired on 30 September 2023.

Google’s own notice states it plainly: Optimize and Optimize 360 are no longer available as of that date, and any experiments still active on it ended. Google’s stated reason was that the product "did not have many of the features and services that our customers request and need for experimentation testing", and that it would invest in third-party integrations for Google Analytics instead.

What that establishes is narrow: one widely used free first-party tool ended without a first-party successor. This is not a survey of what free or open-source alternatives exist today. It says nothing about what testing costs in general, or about who can still test.

We are not publishing a price table here. Vendor pricing for this category is mostly quote-based and we found no independent published range, and an invented one would be worse than none. What is stable enough to say is the shape: the statistics engine differs between vendors, and which engine you are on decides whether the peeking question above applies to you. Optimizely and VWO each publish an engine built to be watched while it runs. Check which engine your own tool uses before you decide whether you are allowed to look.

Your traffic decides the rest. If you sell to other businesses, the same sample requirement takes longer to reach, because it is a count of visitors and you have fewer of them.

B2B and SaaS CRO starts by asking whether your traffic can carry a test at all.

If your traffic does carry one, the page changes that follow are our landing page optimization service.

When not to run an A/B test

With the cost question bounded, the harder question is whether to run one at all. People keep asking it and rarely get an answer. It has one, and it is arithmetic, not opinion.

If you cannot reach the sample the effect requires, the test does not become a smaller test.

The scale is worth holding in mind. IRP Commerce’s July 2026 figures put the all-markets conversion rate at 2.26 percent. Detecting a small relative change on a baseline that size takes tens of thousands of visitors per variant. The exact number depends on the effect you want to detect and the power you want. That arithmetic, and what it does to your results when you launch below it, is worked through in the CRO guide.

When a change is worth making without a test, look first at the form people are trying to finish. That is the subject of our guide to checkout optimization.

One check settles more arguments than any of the above: take your last completed test, count the visitors in each group, and check the split against what you configured. If the two numbers are further apart than chance explains, that result needs a cause found before it can be trusted, and finding that out costs one calculation.