Why most A/B tests lie to you

What peeking actually does

A classical significance test makes you one promise: if there is truly no difference between A and B, the test will only fool you into seeing one about 5% of the time. That promise has fine print almost nobody reads — it holds for exactly one look, at a sample size you fixed in advance.

Check the same test every day and you are running a different procedure: "call it the first time the number crosses the line." Random noise wanders. A truly flat test will drift above the significance threshold and back below it repeatedly on its way to nowhere. If you stop the first time it drifts above, you have not observed a 5%-error event — you have run a search for one. Simulations of daily peeking on flat tests routinely push the real false-positive rate from the advertised 5% to 25–40%.

That is the mechanism behind a pattern many teams know but rarely say out loud: a testing program where a third of tests "win" and the annual conversion rate is flat. The wins were noise, harvested by impatience.

Why a confidence interval beats a p-value here

A p-value collapses your entire experiment into one binary-ish verdict, which invites exactly the wrong question: "has it crossed the line yet?" A confidence interval answers a better one: "given the data so far, what range of true effects is still plausible?"

Early in a test the interval is embarrassingly wide — something like "B is between −14% and +21% versus A." That width is not a flaw; it is the honest state of your knowledge, and seeing it kills the urge to call a winner in week one. As data accumulates, the interval narrows. A result is worth acting on when the interval is both narrow enough to matter and clear of zero.

Intervals also fail more gracefully. A test that ends inconclusive with a p-value tells you almost nothing. The same test ending with "between −2% and +4%" tells you something genuinely useful: whatever this change does, it is small. That is a real finding — it means the lever you pulled is weak, and you should test a bigger swing or a different lever.

If you're going to look, use math that expects looking

Telling people not to look at a running test is advice that has never once worked. The realistic fix is to use statistics that price the looking in. Sequential and always-valid methods — group-sequential boundaries, mixture sequential probability ratio tests, always-valid confidence intervals — are all built on the same trade: your intervals are somewhat wider at any given moment, and in exchange the error guarantee survives you checking every hour.

This is the approach we chose for Looplift. Results are reported as Wilson confidence intervals, and the verdict — win, loss, or not yet decidable — is computed to be peeking-safe. When the product says a test has not earned a call, that is the statistics talking, not a loading state. You can stare at the dashboard all day without corrupting your own experiment, which, realistically, is what everyone does anyway.

When to stop a test

Given all of the above, stopping rules become simple enough to write down. Four cases cover almost everything:

• Stop for a winner when your peeking-safe verdict says so — not when a raw p-value first dips under 0.05, and not because Friday's numbers look exciting.
• Stop for futility when the interval has narrowed around zero. "This change does roughly nothing" is a legitimate conclusion; grinding on for weeks to upgrade it to a formal loss is a waste of traffic.
• Stop for harm early: if the variant is credibly hurting revenue or a guardrail metric, kill it. Protecting the business outranks statistical closure.
• Otherwise, run to the sample size you estimated before launch. If your traffic cannot reach that sample in a few weeks, the honest move is to test a bolder change with a larger expected effect — not to run a subtle tweak forever.

Flicker is not a cosmetic problem. It is a measurement bias.

Everything above assumes the numbers arriving at your dashboard are clean. On a client-side test they may not be, and the reason is not the one usually given.

A client-side test paints the original page, then swaps in the variant. The standard complaint about this is that it looks unprofessional. That is true and it is the least of it. Look at who pays the cost: visitors in the control arm see the page they were always going to see. Visitors in the variant arm see the original, then a flash, then the variant. The flash is not a rendering artefact spread evenly over your traffic — it is a second treatment, applied to one arm only.

So your experiment is no longer testing the headline. It is testing the headline plus a visual stutter, against no headline change and no stutter. If the variant loses narrowly, you cannot say which of the two caused it. If it wins narrowly, it won despite a handicap the control never carried, and you have understated the effect.

The bias is also largest exactly where it hurts most. The gap between first paint and the swap is a function of device speed and connection quality, so it is widest on a mid-range Android on mobile data — the same visitors whose purchase intent is most fragile. A test that looks fine on the desktop you QA’d it on can be meaningfully biased for the segment that decides your mobile conversion rate.

A week is not an arbitrary number

The sample-size gate above has a sibling that teams skip more often, because on a busy store it feels like waiting for nothing: run the test for at least one full business cycle before you read it.

Traffic is not a homogeneous stream. A storefront on a Tuesday and the same storefront on a Saturday have different visitors with different intent — different traffic sources, different device mix, different distance from payday. A test that reaches its sample target in six hours has measured Tuesday morning, thoroughly. It has not measured your store.

Novelty makes it worse in the other direction. Returning visitors react to a changed page partly because it changed, and that reaction decays. Read the test inside that window and you have measured a response to novelty that will not be there in a month, which is the mechanism behind the depressingly common pattern of a winner that fails to replicate after rollout.

Looplift enforces 168 hours — seven days, every weekday and both weekend days contributing — as a gate alongside the sample target. Both have to be satisfied before a verdict is available. On a high-traffic store the runtime gate is usually the binding one, and that is the point: it exists precisely for the case where the data arrives faster than the week does.

Frequentist or Bayesian is the wrong argument

Vendors market the choice as if one school were simply correct. It is worth knowing what actually separates them, because the distinction that matters commercially is not the one in the brochure.

A frequentist confidence interval answers: if I repeated this experiment many times, how often would an interval built this way contain the true effect? A Bayesian credible interval answers a question closer to what you wanted to ask: given this data and my prior beliefs, where is the true effect likely to be? The Bayesian phrasing is more intuitive, and its priors are a real input — with an honest prior it is a better description of a small experiment, and with a flattering one it is a way to reach a conclusion faster.

Here is the part that decides more outcomes than the school does: whether the tool lets you stop when you like. A Bayesian dashboard that invites you to check probability-to-beat-control every morning and act on it has the same optional-stopping problem described at the top of this post, dressed differently. A frequentist test with a pre-committed sample size and a frozen decision does not. The guarantee comes from the stopping rule, not the mathematics.

Looplift is frequentist — Wilson score intervals per arm, a Wilson interval on the difference, and a two-proportion z-test — chosen because it composes cleanly with a pre-committed stopping rule and because it is easy to state exactly what it does and does not promise. That is not a claim that Bayesian methods are worse. It is a claim that a stopping rule you cannot override is worth more than either.

Conversion rate is the cheap number

One last way a clean test misleads: measuring the wrong thing carefully.

Conversion rate is the default everywhere because it is a proportion, and proportions are statistically well-behaved — bounded, low variance, quick to resolve. It is also the metric most able to move in the opposite direction to your bank balance. A change that nudges visitors toward a cheaper product, or leans on urgency to convert browsers who would have bought more later, can lift conversion rate and lower revenue at the same time.

Revenue per visitor is the number that cannot do this, because it carries both halves: how many bought, and how much they spent. It is also genuinely harder to measure, and it is worth understanding why rather than treating it as a preference. Order values have a long right tail — a handful of large baskets carry a disproportionate share of revenue — so revenue per visitor has far higher variance than a conversion rate computed on the same traffic, and higher variance needs more data to resolve. This is the real reason tools default to conversion rate, and it is a reason about convenience, not about correctness.

The practical answer is not to pick one. Read the conversion rate to understand what the change did to behaviour, read revenue per visitor to decide whether to ship it, and treat a variant that lifts one while lowering the other as the interesting result it is — something worth understanding before rollout, not a rounding error.

The habit that fixes most of this

Before launch, write down three things: the metric, the minimum effect worth shipping, and the planned sample size. Thirty seconds of writing removes the two biggest sources of self-deception — redefining success after seeing the data, and stopping on a lucky day. Looplift enforces this by making an experiment declare its conversion goal before it can launch, and by recording the decision when you conclude one.

None of this makes testing slower in any way that matters. It makes testing mean something. A smaller number of results you can actually trust compounds; a large pile of noise-wins does not — and the flat year-end conversion rate is how the pile eventually introduces itself.