How many orders before your numbers mean anything?

The number that ends most tests before they start

Conversion rates are small percentages, and small percentages are statistically expensive to measure. That is the whole problem, and it is not a matter of opinion or of using a better tool.

The standard sample size formula for comparing two conversion rates, at the usual 95% confidence and 80% power, is:

n = 15.7 × p × (1 - p) / d²

// p = your current conversion rate, as a decimal
// d = the absolute improvement you want to detect

// Example: 2% now, hoping to detect a rise to 2.2%
p = 0.02
d = 0.002                     // 2.2% - 2.0%, in absolute terms

n = 15.7 × 0.02 × 0.98 / 0.000004
n = 76,900 sessions per variant

Sessions needed per variant

Note the squared term in the denominator. Halving the size of the effect you want to detect does not double your sample requirement, it quadruples it. That single fact is why small stores cannot test small improvements, and no amount of patience fixes it, because your traffic is also being spent on seasonality while you wait.

The whole problem, in one table

Assuming a 2% conversion rate, which is unremarkable for ecommerce.

Now put your own traffic against that. A store doing 3,000 sessions a month needs just over four years to detect a 10% improvement, and just over a year to detect a 20% one. In both cases the market, your products and your marketing will have changed several times over, which means the test was never going to answer the question anyway.

Why "we increased conversions by 30%" is usually noise

Look at the third row of the table. To honestly detect a 50% relative lift you need about 120 orders across both variants. Below that, a 30% or 50% swing is entirely ordinary random variation.

Which means a case study reporting a large lift is telling you one of three things, and you cannot tell which from the headline:

• The sample was large, and the result is real. Rare, and usually stated proudly, with numbers.
• The sample was small, and they reported the noise. Extremely common, and invisible unless the sample is disclosed.
• The starting point was broken. Also common, and the most honest of the three. If the old checkout had an unusable button on mobile, a 40% lift is real and it is a bug fix, not an optimisation you can copy.

Peeking, and why it feels like diligence

This is the mistake that catches careful people, because it is indistinguishable from paying attention.

You start a test. You check it daily. On day nine, variant B is ahead and the tool says 95% confident. You stop the test and ship B.

That result is not 95% confident. The 95% figure assumes you looked once, at a sample size fixed in advance. If you check repeatedly and stop the moment it crosses the line, you have given random noise many chances to cross the line, and it eventually will. Run two identical variants against each other and check daily and you will still get a "significant" result a large share of the time. The test detected nothing except your willingness to stop looking.

The discipline is unglamorous and it is the whole of it:

• Calculate the sample size before you start, using the formula above.
• Decide the stopping point in advance, and write it down.
• Do not stop early because it looks good. Stop early only if something is broken.
• Run whole weeks. Tuesday traffic is not Saturday traffic, and a test ending mid-week is skewed by whichever days it over-sampled.
• Test one thing. If you change the layout, the button copy and the field order together, a win tells you nothing about which of the three did it, and a loss may be hiding two wins and a disaster.

What to do when you are too small to test

Which, to be clear, is most stores. This is a normal condition and not a failure. The alternative is not guessing, it is a different and older method.

1. Fix defects rather than testing preferences. A button that cannot be tapped on a phone does not need an experiment, it needs fixing. Defects produce step changes large enough to see without statistics, and they are the most common cause of a genuinely bad conversion rate. Walk your own checkout on a real phone with a realistic cart.
2. Follow the research instead of your own sample. This is exactly what large-sample usability research is for. The Baymard Institute's benchmarking of 344 sites and its testing with over 4,400 participants can afford a sample you never will. Borrowing a conclusion drawn from that is more reliable than a conclusion drawn from your own 200 orders.
3. Watch session recordings. Twenty recordings of real people checking out will teach you more than any test you have the traffic to run. It is qualitative, it does not prove anything, and it will show you problems you did not know existed.
4. Measure the gate, not the site. Checkout completion rate moves on a much larger base than site-wide conversion, so a change there is visible sooner. The four gates and how to measure them are in your traffic is fine, here is where the money leaks.
5. Accept directional evidence and keep moving. If a change is supported by research, does not hurt your numbers over a quarter, and makes the checkout obviously less annoying, ship it. Demanding proof you cannot afford is a way of never improving anything.

The things that are easier to measure

Not everything needs this much traffic. Sample size requirements fall fast as the base rate rises, so metrics further down the funnel are cheaper to measure than site-wide conversion.

• Checkout completion rate. Base rate around 30% rather than 2%, so detecting a change needs dramatically fewer sessions. This is the number to run your experiments on.
• Order bump take rate. If a bump is accepted by 12% of checkouts, that is a large base rate and a change in it is visible quickly. Watch checkout completion alongside it, as covered in WooCommerce order bumps that actually convert.
• Average order value. A continuous measure rather than a proportion, so it needs far fewer orders, though its variance is high if you sell a wide price range.
• Field level completion. If 8% of shoppers fail validation on one field, that is a large, obvious signal and you do not need statistics to act on it.

The general rule: the closer the metric sits to the change you made, the less traffic you need to see it move, and the more confident you can be that you caused it.

A short list of claims to distrust

• Any percentage without a sample size. Covered above, and it is the main one.
• "We tested this on our own store." One store, one audience, one product category. Possibly true there and not transferable to you.
• A test that ran for exactly as long as it took to win. That is peeking, described admiringly.
• Month over month comparisons. Not a test. Seasonality, marketing changes and traffic mix all moved too.
• Any figure a vendor quotes about their own product, ours included, unless the method is published alongside it.
• The widely quoted 35% checkout uplift figure. It is real and it is Baymard's, but it estimates the total available from fixing an average of 32 distinct issues on a large site against a 110-point standard. It is not what one change does, and it is routinely quoted as though it were.