How to A/B Test When You Don't Have Much Traffic

A/B testing has a dirty secret, which is that it’s a rich company’s tool. The famous stories, the button color experiments, the forty shades of blue, all come from companies with millions of visitors, and the method quietly assumes that scale. Then the method gets taught to everyone else, and a founder with two hundred visitors a week dutifully sets up an experiment, runs it for a month, and learns nothing. Worse than nothing: they learn something false, because with small numbers, whichever variant is randomly ahead looks like a winner.

The core problem is arithmetic, and it’s worth feeling it rather than just nodding at it. Detecting a small difference between two versions requires enough visitors that random luck averages out, and the smaller the difference, the more visitors you need, not proportionally more, but explosively more. Distinguishing a page that converts at three percent from one that converts at three and a half can take tens of thousands of visitors per variant. If you get a few hundred visitors a week, that test runs for a year, during which your product, your audience, and your seasonality all change under it, invalidating the comparison anyway. The test isn’t merely slow. It’s unrunnable.

Most low-traffic teams respond by running it anyway and stopping when a result “looks clear,” which is the statistical equivalent of flipping a coin twelve times, seeing eight heads, and declaring the coin biased. The dashboard will happily display a percentage difference between variants. It cannot display the fact that the difference is noise, so the team ships the noise, tells itself the story, and repeats.

What should you do instead? The honest answer has several parts, and the first is the most counterintuitive: test bigger things. The arithmetic that makes small differences undetectable is symmetric; big differences are detectable with far fewer visitors. If your traffic is small, you can’t afford to learn whether a reworded button helps by two percent, but you can afford to learn whether a completely different page, a new headline, a restructured offer, a different price, changes conversion by half. So invert the usual advice. The standard doctrine says change one small thing at a time so you know what caused the effect. At low traffic, that doctrine guarantees you’ll never detect any effect at all. Make big swings. You lose some ability to attribute the result to a single element, but you gain the ability to get a result, and at small scale that’s the trade to make.

The second part is to stop demanding that every decision be an experiment. A/B testing exists to settle questions where reasonable people could disagree and the answer is genuinely unknowable in advance. Plenty of website defects are not like that. A page whose call to action is invisible below the fold, a checkout that errors on mobile, a headline that doesn’t say what the product is, these don’t need a randomized trial any more than a leaking pipe needs one. This is a large part of why I built GazeSite the way I did: it audits a page against known failure modes, unclear calls to action, illegible text, broken mobile layout, missing trust signals, and most sites have a backlog of these. Fixing known defects is not testing. It’s maintenance, and no sample size is required to justify it. The experiments can wait until the obvious things are done, and at low traffic, the obvious things are usually worth more than any experiment you could run.

The third part is to use evidence that doesn’t need volume. Statistics needs numbers, but understanding doesn’t. Five recordings of real people trying to use your signup flow will show you where they hesitate, what they misread, what they never notice, and the failures you see will be blunt, not subtle. Talking to a handful of recent customers about what almost stopped them from buying yields objections you can answer on the page directly. This kind of evidence is unfashionable because it doesn’t produce a percentage, but a founder who has watched ten strangers use the product knows things about it that no dashboard shows. Small-scale teams treat qualitative evidence as a consolation prize for not having traffic. It’s closer to the opposite: it’s the tool that tells you what to change, where an A/B test can only tell you whether a change you already thought of worked.

Fourth, when you do compare versions, compare them sequentially and hold the conclusion loosely. Run the old page for a period, the new one for a similar period, and look at the difference. Yes, this is methodologically weaker than a randomized split; time-varying factors can confound it. But a large effect that persists across a switch is real information, and pretending you’ll get cleaner information from an underpowered split test is self-deception with better branding. The discipline that matters at small scale isn’t randomization. It’s honesty about uncertainty: acting on big clear differences, refusing to act on small murky ones.

The consoling thought in all this is that low traffic, for all its statistical poverty, comes with an advantage: you can still see individual users. At scale, people become rates. At your scale, they’re still people, and you can watch them, talk to them, and fix what visibly fails them. The companies with the traffic to run real experiments would pay dearly to see what you can see for free. Use that, and save the coin-flipping for when you have enough coins.