Crawl Budget for Small Sites: Mostly a Myth, Sometimes Not
Small sites almost never have a crawl budget problem, but they sometimes have a crawl quality problem that looks like one.
Every field accumulates a layer of borrowed anxieties: problems that are real for someone, somewhere, that everyone else adopts anyway because worrying feels like diligence. In SEO, crawl budget is the classic example. I’ve talked to owners of forty-page websites who wanted to know if their crawl budget was being wasted. The honest answer is that a forty-page site could be crawled in its entirety during the time it took to ask the question.
Crawl budget is a real concept. Search engines have finite machines and finite politeness — they won’t hammer your server with requests — so for any given site there’s a rough ceiling on how many pages they’ll fetch in a given period. If your site has ten million pages and the crawler fetches a hundred thousand a day, you have a genuine allocation problem: which hundred thousand? Big e-commerce sites, news archives, and sites that generate pages combinatorially think hard about this, and they should. That’s who the crawl budget literature is written for. Google’s own guidance says as much: sites with fewer than a few thousand pages, updated at a normal pace, don’t need to think about crawl budget at all. The ceiling is so far above your head that you will never touch it.
So why do small sites sometimes look like they have crawling problems? Because there’s a neighboring problem that produces similar symptoms — pages not getting indexed, new content taking forever to appear in search — and it isn’t about budget at all. It’s about what I’d call crawl quality: whether the crawler, on its perfectly adequate visits, can actually find and make sense of your pages.
The most common version is the orphan page. You published a page, it exists, it loads, but nothing links to it — it was announced in a newsletter, or linked from a page that has since been redesigned away. Crawlers discover pages mostly by following links. A page no link points to is, from the crawler’s perspective, a rumor. No budget increase would help, because budget was never the constraint; discoverability was. The fix is a link from somewhere the crawler already goes, and a sitemap that’s actually up to date rather than the one your site generator wrote two years ago and forgot.
The second version is self-inflicted multiplication. A small site can accidentally present itself as a big one. URL parameters are the usual culprit: a catalog page that’s reachable with every combination of sort order, filter, and tracking tag becomes, to a crawler, hundreds of distinct URLs containing the same products. Now the crawler is spending its visits on duplicates, and this is the one case where small-site crawl budget worries have a grain of truth — not because the budget is small but because you’ve filled it with noise. The remedy isn’t more crawling; it’s canonical tags that say plainly which URL is the real one, so the crawler can collapse the duplicates and move on. This is why a missing or wrong canonical is worth flagging even on a modest site. It’s not pedantry. It’s the difference between the crawler seeing forty pages and seeing four hundred smeared copies of them.
The third version is the wall. Sometimes the crawler arrives with ample budget and simply can’t get in — a robots.txt rule left over from the staging environment that blocks everything, a page whose content only appears after JavaScript that fails, a server that answers errors to crawlers while working fine in your browser. These failures are silent from the owner’s side. The site works when you check it, so you assume it works for everyone. When I built GazeSite I made it fetch pages the way an outside visitor does, precisely because of this class of problem: what you see from inside your own network and habits is not evidence of what a crawler sees from outside.
Here’s the practical upshot. If you run a small site and your pages aren’t showing up in search, resist the pull of the exotic explanation. You do not need to “optimize crawl budget.” You need to answer three ordinary questions. Can every page you care about be reached by following links from your homepage? Does each page exist at exactly one canonical URL, with the variants pointing to it? And does the page, fetched cold from outside, actually return its content? In my experience nearly every small-site indexing mystery dissolves into one of those three.
There’s a general lesson hiding in this, which is why I find the crawl budget myth worth writing about. Borrowed problems are attractive because they come with sophisticated-sounding solutions, and sophisticated solutions feel like progress. But the failure modes of small sites are almost always primitive: a missing link, a duplicated URL, a blocked path, a missing H1 or title that leaves the crawler with a page it can fetch but can’t summarize. The unglamorous diagnosis is usually the right one. Fix the primitive things first. For a small site, the primitive things are the whole game.
Test your own website
Type your website address and press Test. In about ten seconds you see a grade for your homepage. Then pick what to order. The first option is free.
More articles
Before paying a stranger on the internet, buyers run a fast, mostly unconscious audit — and most sites fail it on details the owner stopped seeing years ago.
Read →Writing at an 8th-grade level isn't dumbing down; it's removing the tax you charge readers for understanding you.
Read →An explainer video embedded on the homepage feels like it solves the clarity problem. The play-rate data says it solves it for roughly a third of visitors, at best.
Read →