Pagination, Infinite Scroll, and Crawlability

There’s a class of bugs that never shows up in your own testing because you’re the wrong kind of visitor. Infinite scroll is the classic example. You open your blog index, you scroll, more posts load, everything works. But the visitor that matters most for discovery isn’t a person with a mouse wheel. It’s a crawler. And a crawler doesn’t scroll.

This surprises people, so it’s worth being precise about what a crawler actually does. Googlebot fetches your page, and these days it does render JavaScript, which lulls people into thinking anything a browser can show, Google can see. But rendering is not interacting. The crawler doesn’t scroll to the bottom of the viewport and wait for your intersection observer to fire. It doesn’t click “Load more.” It renders the page roughly as it arrives, extracts the links it finds in the DOM, and moves on. If posts eleven through two hundred only enter the DOM when a human scrolls, then as far as Google is concerned, your site has ten posts.

The damage is quiet, which is what makes it dangerous. Nothing errors. Your analytics look fine, because humans can still reach everything. What you lose is the long tail: the three-year-old post that would have ranked for some specific query, that nobody links to from anywhere except page fourteen of your archive. If page fourteen doesn’t exist as a page, that post is an orphan. Crawlers find pages by following links from other pages, and an orphan has no links pointing at it. A sitemap helps, and you should have one, but a sitemap is a hint, not a substitute for structure. Google has said for years that pages discovered only through sitemaps, with no internal links, tend to be treated as unimportant. That matches how I’d design it too. Links are how a site expresses what it cares about.

The old-fashioned answer is pagination, and I want to defend it, because it’s become unfashionable for reasons that have nothing to do with whether it works. Numbered pages with real URLs give every item in a long list a crawlable path. Page 2 links to page 3, and a crawler walking those links eventually touches everything. Each page has its own address, which means someone can link to it, bookmark it, open it in a tab. The back button works. These sound like small things, but they’re the connective tissue of the web, and infinite scroll cuts every one of them.

The good news is you don’t actually have to choose. The pattern that works is to build paginated pages first — real URLs, real anchor tags pointing to the next page — and then layer infinite scroll on top with JavaScript for humans. When the visitor nears the bottom, you fetch the next page’s content and append it, ideally updating the URL bar with the History API so the address reflects where they are. Humans get the smooth experience. Crawlers, which just read the anchors, get the ladder of pages. This is plain old progressive enhancement, and it remains the right instinct: build the version that works for the dumbest possible client, then make it nicer for the smart ones.

Two details trip people up even when they get the big structure right. The first is the link itself. A “next page” control implemented as a button with a click handler is invisible to crawlers; they follow <a href> elements, not click handlers. If your “Load more” button fetches data and appends it but there’s no anchor with a real URL underneath, you’ve built infinite scroll with extra steps. The second is canonicals. Some sites point every paginated page’s canonical tag at page one, reasoning that page one is the “real” version. But a canonical tag tells Google the pages are duplicates, and page seven of your archive is not a duplicate of page one — it contains entirely different links. Consolidate them and you’ve asked Google to ignore the very pages that lead to your older content. Each paginated page should canonicalize to itself.

I think about crawlability a lot because GazeSite, the auditing tool I built, looks at websites the way a machine does: it fetches the page, renders it, and works with what’s actually there. That perspective is clarifying. When you audit a site from the raw fetch rather than from memory of how you built it, you see what a crawler sees, and the gap between the two is often startling. A site’s owner knows there are four hundred articles. The rendered homepage links to six.

The deeper lesson isn’t really about scrolling. It’s that a website has two audiences, and only one of them is human. Machines read your site constantly — search crawlers, link preview generators, and now the AI systems that summarize the web for people who never visit it. Every one of them navigates by URLs and links, not by gestures. When you adopt an interaction pattern, it’s worth asking a simple question: can this content be reached by following links alone? If the answer is no, you’ve made something that works beautifully for the visitors you can see and doesn’t exist for the ones you can’t. Infinite scroll is fine. Infinite scroll as the only path to your content is a slow leak in the hull, and you won’t notice the water until you wonder why nobody finds the old stuff anymore.