Sitemaps and robots.txt Without the Mystery

There are two files on most websites that owners treat like electrical panels: they know the files matter, they suspect touching them could break something important, and so they never look inside. The files are the sitemap and robots.txt. I want to convince you that both are boring, and that boring is good news.

Start with robots.txt. It is a plain text file that lives at the root of your domain, at a fixed address: your site followed by /robots.txt. You can look at anyone’s right now. Type any big company’s domain into your browser, add /robots.txt, and read it. It will be a handful of lines saying which crawlers are addressed and which paths they are asked not to visit. That is the whole mechanism. A crawler shows up, reads the file, and decides what to fetch.

Notice the word “asked.” Robots.txt is a request, not a lock. Well-behaved crawlers, including the ones from search engines, honor it. Bad actors ignore it completely. This is the single most important thing to understand about the file, because it kills two common mistakes at once. The first mistake is using robots.txt as security, putting your admin pages in it and thinking they are now hidden. You have actually done the opposite: you have published a list of your sensitive paths in a public file. The second mistake is thinking robots.txt removes pages from search results. It does not. It only stops crawling. A page blocked in robots.txt can still appear in results if other sites link to it; the search engine just shows it without a description, because it was never allowed to read the page. If you want a page out of the index, the right tool is a noindex directive on the page itself, and for the search engine to see that directive, it has to be allowed to crawl the page. People block the page and add noindex and wonder why nothing happens. The block prevents the engine from ever reading the instruction.

The sitemap is even simpler. It is a list of your pages, usually in an XML format, that says: here are my URLs, and here is roughly when each one last changed. That’s it. It is not a ranking factor. Listing a page in your sitemap does not make it rank higher, and search engines feel free to crawl pages that aren’t in it and to skip pages that are. What a sitemap actually does is help discovery. Search engines find pages mostly by following links. If your site’s internal linking is good, they will find everything anyway. If it is not, if you have pages reachable only through a search box or a filter, the sitemap is how those pages get found at all. It is a safety net, not an engine.

So what should you actually do? For most sites, almost nothing, and that is the point. If you use a static site generator or any mainstream CMS, your sitemap is probably generated automatically. Your job is to check three things. First, that the sitemap exists and loads. Second, that it contains the pages you care about and not the ones you don’t: no dead URLs, no redirects, no pages marked noindex. A sitemap full of junk trains crawlers to trust it less. Third, that robots.txt points to the sitemap with a Sitemap line, which is the standard way to advertise it.

For robots.txt, the checklist is shorter and mostly about not shooting yourself. The catastrophic failure mode is a single line, Disallow followed by a slash, addressed to all agents. That asks every crawler to skip your entire site. It sounds too dumb to happen, but it happens constantly, because staging environments block everything on purpose, and then the staging robots.txt ships to production along with everything else. I have seen sites go quietly dark in search for weeks this way, and nobody notices because the site itself works fine. The pages load, customers who already know the site keep coming, and the decline shows up only in a graph nobody is watching. This is exactly the kind of failure that argues for checking these files on a schedule rather than once. It is part of why I built recurring checks into GazeSite: not because the check is hard, but because nobody remembers to do it in the week that matters.

The deeper point is about the category these files belong to. They are infrastructure, and infrastructure rewards understanding over ritual. Most advice about sitemaps and robots.txt is ritual: magic incantations to copy, priorities and change frequencies to tune, none of which modern search engines pay much attention to. Once you know what the files actually are, a list of pages and a set of polite requests, you can reason about them instead of copying spells. You will know that blocking a page and noindexing it are different actions with different effects. You will know that a sitemap can’t rescue a site with bad linking, only supplement a good one. And you will know that the scariest thing about these files was never their complexity. It was that nobody explained they don’t have any.