Robots.txt is one of the oldest and simplest files on the web, and also one of the most misunderstood. It is a plain-text note to crawlers at the front door of a site, suggesting where they should and should not go. Knowing exactly what it can and cannot do prevents both wasted effort and genuinely damaging mistakes.
What the file is
Robots.txt lives at the root of a site, at a fixed, well-known path, so crawlers know where to look before they start. It is a plain-text file with a simple structure: one or more groups, each naming a user-agent — the crawler the rules apply to — followed by rules that allow or disallow particular paths. A well-behaved crawler reads this file first and adjusts its behaviour accordingly.
The basic syntax
Each group begins with a User-agent line identifying which crawler it addresses, with an asterisk meaning “all crawlers.” Below it, Disallow lines list paths the crawler should not request, and Allow lines can carve out exceptions within a disallowed area. An empty disallow, or none at all, means everything is permitted. The file can also point to the site’s sitemap, helping crawlers find the list of pages you do want discovered.
What it genuinely controls
Robots.txt manages crawling — which URLs a cooperative crawler will fetch. Its main practical use is to keep crawlers away from areas that waste their time or your server’s: endless faceted-search combinations, internal scripts, or duplicate paths that add no value. By steering crawlers toward what matters, it can help them spend their limited attention on your important pages.
What it does not do
Here is where the misconceptions live. Robots.txt is a request, not a lock — it relies on crawlers choosing to obey it, and not all do. It provides no security whatsoever; a disallowed path is still publicly reachable by anyone who knows the address, and listing sensitive paths there actually advertises them. Crucially, disallowing a page does not reliably keep it out of search results: if other sites link to it, it can still appear, just without a useful description. To keep a page out of search, the right tool is a noindex directive, not a disallow.
The mistake that blocks a whole site
The most damaging error is a stray rule that disallows everything — a single line forbidding the root path tells every crawler to stay out entirely. This sometimes survives from a staging site into production, quietly preventing a whole site from being crawled. Because the effect is invisible until traffic disappears, it is worth checking the live robots.txt whenever a site launches or moves.
Using it well
Treat robots.txt as traffic guidance, not a gate. Use it to keep crawlers out of low-value or wasteful areas and to advertise your sitemap. Do not use it to hide private data or to remove pages from search — those need proper access controls and noindex respectively. Kept simple and checked after any big change, it does its modest job reliably.