Search and SEO
What is robots.txt?
robots.txt is a plain-text file at the root of a domain that tells crawlers which paths they may fetch. It controls crawling, not indexing, and it is a request rather than an enforcement mechanism — well-behaved crawlers obey it and hostile ones ignore it entirely.
Also called: robots file · robots exclusion protocol
What it can and cannot do
It can stop a crawler fetching a URL. It cannot remove a URL from search results — a disallowed page with inbound links can still be listed without a description, and blocking it prevents the crawl that would have read your noindex.
It also cannot protect anything. The file is public, so a list of disallowed paths is a directory of exactly what you wanted hidden. Use authentication for that.
What belongs in it
- A Sitemap: directive pointing at your sitemap index
- Disallow rules for genuinely crawl-wasteful paths — internal search, infinite parameter combinations
- Named groups for AI crawlers, which are increasingly the consumers that matter
- Nothing that also carries a noindex tag
The trap almost nobody checks
robots.txt is served by your server. Anything that refuses a request before your server sees it — an edge proxy, a WAF, a CDN bot rule — never consults it. Shared hosts routinely return 403 to AI crawler user-agents while the robots.txt sitting behind them says Allow: / to everyone.
The only way to know is to fetch your site with each crawler user-agent and read the status codes.
Where this is covered in depth
A definition can only go so far. robots.txt in 2026, including every AI crawler worth naming covers this properly — 3 minutes, free, no email required.
Who wrote this
Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan. These definitions come from client work rather than from a content brief — where an entry describes a mistake, it is usually one found on a real site. More about how I work.