robots.txt in 2026, including every AI crawler worth naming

The distinction that causes most of the damage

robots.txt is a crawl directive. It says "do not fetch this". It says nothing about whether a URL may appear in search results, and a disallowed URL with inbound links can and does appear as a bare result with no description.

To remove a page from the index you need a noindex meta tag or header — and a crawler must be allowed to fetch the page to see it. Disallowing and expecting deindexing is a contradiction that leaves the URL stranded in both directions.

The groups worth declaring in 2026

Under RFC 9309 a crawler obeys only its single most specific matching group. Explicit groups therefore act as insurance: a later Disallow added under User-agent: * cannot silently apply to a crawler you meant to keep open.

For a business selling services, allowing all of these is almost always correct. The buyer who asks an assistant "who builds Stripe checkouts for UK retailers" is a buyer with intent, and being absent from that answer is not a privacy win.

The four mistakes I find most often

1. Disallow: / left in place after launch, copied from staging. It is a two-character difference from working, and it takes a site off search entirely.
2. Blocking /assets/ or /js/, which prevents rendering and makes Google index a broken layout.
3. Using robots.txt to hide private URLs. The file is public, so it is a directory of exactly what you wanted hidden. Use authentication.
4. Assuming robots.txt is the last word. Edge proxies, WAFs and CDN bot rules all run before it, and they routinely block crawlers the file explicitly allows.

Verify the file is even reachable as a bot

Fetch robots.txt with each crawler user-agent, not just in a browser. If the host returns 403 to GPTBot at the edge, your carefully written Allow directives are never read — the crawler is refused before Apache or nginx sees the request.

That is the exact situation this site launched into: a robots.txt that welcomed every AI crawler, on a host whose edge proxy refused seven of them. The file was perfect. It was also never delivered.

A sane baseline

User-agent: *
Allow: /
Disallow: /thank-you/
Disallow: /*.zip$

# Named groups: insurance against a future Disallow under *
User-agent: Googlebot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap-index.xml

Questions people ask about this

Should I block AI crawlers?

For a publisher whose revenue depends on people arriving to read the article, there is a real argument. For a business selling services, blocking them removes you from the answers your buyers are increasingly asking for, and protects content you published in order to be read.

Does robots.txt stop a page appearing in Google?

No. It stops the crawl. The URL can still be listed without a description, sourced from links. Deindexing requires noindex — and therefore requires the crawl you just blocked.

Do I need a separate llms.txt?

It is not a standard any major crawler currently obeys, and it costs almost nothing to publish. Treat it as a well-structured summary for agents that do read it, not as an access control.

Who wrote this

Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan — fifteen live client sites, six of them stores taking real payments. The examples in these guides come from those builds and from the audit that rebuilt this site, not from a content brief. More about how I work, or read the case studies.