Technical SEO
How to write a robots.txt file
Put it at the root of the domain, allow everything you want crawled, disallow only paths that genuinely should not be requested, and add a Sitemap line. Never use it to hide a page from search — that is what noindex is for, and the two do different jobs.
Before you start
- Access to the site root
- A list of paths that should not be crawled
The steps
-
Step 1: Create the file at the root
It must be at /robots.txt exactly. A file anywhere else is ignored entirely.
-
Step 2: Start permissive
User-agent: * followed by Disallow: with nothing after it allows everything. Add restrictions deliberately.
-
Step 3: Disallow only what should not be requested
Admin paths, internal search results, cart URLs. Not pages you want kept out of the index.
-
Step 4: Never disallow a page you want deindexed
A blocked page cannot be fetched, so its noindex tag is never read, and it can stay indexed indefinitely.
-
Step 5: Address AI crawlers explicitly
Named groups for GPTBot, ClaudeBot, PerplexityBot and the rest, so your position is stated rather than inferred.
-
Step 6: Add the Sitemap line
An absolute URL, at the end. It applies to every user-agent regardless of grouping.
-
Step 7: Test it
Fetch /robots.txt yourself, and use Search Console’s tester against specific URLs.
The distinction that causes the most damage
robots.txt controls crawling; noindex controls indexing. Disallowing a URL you want removed prevents the crawler seeing the noindex, so the page can remain in the index — sometimes with the description replaced by an apology that no information is available.
When robots.txt cannot help you
When the block is upstream. If a host returns 403 to a crawler at its edge proxy, robots.txt is never delivered to that crawler at all — so what it says is irrelevant. Test with the actual user-agents before assuming the file is doing anything.
The background behind this
robots.txt in 2026, including every AI crawler worth naming covers the reasoning in about 3 minutes — free, ungated, written from client work.
Who wrote this
Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada, Ireland and Pakistan. These procedures are the ones I run on client work, written down rather than invented — including the ones where the honest answer is that the fix is not available on your current hosting.