Technical
noindex vs robots.txt disallow
noindex removes a page from the index; robots.txt disallow prevents it being crawled. They are not interchangeable, and using both on the same URL is self-defeating — the disallow blocks the crawl that would read the noindex.
Side by side
| Compared on | noindex | robots.txt disallow |
|---|---|---|
| Removes from search results | Yes | No |
| Prevents crawling | No | Yes |
| Saves crawl budget | No | Yes |
| Page can still be listed | No | Yes — as a bare URL with no description |
| Requires the page to be fetchable | Yes | No |
When noindex is the right choice
- Thank-you pages and post-conversion destinations
- Internal search results, which are infinite
- Thin filtered views you want removed from the index
- Anything you want gone from search but still reachable by users
When robots.txt disallow is the right choice
- Paths with no value that waste crawl attention at scale
- Infinite parameter spaces on a large site
- Server-side scripts and endpoints that should never be fetched
The order that matters
To remove pages already in the index: noindex first, allow crawling so it is read, wait for them to drop out, and only then consider a disallow to save crawl attention. Doing it the other way strands the URLs in both directions.
What robots.txt cannot do
It cannot protect anything — the file is public, so a list of disallowed paths is a directory of exactly what you wanted hidden. Use authentication for that.
The combination that strands a URL
noindex plus disallow on the same path. The crawler is told not to fetch the page, so it never reads the instruction to remove it, and the URL can sit in results indefinitely as a bare link with no description. Pick one.
The verdict
noindex to control the index, robots.txt to control the crawl. If you want a page gone from search, the answer is almost always noindex with crawling allowed.
If you want the full treatment
robots.txt in 2026, including every AI crawler worth naming covers this in about 3 minutes — free, ungated, written from client work.
Who wrote this comparison
Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan. Where a comparison involves something I sell, the page says so — and every comparison here has to name at least two situations where each side wins, because a comparison one option always wins is an advert with a table in it.