Search and SEO
What is Noindex?
Noindex is an instruction telling search engines not to include a page in their index. It can be set as a meta tag in the HTML head or as an X-Robots-Tag HTTP header. Unlike a robots.txt disallow, it is a genuine directive and it is obeyed.
Also called: noindex meta tag · X-Robots-Tag noindex
Why it matters
Some pages should exist for users but not compete in search: thank-you pages, internal search results, staging environments, thin filtered views. Noindex is how you say so without hiding the page from the people who need it.
The critical interaction with robots.txt
A crawler must be allowed to fetch a page in order to read the noindex on it. If you disallow a URL in robots.txt and also set noindex, the crawler never sees the noindex — and the URL can remain in results as a bare link with no description, indefinitely.
This is the single most common self-inflicted indexing problem there is. Allow the crawl; use noindex to control the index.
How to use it
- Meta tag for HTML pages, X-Robots-Tag header for PDFs and other non-HTML files
- Combine with nofollow only when you also want the links on the page ignored, which is rarer than people assume
- Remove noindex from a page before expecting it to rank — and then be patient, because recovery is not instant
- Never leave staging noindex rules in a production deployment. It is a two-character difference from working
Where this is covered in depth
A definition can only go so far. robots.txt in 2026, including every AI crawler worth naming covers this properly — 3 minutes, free, no email required.
Who wrote this
Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan. These definitions come from client work rather than from a content brief — where an entry describes a mistake, it is usually one found on a real site. More about how I work.