AI search
llms.txt: a fair assessment of a file nobody is obliged to read
llms.txt is a proposed plain-text file summarising a site for language models. No major AI crawler is known to require or obey it, so treat it as a low-cost, well-structured summary rather than a mechanism. It costs twenty minutes and it cannot hurt you.
What it is, precisely
llms.txt is a community proposal for a Markdown file at the root of a site that gives language models a curated summary and a map of the important pages — the idea being that a model working with a limited context window benefits from a condensed, authoritative version rather than crawling everything.
It is not a standard in the sense robots.txt is. There is no RFC, no formal adoption commitment from OpenAI, Anthropic, Google or Perplexity, and no way to verify a model consulted it. Anyone claiming it is required is overselling.
Why publish one anyway
- It costs almost nothing, especially if it is generated from data you already maintain
- It is a genuinely useful artefact for humans evaluating you — several people have told me they read it first
- It forces you to write a clean, factual summary of what you sell and at what price, which improves the rest of the site
- If adoption grows, you are already there; if it does not, you have lost twenty minutes
What to put in it
The most common mistake is writing a CV. A model does not need your career history — it needs the facts a buyer would ask for, in extractable form.
- One-line description of what the business does and for whom
- Services, each with a URL, a price floor and a typical timeline
- Pricing and commercial terms — deposit, payment methods, what is included
- Proof: live client work with URLs
- Practical facts: location, markets served, working hours, response time, languages
- Common questions with direct answers
- Contact routes
- A note that the site is static HTML and needs no JavaScript execution to read
Generate it, do not maintain it
A hand-written llms.txt drifts from the site within a month, and a confidently wrong file is worse than no file — it publishes stale prices and dead links in the format most likely to be quoted verbatim.
Generate it from the same data that builds your pages. On this site llms.txt, robots.txt, humans.txt and the RSS feed are all produced by one script from one data source, so they cannot disagree with the pages or with each other.
What it will not do
It will not grant access. If your host returns 403 to ClaudeBot, llms.txt is refused along with everything else — the file that explains your site is behind the same wall as the site.
It will not substitute for content. A summary pointing at thin pages is a summary of thin pages, and no amount of structure fixes that.
Questions people ask about this
- Do ChatGPT or Claude actually read llms.txt?
- There is no public commitment from any major provider that they require or prioritise it. Treat any claim otherwise as marketing until a provider documents it.
- Where does it go?
- At the site root — /llms.txt — as plain text or Markdown. Some implementations also publish /llms-full.txt with the complete content inlined.
- Is it a replacement for robots.txt?
- No. robots.txt is an access directive that crawlers do obey. llms.txt is a summary with no access semantics at all. Ship both; they do different jobs.
Who wrote this
Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan — fifteen live client sites, six of them stores taking real payments. The examples in these guides come from those builds and from the audit that rebuilt this site, not from a content brief. More about how I work, or read the case studies.