Is my site visible to ChatGPT and other AI assistants?

Test it rather than assume. Fetch your home page with each AI crawler user-agent and read the status code — every line should be 200. A 403 means an edge proxy is refusing them before your server or robots.txt is ever consulted.

How do I get cited by AI search engines?

Three prerequisites in strict order: AI crawlers must be able to fetch you, your content must be in the HTML without JavaScript, and independent sources must corroborate that you exist. Fail the first and nothing else counts.

Do I need an llms.txt file?

No major AI crawler is known to require or obey llms.txt, so it is not a mechanism. It costs almost nothing if generated from data you already maintain, it is genuinely useful to humans evaluating you, and it cannot hurt — publish one and expect nothing.

AI search

Should I block AI crawlers?

It depends on how you make money. A publisher whose revenue depends on people arriving to read has a genuine argument for blocking. A business selling services does not — blocking removes you from the answer at the moment a buyer is asking who to hire.

Answered by Anas Bin Masud 167 words 3 sections

The decision by business model

You are	Reasonable position
A services business	Allow. Being absent from the answer is not a privacy win
An e-commerce store	Allow. Product discovery increasingly runs through assistants
A publisher monetised by ads	A genuine trade-off worth thinking about
Holding proprietary research	Consider blocking training crawlers, allowing search ones

The distinction most people miss

The crawlers do different jobs. GPTBot builds a training corpus; OAI-SearchBot builds a search index; ChatGPT-User fetches live when a real person asks about your URL. Blocking the third is blocking a buyer with intent.

The Google-Extended note

Google-Extended controls Gemini grounding and AI Overview sourcing, and is separate from Googlebot. Blocking it removes you from those answers without improving Search ranking.

The longer version

This answer is deliberately short. If you want the full treatment, Generative Engine Optimization: being the source an AI answer quotes covers it in about 4 minutes — free, ungated, written from client work.

Who answered this

Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan — fifteen live client sites, six of them stores taking real payments. These answers come from those projects rather than from a content brief. More about how I work, or ask me something directly.