Should I block AI crawlers?

It depends on how you make money. A publisher whose revenue depends on people arriving to read has a genuine argument for blocking. A business selling services does not — blocking removes you from the answer at the moment a buyer is asking who to hire.

How do I get cited by AI search engines?

Three prerequisites in strict order: AI crawlers must be able to fetch you, your content must be in the HTML without JavaScript, and independent sources must corroborate that you exist. Fail the first and nothing else counts.

What should be in my robots.txt?

Allow everything you want crawled, disallow only genuinely wasteful paths such as internal search results, reference your sitemap, and name the AI crawlers explicitly. Remember it controls crawling, not indexing — disallowing a page does not remove it from search.

AI search

Is my site visible to ChatGPT and other AI assistants?

Test it rather than assume. Fetch your home page with each AI crawler user-agent and read the status code — every line should be 200. A 403 means an edge proxy is refusing them before your server or robots.txt is ever consulted.

Answered by Anas Bin Masud 160 words 3 sections

The test

Paste into any terminal with curl

for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot PerplexityBot Applebot CCBot; do
  printf "%-16s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua/1.0" https://YOUR-SITE.com/
done

What the codes mean

200 — reachable. Now check the HTML actually contains content
403 — refused at an edge proxy or WAF. robots.txt cannot override it
401 — staging protection left on in production
429 — rate limiting too aggressive for legitimate crawlers

Why a custom domain does not fix a 403

These rules match the user-agent, not the hostname. Pointing a new domain at the same host produces the same refusal, so it is worth testing before spending money expecting it to help.

The longer version

This answer is deliberately short. If you want the full treatment, How to check whether AI crawlers can actually reach your site covers it in about 3 minutes — free, ungated, written from client work.

Who answered this

Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan — fifteen live client sites, six of them stores taking real payments. These answers come from those projects rather than from a content brief. More about how I work, or ask me something directly.