Do I need an XML sitemap?

Yes, and it should be generated rather than hand-written. A sitemap helps engines discover URLs they might otherwise miss and signals when pages genuinely changed. It cannot make a thin page rank, and it will not rescue a page nothing links to.

Should I block AI crawlers?

It depends on how you make money. A publisher whose revenue depends on people arriving to read has a genuine argument for blocking. A business selling services does not — blocking removes you from the answer at the moment a buyer is asking who to hire.

Is my site visible to ChatGPT and other AI assistants?

Test it rather than assume. Fetch your home page with each AI crawler user-agent and read the status code — every line should be 200. A 403 means an edge proxy is refusing them before your server or robots.txt is ever consulted.

SEO

What should be in my robots.txt?

Allow everything you want crawled, disallow only genuinely wasteful paths such as internal search results, reference your sitemap, and name the AI crawlers explicitly. Remember it controls crawling, not indexing — disallowing a page does not remove it from search.

Answered by Anas Bin Masud 185 words 3 sections

A sane baseline

robots.txt for a business site that wants to be found

User-agent: *
Allow: /
Disallow: /api/
Disallow: /cgi-bin/

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://example.com/sitemap-index.xml

The four mistakes worth avoiding

Disallow: / left in from staging — two characters from working, and it removes the site from search
Blocking /assets/ or /js/, which prevents rendering and makes Google index a broken layout
Using it to hide private URLs. The file is public, so it is a directory of what you wanted hidden
Disallowing a page that also has noindex, which blocks the crawl that would read the noindex

The check almost nobody runs

Fetch robots.txt with each crawler user-agent rather than in a browser. Some hosts return 403 to AI crawlers at their edge, so the file is never delivered to the crawlers it addresses.

The longer version

This answer is deliberately short. If you want the full treatment, robots.txt in 2026, including every AI crawler worth naming covers it in about 3 minutes — free, ungated, written from client work.

Who answered this

Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan — fifteen live client sites, six of them stores taking real payments. These answers come from those projects rather than from a content brief. More about how I work, or ask me something directly.