AI search and working with a developer
What is ClaudeBot?
ClaudeBot is Anthropic’s crawler for Claude. Claude-User is the companion agent that fetches a page live when someone asks Claude about a specific URL, and Claude-SearchBot supports search. All three are declared separately in robots.txt.
Also called: Anthropic crawler · Claude-User
What allowing it does
Makes your pages available to be retrieved and cited when someone asks Claude about your business, your services or a subject you have written about. For a services business that is a referral channel that did not exist five years ago.
How to verify access
Request your site with the ClaudeBot user-agent and read the status. A 403 means an edge proxy is refusing it — and because it happens above your account, no configuration you control will fix it. Changing host will.
What makes a page quotable once it is reachable
- A direct answer in the first 40 to 60 words under the heading
- Self-contained sentences that survive being lifted out of context
- Specific, checkable facts rather than atmosphere
- Structured formats — tables, ordered lists, question and answer pairs
Where this is covered in depth
A definition can only go so far. How to check whether AI crawlers can actually reach your site covers this properly — 3 minutes, free, no email required.
Who wrote this
Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan. These definitions come from client work rather than from a content brief — where an entry describes a mistake, it is usually one found on a real site. More about how I work.