AI search
How to check whether AI crawlers can actually reach your site
Run one curl loop with each AI crawler user-agent and read the status codes. Every line should be 200. A 403 means an edge proxy, WAF or CDN bot rule is refusing the crawler before your server or robots.txt is ever consulted — and your robots.txt cannot fix it.
The test
for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User \
PerplexityBot Applebot Amazonbot CCBot Googlebot Bingbot; do
printf "%-18s " "$ua"
curl -s -o /dev/null -w "%{http_code}\n" -A "$ua/1.0" https://YOUR-SITE.com/
doneOn Windows without curl, the same test runs in PowerShell with Invoke-WebRequest and a -UserAgent argument. The point is the user-agent header — a browser request tells you nothing, because browser requests are precisely the ones that are allowed.
Reading the result
| Code | Meaning | What to do |
|---|---|---|
| 200 | The crawler can fetch the page | Nothing. Check the HTML actually contains content. |
| 403 | Refused, almost always at an edge proxy or WAF | Fix the bot rule, or change host. robots.txt cannot override it. |
| 401 | Authentication required | Staging protection left on in production. |
| 429 | Rate limited | Bot rate rules too aggressive; whitelist the crawlers you want. |
| 503 | Temporarily unavailable | Under-provisioned host, or a security plugin throttling non-browser agents. |
| 302 to a challenge page | Bot management interstitial | The crawler receives a challenge, not your content. Same effect as a block. |
Why a 403 happens when your robots.txt says Allow
robots.txt is a file your server hands over after it decides to serve the request. Anything that refuses the request before that point never consults it. That includes shared-host edge proxies, WAF rules, CDN bot-management defaults and security plugins that treat every non-browser user-agent as hostile.
This is the trap, and it is invisible from a browser: the site works perfectly for you, your robots.txt is textbook, and seven AI crawlers have never once received a byte. I found exactly this on the free host this site launched on — openresty at the edge returning 403 to GPTBot, ClaudeBot, PerplexityBot, Applebot, Amazonbot, meta-externalagent and Bytespider.
Fixing it
- Shared or free hosting: usually not fixable — the rule is the provider’s, applied above your account. Moving host is the fix.
- Cloudflare: check the AI bot controls under Security, plus any custom WAF rules matching user-agent strings. "Block AI scrapers" is on by default in some plans.
- Security plugins (WordPress especially): most ship a bad-bot list that includes legitimate AI crawlers by default.
- Server config: look for user-agent matching in .htaccess, nginx conf, or ModSecurity rules inherited from a template.
- After any change, re-run the loop. Believe the status codes, not the settings page.
Passing the test is necessary, not sufficient
A 200 that returns an empty container is the same outcome as a 403, arrived at differently. After access is confirmed, check what the response body actually contains — fetch the page with curl and count the words. If your content only exists after JavaScript runs, an assistant reads almost nothing about you.
Questions people ask about this
- Which AI crawlers matter most?
- GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity, Google-Extended for Gemini and AI Overviews, and Applebot-Extended for Apple Intelligence. ChatGPT-User and Claude-User matter disproportionately, because they fire when a real person asks an assistant about your specific URL.
- Should I block Bytespider or other aggressive crawlers?
- Blocking on bandwidth grounds is a legitimate operational decision, and it is different from blocking the crawlers that feed answer engines your buyers use. Decide per crawler, not with one rule.
- Does allowing AI crawlers hurt my SEO?
- No. Google-Extended is separate from Googlebot and blocking it has no effect on Search ranking — it only removes you from Gemini grounding and AI Overview sourcing.
Who wrote this
Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan — fifteen live client sites, six of them stores taking real payments. The examples in these guides come from those builds and from the audit that rebuilt this site, not from a content brief. More about how I work, or read the case studies.