How to check if AI crawlers can reach your site

Before you start

• A terminal with curl, or any HTTP client that lets you set a user-agent

The steps

1. 1

   Step 1: Open a terminal

   Any terminal with curl available. On Windows, Git Bash or PowerShell both work.

2. 2

   Step 2: Run the loop

   Paste the command below, replacing the URL with your own domain.

3. 3

   Step 3: Read every status code

   You are looking for 200 on every line. Note any that differ.

4. 4

   Step 4: Interpret the failures

   A 403 means an edge denylist. A 401 means staging protection left on. A 429 means rate limiting. A redirect to a challenge page is a block wearing a disguise.

5. 5

   Step 5: Test robots.txt separately

   Fetch /robots.txt with the same user-agents. If that is also 403, the file addressing those crawlers is never delivered to them.

6. 6

   Step 6: Decide what to do

   On shared hosting the rule is usually the provider’s and not fixable from your account. Moving host is the fix; editing robots.txt is not.

The test, in one command

for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User \
          PerplexityBot Applebot Amazonbot CCBot Googlebot Bingbot; do
  printf "%-18s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua/1.0" https://YOUR-SITE.com/
done

Why a browser test proves nothing

The block is applied by user-agent. Your browser sends a browser user-agent, which is precisely the one that is allowed, so the site works perfectly for you while being invisible to the crawlers that matter.

What a 200 does not tell you

That the response contains anything. After confirming access, fetch the page and count the words — if your content only exists after JavaScript runs, an assistant reads almost nothing about your business regardless of the status code.

The background behind this

How to check whether AI crawlers can actually reach your site covers the reasoning in about 3 minutes — free, ungated, written from client work.

Who wrote this

Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada, Ireland and Pakistan. These procedures are the ones I run on client work, written down rather than invented — including the ones where the honest answer is that the fix is not available on your current hosting.