Reference
What is Log file analysis?
Log file analysis reads your server’s access logs to see exactly which URLs crawlers fetched, when they fetched them, and what status code each request received. It is the only direct evidence of crawler behaviour available to you — every other method, including Search Console, is inference from a summary.
Also called: server log analysis · crawl logs
What only logs can tell you
- Which URLs crawlers actually spend their time on, versus what you assumed
- Whether AI crawlers are reaching you at all, and what status they receive
- Response times as experienced by the crawler rather than by your browser
- Whether a redirect chain is being followed or abandoned
What to look for first
- Non-200 responses to legitimate crawlers, especially 403 and 503
- Crawl attention concentrated on parameter combinations nobody wants indexed
- Important pages that are rarely or never fetched
- Spikes that correlate with a deploy or a configuration change
The practical barrier
Many shared hosts make logs awkward to obtain, and some do not retain them. Where that is the case, requesting your site with each crawler user-agent and reading the status codes gives you most of the same answer in two minutes.
Where this is covered in depth
A definition can only go so far. Technical SEO, in the order the problems actually block you covers this properly — 4 minutes, free, no email required.
Who wrote this
Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan. These definitions come from client work rather than from a content brief — where an entry describes a mistake, it is usually one found on a real site. More about how I work.