Search and SEO
What is Crawl budget?
Crawl budget is the number of URLs a search engine is willing and able to fetch from your site in a given period. It is a product of how fast your server responds and how much demand the engine has for your content, and it only becomes a real constraint on large sites.
Also called: crawl rate · crawl demand
When it actually matters
Below roughly ten thousand URLs, crawl budget is almost never your problem — a page that is not indexed is usually not indexed for quality or linking reasons rather than because nobody fetched it.
It becomes real on large e-commerce sites, where faceted navigation can generate hundreds of thousands of crawlable combinations that nobody searched for.
What wastes it
- Faceted navigation left fully crawlable
- Internal search result pages, which are infinite by definition
- Session identifiers and tracking parameters creating unique URLs
- Long redirect chains, each hop costing a fetch
- Slow server responses, which directly reduce the rate a crawler is willing to use
The honest diagnosis
Check server logs for what is actually being fetched before assuming a budget problem. Most sites discover the crawler is spending its time on parameter combinations nobody wants, which is a linking and canonical problem rather than a budget one.
Where this is covered in depth
A definition can only go so far. Technical SEO, in the order the problems actually block you covers this properly — 4 minutes, free, no email required.
Who wrote this
Anas Bin Masud builds e-commerce sites and does technical SEO for businesses in the UK, Canada and Pakistan. These definitions come from client work rather than from a content brief — where an entry describes a mistake, it is usually one found on a real site. More about how I work.