Glossary
What is Crawl budget?
Crawl budget is how many pages a search engine will crawl on your site in a given time. It only becomes a problem on large sites.
Updated
Search engines can't crawl every page of every site all the time. Crawl budget is the amount of crawling a search engine is willing to spend on your site: roughly a mix of how fast your server responds and how much it wants to crawl.
Who needs to care
Mostly large sites, with hundreds of thousands of pages, or sites that generate many URLs automatically. For a typical site, crawling isn't the bottleneck; content quality is.
What wastes it
- Endless URL variations from filters and tracking parameters.
- Duplicate pages without a canonical URL.
- Long redirect chains and error pages.
- Slow servers.
- Low-value pages that crowd out the ones that matter.
What helps
- A clean sitemap with only indexable pages.
- Good internal linking so important pages are close to the home page.
- Fast, reliable hosting.
- Telling search engines about new pages (IndexNow, Search Console).
Related
- IndexingIndexing is when a search engine stores a page in its database so it can appear in results. A page that isn't indexed can't rank.
- SitemapA sitemap is a file that lists the pages of a site you want search engines to find, often with the date each last changed.
- robots.txtrobots.txt is a text file at the root of a site that tells crawlers which paths they may or may not fetch. It controls crawling, not indexing.