Skip to content

Crawl budget

Crawl budget is how much of your site Google will fetch in a given period. Which sites it affects, what wastes it, and when it is not your problem.

Definition

Crawl budget is the number of URLs Googlebot will fetch from a site in a given period. Google describes it as the combination of how much a site can handle without slowing down and how much demand there is to crawl it. It matters mainly for large or frequently changing sites.

Illustration representing crawl budget

Why it matters

Crawl budget is the most over-applied concept in technical SEO. It is a real constraint on large sites and almost never the limiting factor on small ones.

Google's own guidance on managing crawl budget is explicit about who should care: sites with more than a million unique pages with weekly content changes, or more than ten thousand with daily changes. Below that, Google states that crawl budget management is generally unnecessary, and that a site's pages being slow to index usually has a different cause.

Reading that before spending a week on it is the single most useful thing in this entry.

How it works

Google describes two components.

Crawl capacity limit. How fast Googlebot can fetch without degrading the site. If the server answers quickly, the limit rises. If responses slow down or return errors, it falls, because the crawler is designed not to take a site down.

Crawl demand. As Google's documentation on how Search works describes, how much Google wants to crawl, driven by how popular the URLs are and how often they genuinely change. A page that has not changed in two years is refetched rarely, which is correct behaviour rather than neglect.

Budget is spent per URL fetched, which is why duplication is expensive and why it is one of the first things a technical SEO review quantifies. Every parameter variation, every faceted navigation combination and every session identifier is a separate URL, and a crawler that spends its fetches on those is not spending them on your new pages.

Worked example

Where it is actually worth acting, the targets are predictable:

  • Infinite filter combinationsdisallow in robots.txt
  • Internal search result pagesdisallow in robots.txt
  • Calendars generating a URL a daydisallow in robots.txt
  • Parameter duplicatescanonical tag to the clean URL
  • Long redirect chainscollapse to one hop
  • 404s linked from the sitefix the links
  • Slow server responsesthe capacity limit follows this
A conveyor of blocks passing over four side-chutes, three of them shut by shutters, into a collecting bin
A finite number of fetches. Closing the chutes that lead nowhere is what keeps them on the main line.

Note what is not on the list: deleting pages to "save budget". Removing content that serves readers in order to influence a crawler is a trade that almost never pays.

Common mistakes

  • Applying it to a small site.
    Under ten thousand pages, indexing problems are usually about quality, duplication or internal linking, not crawl budget.
  • Blocking pages that carry a canonical.
    A blocked URL is never read, so its canonical tag never takes effect.
  • Assuming crawled means indexed.
    They are separate steps. A page can be fetched and still not be selected for the index.
  • Chasing crawl stats with no indexing problem.
    The report is interesting. It is only actionable when pages you want indexed are not being fetched.
  • Letting the server get slow.
    Capacity is responsive to performance, so page speed work raises the ceiling as a side effect.

How to check

The crawl stats report in Search Console shows requests over time, response codes and average response time. The useful question is narrow: are the URLs you care about being fetched, and is the crawler spending most of its requests on URLs you would rather it ignored. If the answer to both is reassuring, this is not your constraint, and the SEO audit should move on to what is.

Let's build

Got something to build?

Tell us what you're making. We'll tell you - honestly - if we're the right team, and how we'd do it.