Tornado SEO

Crawl budget

By Thibault · Updated

Definition

Crawl budget is the set of URLs Google can and wants to crawl on a site. It depends on two things: how much load your server can take, called the crawl capacity limit, and how much Google wants your pages, called crawl demand.

Who should care about crawl budget?

Crawl budget mostly matters for large sites and for sites that change often. Google names three cases: sites with more than 1 million unique pages that change about once a week, sites with more than 10000 unique pages that change daily, and sites with many URLs marked "Discovered - currently not indexed" in Search Console. Google says these are rough estimates and not exact thresholds. If your site has a few hundred pages, Google will crawl them without trouble, and your effort is better spent elsewhere.

How does crawl budget work?

Crawl budget comes from two forces: crawl capacity and crawl demand. Google treats each hostname as a separate site with its own budget.

  1. Crawl capacity limit: how many parallel connections and how much server time Google can use without overloading you. It goes up when your server answers fast and consistently. It goes down with slow responses, 5xx errors or HTTP 429 rate limiting.
  2. Crawl demand: how much Google wants your URLs. It depends on perceived inventory (every URL Google knows about), popularity (popular URLs get crawled more often) and staleness (Google comes back to catch changes).

The capacity limit is shared across all Google crawlers. Heavy demand from one crawler can leave less room for the others.

What wastes crawl budget?

Crawl budget is wasted on URLs that should never have been crawled in the first place. Take an online store with 20000 products. Filters for color, size and sort order can create hundreds of thousands of near duplicate URLs. Googlebot spends its time on those combinations while new product pages wait. Soft 404s, long redirect chains and duplicate content do the same damage.

How do you optimize crawl budget?

You optimize crawl budget by removing waste and making your server faster. These are the steps Google recommends:

  1. Consolidate duplicate content so Google crawls unique pages.
  2. Block URLs you never want crawled, like faceted filters or sort orders, with robots.txt.
  3. Return 404 or 410 for pages removed for good, and fix soft 404s.
  4. Keep your sitemaps up to date, with a lastmod date on updated pages.
  5. Avoid long redirect chains.
  6. Speed up your server and support HTTP caching, so unchanged pages can return a 304.

Do not use noindex to control crawling. Google still has to fetch the page to see the tag, so the crawl is already spent.

Thibault's take I always start with accessibility before anything else. Open the Crawl Stats report in Search Console and look at where Googlebot spends its requests. On almost every large site I audit, a big share goes to parameters, filters and redirects nobody cares about. And check your JavaScript: if your links only exist after rendering, Google discovers your pages slower. Fix the waste first, then think about getting more budget.

Frequently asked questions

Does crawl budget matter for a small site?

Usually not. Google says its crawl budget guide is meant for large sites or sites that change very often. If your pages get indexed shortly after you publish them, you do not have a crawl budget problem.

How can I increase my crawl budget?

Improve your server first if capacity is the bottleneck, for example when URL Inspection shows "Hostload exceeded". Then raise demand with content that is unique, useful and popular. Google crawls more of what it values.

Does blocking pages in robots.txt give the budget to my other pages?

Not automatically. Google says freed budget is only reused if it was already hitting your capacity limit. Use robots.txt for URLs you never want crawled, not as a temporary trick.

Sources

  1. developers.google.com/crawling/docs/crawl-budget

Stay ahead of every search update

Google, ChatGPT, Gemini, Claude: what changed, why it matters, and what to do about it. Every morning, in 3 minutes, free.

What the...?