In short
The crawl budget refers to the amount of resources Googlebot spends on crawling a website, i.e. how many URLs it fetches in a given period and how often it comes back. The term comes from Google's own documentation and is made up of two factors: the crawl capacity limit that the website can handle technically and the crawl demand that Google has for the content.
For small websites, the crawl budget is practically never a problem; for large online shops with many thousands of URLs, it can decide how quickly new products, price changes or a complete relaunch arrive in the index.
For a long time "crawl budget" was a vague industry term, until Google officially defined it in 2017 in a blog post by Gary Illyes ("What Crawl Budget Means for Googlebot"). Today the concept is an established part of Google's documentation for site owners, including a dedicated guide to managing the crawl budget of large sites.
The two components of the crawl budget
Google describes the crawl budget as the interplay of two variables that should be understood separately, because they can be influenced via different adjustment screws.
Crawl capacity limit: what the server can handle
Googlebot wants to crawl websites without overloading them. It therefore observes how quickly and stably a server responds and continuously adjusts its fetch rate. If the website responds quickly, Google increases the number of parallel connections; if server errors (5xx status codes) pile up or response times rise, the bot throttles back. The capacity limit is therefore not a fixed quota but a dynamic control loop. Practical consequence: a fast, stable server is not only a question of user experience but directly enlarges the technical crawl window. Conversely, weak hosting can slow down the crawling of a large website for weeks without an error message lighting up anywhere.
Crawl demand: what Google wants to fetch at all
The second component is the demand side: how important and how changeable does Google consider a website's URLs to be? Popular, well-linked pages are crawled more often; so are pages that change frequently. A product detail page that has been unchanged for two years and hardly receives any internal links is visited correspondingly rarely by the bot. Crawl demand also explains why crawling suddenly increases massively after a migration: thousands of redirects and new URLs create a processing backlog that Google wants to work through. The separation of crawling and indexing is important: being crawled is the prerequisite but no guarantee that a page will be included in the index or rank.
When crawl budget is really relevant
The uncomfortable truth first: for most websites, crawl budget is not an issue. In its documentation, Google itself draws the line roughly like this: the budget question becomes relevant for websites with around a million unique URLs or more whose content changes weekly, or with around ten thousand URLs or more whose content changes daily. A corporate blog with 300 articles is crawled completely and regularly, however untidy it is. Anyone doing "crawl budget optimisation" there is polishing an adjustment screw that moves nothing.
Online shops, however, slip over this threshold faster than the pure number of products would suggest. Because the number of URLs of a shop is not the number of its products; it is the number of reachable URL variants. And that explodes through technology that makes complete sense from the user's point of view:
- Faceted navigation: every combination of filters for colour, size, brand and price creates its own URL. 200 categories can quickly turn into hundreds of thousands of crawlable addresses.
- Parameter URLs: sort orders (
?sort=price), tracking parameters, session IDs and pagination multiply every page. - Internal search: indexable search results pages are a bottomless pit; every conceivable search entry is a URL.
- Duplicates: without a clean canonical concept, products that can be reached via several category paths exist several times.
- Redirect chains and error pages: every redirect hop and every crawled 404 page uses up a fetch that is then missing for a real page.
It is exactly this mechanism that makes the crawl budget an e-commerce topic: the bot spends its time on filter variants and sorting parameters while the new product line waits days for its first crawl.
Measuring and optimising crawl budget
Measuring: crawl stats and log files
The first place to look is the "Crawl stats" report in Google Search Console (under Settings). For the last 90 days, it shows how many requests Googlebot made, how response times are developing, which status codes it received and how the fetches are distributed across file types and purposes (refresh vs new URLs). Conspicuous patterns, such as a high proportion of 404 or redirect responses, are visible here at a glance.
Anyone who needs more precision cannot avoid a log file analysis. The server logs show every single bot request: which URLs Googlebot actually fetches, how often, and which areas it ignores. Tools such as the Screaming Frog Log File Analyser combine this data with a crawl of the website and answer the decisive question: does what Google crawls match what is supposed to rank? In practice the result is often sobering: it is not unusual for a double-digit percentage of bot requests to go to parameter URLs that should never rank.
Optimising: the effective levers
Optimisation follows a simple logic: offer the bot less rubbish and make the important pages easier to reach.
| Measure | Effect |
|---|---|
| Block filter and parameter URLs via robots.txt | Prevents the crawling of infinite URL spaces at the source |
| Clean canonical concept for duplicates | Bundles signals, reduces redundant crawling |
| Shorten redirect chains to a direct target | Saves one or more hops per request |
| Well-maintained XML sitemaps with correct lastmod data | Directs the bot specifically to new and changed content |
| 404/410 for permanently removed content | Google quickly learns not to visit these URLs any more |
| Improve server performance and response times | Raises the capacity limit, so the bot crawls more per day |
| Flat information architecture, strong internal linking | Increases the crawl demand of the important pages |
One common misconception needs to be cleared up: noindex does not save crawl budget. A page with noindex has to be crawled for Google to see the instruction at all. Anyone who wants to prevent crawling needs robots.txt; anyone who wants to prevent indexing needs noindex. The two tools solve different problems and even conflict when combined: a page blocked via robots.txt can no longer deliver its noindex.
A real-world example: faceted navigation in a fashion shop
The textbook example of wasted crawl budget is the fashion shop with faceted navigation. 50,000 products, filters for brand, colour, size, material and price range, combinable and each reachable as a separate URL. Mathematically, this results in tens of millions of URL combinations. Google's own guide to faceted navigation describes exactly this case and recommends consistently excluding filter URLs that are not meant to rank from crawling (robots.txt or URL fragments) and only developing curated, sought-after filter pages, such as "red trainers", into indexable landing pages. The effect of such a clean-up usually becomes visible in the crawl stats within weeks: bot requests shift from parameter variants to product and category pages, and new content appears in the index faster.
Crawl budget during relaunch and migration
The crawl budget plays a special role when changing platforms. After go-live, Google has to recrawl the entire old URL inventory to discover the 301 redirects and at the same time take in the new inventory; crawl demand effectively doubles for a few weeks. Every legacy issue now has a double effect: redirect chains from earlier relaunches, forgotten parameter URLs and slow servers prolong the transition phase in which rankings fluctuate. Anyone who tidies up before the migration (straightening out chains, cleaning up the URL inventory, checking server performance) measurably shortens the time until the move has been processed. That is also why the temporary sitemap with the old URLs is a standard tool: it directs the available budget specifically to the URLs whose redirects Google has to find.
Outlook: crawl budget in times of AI crawlers
The concept comes from a world in which essentially one bot counted: Googlebot. That world no longer exists in that form. Alongside the classic search engine crawlers, AI crawlers such as GPTBot (OpenAI), ClaudeBot (Anthropic) or PerplexityBot now fetch websites to a considerable extent, partly for training language models and partly for answering user questions live. For the server, this is simply additional load that counts against the same capacity limit: hosting that is already struggling with Googlebot gets a measurably bigger problem from AI traffic.
This new generation of bots is also controlled via robots.txt, in which crawlers can be allowed or blocked individually by user agent. The decision has long been strategic, not just technical: anyone who blocks AI crawlers completely protects server resources and content but forgoes appearing as a source in AI answers. For shops that want visibility in ChatGPT, Perplexity and co., the same logic therefore applies as with the classic crawl budget: fast servers, clean URL structures and no infinite parameter spaces in which an AI bot can get lost too. The homework has stayed the same; only the number of guests who benefit from it has grown.
Frequently asked questions about crawl budget
Is crawl budget a ranking factor?
No. The crawl budget influences whether and how quickly pages can be crawled and thus indexed, not how they are assessed. A page that is not crawled for weeks because of wasted budget cannot rank either, of course. The effect is indirect, but for large shops it is still commercially relevant.
From what size do I need to worry about my crawl budget?
Google's rule of thumb: from around a million URLs with weekly changes or from around ten thousand URLs with daily changes. Online shops should count in reachable URL variants, not in products; with faceted navigation, even a medium-sized shop quickly exceeds these thresholds.
Does noindex save crawl budget?
No. The noindex instruction is in the HTML or HTTP header of the page, so Google has to crawl the page in order to read it. Only robots.txt prevents crawling. In the long term, Google does crawl pages permanently set to noindex less often, but this is not suitable as a saving instrument.
Can I speed up crawling?
Little can be requested directly: the URL inspection in Search Console triggers individual URLs, and sitemaps with current lastmod values signal changes in the inventory. Only the combination of a fast server, a clean URL structure and strong internal linking works sustainably, i.e. a higher capacity limit and clear demand.
How do I recognise that crawl budget is a problem for me?
Typical symptoms: new products or content take many days to be indexed, the "Crawl stats" report shows high proportions of parameter URLs, redirects or 404s, and in the indexing overview the group "Crawled - currently not indexed" keeps growing. The suspicion can only be confirmed cleanly by a log file analysis.