Googlebot and Crawl Budget: How to Help Search Engines See Your Site

A large site — more than a thousand pages — will sooner or later run into the problem of Googlebot not being able to crawl it at the pace of updates. New pages get indexed after two weeks instead of two days, changes to existing pages are picked up with a delay, and some product cards never get indexed at all. This is the crawl budget problem — the crawl allowance Google allocates to your site.

In this article, we break down what crawl budget is, who suffers from it most, and what concrete steps help.

What is crawl budget

It is the volume of pages on your site that Googlebot is willing to crawl in a given period (usually a day). It consists of two parts.

Crawl rate limit — a technical constraint: how many simultaneous requests and how frequently Googlebot can make them to your server. If the server is fast and consistently responds in under 500 ms, the limit is high. If the server is slow or returns 5xx errors, Google automatically reduces its crawl speed to avoid overloading you.

Crawl demand — the demand for crawling: how much Google wants to crawl your site. It depends on:

Domain authority (a new domain → low demand; an established domain with traffic → high demand)
Update frequency (dynamic content → higher demand)
Content quality (thin or duplicate content → lower demand)
Current standing in the index (popular pages are crawled more often)

The final crawl budget = min(rate limit, demand).

Who this matters for — and who it doesn't

Crawl budget is mainly a problem for large and frequently updated sites. If you have a 50-page landing site, don't worry — Google will crawl it without any issues.

Real categories where crawl budget is critical:

E-commerce with thousands of product cards (especially with filters and combinations)
News sites and media publishing dozens of articles per day
Marketplaces with user-generated listings
Aggregators for classifieds, real estate, or job postings
Large blogs and portals with an archive of 1,000+ articles
Multilingual sites with dozens of language versions

If you have 200 pages that are updated once a month, this post may not be for you.

Signs of the problem

How to tell if your site has a crawl budget issue:

Search Console → Coverage → Excluded → "Crawled — currently not indexed" contains hundreds of pages. Google crawled them but decided not to index them. This is often a signal of low quality or wasted crawl budget.
Discovered — currently not indexed — Google knows the page exists but hasn't crawled it yet. If this number keeps growing, you definitely have too little crawl budget.
Server logs show Googlebot visiting the "wrong" pages — catalog filters, second-level pagination pages, technical URLs.
Your sitemap has 5,000 URLs, but only 1,500 are indexed. A large gap is a reason to investigate.
Page updates are picked up slowly — you changed a title on Monday, but the new title only appeared in search results two weeks later.

Where crawl budget "leaks"

Typical culprits:

Faceted navigation in e-commerce

/catalog/dresses/?color=red&size=m&price=2000-3000&sort=popular — with 4 filters and 5 values each, that's 625 URL combinations. With 6 filters — 15,625. And Googlebot tries to crawl every single one of them.

Solution: Block filters from indexation via meta robots noindex or X-Robots-Tag (but not via robots.txt — we want Google to visit the page, see the noindex directive, and leave, rather than waste crawl budget on repeated attempts).

For critical combinations (e.g., the most popular colors), keep them open and build meaningful landing pages with unique content.

Internal search with GET parameters

/search?q=red+spring+dress — a new URL every time. Google crawls it and finds nothing useful.

Solution: In robots.txt, block Disallow: /search? (though robots.txt is not ideal — it's better to use a noindex tag on the search results page itself).

Infinite pagination

/blog/page/2/ ... /blog/page/47/. If page/47 contains articles from 5 years ago, Google wastes crawl budget on them every day.

Solution: A self-canonical on every pagination page (do NOT canonical to page/1 — that will deindex everything except the first page). For very old pages — meta noindex, follow (the page is not indexed, but Google still follows the links on it).

Duplicate URLs with UTM tags and tracking parameters

/article/123?utm_source=twitter, /article/123?fbclid=xxx, /article/123?gclid=xxx. Each one is a new URL in Google's eyes.

Solution: A <link rel="canonical"> pointing to the clean URL (without parameters) on every page. Also — Search Console → URL Parameters (though Google effectively disabled this tool in 2024). Plus Clean-param for Yandex in robots.txt.

Long redirect chains

/old-url → 301 → /old-url-2 → 301 → /old-url-3 → 301 → /new-url. Google burns crawl budget on every step of the chain.

Solution: Audit your redirects twice a year. Shorten all chains longer than one hop into a direct redirect.

Soft 404s

A page returns HTTP 200, but the content is "sorry, nothing found" or an empty product card. Google detects this and wastes crawl budget on repeated checks.

Solution: Return a proper 404 (or 410 if the resource has been permanently removed) for non-existent resources. For empty product cards — add noindex while the item is out of stock.

Practical optimization steps

1. Audit your server logs

This is the most important step, and almost nobody does it. Download your nginx logs for the past month, filter by User-Agent (Googlebot), and see which URLs it visits most frequently.

If you see thousands of requests to /catalog/?filter=... — that's a leak. If the bot frequently hits 404 pages — that's a leak. If the bot visits pagination pages beyond page/30 — that's a leak.

Tools for analysis: Screaming Frog Log Analyzer, GoAccess (CLI), Splunk for large sites. For smaller sites, even grep + awk will do.

2. Sitemap prioritization

In sitemap.xml, each URL has <priority> and <changefreq> fields. Most sites ignore them or leave them at identical default values.

Use them meaningfully:

priority 1.0 — homepage and your top 5–10 pages
priority 0.8 — categories and sections
priority 0.6 — product cards / articles
priority 0.3 — archived / outdated content
changefreq weekly — for frequently updated content
changefreq monthly — for stable content
changefreq yearly — for rarely changing content

Google uses this as a hint, not a strict instruction — but it does have an effect.

3. Remove from the sitemap what shouldn't be in the index

A sitemap is "a list of what I want to see indexed." If it contains pagination, filters, tag pages, or technical URLs — remove them. The sitemap should be "clean."

The ideal target: the number of URLs in your sitemap equals the number of pages you actually want appearing in search results.

4. Strong internal linking

A well-connected site gets crawled faster. Every important page should have 3–5 internal links pointing to it. Click depth from the homepage should not exceed 3.

Category pages, breadcrumbs, related-content blocks, and curated collections — all of these build internal linking. A hidden benefit: Google understands your site's priorities without you having to do anything extra for each individual URL.

5. IndexNow for fresh content

IndexNow is an open protocol that lets you ping search engines about new or updated URLs. It is supported by Bing (and through it, ChatGPT), Yandex, and partially by Google.

When you publish a new article or update an existing one, you send a POST request with the URL. The search engine crawls it within hours, not weeks.

More about setting up IndexNow — in the article on GEO/AEO optimization.

6. Server speed

If your server responds in 200 ms, Google can comfortably make 50 requests per second. If it takes 2 seconds, Google automatically slows down to 5 requests per second to avoid crashing it.

Faster server = automatic increase in crawl budget. No additional configuration needed — Google adjusts on its own.

What doesn't work

A few approaches that seem logical but don't actually help:

Deleting old articles to "free up crawl budget." Old articles that have inbound links should be kept (you can refresh the date if needed). Only delete truly dead content with no traffic and no links pointing to it.
Blocking everything via robots.txt. This closes pages from crawling but doesn't help with budget — Google still knows the pages exist through links and keeps attempting to crawl them.
Asking Google in Search Console to crawl more frequently. This direct setting no longer exists (it was removed in 2024). The only way to influence crawl frequency is indirectly — through quality.

Summary

Crawl budget is a real problem for large sites, even in 2026. But it is solved not by magic, but by methodical work: log audits, a clean sitemap, eliminating duplicate content and faceted navigation issues, strong internal linking, and a fast server.

Want a crawl budget audit for your site? Get in touch. We analyze logs, sitemaps, and Googlebot behavior within 3 business days.