Crawl Budget Optimization for Large Sites in 2026

On a large online shop, a crawler often finds far more URLs than Search Console shows as indexed: filter combinations, sort orders, session parameters and old pagination. Every one of those URLs is a request Googlebot can spend instead of crawling a new product page. Crawl budget optimization is the discipline of making sure Googlebot spends its crawling on the URLs you actually want indexed.

What is Crawl Budget?

Google’s own crawl budget guide (checked October 5, 2026) says it is written mainly for large sites (1 million+ unique pages) whose content changes about once a week, medium or larger sites (10,000+ unique pages) whose content changes daily, and sites with a large share of URLs marked “Discovered - currently not indexed” in Search Console. Google calls these rough estimates, not exact thresholds. The topic matters most for large e-commerce, news, classifieds, programmatic SEO and faceted-navigation sites, where the URL space can run into millions.

Crawl Budget: Key Facts

How Does Google Decide Your Crawl Budget?

Google calculates crawl budget per site dynamically. The crawl rate limit is determined by how your server responds - if Googlebot sees 200 OK responses with low latency, it crawls faster; if it sees 5xx errors or slow response times, it backs off. This is why a slow server actively reduces your crawl budget over time.

Crawl demand is the other half. Google decides which URLs are worth re-crawling based on their popularity (inbound links, search interest) and staleness (when they last changed). A homepage with thousands of backlinks is re-crawled often. A 2018 tag-archive page with one inlink may be re-crawled only rarely.

This means crawl budget is not a fixed quota you can request more of. It is an emergent property of your server health, internal linking signals, and external authority. You influence it by changing the inputs.

Why Faceted Navigation Eats Crawl Budget

Faceted navigation is the single biggest crawl budget drain on e-commerce sites. Every filter combination creates a new URL: /shop/shoes?color=red&size=42&sort=price. Multiply that across 20 filter dimensions and you have an effectively infinite URL space - most of which contains zero unique content compared to the parent category.

Googlebot will dutifully follow these URLs because they look like real pages. It can spend a large share of its visits on sort and filter permutations of the same product list before it reaches your newly added product pages. That is why Google’s guide recommends blocking faceted navigation URLs you do not want crawled.

Crawl Rate Limit vs Crawl Demand: When Each Matters

Choose to improve rate limit when Search Console shows host status warnings or the average response time in Crawl Stats is high or rising. Choose to improve demand when your crawl stats show plenty of capacity but specific page types are not getting crawled.

How to Diagnose Crawl Budget Problems (Step by Step)

Start in Search Console; you need access to the property. The goal is a clear answer to which URL patterns are draining crawling.

Open the Crawl Stats Report

In Google Search Console, click Settings (gear icon, bottom left), then Crawl Stats. You will see total crawl requests, average response time, and host status for the last 90 days.

Check Response Code Distribution

Scroll to By Response. On a healthy site, most crawl requests return 200 OK. A large share of 301 or 404 responses means Googlebot is spending visits on URLs that no longer matter - fix the internal links and sitemaps that still point at them. 5xx responses are the worst: Google crawls less when your server returns errors.

Check Crawl Purpose Distribution

The "By Purpose" view splits crawl into Discovery (new URLs Google found) and Refresh (re-crawling known URLs). On large sites with stable architecture, Refresh should dominate. If Discovery stays high on a site whose content rarely changes, something keeps generating new URLs - usually parameters or session IDs.

Run a Full-Site Crawl and Diff Against GSC

Use a crawler like Daylytix or Screaming Frog to list the URLs your internal links expose (on a very large site, crawl one section or template at a time). Compare the count to the indexed page count in Search Console. If the crawl finds many times more URLs than Search Console shows as indexed, your faceted navigation or pagination is likely creating waste.

Identify the Top Wasted Patterns

Group the crawled URLs by template (e.g. /shop/*?sort=*, /tag/*, /page/*). Sort by URL count descending. Often a handful of patterns account for most of the waste.

The 2026 Playbook: How to Reclaim Wasted Crawl Budget

The exact intervention depends on what your audit found. These are the standard moves in order of leverage.

Block Parameter URLs in robots.txt

For sort, filter, view, and session parameters that never generate indexable content, add explicit Disallow rules. Example: Disallow: /*?sort=. This is the cleanest fix because Googlebot will not even fetch the URL.

Add rel="canonical" Pointing to the Clean URL

When you cannot block the parameter (because users still need that URL to work), use canonical tags. Every parameter variant should canonical to the parameter-less version. Google still crawls these, but it can treat them as duplicates and consolidate signals on the clean URL.

Use noindex on Thin and Auto-Generated Pages

Tag archives, author archives, year/month archives, and any auto-generated page with minimal unique content should carry <meta name="robots" content="noindex,follow">. This keeps them out of the index, but it does not save crawling: Google’s guide notes that it still requests noindex pages and only then drops them. If the goal is to stop crawling, block the URL pattern in robots.txt instead.

Fix Internal Links Pointing at Disallowed URLs

If you block ?sort=price in robots.txt but your category pages still link to those URLs, Google sees a contradiction. Update internal links to point at the canonical version. The crawler should never have a reason to ask for a blocked URL.

Improve Server Response Time

Google’s guide says that when a site responds consistently and its response times stay stable or improve, the crawl capacity limit goes up; when it slows down or returns server errors, the limit goes down. The biggest wins are usually: a CDN at the edge for static assets, full-page caching at the application layer, and database query optimisation for the slowest 5% of pages.

Submit a Cleaner Sitemap

Your XML sitemap should contain only canonical, indexable, status-200 URLs. Stripping out parameter variants, redirects, and noindex pages from the sitemap concentrates Google's discovery effort on what you actually want indexed.

Common Mistakes That Make Crawl Budget Worse

Mistake 1: Using nofollow on Internal Links to "Save" Crawl Budget

Why it happens: An old SEO myth from 2008 ("PageRank sculpting"). Why it backfires: Google treats internal nofollow as a hint, not a directive, and you lose the ability to flow link equity through your own site. What to do instead: Use robots.txt or noindex on the destination, not nofollow on the link.

Mistake 2: Blocking JavaScript and CSS in robots.txt

Why it happens: Teams trying to reduce crawl load by blocking static asset directories. Why it backfires: Googlebot needs to render the page to evaluate it. Blocked CSS/JS means Google sees a broken version and may not understand the page correctly. What to do instead: Leave assets crawlable; address bandwidth with a CDN.

Mistake 3: Submitting Massive Sitemaps with Every URL

Why it happens: CMSes auto-generate sitemaps from the database without filtering. Why it backfires: A sitemap with 500,000 URLs, half of which are noindex or 404, sends Googlebot to URLs you do not want indexed and weakens the sitemap as a list of your important pages. What to do instead: Build the sitemap from a clean indexable-URL query, not every URL in the database.

Limitation: You Cannot Force Google to Crawl Faster

Google retired the crawl rate limiter tool in Search Console on January 8, 2024. There is no slider, no API call, no manual override. The only way to get more crawls is to earn them through better signals (faster server, more authority, fresher content) - or wait for Google to discover that your server can handle more.

TL;DR: Crawl Budget Optimization Summary

Frequently Asked Questions

What is crawl budget?

Crawl budget is the number of URLs Googlebot is willing and able to crawl on your site within a given period. It is determined by your crawl rate limit (how fast Google can crawl without hurting your server) and your crawl demand (how much Google wants the content).

Does crawl budget matter for small sites?

Usually not. Google's crawl budget guide is aimed at large sites (roughly 1 million+ pages), medium sites (10,000+ pages) whose content changes daily, and sites with many URLs stuck as Discovered - currently not indexed. Crawling is also not a ranking signal in itself: Google has to crawl a page to show it, but more crawling does not make a page rank higher.

How do I check my crawl budget in Google Search Console?

Open Google Search Console, go to Settings, then Crawl Stats. You will see total crawl requests, average response time, and host status, broken down by response, file type, purpose and Googlebot type.

Should I block parameter URLs in robots.txt?

Yes, when those parameters do not generate unique indexable content. Sort and filter parameters that only reorder or narrow the same content are good candidates for a robots.txt Disallow. Do not add noindex to the same URLs: Google cannot see a noindex on a page it is not allowed to fetch. If users and links need a parameter URL to stay crawlable, point it to the clean URL with rel=canonical instead.

How long until crawl budget changes show up?

There is no fixed timeline. Crawl Stats covers the last 90 days, so compare the weeks before and after the change; indexing improvements show later in the Page indexing report. Better crawling helps new and updated pages get found sooner, but it does not raise rankings by itself.

Related Topics

• Technical SEO Audit Checklist for 2026: The full audit that includes crawl budget alongside Core Web Vitals, indexability, and structured data.
• Internal Linking Architecture: Internal link signals are half of crawl demand - get this right and Googlebot prioritises the pages you want.
• Core Web Vitals in 2026: Server response time (TTFB) feeds into LCP, and a fast, stable server also lets Google crawl more.
• Content Pruning: Removing thin pages frees up crawl budget for the ones that actually rank.

Getting Started

Start with Search Console Crawl Stats - even before running any tool, the response code and purpose breakdowns show where Googlebot spends its visits.

Then run a full-site crawl and compare the URL count to your indexed count in GSC. The bigger the gap between the two, the more room there is to clean up. Run a free audit with Daylytix - the report groups problem URLs by template, so you can see which patterns to fix first.