Crawl Budget & Indexation
crawl budget & indexation crawl budget and indexation engineering shapes google's fetch priority through h crawl budget and indexation engineering
Introduction
Crawl budget and indexation engineering optimizes which URLs Googlebot fetches and indexes — critical for million-URL catalogs where faceted navigation, calendar archives, and API-generated pages explode crawl waste on thin duplicates.
Business problem
Large sites hit crawl budget limits: Google doesn't recrawl every URL daily. Low-value pages consume budget while money PDPs and fresh content wait — organic revenue stalls despite healthy domain authority.
- Facets: color×size×brand combinations create millions of thin URLs.
- Internal links: Orphan and infinite paginated archives trap crawlers.
- Soft 404s: 200 OK empty pages inflate index bloat.
Why this feature exists
Google Search allocates crawl rate based on site health, server response, and URL importance signals. HTML controls — internal links, noindex, canonical, robots.txt — shape what enters the index and how often it's refreshed.
- Crawl budget: Relevant mainly for large/high-change sites — not small blogs.
- Index bloat: Too many low-quality indexed pages dilutes site-wide quality signals.
- Sitemap: Hint for discovery — not a guarantee of indexation.
Browser internals
Googlebot scheduler prioritizes URLs by PageRank internal flow, freshness signals, and sitemap submission. Server 5xx and slow TTFB reduce crawl rate. Meta robots noindex still consumes fetch unless blocked earlier.
- Render queue: JS-heavy URLs cost more budget — prefer static HTML for catalog shells.
- Redirect chains: Multiple hops waste fetch quota per URL.
Rendering workflow
HTML architecture for crawl efficiency: Clean internal link graph, pagination with rel=next hints or view-all canonical strategy, faceted params blocked via robots/noindex, XML sitemap listing only canonical money URLs.
- Navigation: Facets as JS-only state without unique URLs — best for budget.
- Pagination: Self-canonical per page or view-all master — pick one strategy sitewide.
- Status codes: Real 404 for gone products — not soft 404 with 200.
Feature deep dive
Crawl/indexation toolkit in HTML and site architecture:
- robots.txt: Disallow crawl of /filter/, /search?, /api/ — saves budget.
- noindex: On thin facet URLs that must exist for UX.
- Internal links: XML sitemap + prominent nav to tier-1 URLs.
- Log analysis: Compare Googlebot hit frequency vs revenue URL tiers.
<!-- Faceted URL: noindex + canonical to category master --><head><meta name="robots" content="noindex, follow"><link rel="canonical" href="https://shop.example.com/shoes/running"></head><!-- Gone product: HTTP 404 + helpful HTML (not 302 to home) --><main><h1>Product discontinued</h1><p><a href="/shoes/running">Browse similar running shoes</a></p></main>
Accessibility analysis
Crawl optimizations must not harm users: noindex pages still need navigable HTML for humans who bookmark facet URLs. Soft 404s hurt users relying on screen reader page title announcements.
- Pagination: Accessible nav with current page announced — not infinite scroll only without load-more button.
SEO impact
Central to large-site SEO. Search Console Pages report: indexed vs not indexed reasons. Fix "Crawled - currently not indexed" and "Discovered - currently not indexed" buckets by improving quality signals and internal links.
- Crawl stats: Search Console crawl stats report — spike in 404 or response time hurts rate.
- Index coverage: Target index ratio aligned with revenue URL set.
Security considerations
Indexed admin URLs leak session tokens in query strings historically — noindex + auth + disallow. Parameter pollution URLs indexed from log files — monitor Search Console for unknown paths.
- Sensitive paths: /admin/, /internal/ in robots.txt + noindex + login wall.
Performance impact
Server capacity during crawl spikes: slow HTML TTFB reduces Googlebot crawl rate — fast edge HTML increases healthy recrawl of money pages.
- 5xx during deploy: Temporary crawl rate reduction — use graceful deploys.
- Infinite scroll: Without pagination URLs, content may not be linked for crawl.
Real production example
Marketplace facet strategy — URL params noindex, category pages index:
- Logs: BigQuery Googlebot log analysis — top crawled URL patterns monthly.
- KPI: Indexed URL count / revenue URL count ratio tracked weekly.
<!-- robots meta injected when ?color= or ?sort= present --><meta name="robots" content="noindex, follow"><link rel="canonical" href="https://market.example.com/laptops"><!-- Sitemap: only canonical category + PDP URLs --><url><loc>https://market.example.com/laptops/dell-xps-15</loc></url>
Enterprise usage
News publisher with 20 years archives: noindex paginated year pages beyond 2 years, keep canonical hub pages indexed, sitemap segmented by section with lastmod from CMS.
- Staging: Never link staging from prod — accidental discovery wastes budget.
- API HTML: Programmatic SEO pages gated by quality score before index allow.
Common production failures
Facet explosion: 8M indexed filter URLs — money PDP crawl frequency dropped 70%. Infinite calendar prev-month links created uncrawlable depth.
- Soft 404: Empty search results returned 200 — 400k junk indexed pages.
- Sitemap noise: Every variant in sitemap — Google ignored sitemap trust.
Architecture review questions
- Are facet/session/param URLs noindexed or absent entirely?
- Does sitemap contain only canonical revenue URLs?
- Are discontinued products returning true 404 with helpful HTML?
- Internal link depth to money pages ≤ 3 clicks from home?
- Googlebot crawl stats — healthy response time and low 5xx?
Hands-on project
Project: Audit a faceted category template: implement noindex for param URLs, canonical to category, trim sitemap — document expected index count change.
- Measure: Search Console indexed pages before/after 30 days.
- Tool: Screaming Frog or comparable crawl + log file analysis sample.
Interview questions
When does crawl budget actually matter?(Advanced)
Large sites (100k+ URLs), frequent updates, or server constraints. Small sites rarely budget-limited. Focus on index quality — removing low-value indexed pages often helps rankings more than 'saving' crawl on small sites.
Follow-up: How identify low-value indexed URLs?
Faceted nav: block in robots.txt or noindex?(Advanced)
noindex,follow on facet URLs preserves link equity flow from external links to facets while excluding from index. robots.txt disallow prevents crawl — Google can't see noindex on uncrawled URL. Often combine: disallow worst patterns, noindex remainder.
Follow-up: Client-side facets without URL change — SEO tradeoff?
Difference between 'Crawled - not indexed' and 'Discovered - not indexed'?(Advanced)
Crawled: Google fetched but chose not to index — usually quality/thin/duplicate signals. Discovered: known but not crawled yet — budget/priority. Fix crawled-not-indexed with better content/links; discovered-not-indexed with internal linking and sitemap priority.
Follow-up: Role of lastmod in sitemap?
Try it yourself
Edit the HTML, CSS, or JS panels — the preview updates as you type.
Try it yourself
Summary
Crawl budget and indexation engineering shapes Google's fetch priority through HTML internal linking, robots/noindex/canonical strategy, sitemap hygiene, and HTTP semantics — essential for large catalogs where thin URL explosion starves money pages.