Crawlability & robots.txt: Controlling How Google Explores Your Site

Welcome to technical SEO — the discipline that makes everything you built in Module 3 reachable. We start at the front gate: crawlability, and the small text file that controls it. Small file, outsized consequences: one wrong line in robots.txt can remove an entire site from Google.

Crawl Budget: Google’s Attention Is Finite

From Lesson 1.1 you know Googlebot discovers pages by following links and sitemaps. What that lesson only touched: Google allocates each site a crawl budget — roughly, how many URLs it will fetch, how often — based on two factors:

• Crawl capacity: how fast your server responds without strain. Slow or error-prone servers get crawled gently — meaning less often.
• Crawl demand: how much Google wants your URLs — driven by popularity (links, traffic) and how often your content meaningfully changes.

For a site under a few thousand pages, crawl budget is rarely the bottleneck — Google will get to everything eventually. The real issue for small and medium sites is crawl waste: when Googlebot spends its visits on junk URLs (endless filter combinations, search result pages, parameter duplicates), your new and updated content waits longer to be discovered. Crawlability work is mostly about eliminating that waste — and robots.txt is the primary tool.

robots.txt: The File and Its Syntax

robots.txt is a plain text file at your domain root — always exactly at yourdomain.com/robots.txt — that tells crawlers which parts of the site not to visit. Every legitimate crawler checks it before crawling. The entire syntax worth knowing fits in one annotated example:

# Rules for all crawlers ("*" = everyone)
User-agent: *
# Block WordPress admin area…
Disallow: /wp-admin/
# …but allow the one file front-end features need
Allow: /wp-admin/admin-ajax.php
# Block internal search result pages (classic crawl waste)
Disallow: /?s=
Disallow: /search/

# Rules for one specific crawler
User-agent: GPTBot
Disallow: /premium-content/

# Where your sitemap lives (Lesson 4.2)
Sitemap: https://yourdomain.com/sitemap_index.xml

The four directives:

• User-agent: which crawler the following rules apply to. * means all; a named agent (Googlebot, GPTBot) gets its own rule group.
• Disallow: the path prefix not to crawl. Disallow: / blocks the entire site — the famous one-line catastrophe, usually a staging-site setting accidentally carried to production.
• Allow: an exception within a disallowed path (as with admin-ajax.php above).
• Sitemap: the full URL of your XML sitemap — the one directive that helps crawlers rather than restricting them.

What to Block — and What Never to Block

The Classic Confusion: robots.txt vs noindex

This trips up even experienced site owners, so let’s settle it permanently. They control different pipeline stages (Lesson 1.1):

• robots.txt controls crawling — whether Googlebot fetches the URL at all.
• noindex (a meta tag on the page: <meta name="robots" content="noindex">) controls indexing — whether the fetched page enters the index.

Here’s the trap: a robots.txt-blocked URL can still be indexed. If other sites link to it, Google may index the bare URL without ever crawling it — appearing in results with no description (“No information is available for this page”). Worse: to remove a page from the index, Google must be able to crawl it and see the noindex tag. Block it in robots.txt and Google can never read the instruction — the page stays stuck in the index indefinitely.

The 2026 Decision: AI Crawlers

A new robots.txt question now faces every site owner: AI companies crawl the web with their own user-agents — GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google’s AI training control), PerplexityBot and others — and robots.txt is where you allow or refuse them.

The trade-off, honestly stated:

• Allowing AI crawlers makes your content available for AI training and AI search products. As Lesson 1.4 established, AI search surfaces (with citations) are a growing discovery channel — blocking can mean invisibility in the answers your customers increasingly read.
• Blocking them protects content you’d rather not feed AI systems (premium material, proprietary data).
• Critical detail: Google-Extended controls only AI training (like Gemini) — blocking it does not remove you from AI Overviews or AI Mode, which use the normal Googlebot crawl. There is currently no way to stay in Google’s classic results while opting out of its AI answers; they share one crawler.

For most content-and-visibility businesses, the pragmatic 2026 default is to allow AI crawlers on public content — visibility in AI answers is the emerging equivalent of ranking (Module 9 covers the strategy). Block selectively where you have genuinely protected material.

Testing and Monitoring

1. Read your own file today: visit yourdomain.com/robots.txt. WordPress + Rank Math generates a sensible default you can edit inside the plugin; verify no leftover Disallow: / and that the Sitemap line is present.
2. GSC robots.txt report (Settings → robots.txt): shows the versions Google has fetched and any parsing problems.
3. Per-URL check: the URL Inspection tool at the top of Search Console tells you whether a specific URL is crawlable, and if blocked, by which rule.
4. After any change: spot-check three URLs — one that must be crawlable, one that should be blocked, one asset (CSS/JS) — through URL Inspection. Two minutes against catastrophic-mistake insurance.