Robots.txt for SEO and AI Crawlers: A Practical Guide

Robots.txt is a small text file with a lot of power. It tells crawlers which parts of your site they may request. Used well, it keeps search engines away from endless filter URLs and internal search pages. Used carelessly, one line can hide a whole site from Google. It is now also where you decide which AI crawlers may read your content, and for what.

This guide covers what robots.txt does and does not do, how Google reads it, what to block, the mistakes that cost traffic, and a clear policy for AI crawlers.

What robots.txt does, and what it does not

Robots.txt controls crawling: whether a bot fetches a URL. It does not control indexing. Google says plainly that robots.txt is not a mechanism for keeping a web page out of Google. A blocked URL can still appear in search results if other pages link to it, just without a description, because Google never fetched the page to read it.

• To keep a page out of search results, use a noindex robots meta tag or protect it with a password.
• To stop crawlers spending time on low-value URLs, use robots.txt.

Never combine the two on the same page. If robots.txt blocks a page, Google cannot crawl it to see its noindex tag, and the URL can stay in the index.

Where the file lives and how Google fetches it

• Location: the file must sit at the top level of the host, for example https://example.com/robots.txt. It applies only to that host, protocol and port, so shop.example.com needs its own file.
• Size: Google reads the first 500 KiB and ignores the rest.
• Caching: Google generally caches the file for up to 24 hours, so a change can take a day to apply.
• No file is fine: if robots.txt returns a 404 or another 4xx error (except 429), Google treats the site as if there were no restrictions.
• A server error is not fine: if the file returns a 5xx error, Google stops crawling the site for the first 12 hours while it keeps trying, then uses the last good copy for up to 30 days.

The syntax, in one small file

User-agent: *
Disallow: /search/
Disallow: /*?sort=
Allow: /search/help

Sitemap: https://example.com/sitemap.xml

• User-agent starts a group and names the crawler it applies to. * means every crawler without a group of its own. You can list several user-agent lines above one set of rules.
• Disallow blocks paths that start with the value. Allow makes an exception inside a blocked path.
• Wildcards: * matches any characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDF files.
• Sitemap points to your XML sitemap with a full URL. It is not tied to any user-agent group.
• Paths are case-sensitive: Disallow: /Admin/ does not block /admin/.

Google supports only user-agent, allow, disallow and sitemap. It ignores other lines, such as Crawl-delay, although some other crawlers read it.

How Google decides when rules conflict

Two rules decide most cases:

1. One group per crawler. A crawler follows only the group with the most specific user-agent that matches it and ignores all the others. If your file has a User-agent: Googlebot group, Googlebot ignores everything under User-agent: *.
2. The longest rule wins. Within the group, the most specific rule, the one with the longest path, applies. When an Allow and a Disallow rule are equally specific, Google uses the least restrictive one.

What to block, and what to leave alone

Usually worth blocking:

• Internal search results (/search/, ?s=).
• Filter and sort parameters that create endless URL combinations.
• Cart, checkout and account pages.
• Admin areas and test or staging paths on the live host.

Leave crawlable:

• CSS, JavaScript and images that pages need. Google warns that blocking resources it needs means it cannot analyze those pages properly.
• Any page you want to keep out of the index with noindex.
• Pages that only look unimportant but carry links or traffic, such as tag pages that rank. Check Search Console before you block a section.

Keep the file short. Every rule is one more chance to block something you did not mean to.

Mistakes that cost traffic

1. A leftover Disallow: /. Copied from a staging site during a launch, it blocks the whole domain. Check the live file after every deploy.
2. Blocking a page you noindexed. Google cannot see the noindex, so the URL can stay in results.
3. A crawler group that drops your rules. Adding User-agent: Googlebot with one rule means Googlebot no longer follows anything under User-agent: *. Repeat the rules it still needs.
4. Blocking CSS and JavaScript. An old habit that stops Google from rendering pages the way visitors see them.
5. A robots.txt that errors. A misconfigured server or firewall that answers with a 5xx error pauses crawling of the whole site.
6. Using robots.txt to remove pages. A Disallow does not take a page out of the index. Use noindex, a 404 or 410, or the Removals tool in Search Console for urgent cases.

AI crawlers: who is who

Most AI companies now run separate crawlers for search, for answering a user's request, and for training models. Their documentation says what each one does:

Anthropic states that its crawlers honor robots.txt. OpenAI notes that changes to OAI-SearchBot rules can take about 24 hours to apply. Apple says pages that disallow Applebot-Extended stay discoverable in Siri, Spotlight and Safari.

A sensible AI crawler policy

The split above lets you choose. Many sites want to appear and be cited in AI answers while keeping their content out of model training. A robots.txt for that looks like this:

# Search engines and everyone else
User-agent: *
Disallow: /search/

# AI search: allowed, with the same limits
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /search/

# Model training: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Note the repeated Disallow: /search/: the AI search crawlers have their own group, so they would otherwise ignore the rules under *. If you would rather allow training too, leave the last group out. And if a crawler is missing from this list, it simply follows your * group.

How to test your robots.txt

• Open the file yourself at /robots.txt after every deploy and check it returns 200 with the content you expect.
• Search Console's robots.txt report (in the property settings, for domain properties) shows the robots.txt files Google found for your top hosts, when it last fetched them, and any errors or warnings. After an important fix you can request a recrawl of the file there.
• URL Inspection in Search Console tells you whether a specific URL is blocked by robots.txt.

In Daylytix, the Robots.txt Analyzer fetches your live file and flags a Disallow: / for all crawlers or for Googlebot, blocked CSS, JavaScript and image folders, duplicate rules, a missing * group or Sitemap line, and Crawl-delay values Google ignores. It can also generate a new file. Every audit also runs an AI crawlers check that lists which AI and search crawlers your robots.txt allows or blocks, grouped by search, user-requested and training crawlers.

Frequently asked questions

What does robots.txt do for SEO?

It tells crawlers which URLs they may fetch, so search engines spend their time on pages that matter instead of filters, internal search and admin paths. It controls crawling, not indexing.

Does robots.txt stop a page from being indexed?

No. A blocked URL can still appear in search results without a description if other pages link to it. To keep a page out of the index, use a noindex tag or password protection, and do not block that page in robots.txt.

Where should the robots.txt file be?

At the top level of each host, such as https://example.com/robots.txt. A subdomain like shop.example.com needs its own file, because the rules apply only to the host, protocol and port where the file sits.

Should I block AI crawlers?

Decide per purpose. Blocking AI search crawlers such as OAI-SearchBot or PerplexityBot keeps you out of those answers, while blocking training crawlers such as GPTBot, ClaudeBot or Google-Extended only opts you out of model training.

Does Google support Crawl-delay?

No. Google supports only user-agent, allow, disallow and sitemap in robots.txt, and ignores Crawl-delay. Some other crawlers, such as Anthropic's, do read it.

Related reading

• The Technical SEO Audit Checklist for 2026
• Crawl Budget Optimization for Large Sites
• What Is llms.txt?
• How to Find and Fix Keyword Cannibalization