Robots.txt is a small text file with an outsized ability to help — or quietly wreck — your SEO. It tells crawlers which parts of your site they may request. Used well, it manages crawl budget and keeps junk out of the crawl; used carelessly, it can hide your whole site from Google. Here’s how to get it right.
What robots.txt does (and doesn’t) do
Robots.txt controls crawling — whether a bot fetches a URL. It does not reliably control indexing. A page blocked in robots.txt can still appear in search results (without a description) if it’s linked elsewhere, because Google never crawled it to see a noindex tag. This distinction trips up a lot of sites.
robots.txt vs noindex
- Use robots.txt to stop crawlers wasting time on low-value areas (faceted URLs, internal search, admin paths).
- Use a noindex meta tag to keep a page out of the index — and crucially, don’t block that page in robots.txt, or Google can’t crawl it to see the noindex.
The rules that matter
A robots.txt file lists user-agents and their Allow/Disallow rules, plus a link to your sitemap. Remember that a URL with no matching rule is allowed by default — robots.txt is an opt-out, not an opt-in. Keep it minimal; every rule is a chance to block something you didn’t mean to.
Tip: The most damaging robots.txt mistake is a stray Disallow: / — often left over from a staging site — which blocks your entire domain. After any deploy, check that your live robots.txt isn’t disallowing everything.
Handling AI crawlers
Robots.txt is now also where you decide whether AI crawlers can access your site. Blocking citation-capable bots (like OAI-SearchBot or PerplexityBot) means those engines can’t cite you — increasingly a visibility cost, not a saving. Decide deliberately which AI crawlers to allow rather than blocking by default.
Check it regularly
Because one wrong line can de-index a site, robots.txt is worth checking on a schedule — especially after migrations and deploys. An audit will flag if your robots.txt is blocking important content or citation-capable crawlers.
Frequently asked questions
What does robots.txt do for SEO?
It tells crawlers which URLs they may fetch, helping manage crawl budget and keep low-value areas out of the crawl. It controls crawling, not indexing.
What’s the difference between robots.txt and noindex?
Robots.txt blocks crawling; noindex keeps a page out of the index. To noindex a page, don’t also block it in robots.txt, or Google can’t crawl it to see the tag.
Should I block AI crawlers in robots.txt?
Only deliberately. Blocking citation-capable AI crawlers means those engines can’t cite you, which is usually a visibility cost rather than a benefit.