Crawl vs index — the distinction that matters
Almost every robots.txt mistake comes from conflating two things:
| robots.txt | Meta robots / X-Robots-Tag |
|---|---|
| Controls crawling (fetching) | Controls indexing (appearing in results) |
Disallow: blocks the bot’s request | noindex removes the page from results |
| Page can still be indexed via links | Requires the page to be crawlable to be seen |
Internalize this and you’ll use each tool correctly: robots.txt to steer crawlers away from low-value or heavy sections, noindex to actually keep a page out of search.
Anatomy of the file
A robots.txt lives at your domain root (https://example.com/robots.txt — nowhere else) and is grouped by user-agent:
User-agent: *
Disallow: /admin/
Allow: /admin/public/
Sitemap: https://example.com/sitemap.xmlDisallow blocks paths, Allow carves exceptions, and the Sitemap directive (placeable outside any user-agent block) points crawlers to your sitemap so they discover pages that aren’t well-linked.
Don’t block your own CSS and JS
A self-inflicted SEO wound: disallowing your CSS or JavaScript directories. Google renders pages to evaluate them, so if it can’t fetch your stylesheets and scripts, it sees a broken layout and may rank you lower — Search Console will flag this. Always allow crawlers to reach the assets needed to render the page.
It’s advisory, not enforcement
robots.txt is a request, not a lock. Reputable crawlers (Googlebot, Bingbot, GPTBot) obey it; malicious scrapers ignore it entirely. For anything you must truly protect, use authentication or server-level blocking — robots.txt is your first line of guidance, not a security boundary. Pair this with the Privacy Policy Generator and Meta Tag Generator to round out your site’s technical SEO setup.