Robots.txt Generator & Validator

The robots.txt file must be placed at the root of your domain: https://example.com/robots.txt. It must be accessible at that exact URL. Placing it in a subdirectory will not work — crawlers only look for it at the domain root. The file must be served with a text/plain Content-Type header.

No. Robots.txt blocks crawling, not indexing. If other pages link to a URL that is disallowed in robots.txt, search engines may still index the URL based on anchor text and link context — they just won't crawl the page content. To prevent indexing, use the noindex meta tag or X-Robots-Tag HTTP header instead.

Add separate User-agent blocks for each AI crawler you want to block: User-agent: GPTBot followed by Disallow: /. This tool includes presets for major AI crawlers including GPTBot (OpenAI), ChatGPT-User, Google-Extended (Gemini), CCBot (Common Crawl), anthropic-ai, and Bytespider (TikTok). Note that compliance is voluntary — well-behaved bots respect robots.txt, but malicious scrapers may ignore it.

Yes. Google and Bing support the * wildcard and the $ end-of-URL anchor. For example, Disallow: /*.pdf$ blocks all URLs ending in .pdf, and Disallow: /*/private/ blocks any URL containing /private/ in the path. However, not all crawlers support these extensions — the original robots.txt specification only defines exact prefix matching.

The Sitemap directive tells crawlers where to find your XML sitemap: Sitemap: https://example.com/sitemap.xml. This is important because it helps search engines discover pages that might not be reachable through internal links alone. The Sitemap directive can be placed outside any User-agent block and applies globally.

Because Googlebot has its own group. Under RFC 9309 a crawler obeys only the single most specific group that names it, and ignores every other group completely — including "User-agent: *". So a file that says "User-agent: Googlebot / Allow: /" to welcome Google, and then puts the real restrictions under "User-agent: *", has exempted Googlebot from all of them. Either repeat the rules inside the named group, or list the named agents alongside * in one group. The validator on this page flags exactly this pattern.

The more specific rule, measured by the number of characters in the path pattern. "Disallow: /folder/" and "Allow: /folder/public/" both match /folder/public/page, but the Allow pattern is longer, so the URL is crawlable. When two patterns are exactly the same length, Allow wins. The tester on this page shows which rule won and why.

Not for Google — it has never supported the directive and ignores it silently. Bing and Yandex do honour it. If you need Googlebot to slow down, use the crawl rate setting in Search Console, or return 503/429 responses. The validator warns when it sees Crawl-delay so you do not assume it is working.

No — this is the single most common robots.txt misconception. robots.txt controls CRAWLING (whether bots fetch the page), not INDEXING (whether it appears in search results). If another site links to a URL you've disallowed, Google can still INDEX that URL and show it in results — just with no description, since it couldn't crawl the content. So disallowing a page you want hidden can backfire: it stays in the index AND Google can't see the noindex tag that would actually remove it. To truly keep a page out of search, ALLOW crawling and add <meta name='robots' content='noindex'> (or an X-Robots-Tag: noindex header) so Google can read the directive. Use robots.txt to manage crawl budget and block bots from heavy/irrelevant paths — not as a privacy or de-indexing tool.

Add a separate User-agent block for each AI crawler with Disallow: /. The major ones to name: GPTBot (OpenAI training), ChatGPT-User (ChatGPT browsing), Google-Extended (Gemini/Bard training, does NOT affect Google Search ranking), CCBot (Common Crawl, feeds many AI datasets), anthropic-ai / ClaudeBot (Anthropic), PerplexityBot, and Bytespider (TikTok). For example: 'User-agent: GPTBot' then 'Disallow: /'. This generator includes presets for these. Two caveats: compliance is VOLUNTARY — well-behaved bots honor it, but you can't force a scraper that ignores robots.txt, so for hard enforcement also block their user-agent strings at your server/CDN; and blocking Google-Extended opts you out of AI training without hurting your normal Google Search visibility, which many sites want.