# AI robots.txt generator: block AI crawlers by purpose > Build robots.txt lines that block AI crawlers by purpose: training, search, or fetches a user asks for. It sends no request, and allowed crawlers stay unnamed. - Source: /tools/ai-robots-txt-generator - Site: AgenticWebScore - Language: English --- Pick the AI crawlers to block, or start from a preset. The page writes the robots.txt group for them, and it sends no request to your site or anyone else. The generator is a plain form. It reads your choices from the address of this page, sends no request to your site or anyone else, and stores nothing. ## The presets Add the group to your robots.txt, with a blank line before it. If the file already has a group that names one of these crawlers, RFC 9309 combines the two, so remove the old one. ### Block training, keep search Blocks the 7 tokens filed under model training and datasets. Search, answers, and fetches a user asks for stay open. [Open this preset](/tools/ai-robots-txt-generator?preset=block-training). ```txt User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Amazonbot User-agent: Meta-ExternalAgent Disallow: / ``` ### Block all listed AI crawlers Blocks all 18 tokens the ai-rules check resolves. [Open this preset](/tools/ai-robots-txt-generator?preset=block-all). ```txt User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot User-agent: anthropic-ai User-agent: cohere-ai User-agent: DuckAssistBot User-agent: MistralAI-User User-agent: Meta-ExternalAgent Disallow: / ``` ### Allow all No group to add. An AI crawler your robots.txt does not name follows your `*` group. [Open this preset](/tools/ai-robots-txt-generator?preset=allow-all). ## Why an allowed crawler stays unnamed An allowed crawler stays unnamed: a group that names it replaces your `*` group for it (RFC 9309), so it would skip the rules you keep there. The fix note of [ai-rules](/checks/ai-rules) says the same, and what a block earns: A crawler obeys the one group that names it and skips the * group (RFC 9309, 2.2.1), so a named group repeats every * rule it should keep. This convention counts a named group that allows / for its crawler. Whether to block a crawler is your choice, and a block earns no points. ## Content-Signal Content-Signal states search, ai-input and ai-train preferences in robots.txt. No crawler documents honouring it. - `search`: building a search index and returning links and short excerpts. - `ai-input`: putting content into an AI model at request time, such as grounding or a generated search answer. - `ai-train`: training or fine-tuning AI models. The generator writes the line on its own. Add the line to the site's own User-agent: * group. Each value is the site owner's choice, and a signal left out states no preference. ```txt Content-Signal: search=yes, ai-input=yes, ai-train=no ``` The definitions are the policy's, at [contentsignals.org](https://contentsignals.org/). [Content-Signal in robots.txt explained](/learn/content-signal-robots-txt) says who reads the line today. ## The AI crawler tokens These are the 18 tokens the [ai-rules](/checks/ai-rules) check resolves, by what each operator says it fetches for, read on 27 October 2026. | Token | Operator | Fetches for | Purpose | Source | | --- | --- | --- | --- | --- | | `GPTBot` | OpenAI | Model training | Model training and datasets | [OpenAI](https://developers.openai.com/api/docs/bots) | | `OAI-SearchBot` | OpenAI | ChatGPT search | Search and answers | [OpenAI](https://developers.openai.com/api/docs/bots) | | `ChatGPT-User` | OpenAI | Actions a user starts | Fetches a user asks for | [OpenAI](https://developers.openai.com/api/docs/bots) | | `ClaudeBot` | Anthropic | Model training | Model training and datasets | [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | | `Claude-User` | Anthropic | Questions a user asks | Fetches a user asks for | [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | | `Claude-SearchBot` | Anthropic | Search results | Search and answers | [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | | `PerplexityBot` | Perplexity | Search results, not training | Search and answers | [Perplexity](https://docs.perplexity.ai/guides/bots) | | `Perplexity-User` | Perplexity | Questions a user asks | Fetches a user asks for | [Perplexity](https://docs.perplexity.ai/guides/bots) | | `Google-Extended` | Google | Gemini training and grounding, as a control token with no crawler of its own | Model training and datasets | [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) | | `Applebot-Extended` | Apple | Training Apple's models, as a control token that does not crawl | Model training and datasets | [Apple](https://support.apple.com/en-us/119829) | | `CCBot` | Common Crawl | An open crawl archive | Model training and datasets | [Common Crawl](https://commoncrawl.org/faq) | | `Bytespider` | ByteDance | Not documented by the operator | Not documented by an operator | None found | | `Amazonbot` | Amazon | Amazon products, and may train Amazon AI models | Model training and datasets | [Amazon](https://developer.amazon.com/amazonbot) | | `anthropic-ai` | Anthropic | Not on Anthropic's current page | Not documented by an operator | None found | | `cohere-ai` | Cohere | Cohere lists no crawler | Not documented by an operator | [Cohere](https://docs.cohere.com/docs/cohere-web-crawlers) | | `DuckAssistBot` | DuckDuckGo | AI-assisted answers, not training | Search and answers | [DuckDuckGo](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) | | `MistralAI-User` | Mistral AI | User actions in Vibe, not training | Fetches a user asks for | [Mistral AI](https://docs.mistral.ai/robots/) | | `Meta-ExternalAgent` | Meta | Model training and product indexing | Model training and datasets | [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) | Once the file is live, test it with the [AI crawler checker](/tools/ai-crawler-checker). ## Read next - [How to block AI crawlers in robots.txt](/learn/ai-crawler-blocking) - [AI crawler list 2026: robots.txt tokens](/learn/ai-crawler-user-agents) - [Content-Signal in robots.txt explained](/learn/content-signal-robots-txt) - [`ai-rules`](/checks/ai-rules): AI crawler policy in robots.txt - [`robots-txt`](/checks/robots-txt): robots.txt crawl policy - [`content-signal`](/checks/content-signal): Content-Signal preferences