Agent readiness tools

AI robots.txt generator

Pick the AI crawlers to block, or start from a preset. The page writes the robots.txt group for them, and it sends no request to your site or anyone else.

The generator is a plain form. It reads your choices from the address of this page, sends no request to your site or anyone else, and stores nothing.

The presets

Add the group to your robots.txt, with a blank line before it. If the file already has a group that names one of these crawlers, RFC 9309 combines the two, so remove the old one.

Block training, keep search

Blocks the 7 tokens filed under model training and datasets. Search, answers, and fetches a user asks for stay open. Open this preset.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
Disallow: /

Block all listed AI crawlers

Blocks all 18 tokens the ai-rules check resolves. Open this preset.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: cohere-ai
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: Meta-ExternalAgent
Disallow: /

Allow all

No group to add. An AI crawler your robots.txt does not name follows your * group. Open this preset.

Why an allowed crawler stays unnamed

An allowed crawler stays unnamed: a group that names it replaces your * group for it (RFC 9309), so it would skip the rules you keep there.

The fix note of ai-rules says the same, and what a block earns:

A crawler obeys the one group that names it and skips the * group (RFC 9309, 2.2.1), so a named group repeats every * rule it should keep. This convention counts a named group that allows / for its crawler. Whether to block a crawler is your choice, and a block earns no points.

Content-Signal

Content-Signal states search, ai-input and ai-train preferences in robots.txt. No crawler documents honouring it.

search: building a search index and returning links and short excerpts.
ai-input: putting content into an AI model at request time, such as grounding or a generated search answer.
ai-train: training or fine-tuning AI models.

The generator writes the line on its own. Add the line to the site's own User-agent: * group. Each value is the site owner's choice, and a signal left out states no preference.

Content-Signal: search=yes, ai-input=yes, ai-train=no

The definitions are the policy's, at contentsignals.org. Content-Signal in robots.txt explained says who reads the line today.

The AI crawler tokens

These are the 18 tokens the ai-rules check resolves, by what each operator says it fetches for, read on 27 October 2026.

The AI crawler tokens — Token, Operator, Fetches for, Purpose, Source
Token	Operator	Fetches for	Purpose	Source
`GPTBot`	OpenAI	Model training	Model training and datasets	OpenAI
`OAI-SearchBot`	OpenAI	ChatGPT search	Search and answers	OpenAI
`ChatGPT-User`	OpenAI	Actions a user starts	Fetches a user asks for	OpenAI
`ClaudeBot`	Anthropic	Model training	Model training and datasets	Anthropic
`Claude-User`	Anthropic	Questions a user asks	Fetches a user asks for	Anthropic
`Claude-SearchBot`	Anthropic	Search results	Search and answers	Anthropic
`PerplexityBot`	Perplexity	Search results, not training	Search and answers	Perplexity
`Perplexity-User`	Perplexity	Questions a user asks	Fetches a user asks for	Perplexity
`Google-Extended`	Google	Gemini training and grounding, as a control token with no crawler of its own	Model training and datasets	Google
`Applebot-Extended`	Apple	Training Apple's models, as a control token that does not crawl	Model training and datasets	Apple
`CCBot`	Common Crawl	An open crawl archive	Model training and datasets	Common Crawl
`Bytespider`	ByteDance	Not documented by the operator	Not documented by an operator	None found
`Amazonbot`	Amazon	Amazon products, and may train Amazon AI models	Model training and datasets	Amazon
`anthropic-ai`	Anthropic	Not on Anthropic's current page	Not documented by an operator	None found
`cohere-ai`	Cohere	Cohere lists no crawler	Not documented by an operator	Cohere
`DuckAssistBot`	DuckDuckGo	AI-assisted answers, not training	Search and answers	DuckDuckGo
`MistralAI-User`	Mistral AI	User actions in Vibe, not training	Fetches a user asks for	Mistral AI
`Meta-ExternalAgent`	Meta	Model training and product indexing	Model training and datasets	Meta

Once the file is live, test it with the AI crawler checker.

Choose the crawlers to block