Content-Signal in robots.txt: search, ai-input and ai-train explained

Content-Signal is a line inside a robots.txt group. It says, yes or no, whether crawlers may use your content for search, as AI input, or for AI training. Cloudflare published it as the Content Signals Policy on 24 September 2025, under a CC0 licence. The line enforces nothing, and none of the operator pages in AI crawler user agents in robots.txt mentions it.

What search, ai-input and ai-train mean

The definitions are the policy's own, from contentsignals.org:

• search: building a search index and returning search results, such as links and short excerpts. The policy adds that search “does not include providing AI-generated search summaries”.
• ai-input: putting content into one or more AI models at request time, such as retrieval augmented generation, grounding, or generative AI search answers.
• ai-train: training or fine-tuning AI models.

Each value is yes or no. A value you leave out states nothing: the policy text says the site then neither grants nor restricts that use through a content signal. So Content-Signal: ai-train=no refuses training and says nothing about search.

Where the line goes in robots.txt

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

The line belongs to the group it sits in. The contentsignals.org page also shows a path-scoped form, such as Content-Signal: /blog/ ai-train=no, search=yes, ai-input=no.

Group selection follows RFC 9309: a crawler with a group of its own does not read the * group. A Content-Signal line in * therefore says nothing to GPTBot once GPTBot has its own group. Repeat the line in every named group.

This site does that. Its robots.txt holds 18 groups and 18 Content-Signal lines:

curl -sS https://agenticwebscore.com/robots.txt | grep -i -B3 '^content-signal:' | head -9

User-agent: *
Disallow: /api/
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes
--
User-agent: GPTBot
Disallow: /api/
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes

Content-Signal lines for 4 common policies

The AI robots.txt generator writes this line from the three answers you pick.

The generator on contentsignals.org offers 4 default policies. Three of them produce a Content-Signal line in the * group:

# Allow Search Only
Content-Signal: ai-train=no, search=yes, ai-input=no

# Allow Search & AI Input
Content-Signal: ai-train=no, search=yes, ai-input=yes

# Allow Search, AI Input & AI Training
Content-Signal: ai-train=yes, search=yes, ai-input=yes

The fourth, Disallow All, produces no Content-Signal line. It writes Disallow: / for every crawler, and the site warns that it may cause search engines like Google to exclude the site from results. The second line is the one this site uses. Whichever you pick, write all 3 values, so no use is left to a reader's guess.

The policy text that comes with it

Cloudflare's version ships as comments above the groups. Its opening line says that, by accessing the site, “you agree to abide by the following content signals.” It ends by calling any restriction an express reservation of rights under Article 4 of EU Directive 2019/790.

The contentsignals.org page limits what it claims for that text. It says robots.txt files do not stop crawler operators from taking content at a technical level. Courts and regulators, it adds, may find that such files impose no enforceable legal obligation, and it tells readers to consult a lawyer. Treat the line as a stated preference, not as a control.

Who reads Content-Signal today

Cloudflare writes it. Cloudflare's managed robots.txt adds Content-signal: search=yes, ai-train=no and no ai-input value. The launch post says the managed file was already on for more than 3.8 million domains. The current documentation describes a fourth field in testing, use, with the values immediate, reference and full, and shows use=reference in the managed file.

No crawler operator documents reading it. On 25 October 2026, the pages of OpenAI, Anthropic, Perplexity, Apple, Amazon, Meta, Common Crawl, DuckDuckGo and Mistral AI did not mention it. Google's robots.txt page says its parser ignores rules other than allow, disallow and user-agent. Cloudflare notes that Google Search Console may report “Syntax not understood” for the line, and says it saw no effect on crawling.

Scanners read it. AgenticWebScore's Content-Signal check is a scored agent convention in the recommended tier. It reads the lines in robots.txt and a Content-Signal header on the homepage. Its own rubric text says no crawler documents honouring it.

What the scan records show

The scan records of 20 October 2026 split 2 to 3 among the 5 measured sites:

• Cloudflare states search=yes, ai-input=yes, ai-train=yes, in its Cohere-ai group only. A crawler that reads the * group sees no signal.
• Resend states the same 3 values as yes in its * group.
• Vercel, GitHub and MDN have no Content-Signal line.

This site states ai-train=no, search=yes, ai-input=yes in every group.

How Content-Signal differs from the IETF aipref drafts

The IETF AI Preferences working group is writing the standards-track version of the same idea, in 2 drafts. Neither mentions Content-Signal, and neither is an RFC yet.

draft-ietf-aipref-vocab-08 (14 September 2026) defines 3 categories with the labels train-ai, ai-use and search. Its own note says the vocabulary section does not yet have working group consensus. draft-ietf-aipref-attach-05 (19 August 2026) attaches them with a Content-Usage rule in robots.txt and a Content-Usage HTTP header:

User-Agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /ai-ok/ train-ai=y

The 2 formats are not interchangeable:

A file can carry both lines. Neither format defines the other's rule, so a parser built for one reads the other as an unknown line.

Should you add the line

It costs 1 line per group, and it states your choice in a format some tools read. It binds no crawler. If you add it, set all 3 values, put it in every group, and link a page that explains the policy.

To refuse training in a way honest crawlers act on, give the training crawlers named Disallow groups. AI crawler user agents in robots.txt shows example policies. Blocking AI crawlers without locking out agents covers enforcement at the edge. To see what your robots.txt states, group by group, scan your site.