Content-Signal usage signals

What is measured

The scanner parses https://<domain>/robots.txt ($robots in the probe map) for a Content-Signal directive inside a user-agent group, per the Content Signals Policy (contentsignals.org). Pass = at least one Content-Signal line that sets all three signals - search, ai-input and ai-train - to yes or no. Partial = the directive appears but omits one or more of the three signals, or sits outside any user-agent group. Fail = no Content-Signal line in robots.txt.

Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

              DOMAIN=example.com
curl -sS "https://$DOMAIN/robots.txt" | grep -i -B1 '^content-signal:'
            

The fix

This is the artifact to ship. It is the complete change, not a description of one.

              # Content Signals Policy for example.com
#
# search   = building a search index and returning links and short excerpts
# ai-input = feeding content into a model at answer time (RAG, grounding)
# ai-train = training or fine-tuning a model
# yes grants the use, no withholds it; an omitted signal does neither.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

# Per-crawler preferences go inside that crawler's own group.
User-agent: GPTBot
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
            

Standards referenced

• contentsignals.org

Related checks