AgenticWebScore
Scan

Agent readiness tools

AI crawler checker

Enter a domain to read its robots.txt the way each AI crawler does. The report opens on the crawler rules, Content-Signal, and the homepage they reach.

Free. No account and no email address. A site that blocks every automated client is reported as not measured, not as a low score.

The AI crawler checker runs the full scan of 60 checks, not one request. The five exemplar scans of 20 October 2026 sent 29 to 89 requests each and took 3 to 8 seconds. A scan that reaches its 60 s deadline is reported as not measured.

A stored result: github.com, measured on 20 October 2026. Its report opens on the same panel yours will.

What is checked

The report opens on 4 checks of rubric v0.9. Each is described here in the rubric's own words.

AI crawler policy in robots.txt

A crawler follows the robots.txt group that names it, otherwise the * group. A named group states the site's rules for that crawler.

ai-rules is a scored check. Agent conventions: 30% of the score. A convention never holds a level.

The scanner reads the fetched /robots.txt and resolves, for each of 18 AI crawler tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, anthropic-ai, cohere-ai, DuckAssistBot, MistralAI-User, Meta-ExternalAgent), the group that governs it under RFC 9309 2.2.1: the groups that name the token, merged, or else the * groups, merged. By the longest matching rule, each token is allowed, partly disallowed or disallowed, and the detail lists the disallowed tokens first. Present: at least 1 token has its own group with rules, and that group allows /. Not used: every token follows the * group, or no group; every named group disallows /, which blocks its crawler instead of giving it rules to read the site by; or robots.txt is not served (a 4xx other than 401, 403, 406 and 429, or an HTML page), so every crawler may fetch everything. Served with an error: a token named in a group with 0 rules, which then skips the * rules, or a token named only in comments. Not measured on 401, 403, 429, a 5xx, 406, which refuses the scan's Accept header, or no answer. Present earns the check's points in the agent-conventions part; Not used and Served with an error earn none.

Run the same measurement against your own site. Set the origin on the first line:

ORIGIN=https://example.com
curl -sSL -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/plain, */*;q=0.5' "$ORIGIN/robots.txt" | grep -Ein '^[[:space:]]*(user-agent|allow|disallow)[[:space:]]*:'

Read the ai-rules check page for its sources and the change to make.

robots.txt crawl policy

Crawlers read robots.txt before they fetch a page (RFC 9309). Its Sitemap line tells them where the URL list is.

robots-txt is a scored check. Web standards: 70% of the score. A check in it can hold a level.

The scanner fetches /robots.txt on the measured origin ($robots in the probe map) and parses it per RFC 9309, where a group is one or more User-agent lines followed by their Allow and Disallow rules. Pass = a text file that parses, including a file with only a User-agent: * group, no Sitemap line or no rules at all; also a 404 or any other 4xx except 401, 403 and 429 and 406, because under RFC 9309 every crawler may then fetch everything. The pass counts the groups and the rules across all groups. Partial = rules placed before the first User-agent line, which RFC 9309 ignores, or a User-agent: * group that disallows /, which the detail reports as the site's choice. Fail = a 5xx status, after which RFC 9309 has a crawler assume the whole site is disallowed; an HTML page served for the path, judged by the opening of the body or by an HTML type on a body that opens with a tag; or a type other than text. Not measured = 401, 403 or 429, which describe this scanner's access; 406, which refuses the scan's Accept header; or no response.

Run the same measurement against your own site. Set the origin on the first line:

ORIGIN=https://example.com
curl -sSL -D - -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/plain, */*;q=0.5' "$ORIGIN/robots.txt" | sed -n '1,60p'

Read the robots-txt check page for its sources and the change to make.

Content-Signal preferences

Content-Signal states search, ai-input and ai-train preferences in robots.txt. No crawler documents honouring it.

content-signal is a scored check. Agent conventions: 30% of the score. A convention never holds a level.

The scanner reads the Content-Signal lines of /robots.txt on the measured origin ($robots in the probe map) and the Content-Signal header of the homepage response ($home), per the Content Signals Policy. Present = a line or header that states search, ai-input or ai-train as yes or no; the detail names each value and the group the line sits in, and a signal not stated is no preference. Served with an error = Content-Signal lines with no yes or no value for any of the 3 signals. Not used = no Content-Signal line in robots.txt and no header. Not measured = /robots.txt answered 401, 403, 429 or a 5xx status, or 406, which refuses the scan's Accept header, or did not answer, and the homepage carries no header. Present earns the check's points in the agent-conventions part; Served with an error and Not used earn none.

Run the same measurement against your own site. Set the origin on the first line:

ORIGIN=https://example.com
curl -sSL -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/plain, */*;q=0.5' "$ORIGIN/robots.txt" | grep -i -B3 '^content-signal:'
curl -sSL -o /dev/null -D - -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7' "$ORIGIN/" | grep -i '^content-signal:'

Read the content-signal check page for its sources and the change to make.

Readable homepage

The homepage is the first URL every agent requests. An error or an empty page gives it nothing to read.

reachable is a scored check. Web standards: 70% of the score. A check in it can hold a level.

The scanner sends one GET to / and follows its redirects. A same-site redirect (apex to www, http to https, or both) moves the scan to the origin it lands on, and the detail names the measured URL; a redirect to another site is not measured. Pass = a 2xx response whose body is HTML, judged by the Content-Type or by the opening of the body after any comments, with at least 25 words of visible text. Partial = a 2xx HTML page with fewer than 25 words of visible text and no script. Fail = a non-2xx status that does not block the scan, such as 404, 410 or 500; a 2xx body that is not HTML; or an HTML page with fewer than 25 words of visible text and at least 1 script, which a client that runs no JavaScript reads as almost empty. Not measured = 401, 402, 403, 406, 429 or 503, no response, a JavaScript-only shell (a 2xx page with fewer than 60 words of visible text in more than 4096 characters of markup, a rule applied before the others), or a scan cut at the whole-scan deadline: the whole scan is then reported as not measured, with no score. A refusal is a fact about the scanner's vantage point, not about the site. Partial also = a same-site move to another origin, such as apex to www or http to https, answered with 302 or 307 at any hop, however readable the page: the detail cites Bing Webmaster Guidelines §7, which asks for 301 when a move is permanent and keeps 302 for changes shorter than 2 days. 301 and 308 pass. A 302 or 307 first redirect inside one origin, such as / to /en-US/, is not a move of the site: the detail notes that a temporary redirect is a weaker canonical signal than 301 or 308, and no points are lost.

Run the same measurement against your own site. Set the origin on the first line:

START=https://example.com
curl -sSL -o /dev/null -w 'status=%{http_code} final=%{url_effective} redirects=%{num_redirects} type=%{content_type} bytes=%{size_download}\n' -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7' "$START/"

Read the reachable check page for its sources and the change to make.

Questions about the AI crawler checker

Do AI crawlers honour Content-Signal?

Content-Signal states search, ai-input and ai-train preferences in robots.txt. No crawler documents honouring it. Cloudflare's scanner scores it. The IETF AIPREF drafts are the standards track.

Does blocking an AI crawler cost points?

A crawler obeys the one group that names it and skips the * group (RFC 9309, 2.2.1), so a named group repeats every * rule it should keep. This convention counts a named group that allows / for its crawler. Whether to block a crawler is your choice, and a block earns no points.

How do these checks count in the score?

robots-txt and reachable are in the web-standards part: 70% of the score, and the one part whose checks hold a level. ai-rules and content-signal are in the agent-conventions part: 30% of the score, and none of its checks holds a level. The methodology says what each state earns.

Why does it run the full scan?

The form starts the same scan as /scan, so the result is a scan record with a fingerprint you can verify. The report opens on ai-rules, robots-txt, content-signal, and reachable, and the full report follows. A stored record younger than 6 hours opens instead of a new scan.

Read next

Related checks in the rubric:

The methodology is the scoring contract for all 60 checks, including the rule for a site that is not measured.