AgenticWebScore
Scan

Bot Access Control · Essential · High impact

AI usage policy page

States the rules agents must follow and reduces licensing ambiguity.

  • Essential
  • High impact
  • Adopted standard
AI usage policy page — Property, Value
Property Value
Check idai-policy
DimensionBot Access Control (18%)
TierEssential — pool 80 pts
Weight inside tier2
ImpactHigh
MaturityAdopted — A ratified standard with real deployment.
ConditionalNo
Publishing skillai-policy
Standards—

What is measured

This check has no fixed probe path, so the scanner fetches the home page, collects its anchors and matches href or anchor text against AI-policy patterns (/ai-policy, /ai-usage-policy, /ai, /policies/ai, "AI usage policy") before fetching the match. It passes when a linked page states at least one permitted and one prohibited use of the content plus a contact route, returns partial when the page exists but is unlinked or carries no explicit permitted and prohibited statements, and fails when no policy page can be found.

Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

              DOMAIN="example.com"
LINK=$(curl -sS "https://$DOMAIN/" | grep -Eio 'href="[^"]*(ai|artificial)[^"]*polic[^"]*"' | head -1 | cut -d'"' -f2); LINK=${LINK#https://$DOMAIN}; curl -sSL "https://$DOMAIN${LINK:-/ai-policy}" -o /tmp/ai-policy.html -w 'HTTP %{http_code} %{content_type}\n'; grep -Eio 'training|crawl|scrap|licen[cs]e|commercial|contact' /tmp/ai-policy.html | sort -u
            

The fix

This is the artifact to ship. It is the complete change, not a description of one.

              <!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>AI usage policy</title>
  <link rel="canonical" href="https://example.com/ai-policy">
</head>
<body>
  <h1>AI usage policy</h1>
  <p>Effective 2026-10-08. These rules govern automated access to example.com.</p>
  <h2>Permitted</h2>
  <ul>
    <li>Indexing, retrieval and citation by identified search and answer engines.</li>
    <li>Summarising public pages with attribution and a link to the source.</li>
  </ul>
  <h2>Not permitted</h2>
  <ul>
    <li>Training, fine-tuning or distilling a model on our content.</li>
    <li>Bulk scraping, resale or republishing content as a substitute for the original.</li>
  </ul>
  <h2>Contact</h2>
  <p>Licensing and corrections: <a href="mailto:agents@example.com">agents@example.com</a></p>
</body>
</html>
            

Related checks

Related checks — Check, Tier, Impact, Same skill
Check Tier Impact Same skill
webbotauth — Web Bot Auth key directoryEssentialCritical—
agent-permissions — Agent permission model (agent-permissions.json)EssentialCritical—
ai-rules — Named AI bot rules in robots.txtEssentialHigh—
bot-page — Crawler and agent policy pageRecommendedMedium—
security-txt — security.txt (RFC 9116)RecommendedLow—
crawl-delay — Crawl rate declarationEmergingLow—
ai-txt — /ai.txt access declarationEmergingLow—