Bot Access Control · Essential · High impact
AI usage policy page
States the rules agents must follow and reduces licensing ambiguity.
- Essential
- High impact
- Adopted standard
| Property | Value |
|---|---|
| Check id | ai-policy |
| Dimension | Bot Access Control (18%) |
| Tier | Essential — pool 80 pts |
| Weight inside tier | 2 |
| Impact | High |
| Maturity | Adopted — A ratified standard with real deployment. |
| Conditional | No |
| Publishing skill | ai-policy |
| Standards | — |
What is measured
This check has no fixed probe path, so the scanner fetches the home page, collects its anchors and matches href or anchor text against AI-policy patterns (/ai-policy, /ai-usage-policy, /ai, /policies/ai, "AI usage policy") before fetching the match. It passes when a linked page states at least one permitted and one prohibited use of the content plus a contact route, returns partial when the page exists but is unlinked or carries no explicit permitted and prohibited statements, and fails when no policy page can be found.
Reproduce it yourself
You do not have to take our word for any of this. Run the same measurement against your own domain:
DOMAIN="example.com"
LINK=$(curl -sS "https://$DOMAIN/" | grep -Eio 'href="[^"]*(ai|artificial)[^"]*polic[^"]*"' | head -1 | cut -d'"' -f2); LINK=${LINK#https://$DOMAIN}; curl -sSL "https://$DOMAIN${LINK:-/ai-policy}" -o /tmp/ai-policy.html -w 'HTTP %{http_code} %{content_type}\n'; grep -Eio 'training|crawl|scrap|licen[cs]e|commercial|contact' /tmp/ai-policy.html | sort -u
The fix
This is the artifact to ship. It is the complete change, not a description of one.
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>AI usage policy</title>
<link rel="canonical" href="https://example.com/ai-policy">
</head>
<body>
<h1>AI usage policy</h1>
<p>Effective 2026-10-08. These rules govern automated access to example.com.</p>
<h2>Permitted</h2>
<ul>
<li>Indexing, retrieval and citation by identified search and answer engines.</li>
<li>Summarising public pages with attribution and a link to the source.</li>
</ul>
<h2>Not permitted</h2>
<ul>
<li>Training, fine-tuning or distilling a model on our content.</li>
<li>Bulk scraping, resale or republishing content as a substitute for the original.</li>
</ul>
<h2>Contact</h2>
<p>Licensing and corrections: <a href="mailto:agents@example.com">agents@example.com</a></p>
</body>
</html>
Related checks
| Check | Tier | Impact | Same skill |
|---|---|---|---|
webbotauth — Web Bot Auth key directory | Essential | Critical | — |
agent-permissions — Agent permission model (agent-permissions.json) | Essential | Critical | — |
ai-rules — Named AI bot rules in robots.txt | Essential | High | — |
bot-page — Crawler and agent policy page | Recommended | Medium | — |
security-txt — security.txt (RFC 9116) | Recommended | Low | — |
crawl-delay — Crawl rate declaration | Emerging | Low | — |
ai-txt — /ai.txt access declaration | Emerging | Low | — |