Bot Access Control · Emerging · Low impact
/ai.txt access declaration
Some agent ecosystems read an /ai.txt contract alongside robots.txt.
- Emerging
- Low impact
- Draft standard
| Property | Value |
|---|---|
| Check id | ai-txt |
| Dimension | Bot Access Control (18%) |
| Tier | Emerging — pool 5 pts |
| Weight inside tier | 1 |
| Impact | Low |
| Maturity | Draft — A published specification that is not yet ratified. |
| Conditional | No |
| Publishing skill | ai-txt |
| Standards | — |
What the scanner requests
-
/ai.txt
What is measured
The scanner requests /ai.txt and expects HTTP 200 with a text/plain body, not HTML, that reads as an access declaration for automated clients. It passes when the file states both what may be done with the content and what may not, with a contact route, returns partial when the file exists but is HTML, is served with the wrong content type, or is a one-line notice with no usage terms or contact, and fails on a non-200.
Reproduce it yourself
You do not have to take our word for any of this. Run the same measurement against your own domain:
DOMAIN="example.com"
curl -sSL -o /tmp/ai.txt -w 'HTTP %{http_code} %{content_type}\n' "https://$DOMAIN/ai.txt"; grep -Ein 'allow|disallow|permit|prohibit|train|contact' /tmp/ai.txt | head -20
The fix
This is the artifact to ship. It is the complete change, not a description of one.
# ai.txt - access declaration for example.com
# Effective 2026-10-08. Read alongside robots.txt; robots.txt governs crawling.
Site: https://example.com
Contact: agents@example.com
Policy: https://example.com/ai-policy
Allow:
- Indexing, retrieval and citation by identified search and answer engines.
- Summarising public pages with attribution and a link to the source.
Disallow:
- Training, fine-tuning or distilling a model on this content.
- Bulk scraping, resale or redistribution of content or data.
- Account changes or purchases without human approval.
Rate: 1 request per second, descriptive User-Agent, honour robots.txt.
Related checks
| Check | Tier | Impact | Same skill |
|---|---|---|---|
webbotauth — Web Bot Auth key directory | Essential | Critical | — |
agent-permissions — Agent permission model (agent-permissions.json) | Essential | Critical | — |
ai-rules — Named AI bot rules in robots.txt | Essential | High | — |
ai-policy — AI usage policy page | Essential | High | — |
bot-page — Crawler and agent policy page | Recommended | Medium | — |
security-txt — security.txt (RFC 9116) | Recommended | Low | — |
crawl-delay — Crawl rate declaration | Emerging | Low | — |