# Crawler and agent policy page

> Crawler and agent policy page: Explains which agents are permitted and how they are verified. The fix: publish a crawler and agent policy page.

- Source: /checks/bot-page
- Site: AgenticWebScore
- Language: English

---
Explains which agents are permitted and how they are verified.

| Property | Value |
| --- | --- |
| Check id | `bot-page` |
| Dimension | Bot Access Control (18%) |
| Tier | Recommended — pool 20 pts |
| Weight inside tier | 1 |
| Impact | Medium |
| Maturity | Adopted — A ratified standard with real deployment. |
| Conditional | No |
| Publishing skill | `bot-policy` |
| Standards | — |

## What is measured

With no fixed probe path, the scanner lists the home page's anchors, matches href or anchor text against bot-policy patterns (/bot, /bots, /bot-policy, /agent-policy, /agents, "for agents") and reads the page it finds. It passes when that page names the agents it welcomes and refuses and explains how a claimed identity is verified (signed requests checked against a published Web Bot Auth key directory, or published IP ranges) with a contact, returns partial when it lists agents but never explains verification, and fails when no such page is linked or reachable.

## Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

```bash
DOMAIN="example.com"
LINK=$(curl -sS "https://$DOMAIN/" | grep -Eio 'href="[^"]*(bot|agent|crawler)[^"]*"' | head -1 | cut -d'"' -f2); LINK=${LINK#https://$DOMAIN}; curl -sSL "https://$DOMAIN${LINK:-/bot}" -o /tmp/bot-page.html -w 'HTTP %{http_code} %{content_type}\n'; grep -Eio 'verified|signature|key directory|user-agent|ip range|allow|deny|contact' /tmp/bot-page.html | sort -u
```

## The fix

This is the artifact to ship. It is the complete change, not a description of one.

```html
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Crawler and agent policy</title>
</head>
<body>
  <h1>Bot and agent policy</h1>
  <h2>How we verify identity</h2>
  <p>Signed requests are checked against our key directory at <a href="/.well-known/http-message-signatures-directory">/.well-known/http-message-signatures-directory</a>; unsigned clients are identified by User-Agent and published IP ranges only.</p>
  <h2>Welcome</h2>
  <ul>
    <li>Search and answer engines that identify themselves and cite the source.</li>
    <li>Agents acting for a person on a specific task, at 1 request per second.</li>
  </ul>
  <h2>Refused</h2>
  <ul>
    <li>Training and dataset crawlers; see the AI usage policy at /ai-policy.</li>
    <li>Clients that ignore robots.txt, spoof a User-Agent or exceed the stated rate.</li>
  </ul>
  <h2>Contact</h2>
  <p><a href="mailto:agents@example.com">agents@example.com</a></p>
</body>
</html>
```

> The scanner finds this page by following a link from your home page, so publish it at /bot and link it (a footer link is enough) — an unlinked page is scored as missing.

## Related checks

| Check | Tier | Impact | Same skill |
| --- | --- | --- | --- |
| `webbotauth` — Web Bot Auth key directory | Essential | Critical | — |
| `agent-permissions` — Agent permission model (agent-permissions.json) | Essential | Critical | — |
| `ai-rules` — Named AI bot rules in robots.txt | Essential | High | — |
| `ai-policy` — AI usage policy page | Essential | High | — |
| `security-txt` — security.txt (RFC 9116) | Recommended | Low | — |
| `crawl-delay` — Crawl rate declaration | Emerging | Low | — |
| `ai-txt` — /ai.txt access declaration | Emerging | Low | — |