# Crawl rate declaration

> Crawl rate declaration: Cloudflare's Verified Bots policy drops AI crawlers that ignore crawl-delay. The fix: declare a crawl rate.

- Source: /checks/crawl-delay
- Site: AgenticWebScore
- Language: English

---
Cloudflare's Verified Bots policy drops AI crawlers that ignore crawl-delay.

| Property | Value |
| --- | --- |
| Check id | `crawl-delay` |
| Dimension | Bot Access Control (18%) |
| Tier | Emerging — pool 5 pts |
| Weight inside tier | 1 |
| Impact | Low |
| Maturity | Adopted — A ratified standard with real deployment. |
| Conditional | No |
| Publishing skill | `crawl-delay` |
| Standards | — |

## What is measured

The scanner reads /robots.txt (there is no separate probe path) and looks for a Crawl-delay directive together with the user-agent group it belongs to. It passes when at least one Crawl-delay line holds a positive number of seconds inside a group that names an AI crawler or is the wildcard *, returns partial when Crawl-delay appears only under a group for a non-AI agent or the value is zero or non-numeric, and fails when /robots.txt is missing or contains no Crawl-delay line.

## Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

```bash
DOMAIN="example.com"
curl -sSL "https://$DOMAIN/robots.txt" | grep -in -B3 'crawl-delay'
```

## The fix

This is the artifact to ship. It is the complete change, not a description of one.

```text
# robots.txt - https://example.com/robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Crawl-delay: 1

# Answer engines: one request per second.
User-agent: OAI-SearchBot
Allow: /
Crawl-delay: 1
User-agent: PerplexityBot
Allow: /
Crawl-delay: 1

# Training crawlers: reachable but throttled hard.
User-agent: GPTBot
Disallow: /admin/
Crawl-delay: 10
User-agent: ClaudeBot
Disallow: /admin/
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml
```

> Crawl-delay is a widely supported extension rather than part of the standard robots.txt grammar and its value is in seconds, and because a crawler obeys the group that matches its own user-agent, the directive must be repeated in every named group where you want it enforced.

## Related checks

| Check | Tier | Impact | Same skill |
| --- | --- | --- | --- |
| `webbotauth` — Web Bot Auth key directory | Essential | Critical | — |
| `agent-permissions` — Agent permission model (agent-permissions.json) | Essential | Critical | — |
| `ai-rules` — Named AI bot rules in robots.txt | Essential | High | — |
| `ai-policy` — AI usage policy page | Essential | High | — |
| `bot-page` — Crawler and agent policy page | Recommended | Medium | — |
| `security-txt` — security.txt (RFC 9116) | Recommended | Low | — |
| `ai-txt` — /ai.txt access declaration | Emerging | Low | — |