Crawl rate declaration

What is measured

The scanner reads /robots.txt (there is no separate probe path) and looks for a Crawl-delay directive together with the user-agent group it belongs to. It passes when at least one Crawl-delay line holds a positive number of seconds inside a group that names an AI crawler or is the wildcard *, returns partial when Crawl-delay appears only under a group for a non-AI agent or the value is zero or non-numeric, and fails when /robots.txt is missing or contains no Crawl-delay line.

Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

              DOMAIN="example.com"
curl -sSL "https://$DOMAIN/robots.txt" | grep -in -B3 'crawl-delay'
            

The fix

This is the artifact to ship. It is the complete change, not a description of one.

              # robots.txt - https://example.com/robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Crawl-delay: 1

# Answer engines: one request per second.
User-agent: OAI-SearchBot
Allow: /
Crawl-delay: 1
User-agent: PerplexityBot
Allow: /
Crawl-delay: 1

# Training crawlers: reachable but throttled hard.
User-agent: GPTBot
Disallow: /admin/
Crawl-delay: 10
User-agent: ClaudeBot
Disallow: /admin/
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml
            

Related checks