Retry-After and 429 for AI crawlers: slow them down instead of blocking them

To slow a crawler without losing it, answer its excess requests with 429 Too Many Requests and a Retry-After header. A 403 or a robots.txt Disallow removes the crawler, and a rate limit asks it to wait. Google documents this for Googlebot as a short-term measure. The OpenAI, Anthropic and Perplexity pages say nothing about 429, so for those crawlers it is a request you can measure and cannot count on.

What do 429, 503 and Retry-After mean?

RFC 6585 defines 429 as too many requests in a given amount of time. The response should explain the condition and may carry Retry-After. The RFC leaves open how a server counts: per resource, across the server, or across a set of servers. It also says a cache must not store a 429.

RFC 9110 defines the header. Its value is either a number of seconds or an HTTP date:

HTTP/1.1 429 Too Many Requests
Retry-After: 120

A 503 says the service is temporarily unable to handle the request, because of overload or maintenance. RFC 9110 allows Retry-After there too, and says it then tells the client how long the service is expected to be unavailable. For a crawler the two differ in meaning only: 429 is about this client's rate, and 503 is about the service.

Which crawlers say they honour it?

Google is the only operator below that documents a status code. The others document a robots.txt line or a dashboard, or nothing.

Google's crawl rate page tells you to return 500, 503 or 429 to cut its crawl rate in an emergency. It adds that Retry-After can carry seconds or an HTTP date. Its changelog says it added the header's details to that page on 6 October 2026, and that the header was already documented elsewhere. Google's robots.txt page lists crawl-delay among fields it does not support.

Bing's report page says Bingbot honours Crawl-delay, and describes it as a window of 1 to 20 seconds in which Bingbot fetches one URL. Anthropic supports Crawl-delay and calls it non-standard. The OpenAI and Perplexity pages describe tokens, user agents and IP ranges. Perplexity's page also shows how to skip rate limiting rules for requests that match its user agent and its published addresses.

Cloudflare measures what the operators do not document. Its post of 16 October 2026 says a 429 asks a visitor to reduce its request rate. Cloudflare checks whether a verified crawler slows down, pauses, or keeps the same pace, and lets that move the crawler's trust band. The post does not mention Retry-After.

Why does 429 beat a block for a crawler you want?

A block removes the relationship. A rate limit keeps it. Google's robots.txt page says not to use 401 and 403 to limit the crawl rate. Bing's crawl error alerts suggest Crawl Control instead of blocking requests.

The cost of blocking depends on the token. Refusing OAI-SearchBot removes you from ChatGPT search answers, as OpenAI's crawlers explains. A refusal for a training crawler is a policy, and robots.txt is where policy lives. How to block AI crawlers shows the file.

A rate limit is for load. Two limits from Google apply:

• It acts on the whole hostname. Many 500, 503 or 429 responses cut the crawl rate for every URL on that hostname, errors and content alike.
• It is short-term. Google advises against more than 1–2 days. A URL that shows these codes for several days may be dropped from the index.

How do you rate limit AI crawlers in nginx?

Limit by user agent token, answer with 429, send Retry-After on that response only, and leave robots.txt alone. This configuration is written from the nginx documentation and was not run for this page.

# http {} context
map $http_user_agent $crawler_key {
    default "";
    "~*(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Amazonbot|Bytespider|Meta-ExternalAgent)" $binary_remote_addr;
}

limit_req_zone $crawler_key zone=crawlers:10m rate=30r/m;
limit_req_status 429;

server {
    # An exact match beats the prefix location below, so robots.txt is never limited.
    location = /robots.txt {
        # serve it as you do today
    }

    location / {
        limit_req zone=crawlers burst=10 nodelay;
        error_page 429 = @slow_down;
        # your usual handling
    }

    location @slow_down {
        default_type text/plain;
        add_header Retry-After 30 always;
        return 429 "Too many requests. Retry after 30 seconds.\n";
    }
}

The map gives every other client an empty key, and nginx does not count requests with an empty key. A refused request gets 503 by default, so limit_req_status changes it. The pattern lists AI crawlers only, not Googlebot or Bingbot.

burst=10 nodelay allows bursts of up to 10 requests, then admits about one every 2 seconds. nginx cannot compute the wait, so Retry-After is a fixed value above the refill time.

Without always, add_header skips error statuses. With it in location /, every 200 would carry the header too. The named location confines it to the 429.

To see the effect before you enforce it, add limit_req_dry_run on; to location / and log $limit_req_status. It reads REJECTED_DRY_RUN for the requests that would have been refused:

log_format crawl '$remote_addr [$time_local] "$request" $status "$http_user_agent" $limit_req_status';

Behind a CDN, $binary_remote_addr is the proxy's address. Restore the client address first, as Find AI crawlers in your server logs shows.

What does the same rule look like in application code?

This function uses the web-standard Request and Response, so it needs no framework. It keeps a token bucket per client address, holds 10 tokens, adds 1 every 2 seconds, and computes the wait for Retry-After. The test below ran it in Node.

const CRAWLERS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Amazonbot|Bytespider|Meta-ExternalAgent/;
const RATE = 0.5; // tokens added per second: one request every 2 s on average
const BURST = 10; // bucket size: requests allowed at once

const buckets = new Map(); // client address -> { tokens, at }

/** Returns a 429 Response when this client is over its limit, otherwise null. */
export function limitCrawlers(request, clientAddress) {
  if (new URL(request.url).pathname === '/robots.txt') return null; // never limit robots.txt
  if (!CRAWLERS.test(request.headers.get('user-agent') ?? '')) return null;

  const now = Date.now() / 1000;
  const bucket = buckets.get(clientAddress) ?? { tokens: BURST, at: now };
  bucket.tokens = Math.min(BURST, bucket.tokens + (now - bucket.at) * RATE);
  bucket.at = now;
  buckets.set(clientAddress, bucket);

  if (bucket.tokens >= 1) {
    bucket.tokens -= 1;
    return null;
  }
  if (buckets.size > 10_000) {
    for (const [key, b] of buckets) if (now - b.at > BURST / RATE) buckets.delete(key); // idle long enough to be full again
  }
  const wait = Math.ceil((1 - bucket.tokens) / RATE);
  return new Response('Too Many Requests\n', {
    status: 429,
    headers: { 'Retry-After': String(wait), 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

Call it first in your request handler and return its response when it is not null. The buckets live in one process, so several instances need a shared store.

Behind a small Node server, it answered 13 requests sent as GPTBot like this. The command prints each response's status line and Retry-After, and uniq -c counts runs:

for i in $(seq 1 13); do
  curl -s -o /dev/null -D - -A 'GPTBot/1.4' http://localhost:3917/docs | grep -iE '^(HTTP|retry-after)' | tr -d '\r' | paste -sd' ' -
done | uniq -c

  10 HTTP/1.1 200 OK
   3 HTTP/1.1 429 Too Many Requests retry-after: 2

/robots.txt as GPTBot, and /docs with a browser user agent, still answered 200. A client that sleeps for Retry-After and tries again behaves differently:

for i in $(seq 1 14); do
  code=$(curl -s -o /dev/null -D h.txt -w '%{http_code}' -A 'GPTBot/1.4' http://localhost:3917/docs)
  wait=$(tr -d '\r' < h.txt | awk -F': ' 'tolower($1)=="retry-after" {print $2}')
  echo "$code ${wait:+Retry-After $wait}"
  [ -n "$wait" ] && sleep "$wait"
done

  10 200
   1 429 Retry-After 2
   1 200
   1 429 Retry-After 2
   1 200

After the burst of 10, the loop gets one request per refill, which is the configured rate. That is the pattern Cloudflare's post looks for: a client that slows down after a 429.

What should a rate limit leave alone?

Leave /robots.txt out of the limit. RFC 9309 says a 4xx answer means the file is unavailable and the crawler may access any resource. A 5xx answer means it must assume complete disallow.

Google treats 4xx errors other than 429 as no robots.txt, and its page does not say how it reads a 429 there. A limit that hits the file gives crawlers a rule you did not write.

Three smaller cautions:

• Do not treat the user agent as identity. The client chooses it, so a forged GPTBot gets the limit too. For a signed agent, the IETF Web Bot Auth draft lets an origin limit by Signature-Agent URL, once a verifier has resolved it.
• Check your cache. RFC 6585 forbids storing a 429. A cache that stores one serves it to every client.
• Keep Googlebot out of a standing limit. Google documents 429 as an emergency valve for 1–2 days.

How do you check that the crawler slowed down?

Count its requests per minute, and its 429 answers, before and after. Find AI crawlers in your server logs has the per-minute and per-status recipes.

Three outcomes are possible. The crawler slows to the rate you set, pauses for the Retry-After period, or keeps its pace and collects 429 responses. Cloudflare lowers a crawler's trust band for the third. If a crawler ignores 429, the next lever is a block, and only after you have decided you do not want it.

Personal agents with no user agent covers rate limits for traffic that has no token to key on.

What does the scanner do with a 429?

The scanner reports it as not measured. A 429 describes the scanner's access, not your site.

• On the homepage, a 429 makes the whole scan not measured, with no score. The reachable check groups it with 401, 402, 403, 406 and 503.
• On /robots.txt, a 429 leaves the robots-txt and ai-rules checks not measured. A 5xx there is different: robots-txt fails it, because RFC 9309 has a crawler assume the whole site is disallowed.
• On other documents, such as /llms.txt and the sitemap, a 429 also leaves the check not measured. A 429 on the markdown request leaves content negotiation not measured.

The scanner does not wait for Retry-After and does not retry. In the 5 exemplar records of 20 October 2026, a scan sent 29–89 requests and took 3.0–7.5 seconds. A per-address limit that ignores the user agent can trip on that. The scanner identifies itself as AgenticWebScore/0.3 (+https://agenticwebscore.com/bot), so its requests are easy to find in a log.

One check does read Retry-After. Rate limits and quotas documented is unscored and experimental. It is asked only of a site identified as an API platform. The scanner's own limits answer 429 with Retry-After in seconds, as the bot page states.

To see how the scanner reads your site today, scan your site.