# Retry-After and 429 for AI crawlers: slow them down instead of blocking them > How to slow an AI crawler with 429 or 503 and Retry-After instead of blocking it, what Google, Bing and Cloudflare document, and nginx and middleware code. - Source: /learn/retry-after-429-ai-crawlers - Updated: 2026-11-01 - Reading time: 8 minutes - Tags: 429, Retry-After, rate limiting, AI crawlers, nginx --- To slow a crawler without losing it, answer its excess requests with `429 Too Many Requests` and a `Retry-After` header. A `403` or a robots.txt `Disallow` removes the crawler, and a rate limit asks it to wait. Google documents this for Googlebot as a short-term measure. The OpenAI, Anthropic and Perplexity pages say nothing about `429`, so for those crawlers it is a request you can measure and cannot count on. ## What do 429, 503 and Retry-After mean? [RFC 6585](https://www.rfc-editor.org/rfc/rfc6585.html) defines `429` as too many requests in a given amount of time. The response should explain the condition and may carry `Retry-After`. The RFC leaves open how a server counts: per resource, across the server, or across a set of servers. It also says a cache must not store a `429`. [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html) defines the header. Its value is either a number of seconds or an HTTP date: ```http HTTP/1.1 429 Too Many Requests Retry-After: 120 ``` A `503` says the service is temporarily unable to handle the request, because of overload or maintenance. RFC 9110 allows `Retry-After` there too, and says it then tells the client how long the service is expected to be unavailable. For a crawler the two differ in meaning only: `429` is about this client's rate, and `503` is about the service. ## Which crawlers say they honour it? Google is the only operator below that documents a status code. The others document a robots.txt line or a dashboard, or nothing. | Operator | `429` or `503` with `Retry-After` | What it documents instead | | --- | --- | --- | | Google | Yes, for `500`, `503` and `429`, for 1–2 days at most | `Crawl-delay` is not supported | | Bing | Not on the pages read | `Crawl-delay` and the Crawl Control tool | | Anthropic | Not on the page read | `Crawl-delay`, which it calls non-standard | | OpenAI | Not on the page read | robots.txt tokens | | Perplexity | Not on the page read | robots.txt tokens | Google's [crawl rate page](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate) tells you to return `500`, `503` or `429` to cut its crawl rate in an emergency. It adds that `Retry-After` can carry seconds or an HTTP date. Its changelog says it added the header's details to that page on 6 October 2026, and that the header was already documented elsewhere. Google's [robots.txt page](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) lists `crawl-delay` among fields it does not support. Bing's [report page](https://www.bing.com/webmasters/help/how-to-report-an-issue-with-bingbot-25c19802) says Bingbot honours `Crawl-delay`, and describes it as a window of 1 to 20 seconds in which Bingbot fetches one URL. Anthropic [supports `Crawl-delay`](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) and calls it non-standard. The [OpenAI](https://developers.openai.com/api/docs/bots) and [Perplexity](https://docs.perplexity.ai/guides/bots) pages describe tokens, user agents and IP ranges. Perplexity's page also shows how to skip rate limiting rules for requests that match its user agent and its published addresses. Cloudflare measures what the operators do not document. Its [post of 16 October 2026](https://blog.cloudflare.com/verified-bots-accountability/) says a `429` asks a visitor to reduce its request rate. Cloudflare checks whether a verified crawler slows down, pauses, or keeps the same pace, and lets that move the crawler's trust band. The post does not mention `Retry-After`. ## Why does 429 beat a block for a crawler you want? A block removes the relationship. A rate limit keeps it. Google's [robots.txt page](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) says not to use `401` and `403` to limit the crawl rate. Bing's [crawl error alerts](https://www.bing.com/webmasters/help/crawl-error-alerts-e29a3f3e) suggest Crawl Control instead of blocking requests. The cost of blocking depends on the token. Refusing `OAI-SearchBot` removes you from ChatGPT search answers, as [OpenAI's crawlers](/learn/openai-crawlers-gptbot-oai-searchbot-chatgpt-user) explains. A refusal for a training crawler is a policy, and robots.txt is where policy lives. [How to block AI crawlers](/learn/ai-crawler-blocking) shows the file. A rate limit is for load. Two limits from Google apply: - **It acts on the whole hostname.** Many `500`, `503` or `429` responses cut the crawl rate for every URL on that hostname, errors and content alike. - **It is short-term.** Google advises against more than 1–2 days. A URL that shows these codes for several days may be dropped from the index. ## How do you rate limit AI crawlers in nginx? Limit by user agent token, answer with `429`, send `Retry-After` on that response only, and leave `robots.txt` alone. This configuration is written from the nginx documentation and was not run for this page. ```nginx # http {} context map $http_user_agent $crawler_key { default ""; "~*(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Amazonbot|Bytespider|Meta-ExternalAgent)" $binary_remote_addr; } limit_req_zone $crawler_key zone=crawlers:10m rate=30r/m; limit_req_status 429; server { # An exact match beats the prefix location below, so robots.txt is never limited. location = /robots.txt { # serve it as you do today } location / { limit_req zone=crawlers burst=10 nodelay; error_page 429 = @slow_down; # your usual handling } location @slow_down { default_type text/plain; add_header Retry-After 30 always; return 429 "Too many requests. Retry after 30 seconds.\n"; } } ``` The `map` gives every other client an empty key, and nginx does not count requests with an empty key. A refused request gets `503` by default, so `limit_req_status` changes it. The pattern lists AI crawlers only, not Googlebot or Bingbot. `burst=10 nodelay` allows bursts of up to 10 requests, then admits about one every 2 seconds. nginx cannot compute the wait, so `Retry-After` is a fixed value above the refill time. Without `always`, `add_header` skips error statuses. With it in `location /`, every `200` would carry the header too. The named location confines it to the `429`. To see the effect before you enforce it, add `limit_req_dry_run on;` to `location /` and log `$limit_req_status`. It reads `REJECTED_DRY_RUN` for the requests that would have been refused: ```nginx log_format crawl '$remote_addr [$time_local] "$request" $status "$http_user_agent" $limit_req_status'; ``` Behind a CDN, `$binary_remote_addr` is the proxy's address. Restore the client address first, as [Find AI crawlers in your server logs](/learn/find-ai-crawlers-in-server-logs) shows. ## What does the same rule look like in application code? This function uses the web-standard `Request` and `Response`, so it needs no framework. It keeps a token bucket per client address, holds 10 tokens, adds 1 every 2 seconds, and computes the wait for `Retry-After`. The test below ran it in Node. ```js const CRAWLERS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Amazonbot|Bytespider|Meta-ExternalAgent/; const RATE = 0.5; // tokens added per second: one request every 2 s on average const BURST = 10; // bucket size: requests allowed at once const buckets = new Map(); // client address -> { tokens, at } /** Returns a 429 Response when this client is over its limit, otherwise null. */ export function limitCrawlers(request, clientAddress) { if (new URL(request.url).pathname === '/robots.txt') return null; // never limit robots.txt if (!CRAWLERS.test(request.headers.get('user-agent') ?? '')) return null; const now = Date.now() / 1000; const bucket = buckets.get(clientAddress) ?? { tokens: BURST, at: now }; bucket.tokens = Math.min(BURST, bucket.tokens + (now - bucket.at) * RATE); bucket.at = now; buckets.set(clientAddress, bucket); if (bucket.tokens >= 1) { bucket.tokens -= 1; return null; } if (buckets.size > 10_000) { for (const [key, b] of buckets) if (now - b.at > BURST / RATE) buckets.delete(key); // idle long enough to be full again } const wait = Math.ceil((1 - bucket.tokens) / RATE); return new Response('Too Many Requests\n', { status: 429, headers: { 'Retry-After': String(wait), 'Content-Type': 'text/plain; charset=utf-8' }, }); } ``` Call it first in your request handler and return its response when it is not null. The buckets live in one process, so several instances need a shared store. Behind a small Node server, it answered 13 requests sent as `GPTBot` like this. The command prints each response's status line and `Retry-After`, and `uniq -c` counts runs: ```sh for i in $(seq 1 13); do curl -s -o /dev/null -D - -A 'GPTBot/1.4' http://localhost:3917/docs | grep -iE '^(HTTP|retry-after)' | tr -d '\r' | paste -sd' ' - done | uniq -c ``` ```txt 10 HTTP/1.1 200 OK 3 HTTP/1.1 429 Too Many Requests retry-after: 2 ``` `/robots.txt` as `GPTBot`, and `/docs` with a browser user agent, still answered `200`. A client that sleeps for `Retry-After` and tries again behaves differently: ```sh for i in $(seq 1 14); do code=$(curl -s -o /dev/null -D h.txt -w '%{http_code}' -A 'GPTBot/1.4' http://localhost:3917/docs) wait=$(tr -d '\r' < h.txt | awk -F': ' 'tolower($1)=="retry-after" {print $2}') echo "$code ${wait:+Retry-After $wait}" [ -n "$wait" ] && sleep "$wait" done ``` ```txt 10 200 1 429 Retry-After 2 1 200 1 429 Retry-After 2 1 200 ``` After the burst of 10, the loop gets one request per refill, which is the configured rate. That is the pattern Cloudflare's post looks for: a client that slows down after a `429`. ## What should a rate limit leave alone? Leave `/robots.txt` out of the limit. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) says a 4xx answer means the file is unavailable and the crawler may access any resource. A 5xx answer means it must assume complete disallow. Google treats 4xx errors other than `429` as no robots.txt, and its page does not say how it reads a `429` there. A limit that hits the file gives crawlers a rule you did not write. Three smaller cautions: - **Do not treat the user agent as identity.** The client chooses it, so a forged `GPTBot` gets the limit too. For a signed agent, the IETF Web Bot Auth draft lets an origin limit by `Signature-Agent` URL, once a verifier has resolved it. - **Check your cache.** RFC 6585 forbids storing a `429`. A cache that stores one serves it to every client. - **Keep Googlebot out of a standing limit.** Google documents `429` as an emergency valve for 1–2 days. ## How do you check that the crawler slowed down? Count its requests per minute, and its `429` answers, before and after. [Find AI crawlers in your server logs](/learn/find-ai-crawlers-in-server-logs) has the per-minute and per-status recipes. Three outcomes are possible. The crawler slows to the rate you set, pauses for the `Retry-After` period, or keeps its pace and collects `429` responses. Cloudflare lowers a crawler's trust band for the third. If a crawler ignores `429`, the next lever is a block, and only after you have decided you do not want it. [Personal agents with no user agent](/learn/personal-ai-agents-no-user-agent) covers rate limits for traffic that has no token to key on. ## What does the scanner do with a 429? The scanner reports it as not measured. A `429` describes the scanner's access, not your site. - **On the homepage,** a `429` makes the whole scan not measured, with no score. The [reachable check](/checks/reachable) groups it with `401`, `402`, `403`, `406` and `503`. - **On `/robots.txt`,** a `429` leaves the [`robots-txt`](/checks/robots-txt) and [`ai-rules`](/checks/ai-rules) checks not measured. A `5xx` there is different: `robots-txt` fails it, because RFC 9309 has a crawler assume the whole site is disallowed. - **On other documents,** such as `/llms.txt` and the sitemap, a `429` also leaves the check not measured. A `429` on the markdown request leaves content negotiation not measured. The scanner does not wait for `Retry-After` and does not retry. In the 5 exemplar records of 20 October 2026, a scan sent 29–89 requests and took 3.0–7.5 seconds. A per-address limit that ignores the user agent can trip on that. The scanner identifies itself as `AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)`, so its requests are easy to find in a log. One check does read `Retry-After`. [Rate limits and quotas documented](/checks/rate-limits-declared) is unscored and experimental. It is asked only of a site identified as an API platform. The scanner's own limits answer `429` with `Retry-After` in seconds, as [the bot page](/bot) states. To see how the scanner reads your site today, [scan your site](/scan).