# AI crawler blocking: how to refuse scrapers without locking out agents

> Blocking AI crawlers by IP locks out the agents you want. Named user-agent rules and Content Signals let you refuse training without refusing retrieval.

- Source: /learn/ai-crawler-blocking
- Updated: 2026-10-12
- Reading time: 8 minutes
- Tags: robots.txt, AI crawlers, Content Signals, bot access

---
Refusing crawlers is a policy decision, and it is a reasonable one. Blocking them at the network layer is an implementation choice that quietly makes a different decision: it refuses anything that looks automated, including the agent a customer just asked to check your price. Those two goals are compatible only if the mechanism you pick can tell one from the other. Most cannot.

## Three legitimate reasons to refuse, and one that is not

**Cost.** A crawler that walks every URL under every filter combination spends bandwidth and origin capacity. Refusing it is ordinary operational hygiene.

**Training.** Using your content to train or fine-tune a model is a use you may not have agreed to. This is a rights decision, not a security one.

**Competitive copying.** A competitor mirroring your catalogue, pricing or editorial is straightforwardly harmful. Refusing it is self-interested and defensible.

The reason that is not legitimate is "block everything automated." Automation is not a category of intent. The same HTTP client shape covers a price-checking agent acting for a paying customer, a search indexer that sends you traffic, an uptime monitor, and a scraper. A rule that refuses all four has not made a policy decision; it has declined to make one, and the agent ecosystem will read it as an absence of terms.

## Three mechanisms, only one of which identifies anyone

| Mechanism | What it keys on | Can it separate a signed agent from a scraper |
| --- | --- | --- |
| IP-level block | Source network | No |
| User-agent rule | A string the client chooses | No |
| Signed-agent policy | A key the client holds | Yes, for the operator only |

**A datacenter-IP block cannot distinguish a signed agent from a scraper.** This is the point that gets lost. Signed-agent schemes such as [Web Bot Auth](/checks/webbotauth) verify a signature on the request, not an address. They exist precisely because agent operators run on the same clouds, the same regions and often the same providers as scrapers. There is no address range reserved for well-behaved agents, no registry to fetch, and no convention by which an operator's egress addresses belong only to it. A rule that denies datacenter ranges denies signed agents by construction. It is not a strict setting that happens to catch a few good actors; the good actors are the target.

There is a second casualty. A site behind such a block cannot be measured, because a measurement tool running from a datacenter cannot reach it. The honest output of that scan is **not measured** — not a low score. Scoring a site down for a response the scanner never received would report a number that describes our vantage point rather than the site. [The methodology](/methodology) treats a check that is not measured as no finding about the site: it earns no credit, and the confidence falls. [Homepage open to agent access](/checks/reachable) is reported with the status code that produced the result, so a reader can disagree with the interpretation without re-running anything.

## Refuse by policy, not by address

The alternative is to write the refusal down where it belongs — in the document whose entire purpose is stating crawler preferences. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the Robots Exclusion Protocol, is a Standards Track RFC, so this is a specified format rather than a convention. Three properties are worth internalising.

**A user-agent string is unauthenticated.** RFC 9309 is explicit that the product token "SHOULD be a substring of the identification string that the crawler sends", and that string is a request header the client composes. It is not a credential. Forging `GPTBot` costs one line of code in any HTTP client. Such rules are declarations addressed to honest operators, and their value is that honest operators honour them — not that dishonest ones cannot bypass them. RFC 9309 also states plainly that the protocol "is not a form of access authorization", which is the same admission from the other direction.

**Named groups beat wildcards, and are harder to get right.** A group for a specific product token applies instead of the wildcard when it matches, per RFC 9309's matching rules. That means adding a group for a crawler you want to allow inside a wildcard `Disallow` is an act of precision — get the token wrong and the wildcard silently applies instead. The same RFC specifies longest-match for paths, so a narrow `Allow` can override a broader `Disallow` only if it is more specific.

**The wildcard is a policy, and it should be documented as one.** `User-agent: *` is not a leftover. If it disallows everything except a short allowlist, say so in a comment and link to a page explaining the reasoning, because most crawlers will hit that rule.

Content Signals go further by separating *collection* from *use*. Cloudflare's [Content Signals Policy](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) documentation defines three categories carried in a `Content-signal` line inside robots.txt: `search` for building a search index and returning links and short excerpts, `ai-input` for feeding content into a model at request time — retrieval, grounding, generative answers — and `ai-train` for training or fine-tuning. Cloudflare's managed file ships `Content-signal: search=yes, ai-train=no` for its default stance, and documents that a missing signal states neither permission nor restriction. Google Search Console may report `Syntax not understood` for these directives, which Cloudflare attributes to a parser not recognising a newer line rather than to a problem with the line.

One caution on provenance: Content Signals is a published policy convention, not an IETF standard. The IETF's [AI Preferences working group](https://datatracker.ietf.org/wg/aipref/about/) is standardising a vocabulary for AI-related preferences and how to attach them to content, including via robots.txt and HTTP header fields. Until those documents are published as RFCs, treat any `Content-signal` directive as a convention with real deployments and no standards-track guarantee. [The Content-Signal check](/checks/content-signal) asks whether the directive is present and parses, which is the strongest claim the evidence supports.

## The asymmetry that matters

Blocking training and blocking *retrieval on behalf of a user* are different decisions, and most sites conflate them because a single wildcard rule covers both.

Training is a bulk, offline, one-way use of your content whose value does not accrue to you and whose timing you do not control. Refusing it costs you nothing today; nobody arrives at your site because a model was trained on it.

Retrieval is a live request in service of a specific person trying to buy, book or find something. The agent is not consuming your content; it is reading it to act. Refusing retrieval means that when a customer asks an agent to compare your price against a competitor's, the agent reports that your site was unavailable and the competitor wins the comparison by default.

The mechanisms to express both exist and are not the same mechanism. `ai-train=no` with `search=yes` says "do not train on this, do index and link it." A named group for a retrieval crawler says "you specifically may fetch these paths." Treating them as one switch is the most common configuration error in this space — and it is the one Cloudflare's documentation sidesteps when it notes that mixed-use crawlers are excluded from the edge block applied to training crawlers, because a crawler that does both cannot be categorised by name alone.

## A worked robots.txt

Under 25 lines, covering a named training refusal, a wildcard policy documented in place, and the rationale published separately.

```txt
# Policy and rationale: https://example.com/ai-policy
# Training is refused. Retrieval on a user's behalf is permitted.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Everything else: read the public site, leave the rest alone.
User-agent: *
Content-signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /account/
Disallow: /cart/
Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=

Sitemap: https://example.com/sitemap.xml
```

Four things carry weight here. The comment on line one points at the page that explains the reasoning, so a crawler operator who disagrees can find whom to argue with. The wildcard group separates `ai-train` from `ai-input` rather than refusing both. The disallowed paths are the ones where crawling is either pointless or harmful — session-scoped URLs, parameter permutations that multiply one page into thousands, and anything behind a login. And the named groups come first, because RFC 9309 merges matching groups and a group that appears to be overridden is not overridden at all. That file is measured by [robots.txt with agent directives](/checks/robots-txt) and [named AI crawler rules](/checks/ai-rules), which look for exactly these structures.

## How to verify the rules do what you think

Do not read the file. Read what the server actually returns, because those are frequently different documents.

1. **Fetch it as a plain client** — not through a validation tool with its own opinion. A CDN may prepend a managed block to your robots.txt — Cloudflare's documentation describes merging its managed directives ahead of the origin file — and the result is what crawlers parse, not what your repository contains.
2. **Run the RFC 9309 algorithm on the paths you care about,** using the same product token each crawler will send. Longest match wins; if an `Allow` and a `Disallow` are equivalent in length, the `Allow` applies. This catches most misconfigurations, because most misconfigurations are a rule that does not match the path its author believed it matched.
3. **Request a disallowed path with the exact user-agent string.** A `200` where you expected a refusal means the rule is not doing what you think. It also tells you which layer enforces: if the origin serves the page and your edge blocks it, robots.txt is decorative and your edge rules are the real policy.
4. **Request a permitted path with the same string.** A `403` on a path you meant to allow is the failure that costs you the agent traffic you wanted. Test it the same way you test the block.
5. **Publish the human-readable rationale** on a page a crawler operator can find, and describe there how a claimed identity is verified — that is the difference between [an agent identity page](/checks/bot-page) and a list of names.

## What remains unenforceable

Two honest limits. First, robots.txt is advisory: a crawler that ignores it faces no protocol-level consequence, which RFC 9309 concedes by not being an access-control mechanism. Second, a user-agent rule cannot authenticate, so any policy that depends on a name being honest works only against operators who choose to be honest. If you need enforcement against an operator who has decided not to be, the options are edge rules, rate limits, or the signed-agent mechanisms that verify a key rather than a string — still drafts in the [Web Bot Auth working group](https://datatracker.ietf.org/wg/webbotauth/about/), with implementations that vary in what they accept and what they do with a valid signature.

The defensible position is to separate the decisions and choose an instrument for each. Refuse training by name and by signal. Refuse bulk crawling by rate and by path. Keep retrieval on a user's behalf open, because it is the traffic that arrives with a customer attached. And where a site is blocked at the network layer, report that as not measured rather than scoring it.