Crawler and agent policy

How AgenticWebScore identifies itself to other sites, and how other agents are welcome to use this one.

How we identify ourselves as a client

When this service scans a domain it fetches only publicly reachable paths, and it says who it is on every request:

              User-Agent: AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)
From: hello@agenticwebscore.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7
Accept-Language: en
            

A document, such as /llms.txt or a path under /.well-known/, is requested with Accept: application/json, text/plain;q=0.9, text/markdown;q=0.9, */*;q=0.8 instead.

Headers Cloudflare adds

The scanner sets the 4 headers above. On Workers the platform adds the ones below.

Verified means workerd, the runtime the Worker runs on, was observed adding the header, and Cloudflare documents it. Unverified means no request has shown the header. Only a request from the deployed Worker settles an unverified header. CF-EW-Via and CDN-Loop may be added, and no value has been observed.

Where requests come from

Requests leave from Cloudflare's network, not from one server. Cloudflare documents that a site on another Cloudflare zone sees 2a06:98c0:3600::103 in CF-Connecting-IP for a subrequest from a Worker. What a site sees for this scanner's requests is not measured. The address is the same for every Worker, so it does not identify this scanner.

A site can block the scanner in these ways:

• Block by User-Agent: every request the scanner sends carries its own.
• On a site behind Cloudflare, a WAF custom rule with the expression cf.worker.upstream_zone eq "agenticwebscore.com" matches a request from a Worker on that zone. Whether the scanner's requests carry that zone is not measured yet.

What one scan requests

One scan asks for at most 136 URLs. Each source has its own cap, and the caps add up to that ceiling:

• A URL taken from the site's own documents must be on the scanned origin. It is requested with a GET, as written, query string included.
• The ceiling is a bound, not a count: a URL a document names is skipped when the scan has already requested its path.
• A URL that redirects costs 1 more request per hop, for at most 5 hops, so one URL costs at most 6 requests.
• A hop may lead to another host. The address of each hop is checked before it is requested, as the first one is.
• This does not defeat DNS rebinding. Cloudflare resolves the name again when it connects, after our check. Cloudflare documents an outbound filter that reaches public destinations only, without a list of ranges. How that filter answers a name that resolves to a non-public address is not measured.
• In all, one scan sends at most 816 requests. A site that answers every URL without a redirect receives at most 136.
• No JavaScript is executed. Nothing is loaded that a plain HTTP client would not load.
• A 3 MB per-response cap, a 12-second timeout per URL, its redirects included, and at most 5 concurrent requests.
• A scan stops making requests after 60 seconds. A scan cut short is reported as not measured, not scored on part of the site.
• No credential is sent: no cookie, no Authorization header and no API key. A protected path answers the scanner as it answers any anonymous client.
• Private address ranges, cloud metadata endpoints and internal hostnames are refused outright.
• A scan also asks DNS about your hostname. Every lookup goes to cloudflare-dns.com over DNS-over-HTTPS, including the check of each host before it is requested. It also receives 4 DNS-AID questions: SVCB and TXT for _index._agents.<host> and _agent.<host>. Cloudflare's network resolves the host again when it connects.
• At most 500 scan records are kept. The oldest are dropped first once the store is full, and never a record a directory listing points at.

This build also refuses these targets before any request is made:

• An IP address as the target, or as a redirect target. A Worker can request a hostname only.
• An explicit port Cloudflare does not proxy. It proxies 80, 8080, 8880, 2052, 2082, 2086, and 2095 for http, and 443, 2053, 2083, 2087, 2096, and 8443 for https.

What differs with the vantage point

The scanner stands in Cloudflare's network, and some results depend on where it stands.

• A site can block or challenge every request from a Worker with one rule. A homepage that answers 401, 402, 403, 406, 429, or 503 is recorded as not measured. Whether Cloudflare treats requests from a Worker differently from requests from one server is not measured.
• A site behind Cloudflare may answer from its cache, as it does for a visitor. The request headers ask for no-cache.
• Every scan leaves from one location, the Cloudflare location of the object that runs the scans. A redirect, a consent wall, or a language choice that depends on where the visitor is reflects that location.
• The milliseconds in a record are measured from Cloudflare's network to the site.
• Whether a request goes out over IPv4 or IPv6 is not measured. A site that works over only one of them may measure differently.
• The runtime cannot check an Ed448 signature. A signed agent card that uses one is reported as unverified, and the report names the limit.
• The Workers build sends fewer requests at once than the Node build, because a Durable Object limits the connections that wait for a response.

What is kept about your site

A scan stores a record of what it observed on your site, and publishes it at /aws/<domain>. Of the 9 records this service keeps, 6 are about the sites that were scanned. Read what is stored for each one: where it lives and how long it stays.

The record holds public text your site served, including the homepage title and meta description. It can hold a contact address or telephone number that your site publishes. To have it removed, and kept removed, read how removal works.

What agents may do here

This site is designed to be read by machines. You do not need permission to fetch these paths, and you do not need to parse HTML to understand them.

A published surface is one of the 22 paths below: 17 machine-readable documents and 5 HTML policy pages. /index.json and /sitemap.md count the same 22 paths.

• /llms.txt — Structured summary for language models
• /llms-full.txt — Every page in the page registry and every learn article, in one markdown document
• /AGENTS.md — Build, test and contribution contract for coding agents
• /sitemap.md — Every page as a markdown link list
• /sitemap.xml — Machine sitemap with lastmod
• /index.json — Typed index of every page and surface
• /rubric.json — The complete rubric, machine-readable
• /robots.txt — Named AI crawler rules plus Content-Signal
• /ai.txt — Plain-language access declaration
• /auth.md — How an agent authenticates (and what is public)
• /ai-policy — Crawling, training, commercial use, contact
• /bot — Bot identity, rates and contact
• /privacy — What is stored, how long it stays, and what is not collected
• /terms — What a measurement is, what the API allows, and what the badge licence grants
• /abuse — What the scanner requests, how to block it, and how to request removal
• /.well-known/agent-permissions.json — Allow, deny and human-approval rules
• /.well-known/mcp/server-card.json — A declaration only: the planned transport and tools are recorded in x-planned fields, no /mcp endpoint is running, and the scanner reads the card as partial rather than as a server an agent can call
• /.well-known/api-catalog — RFC 9727 catalog of callable APIs. Its targets are absolute URLs on the canonical origin, so a scan of any other host reports them as declared rather than observed, and the scanner scores it partial
• /openapi.json — OpenAPI description of the API endpoints that answer today
• /.well-known/agent-skills/index.json — 60 publishable skill packs covering all 61 checks
• /.well-known/security.txt — RFC 9116 vulnerability disclosure
• /.well-known/dns-aid.json — The DNS-AID records this domain declares, documented in this file and not yet in the zone

A scan result is a path, not a query: /aws/<domain> is the report, or the progress screen while the scan behind it runs. A GET does not start a scan; with no stored record, the page offers a form that does.

The report's downloadable documents are under /aws/<domain>/<file>: status.md, agent-brief.md, bundle.md, prompt.md and report.json. The older /report/<domain> addresses answer a permanent redirect (308) to the same place, so an existing link or fingerprint still resolves.

Rates

Documents have no rate limit, and neither does any page except /verify/<fingerprint>. robots.txt declares no crawl delay. These are limited:

• Scans: 10 per minute per client, and 3 running at once across all clients.
• /api/health: 30 requests per minute per client.
• Badge claims: 10 per minute per client.
• Badge verifications: 10 per minute per client, and 3 per token in any 10 minutes.
• /verify/<fingerprint>, a page: 30 lookups per minute per client. Input that is not a fingerprint spends none.

A scan request answered by a stored record younger than 6 hours spends none of the scan budget. Neither does one that joins a scan already running. A request that declares a site profile reuses only a record measured under that same declared profile. A request that declares none reuses only a record whose profile was detected.

A request over a limit gets HTTP 429 and a Retry-After header in seconds.

To read many measurements, fetch each report's report.json rather than its HTML page.

Identity, not user agents

We do not run a bot-detection wall here, and we do not maintain an allowlist of user-agent strings, because user-agent strings are trivially forged and prove nothing. A signed agent that presents a Web Bot Auth key directory is verifiable; a string that says "GPTBot" is not. This site does not publish a key directory yet — that is the next identity milestone — and it would prefer to read yours.

Contact

• General and abuse reports: hello@agenticwebscore.com
• Security disclosures: see /abuse#disclosure and /.well-known/security.txt
• Machine-readable policies: /ai.txt, /auth.md, /.well-known/agent-permissions.json

Known gaps

A site that measures others for hidden gaps does not get to hide its own. As of rubric v0.7, none of these 15 checks pass on this site.

Most fail outright. A few are partial, not measured, or out of scope for a site that is not a merchant, and the reason says which. Each one stays listed until its reason stops being true.

sandbox-environment and rate-limits-declared are profile checks. They are in scope because the scanner assigns this site the API / developer platform site profile.

A profile needs 3 points and a lead of 2 over every other profile. This site gets 5 points from 2 signals: /openapi.json responds (3 points) and the homepage text contains rate limit (2 points). Without any one of them the scanner identifies no profile, and the 2 profile checks are out of scope.

Checks that depend on the origin

These 2 checks compare absolute URLs on the origin this site's canonical links name with the host that was scanned. A scan at another address, such as a local build, reports them as not passing.