# Abuse and removal > What the scanner requests, how often, and why robots.txt does not stop it. How to block it, have a stored record removed, and report a vulnerability. - Source: /abuse - Site: AgenticWebScore - Language: English --- This page is for the owner of a site this service has measured. It states what the scanner requests, at what rate, and how to block it. It says how to have a record or a listing removed, and how to report a vulnerability. ## What the scanner requests One scan asks for at most 136 URLs on the scanned site. This table lists every source and its cap. _The URLs one scan can ask for_ | Source | URLs at most | | --- | --- | | Fixed probe paths, the same for every site | 103 | | `/` again, asking for markdown | 1 | | Skill artifacts that a skills index lists | 10 | | Policy and Canonical URLs in `security.txt` | 2 | | Links in `llms.txt` | 6 | | GET operation paths in an OpenAPI description | 8 | | Remote transports in an MCP server card | 3 | | Documentation pointers on the homepage | 2 | | Authorization server metadata named by OAuth protected resource metadata | 1 | | Total | 136 | Every source after the second is a URL the site names in its own documents, on the same origin. A URL already requested is not requested again, so a scan usually makes fewer. The ceiling counts URLs, not requests. A URL that redirects costs 1 more request per hop, for at most 5 hops, so one URL costs at most 6 requests. In all, one scan sends at most 816 HTTP requests to the site and the hosts it redirects to. The DNS questions described below are not part of that count. The scanner requests public addresses only. A target that resolves to a private, loopback, link-local, or metadata address is refused before any request is made. This build also refuses these targets before any request is made: - An IP address as the target, or as a redirect target. A Worker can request a hostname only. - An explicit port Cloudflare does not proxy. It proxies 80, 8080, 8880, 2052, 2082, 2086, and 2095 for `http`, and 443, 2053, 2083, 2087, 2096, and 8443 for `https`. This does not defeat DNS rebinding. Cloudflare resolves the name again when it connects, after our check. Cloudflare documents an outbound filter that reaches public destinations only, without a list of ranges. How that filter answers a name that resolves to a non-public address is not measured. Redirects are followed by hand, to another host too, and every hop is resolved and refused if it is private. A response is read to 3 MB and abandoned after 12 s. Every request says who it is: _Scanner request headers_ ```text User-Agent: AgenticWebScore/0.3 (+https://agenticwebscore.com/bot) From: hello@agenticwebscore.com Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7 Accept-Language: en ``` Cloudflare adds headers of its own to each request. [`/bot`](/bot) lists them, and says which are unverified. The report at `/aws/` lists every URL the scanner asked your site for, with the status, byte count, and time of each. A redirect hop is not a row of its own. `report.json` gives the redirect requests each row cost, as `hops`. A scan also asks DNS about your hostname. Every lookup goes to `cloudflare-dns.com` over DNS-over-HTTPS, including the check of each host before it is requested. It also receives 4 DNS-AID questions: SVCB and TXT for `_index._agents.` and `_agent.`. Cloudflare's network resolves the host again when it connects. ## At what rate Up to 5 requests are in flight at once, and there is no delay between them. A scan makes no new request after 60 s. A scan cut at that point is reported as not measured, not scored on part of the site. Service-wide, 3 scans run at once, and one client may start 10 per minute. Over either ceiling the API answers 429 with a `Retry-After` header. Nothing re-scans a site on its own. A scan starts only when someone asks for one, through the scan form, the API, or a badge claim. Such a request reuses a record younger than 6 hours, unless it asks for a fresh scan. A request that declares a site profile reuses only a record measured under that same declared profile. A request that declares none reuses only a record whose profile was detected. A report URL shows the stored record whatever its age, and never starts a scan. A crawler that fetches `/aws/` sends nothing to your site. ## robots.txt The scanner fetches `/robots.txt` as one of its probe paths and reads it to measure your policy. No rule in that file changes what it requests. A `Disallow` line for every agent will not stop this scanner. That is a gap, and it is published here rather than hidden. ## How to block the scanner - Refuse requests whose `User-Agent` is `AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)`. The scanner sets it on every request and does not rotate it. - Requests leave from Cloudflare's network, not from one server. This service publishes no list of the addresses it scans from, so the user-agent rule is the one that works today. - On a site behind Cloudflare, a WAF custom rule with the expression `cf.worker.upstream_zone eq "agenticwebscore.com"` matches a request from a Worker on that zone. Whether the scanner's requests carry that zone is not measured yet. - Cloudflare documents that a site on another Cloudflare zone sees `2a06:98c0:3600::103` in `CF-Connecting-IP` for a subrequest from a Worker. What a site sees for this scanner's requests is not measured. The address is the same for every Worker, so it does not identify this scanner. - A homepage that answers 401, 402, 403, 406, 429, or 503 is recorded as not measured: `score: null`, no grade, and a reason. So is a homepage that does not answer at all. - A blocked host cannot be listed. A claim is refused when the host has no score to publish, so a block is a complete answer. > A block is not a finding. A site that refuses the scanner is not scored low for it. The refusal is recorded as a property of our vantage point rather than of the site. ## Request removal of a stored record Send the hostname and, if you have it, the fingerprint. Say whether you want the scan record deleted, the listing withdrawn, or both. Write to [hello@agenticwebscore.com](mailto:hello@agenticwebscore.com). Removal lasts. The operator adds your hostname to the removal list, the `suppressed` document in the object's storage. Then the operator deletes its scan records, the `records` rows keyed by the host or by the host and a port. The entry covers every port and scheme of that hostname. A host on the list is not scanned again. Its report page says it is on the removal list. It leaves the directory, and its badge shows no score. Deleting the records removes the report, the fingerprint, and the evidence together, because the fingerprint is derived from the record. A copy of the scan still held in memory is never shown, and the next request for the host drops it. A deleted record stays in Cloudflare's point-in-time recovery for 30 days. This service does not read that copy. No route writes the removal list. The operator edits it by hand, in the Cloudflare dashboard, and it keeps the hostname and the time of the request, with no expiry. A listing is withdrawn by removing the meta tag. The next verification that reaches your homepage and finds no token withdraws it. It stops being published 30 days after the last confirmation either way. A claim record is 1 entry in the `badges` document, holding the token and the verification times. It is deleted on request, and the token stops working when it is gone. ## A score you disagree with Every check publishes the request it makes and a command to reproduce it. Run it against your own site and compare the result with the evidence line. A wrong measurement is corrected by a new scan, not by an edit to a stored one. The stored record keeps the time and the rubric version it was measured under. If the site has changed since the scan, press Re-scan now on its report. The new record replaces the old one at the same address. ## How a request is handled There is no automated removal endpoint. A request goes to a person at the contact address, and it is handled by hand. Without anyone acting, nothing re-scans a site, a listing lapses 30 days after its last confirmation, and a token expires after 365 days. ## Reporting a vulnerability Report a security problem in this site, its API, or its scanner to [hello@agenticwebscore.com](mailto:hello@agenticwebscore.com). The same contact is published in `/.well-known/security.txt`. - In scope: the pages and the API of this site, and the scanner's behaviour toward the sites it measures. A way to make the scanner request a private address is in scope. - Out of scope: the measured sites themselves. Report a problem with a measured site to its owner. - What to send: the URL or endpoint, the steps that reproduce the problem, and what you observed. A request and its response are enough. - Test against a host you control. The scanner sends real requests to any public target it is given. A report is handled like a removal request, as [How a request is handled](#response-time) describes. No reward is offered.