Homepage open to agent access

What is measured

The scanner sends one GET to https://<domain>/ - the homepage is the only source for this check - using a non-browser agent user agent, and records the status code, Content-Type and body size. Pass = a 2xx HTML response with a real body. Partial = a 2xx response that is not HTML, a body smaller than roughly 1 KB or with almost no visible text, or a redirect chain that leaves the host before reaching content. Not measured = 401, 402, 403, 406, 429 or 503, no response, a JavaScript-only shell, or a scan cut at the whole-scan deadline: the whole scan is then reported as not measured, with no score. A refusal is a fact about the scanner's vantage point, not about the site. Fail = any other non-2xx status, such as 404, 451 or another 5xx.

Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

              DOMAIN=example.com
curl -sS -o /dev/null -w 'status=%{http_code} type=%{content_type} bytes=%{size_download} redirect=%{redirect_url}\n' -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' "https://$DOMAIN/"
            

The fix

This is the artifact to ship. It is the complete change, not a description of one.

              # nginx: clear the 403/429 that agent user agents hit on the homepage.
# $bot_block and the rate limit come from your existing config; the allow-list
# below is evaluated first so agents are never sent down the block path.
map $http_user_agent $agent_allow {
    default                                                          0;
    "~*(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot)"  1;
    "~*(Google-Extended|Applebot-Extended|CCBot|Amazonbot)"          1;
}

server {
    listen 443 ssl;
    server_name example.com;

    location = / {
        if ($agent_allow) { set $bot_block 0; }
        if ($bot_block)   { return 403; }
        try_files /index.html =404;
    }
}
            

Related checks