# AI crawler access: is your homepage open to agents?

> Homepage open to agent access: If an agent cannot read your site, every other check is moot. The fix: open agent access (clear the 403/429 block).

- Source: /checks/reachable
- Site: AgenticWebScore
- Language: English

---
If an agent cannot read your site, every other check is moot. Cloudflare began blocking AI crawlers by default in 2025, with signed agents routed separately.

| Property | Value |
| --- | --- |
| Check id | `reachable` |
| Dimension | Discoverability (20%) |
| Tier | Essential — pool 80 pts |
| Weight inside tier | 3 |
| Impact | Critical |
| Maturity | Adopted — A ratified standard with real deployment. |
| Conditional | No |
| Publishing skill | `agent-access` |
| Standards | — |

## What is measured

The scanner sends one GET to https://<domain>/ - the homepage is the only source for this check - using a non-browser agent user agent, and records the status code, Content-Type and body size. Pass = a 2xx HTML response with a real body. Partial = a 2xx response that is not HTML, a body smaller than roughly 1 KB or with almost no visible text, or a redirect chain that leaves the host before reaching content. Not measured = 401, 402, 403, 406, 429 or 503, no response, a JavaScript-only shell, or a scan cut at the whole-scan deadline: the whole scan is then reported as not measured, with no score. A refusal is a fact about the scanner's vantage point, not about the site. Fail = any other non-2xx status, such as 404, 451 or another 5xx.

## Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

```bash
DOMAIN=example.com
curl -sS -o /dev/null -w 'status=%{http_code} type=%{content_type} bytes=%{size_download} redirect=%{redirect_url}\n' -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' "https://$DOMAIN/"
```

## The fix

This is the artifact to ship. It is the complete change, not a description of one.

```text
# nginx: clear the 403/429 that agent user agents hit on the homepage.
# $bot_block and the rate limit come from your existing config; the allow-list
# below is evaluated first so agents are never sent down the block path.
map $http_user_agent $agent_allow {
    default                                                          0;
    "~*(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot)"  1;
    "~*(Google-Extended|Applebot-Extended|CCBot|Amazonbot)"          1;
}

server {
    listen 443 ssl;
    server_name example.com;

    location = / {
        if ($agent_allow) { set $bot_block 0; }
        if ($bot_block)   { return 403; }
        try_files /index.html =404;
    }
}
```

> If a CDN or WAF sits in front of nginx - for example Cloudflare's default AI-crawler block - the allow decision must be made there too, because nginx never sees a request the edge rejects.

## Related checks

| Check | Tier | Impact | Same skill |
| --- | --- | --- | --- |
| `robots-txt` — robots.txt with agent directives | Essential | High | — |
| `sitemap-xml` — sitemap.xml valid and current | Essential | High | — |
| `llms-txt` — /llms.txt published and well-structured | Essential | Critical | — |
| `link-headers` — Discovery via Link response headers | Emerging | Medium | — |
| `dns-aid` — DNS-AID discovery record | Emerging | Medium | — |
| `canonical` — Canonical and hreflang discipline | Recommended | Medium | — |
| `llms-full` — /llms-full.txt expanded context | Recommended | Low | — |
| `agents-md` — AGENTS.md with the required sections | Recommended | High | — |