# robots.txt for AI agents: rules, crawl-delay and AI crawlers

> robots.txt with agent directives: The front door for agent traffic (RFC 9309). It is an essential-tier check in the Discoverability dimension.

- Source: /checks/robots-txt
- Site: AgenticWebScore
- Language: English

---
The front door for agent traffic (RFC 9309).

| Property | Value |
| --- | --- |
| Check id | `robots-txt` |
| Dimension | Discoverability (20%) |
| Tier | Essential — pool 80 pts |
| Weight inside tier | 2 |
| Impact | High |
| Maturity | Adopted — A ratified standard with real deployment. |
| Conditional | No |
| Publishing skill | `robots-txt` |
| Standards | RFC 9309 |

## What is measured

The scanner fetches https://<domain>/robots.txt ($robots in the probe map) and parses it per RFC 9309, where a group is a User-agent line followed by its Allow/Disallow rules. Pass = HTTP 200 with a parseable file that names at least one agent crawler as its own group and declares a Sitemap. Partial = the file parses but contains only a blanket User-agent: * group, or places rules before any User-agent line. Fail = 404 or 5xx, a non-text response, or HTML returned for the path by a catch-all route.

## Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

```bash
DOMAIN=example.com
curl -sSi "https://$DOMAIN/robots.txt" | sed -n '1,40p'
```

## The fix

This is the artifact to ship. It is the complete change, not a description of one.

```text
# https://example.com/robots.txt
# RFC 9309: a group is a User-agent line followed by its rules.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /search?

# Name the agent crawlers you welcome, so agent traffic is routed deliberately.
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml
```

> Rules must follow their own User-agent line - a Disallow placed before any User-agent is ignored - and robots.txt is only read at the root of the host, never at a subpath.

## Standards referenced

- `RFC 9309`

## Related checks

| Check | Tier | Impact | Same skill |
| --- | --- | --- | --- |
| `reachable` — Homepage open to agent access | Essential | Critical | — |
| `sitemap-xml` — sitemap.xml valid and current | Essential | High | — |
| `llms-txt` — /llms.txt published and well-structured | Essential | Critical | — |
| `link-headers` — Discovery via Link response headers | Emerging | Medium | — |
| `dns-aid` — DNS-AID discovery record | Emerging | Medium | — |
| `canonical` — Canonical and hreflang discipline | Recommended | Medium | — |
| `llms-full` — /llms-full.txt expanded context | Recommended | Low | — |
| `agents-md` — AGENTS.md with the required sections | Recommended | High | — |