robots.txt with agent directives

What is measured

The scanner fetches https://<domain>/robots.txt ($robots in the probe map) and parses it per RFC 9309, where a group is a User-agent line followed by its Allow/Disallow rules. Pass = HTTP 200 with a parseable file that names at least one agent crawler as its own group and declares a Sitemap. Partial = the file parses but contains only a blanket User-agent: * group, or places rules before any User-agent line. Fail = 404 or 5xx, a non-text response, or HTML returned for the path by a catch-all route.

Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

              DOMAIN=example.com
curl -sSi "https://$DOMAIN/robots.txt" | sed -n '1,40p'
            

The fix

This is the artifact to ship. It is the complete change, not a description of one.

              # https://example.com/robots.txt
# RFC 9309: a group is a User-agent line followed by its rules.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /search?

# Name the agent crawlers you welcome, so agent traffic is routed deliberately.
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml
            

Standards referenced

• RFC 9309

Related checks