How to make your website agent-ready: a 14-step checklist with a curl command for each

To make a website agent-ready, fix what a plain HTTP client receives, starting with a homepage that answers 200. Then work down the list: robots.txt, a sitemap, headings, structured data, caching and a real 404. Last come the files only agents read, such as markdown and /llms.txt, and, for a site with an API, an MCP server.

Each of the 14 steps below gives one curl command, the check that measures it and a primary source. The steps run from what every agent needs to what only a site with an action surface needs.

How to run the commands

Most steps also run on their own from the tools page, which shows the evidence without a terminal.

Set your origin once, with no trailing slash. Every command reads $ORIGIN.

ORIGIN=https://example.com

The check pages send the scanner's User-Agent and Accept-Language headers. These commands leave them out, so a site that treats a scanner differently can answer differently here. The output blocks come from this site on 27 October 2026, trimmed to the lines that matter.

How the steps are ordered

Steps 1, 2, 4, 6 and 7 are web standards that hold levels 1 to 3 of the scanner's L0–L5 scale. Steps 3, 10 and 11 are agent conventions, which earn points but hold no level. Steps 12 to 14 matter for levels 4 and 5.

Step 1: Serve a readable homepage

Answer / with a 2xx status and HTML that holds at least 25 words of visible text. A page that scripts fill in after load reads as empty to a client that runs none. A temporary redirect from the apex to www scores a partial. Google treats a permanent redirect as its strongest canonical signal, and does not use a temporary one as a canonical signal.

Check: reachable, essential tier, level 1. Source: Google on redirects.

curl -sSL -o /dev/null -w 'status=%{http_code} final=%{url_effective} redirects=%{num_redirects} type=%{content_type} bytes=%{size_download}\n' "$ORIGIN/"

status=200 final=https://agenticwebscore.com/ redirects=0 type=text/html; charset=utf-8 bytes=62719

Step 2: Publish robots.txt

Serve /robots.txt as plain text at the root of the host. RFC 9309 lets a crawler fetch anything when the file answers 4xx, and has it assume a complete disallow on 5xx. Google says it stops crawling for the first 12 hours of a 5xx and keeps retrying the file. Declare your sitemap in it.

Check: robots.txt, essential tier, level 2.

curl -sSL -D - "$ORIGIN/robots.txt" | grep -iE '^(HTTP|content-type|user-agent|allow|disallow|sitemap)'

HTTP/2 200
content-type: text/plain; charset=utf-8
User-agent: *
Disallow: /api/
Allow: /
User-agent: GPTBot
Disallow: /api/
Allow: /

Step 3: Name the AI crawlers you have a policy for

A crawler follows the one group that names it and skips the * group. Google says the two kinds of group are not combined. A named group therefore repeats every * rule you want it to keep.

This site's record of 25 October 2026 shows 17 of the 18 tokens the scan resolves with a group of their own. Whether to block a token is your decision, and a block earns no points. The token list, blocking training while keeping search and Content-Signal have their own articles, and so do OpenAI's and Anthropic's crawlers.

Check: ai-rules, an agent convention, recommended tier. Source: RFC 9309 and the Google page above.

curl -sSL "$ORIGIN/robots.txt" | grep -Ein '^[[:space:]]*(user-agent|allow|disallow)[[:space:]]*:'

Step 4: Publish a sitemap

List the URLs you want crawled in /sitemap.xml, as absolute URLs, and add a Sitemap: line to robots.txt. One file holds up to 50 MB uncompressed or 50,000 URLs. Google calls a submission a hint, and uses lastmod only when it is consistently accurate. A missing sitemap scores a partial, not a fail.

Check: sitemap-xml, essential tier, level 2. Source: Google on sitemaps.

curl -sSL -H 'Accept: application/xml, text/xml;q=0.9, */*;q=0.5' "$ORIGIN/sitemap.xml" | grep -Eo '<(urlset|sitemapindex|sitemap|loc|lastmod)[^>]*>[^<]*' | head -6

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<loc>https://agenticwebscore.com/
<lastmod>2026-10-25

Step 5: Name one canonical URL

Put a self-referencing, absolute rel="canonical" link in the head. Google calls canonical tags signals, not directives, and advises against relative paths. The scan passes an absolute, https URL that names the page it reached, and scores a relative href as a partial.

Check: canonical, recommended tier. Source: Google on duplicate URLs.

curl -sSL "$ORIGIN/" | grep -Eio '<link[^>]+canonical[^>]*>'

<link rel="canonical" href="https://agenticwebscore.com/">

Step 6: Put a title and headings in the HTML

Use one <h1>, a descriptive <title> and ranks that do not skip. The W3C tutorial says skipping ranks should be avoided where possible. The headings check notes a skipped level and charges nothing; it fails only a page with no heading at all. The no-js-fallback check fails a page with under 100 characters of visible text outside scripts.

Checks: headings, meta and no-js-fallback, all essential tier, level 3.

curl -sSL "$ORIGIN/" | perl -0777 -ne 's{<!--.*?-->}{}gs; s{<(script|style|template)\b[^>]*>.*?</\1\s*>}{}gsi; while (m{<(h[1-6])\b[^>]*>(.*?)</\1\s*>}gsi) { my ($l,$t)=(lc($1),$2); $t=~s/<[^>]*>//g; $t=~s/\s+/ /g; print "$l ",substr($t,0,60),"\n" }' | head -3

h1  See your site the way agents do.
h2 What this site publishes for agents
h2  What the directory lists

Step 7: Add structured data

Describe the page's entity in JSON-LD. Google recommends JSON-LD and says all 3 formats are equally fine when valid. Its guide to generative AI features says no special schema.org markup is needed for them. So this step serves every client that reads structured data, not one feature.

Check: jsonld, essential tier, level 3. Source: Google on structured data.

curl -sSL "$ORIGIN/" | perl -0777 -ne 'while (m{<script[^>]*application/ld\+json[^>]*>(.*?)</script>}gsi) { print "$1\n" }' | jq -c '[(.. | objects | select(has("@type")) | {type: .["@type"], name: (.name // .headline // null)})][:4]'

[{"type":"Organization","name":"AgenticWebScore"},{"type":"ContactPoint","name":null},{"type":"WebSite","name":"AgenticWebScore"},{"type":"SoftwareApplication","name":"AgenticWebScore"}]

Step 8: Send a validator with the page

Send an ETag or a Last-Modified date. A crawler that holds a copy can then ask whether the page changed, and get a 304 with no body. Google's crawlers support both, prefer the ETag, and use it when a response has both. Google says other caching directives are not supported, and HTTP caching for crawlers covers the CDN pitfalls.

Check: http-caching, recommended tier. Source: Google's crawler overview.

ETAG=$(curl -sS -o /dev/null -D - "$ORIGIN/" | tr -d '\r' | sed -n 's/^[Ee][Tt][Aa][Gg]: *//p' | tail -1)
curl -sS -o /dev/null -w 'status=%{http_code}\n' -H "If-None-Match: $ETAG" "$ORIGIN/"

status=304

Step 9: Answer 404 for pages that do not exist

A missing URL answers 404 or 410. A 2xx page that says "not found" is a soft 404, and Google says it does not index URLs that return a 4xx. In the scan records of 20 October 2026, vercel.com answered a random path with 200 after a redirect to its login page. The other 4 sites of the October comparison answered 404.

Check: not-found-status, recommended tier. Source: Google on status codes.

MISSING="/aws-probe-$(head -c 6 /dev/urandom | od -An -tx1 | tr -d ' \n')"
curl -sS -o /dev/null -w 'status=%{http_code} location=%{redirect_url}\n' "$ORIGIN$MISSING"

status=404 location=

Step 10: Serve markdown on request

Return markdown when a client sends Accept: text/markdown, and add Vary: Accept so a cache keeps the two versions apart. Google's guide says you do not need markdown files for Search, which does not use them, so this step serves other agents. Cloudflare can convert HTML at its edge on Pro, Business and Enterprise plans. The markdown negotiation guide has the details.

Check: markdown-negotiation, an agent convention, recommended tier. Source: Cloudflare's documentation.

curl -sSL -H 'Accept: text/markdown, text/html;q=0.5' -D - -o /dev/null "$ORIGIN/" | grep -iE '^(HTTP/|content-type|vary)'

HTTP/2 200
content-type: text/markdown; charset=utf-8
vary: Accept

Step 11: Publish /llms.txt

Serve a markdown index at /llms.txt: the llmstxt.org proposal requires an H1 with the site name, and allows a blockquote summary and H2 sections of links. Google's guide says Google Search ignores these files. The scan requests up to 6 same-origin links the file names, and a link that answers an error makes the file "served with an error". What llms.txt is and what it is not covers who reads it.

Check: llms-txt, an agent convention, recommended tier. Source: the llmstxt.org proposal above.

curl -sSL -D - "$ORIGIN/llms.txt" | grep -iE '^(HTTP/|content-type:|#)' | head -4

HTTP/2 200
content-type: text/plain; charset=utf-8
# AgenticWebScore
## Core

Step 12: Offer an MCP server or a documented API

Do this only if your site lets an agent act. An MCP server exposes tools that a client lists with tools/list and calls with tools/call, according to the specification. The scan opens one session with 3 POST requests and calls no tool. A documented API, meaning an OpenAPI file and reachable docs, can hold level 4 instead.

Check: mcp-server, essential tier, asked only of a site where the scan saw an API or agent surface. The full handshake is in testing an MCP server with curl.

curl -sSL -o /dev/null -w '%{http_code} allow=%header{allow}\n' "$ORIGIN/mcp"

405 allow=POST

A plain GET answered 405 with Allow: POST is what a running endpoint gives. It is not a failure.

Step 13: Describe the server in a card

A server card tells an agent the endpoint, version and transport before it connects. SEP-2127, which has Final status, lets a card sit at any unreserved URI, and lets an AI catalog at /.well-known/ai-catalog.json list it. The scan tries 4 paths, and the paths are still moving. The server card article covers them. The manifest that lists cards is moving from ai-catalog.json to ard.json; ard.json and ai-catalog.json explained covers both.

Check: mcp-card, unscored. A server found through its card or the AI catalog counts toward level 5.

for p in /.well-known/ai-catalog.json /.well-known/mcp/server-card.json /.well-known/mcp.json /mcp.json; do printf '%s -> ' "$p"; curl -sSL -o /dev/null -w '%{http_code} %{content_type}\n' -H 'Accept: application/json' "$ORIGIN$p"; done

/.well-known/ai-catalog.json -> 404 application/problem+json
/.well-known/mcp/server-card.json -> 200 application/json; charset=utf-8
/.well-known/mcp.json -> 404 application/problem+json
/mcp.json -> 404 application/problem+json

Step 14: Publish a status page and a changelog feed

A status page says whether the service is up. A feed says what changed. RFC 8631 defines the status link relation for a resource that reports a service's status. The scan wants both halves, and a feed whose newest entry is at most 180 days old.

Check: status-page, recommended tier, needed for level 5.

for p in /status /changelog; do printf '%s -> ' "$p"; curl -sSL -o /dev/null -w '%{http_code} %{content_type}\n' "$ORIGIN$p"; done

/status -> 200 text/html; charset=utf-8
/changelog -> 200 application/atom+xml; charset=utf-8

What this checklist does not cover

It covers what a plain HTTP client receives. An agent that drives a browser also reads the accessibility tree, and may call tools a page registers. web.dev's guide covers semantic HTML, labels and layout stability. The article on WebMCP and MCP covers page-registered tools, and Lighthouse's agentic audits show where a browser-based tool and this scanner overlap.

The scan runs no JavaScript. A homepage whose text only scripts add scores as empty, or is reported as not measured when it is a bare shell. By Google's account, steps 10 and 11 do not affect Search visibility. They serve the agents that read those files.

To run every check on your own site and get the evidence for each, scan your site. The methodology lists the weights, and the CI page shows how to run the scan on every deploy.