HTTP caching of the homepage

Sources

• RFC 9111: HTTP Caching, IETF, June 2022
• RFC 9110, 13: conditional requests, IETF
• Google: HTTP caching for crawlers, Google Search Central, December 2024
• Bing Webmaster Guidelines, §3: ETag and lastmod, Microsoft Bing

What is measured

The scanner reads the Cache-Control, ETag and Last-Modified headers of the homepage response ($home in the probe map). When the homepage sends an ETag or a Last-Modified date, the scanner requests the URL the homepage answered at once more, with If-None-Match set to the ETag and If-Modified-Since set to the date, each when it was sent, and reads at most 16 KB of the answer. A validator longer than 1,024 characters, or with a line break, is not sent. Pass = Cache-Control and a validator, and the conditional request answered 304 Not Modified. Partial = a validator, and the conditional request answered a status other than 304 that is not listed under Not measured; a validator and a 304 with no Cache-Control; or Cache-Control with no validator, and then no conditional request is made. Fail = none of the three headers. Cache-Control: private and no-store are policies like any other, and these rules apply to them: no-store with no validator is partial, because a crawler then requests the whole page each time. Not measured = a homepage that answered a status other than 2xx; a validator the scan did not send; or a conditional request that was not made, got no answer, or answered 401, 403, 407, 429 or a 5xx status, or 406, which refuses the scan's Accept header.

Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

              HOMEPAGE=https://example.com/
HEADERS=$(curl -sSL -o /dev/null -D - -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7' "$HOMEPAGE" | tr -d '\r')
printf '%s\n' "$HEADERS" | grep -iE '^(cache-control|etag|last-modified):'
ETAG=$(printf '%s\n' "$HEADERS" | sed -n 's/^[Ee][Tt][Aa][Gg]: *//p' | tail -1)
DATE=$(printf '%s\n' "$HEADERS" | sed -n 's/^[Ll][Aa][Ss][Tt]-[Mm][Oo][Dd][Ii][Ff][Ii][Ee][Dd]: *//p' | tail -1)
curl -sSL -o /dev/null -w 'status=%{http_code}\n' -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7' -H "If-None-Match: $ETAG" -H "If-Modified-Since: $DATE" "$HOMEPAGE"
            

The fix

This is an example written for example.com. Where the scan can build the fix from your own site, the report shows that version instead.

              Cache-Control: <choose: a policy, for example no-cache or max-age=300>
ETag: "<choose: a value that changes when the page changes>"

            

Standards referenced

• RFC 9111
• RFC 9110

Related checks