HTTP caching for crawlers: ETag, Last-Modified and 304

Send an ETag or a Last-Modified date with each page. A crawler that holds a copy sends it back in If-None-Match or If-Modified-Since, and an unchanged page gets 304 Not Modified with no body. Google's crawlers support both validators and use the ETag when a response has both. Bing's guidelines say an ETag helps Bing detect content changes more reliably.

Why Google and Bing use validators

Google's crawler overview says its crawling infrastructure supports heuristic HTTP caching through ETag with If-None-Match, and Last-Modified with If-Modified-Since. It recommends ETag for Google's crawlers, because an ETag has no date format to get wrong. It also says other HTTP caching directives are not supported. Support varies by crawler: Googlebot uses caching when it re-crawls for Search, and Storebot-Google only in certain conditions.

Google's December 2024 post puts a number on the problem. About 0.026% of Google's fetches were cacheable 10 years earlier, and 0.017% were at the time of the post. It asks servers to answer a matching If-None-Match with a 304 and no body, which saves the server from building the page and sending it.

Bing's Webmaster Guidelines make the same point in §3. Accurate lastmod values in a sitemap, and validators such as an ETag, help Bing detect content changes more reliably.

How a conditional request works

RFC 9110 sets 3 rules a server must follow:

• If-None-Match uses the weak comparison, so W/"abc" matches "abc".
• When a request carries both headers, the server must ignore If-Modified-Since. The RFC calls If-None-Match the more accurate condition.
• A 304 must carry the ETag, Cache-Control, Expires, Vary, Date and Content-Location fields a 200 would have carried. It cannot contain content.

Cache-Control: max-age is a separate signal. Google's overview calls it optional, and says a max-age helps a crawler decide when to fetch the URL again.

Which Cache-Control value to send with a validator

Cache-Control says how long a stored copy stays fresh. The validator says how to check a copy once it is stale. A crawler needs both answers, and the common values give different ones:

• max-age=3600 lets a cache reuse the page for an hour without asking. After that, it revalidates.
• max-age=0, must-revalidate makes every reuse a conditional request. With an ETag, the answer is a 304 whenever nothing changed. This site's homepage sends this.
• no-cache asks for the same. RFC 9111 says such a response must not be reused “without forwarding it for validation”.
• No Cache-Control at all leaves a cache to guess. RFC 9111 allows a heuristic lifetime only when no explicit expiration is present, and names 10% of the time since Last-Modified as a typical setting.

For Google's crawlers, only the validators and max-age matter, since the overview says other caching directives are not supported. Pick the max-age that matches how often the page changes, and send a validator either way.

How to test ETag and 304 with curl

The HTTP cache checker runs these requests on your site, together with the 404 and JSON error checks.

Use a GET. curl -I sends HEAD, and a crawler fetching a page sends GET. Print only the headers:

curl -sS -o /dev/null -D - https://agenticwebscore.com/ \
  | grep -Ei '^(HTTP|cache-control|etag|last-modified|vary)'

HTTP/2 200
cache-control: public, max-age=0, must-revalidate
etag: "e92a771d27ded3e3fadeea16d30608ec"
vary: Accept, Cookie

Send the ETag back the way a crawler does:

curl -sS -o /dev/null -D - \
  -H 'If-None-Match: "e92a771d27ded3e3fadeea16d30608ec"' \
  https://agenticwebscore.com/ | grep -Ei '^(HTTP|cache-control|etag)'

HTTP/2 304
cache-control: public, max-age=0, must-revalidate
etag: "e92a771d27ded3e3fadeea16d30608ec"

A server honours only the validator it issued. This site sends no Last-Modified, so a date gets a full answer:

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H 'If-Modified-Since: Sun, 25 Oct 2026 00:00:00 GMT' https://agenticwebscore.com/

200

For any site, read the ETag and send it back in 2 lines:

URL=https://example.com/
ETAG=$(curl -sS -o /dev/null -D - "$URL" | awk 'tolower($1)=="etag:" {sub(/^[^:]*: */,""); sub(/\r$/,""); print}')
curl -sS -o /dev/null -w '%{http_code}\n' -H "If-None-Match: $ETAG" "$URL"

A 304 means the validator works. A 200 means the server ignores it, or the ETag changed between the 2 requests.

CDN and framework pitfalls that break the 304

Compression weakens the ETag. Ask this site for gzip, and the same page comes back with a weak tag:

curl -sS -o /dev/null -D - -H 'Accept-Encoding: gzip' https://agenticwebscore.com/ \
  | grep -Ei '^(etag|content-encoding)'

etag: W/"e92a771d27ded3e3fadeea16d30608ec"
content-encoding: gzip

Cloudflare's ETag reference documents this. When it compresses an origin's uncompressed response, it returns a weak ETag, and some settings remove the strong tag instead. A weak tag still validates under the weak comparison: sent back here, it gets a 304. An origin that compares tags strongly, or a setting that drops the header, loses the 304.

No validator at all. A Cache-Control header with no ETag or Last-Modified gives a crawler nothing to send back. It downloads the whole page each time. Cloudflare's own homepage did this in its scan record of 20 October 2026: Cache-Control: max-age=10 and no validator.

An ETag that changes on every response. A tag built from the whole HTML changes whenever the HTML does. A per-request nonce, a timestamp or a session token in the page makes every tag new, and no request ever matches. Fetch the page twice and compare the 2 tags.

A validator on some representations only. A page negotiated on Accept is 2 or more representations, and each needs its own tag. Google's post gives the example of mobile and desktop versions under the same URL, each with its own ETag. On this site, the markdown version of the homepage, asked for with Accept: text/markdown, carried no ETag on 25 October 2026.

A 200 to every conditional request. A server can send an ETag and still answer every If-None-Match with the full page. The 2-line test above catches it, and the check below gives it a partial pass.

A Last-Modified date in the wrong format. Google asks for the HTTP date format, such as Fri, 4 Sep 1998 19:15:56 GMT, to avoid parsing problems. An ETag avoids the issue, which is why Google recommends it.

What the http-caching check measures

The HTTP caching check is a scored check in the recommended tier of the web-standards part. It reads the homepage's Cache-Control, ETag and Last-Modified headers. When the homepage sends a validator, the scan requests the page once more with If-None-Match or If-Modified-Since. It reads at most 16 KB of that answer.

It passes on Cache-Control, a validator, and a 304. A partial pass covers 3 cases: a validator whose conditional request gets another answer, a 304 with no Cache-Control, and Cache-Control with no validator. It fails when none of the 3 headers is sent.

In the scan records of 20 October 2026, Vercel, Resend and GitHub send an ETag and answer 304. MDN sends both validators with Cache-Control: public, max-age=3600 and answers 304. Cloudflare's homepage is the partial pass.

The 5-site comparison has the rest of each record. To see what your homepage answers to a conditional request, scan your site.