# HTTP caching for crawlers: ETag, Last-Modified and 304

> Why Googlebot and Bingbot send If-None-Match, how to test a 304 Not Modified with curl, and the CDN behaviours that quietly break validators on a page.

- Source: /learn/http-caching-crawlers
- Updated: 2026-10-25
- Reading time: 5 minutes
- Tags: HTTP caching, ETag, crawling, CDN

---
Send an `ETag` or a `Last-Modified` date with each page. A crawler that holds a copy sends it back in `If-None-Match` or `If-Modified-Since`, and an unchanged page gets `304 Not Modified` with no body. Google's crawlers support both validators and use the `ETag` when a response has both. Bing's guidelines say an `ETag` helps Bing detect content changes more reliably.

## Why Google and Bing use validators

Google's [crawler overview](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers) says its crawling infrastructure supports heuristic HTTP caching through `ETag` with `If-None-Match`, and `Last-Modified` with `If-Modified-Since`. It recommends `ETag` for Google's crawlers, because an `ETag` has no date format to get wrong. It also says other HTTP caching directives are not supported. Support varies by crawler: Googlebot uses caching when it re-crawls for Search, and Storebot-Google only in certain conditions.

Google's [December 2024 post](https://developers.google.com/search/blog/2024/12/crawling-december-caching) puts a number on the problem. About 0.026% of Google's fetches were cacheable 10 years earlier, and 0.017% were at the time of the post. It asks servers to answer a matching `If-None-Match` with a 304 and no body, which saves the server from building the page and sending it.

Bing's [Webmaster Guidelines](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a) make the same point in §3. Accurate `lastmod` values in a sitemap, and validators such as an `ETag`, help Bing detect content changes more reliably.

## How a conditional request works

[RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html) sets 3 rules a server must follow:

- `If-None-Match` uses the weak comparison, so `W/"abc"` matches `"abc"`.
- When a request carries both headers, the server must ignore `If-Modified-Since`. The RFC calls `If-None-Match` the more accurate condition.
- A 304 must carry the `ETag`, `Cache-Control`, `Expires`, `Vary`, `Date` and `Content-Location` fields a 200 would have carried. It cannot contain content.

`Cache-Control: max-age` is a separate signal. Google's overview calls it optional, and says a `max-age` helps a crawler decide when to fetch the URL again.

## Which Cache-Control value to send with a validator

`Cache-Control` says how long a stored copy stays fresh. The validator says how to check a copy once it is stale. A crawler needs both answers, and the common values give different ones:

- `max-age=3600` lets a cache reuse the page for an hour without asking. After that, it revalidates.
- `max-age=0, must-revalidate` makes every reuse a conditional request. With an `ETag`, the answer is a 304 whenever nothing changed. This site's homepage sends this.
- `no-cache` asks for the same. [RFC 9111](https://www.rfc-editor.org/rfc/rfc9111.html) says such a response must not be reused “without forwarding it for validation”.
- No `Cache-Control` at all leaves a cache to guess. RFC 9111 allows a heuristic lifetime only when no explicit expiration is present, and names 10% of the time since `Last-Modified` as a typical setting.

For Google's crawlers, only the validators and `max-age` matter, since the overview says other caching directives are not supported. Pick the `max-age` that matches how often the page changes, and send a validator either way.

## How to test ETag and 304 with curl

The [HTTP cache checker](/tools/http-cache-checker) runs these requests on your site, together with the 404 and JSON error checks.

Use a GET. `curl -I` sends HEAD, and a crawler fetching a page sends GET. Print only the headers:

```sh
curl -sS -o /dev/null -D - https://agenticwebscore.com/ \
  | grep -Ei '^(HTTP|cache-control|etag|last-modified|vary)'
```

```http
HTTP/2 200
cache-control: public, max-age=0, must-revalidate
etag: "e92a771d27ded3e3fadeea16d30608ec"
vary: Accept, Cookie
```

Send the `ETag` back the way a crawler does:

```sh
curl -sS -o /dev/null -D - \
  -H 'If-None-Match: "e92a771d27ded3e3fadeea16d30608ec"' \
  https://agenticwebscore.com/ | grep -Ei '^(HTTP|cache-control|etag)'
```

```http
HTTP/2 304
cache-control: public, max-age=0, must-revalidate
etag: "e92a771d27ded3e3fadeea16d30608ec"
```

A server honours only the validator it issued. This site sends no `Last-Modified`, so a date gets a full answer:

```sh
curl -sS -o /dev/null -w '%{http_code}\n' \
  -H 'If-Modified-Since: Sun, 25 Oct 2026 00:00:00 GMT' https://agenticwebscore.com/
```

```text
200
```

For any site, read the `ETag` and send it back in 2 lines:

```sh
URL=https://example.com/
ETAG=$(curl -sS -o /dev/null -D - "$URL" | awk 'tolower($1)=="etag:" {sub(/^[^:]*: */,""); sub(/\r$/,""); print}')
curl -sS -o /dev/null -w '%{http_code}\n' -H "If-None-Match: $ETAG" "$URL"
```

A `304` means the validator works. A `200` means the server ignores it, or the `ETag` changed between the 2 requests.

## CDN and framework pitfalls that break the 304

**Compression weakens the ETag.** Ask this site for gzip, and the same page comes back with a weak tag:

```sh
curl -sS -o /dev/null -D - -H 'Accept-Encoding: gzip' https://agenticwebscore.com/ \
  | grep -Ei '^(etag|content-encoding)'
```

```http
etag: W/"e92a771d27ded3e3fadeea16d30608ec"
content-encoding: gzip
```

Cloudflare's [ETag reference](https://developers.cloudflare.com/cache/reference/etag-headers/) documents this. When it compresses an origin's uncompressed response, it returns a weak `ETag`, and some settings remove the strong tag instead. A weak tag still validates under the weak comparison: sent back here, it gets a 304. An origin that compares tags strongly, or a setting that drops the header, loses the 304.

**No validator at all.** A `Cache-Control` header with no `ETag` or `Last-Modified` gives a crawler nothing to send back. It downloads the whole page each time. Cloudflare's own homepage did this in its [scan record of 20 October 2026](/aws/cloudflare.com): `Cache-Control: max-age=10` and no validator.

**An ETag that changes on every response.** A tag built from the whole HTML changes whenever the HTML does. A per-request nonce, a timestamp or a session token in the page makes every tag new, and no request ever matches. Fetch the page twice and compare the 2 tags.

**A validator on some representations only.** A page negotiated on `Accept` is 2 or more representations, and each needs its own tag. Google's post gives the example of mobile and desktop versions under the same URL, each with its own `ETag`. On this site, the markdown version of the homepage, asked for with `Accept: text/markdown`, carried no `ETag` on 25 October 2026.

**A 200 to every conditional request.** A server can send an `ETag` and still answer every `If-None-Match` with the full page. The 2-line test above catches it, and the check below gives it a partial pass.

**A Last-Modified date in the wrong format.** Google asks for the HTTP date format, such as `Fri, 4 Sep 1998 19:15:56 GMT`, to avoid parsing problems. An `ETag` avoids the issue, which is why Google recommends it.

## What the http-caching check measures

The [HTTP caching check](/checks/http-caching) is a scored check in the recommended tier of the web-standards part. It reads the homepage's `Cache-Control`, `ETag` and `Last-Modified` headers. When the homepage sends a validator, the scan requests the page once more with `If-None-Match` or `If-Modified-Since`. It reads at most 16 KB of that answer.

It passes on `Cache-Control`, a validator, and a 304. A partial pass covers 3 cases: a validator whose conditional request gets another answer, a 304 with no `Cache-Control`, and `Cache-Control` with no validator. It fails when none of the 3 headers is sent.

In the scan records of 20 October 2026, [Vercel](/aws/vercel.com), [Resend](/aws/resend.com) and [GitHub](/aws/github.com) send an `ETag` and answer 304. [MDN](/aws/developer.mozilla.org) sends both validators with `Cache-Control: public, max-age=3600` and answers 304. Cloudflare's homepage is the partial pass.

The [5-site comparison](/learn/five-sites-agent-readiness-october-2026) has the rest of each record. To see what your homepage answers to a conditional request, [scan your site](/scan).