# HTTP caching headers and 304 for crawlers

> HTTP caching of the homepage: Cache-Control says how long the homepage may be reused. The fix: send caching headers on the homepage.

- Source: /checks/http-caching
- Site: AgenticWebScore
- Language: English

---
Cache-Control says how long the homepage may be reused. A validator lets a crawler ask whether it changed, and a 304 answer skips the body.

Standard status: Adopted. A ratified standard with real deployment. Standards: [RFC 9111](https://www.rfc-editor.org/rfc/rfc9111.html) and [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html).

- Kind: Scored check
- Part of the score: Web standards: 70% of the score. A check in it can hold a level.
- Module: Every site
- Who reads it today: Googlebot sends If-None-Match and If-Modified-Since. Every HTTP cache reads Cache-Control.
- Effect on Google Search: Direct. A documented Google Search feature reads it. This is not a ranking factor.

| Property | Value |
| --- | --- |
| Check id | `http-caching` |
| Dimension | Discoverability |
| Part | Web standards |
| Tier | Recommended. Counts its weight once. |
| Weight | 1 |
| Maturity | Adopted. A ratified standard with real deployment. |
| Publishing skill | `http-caching` |
| Standards | RFC 9111, RFC 9110 |

## Sources

- [RFC 9111: HTTP Caching](https://www.rfc-editor.org/rfc/rfc9111.html), IETF, June 2022
- [RFC 9110, 13: conditional requests](https://www.rfc-editor.org/rfc/rfc9110.html#name-conditional-requests), IETF
- [Google: HTTP caching for crawlers](https://developers.google.com/search/blog/2024/12/crawling-december-caching), Google Search Central, December 2024
- [Bing Webmaster Guidelines, §3: ETag and lastmod](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a), Microsoft Bing

## What is measured

The scanner reads the Cache-Control, ETag and Last-Modified headers of the homepage response ($home in the probe map). When the homepage sends an ETag or a Last-Modified date, the scanner requests the URL the homepage answered at once more, with If-None-Match set to the ETag and If-Modified-Since set to the date, each when it was sent, and reads at most 16 KB of the answer. A validator longer than 1,024 characters, or with a line break, is not sent. Pass = Cache-Control and a validator, and the conditional request answered 304 Not Modified. Partial = a validator, and the conditional request answered a status other than 304 that is not listed under Not measured; a validator and a 304 with no Cache-Control; or Cache-Control with no validator, and then no conditional request is made. Fail = none of the three headers. Cache-Control: private and no-store are policies like any other, and these rules apply to them: no-store with no validator is partial, because a crawler then requests the whole page each time. Not measured = a homepage that answered a status other than 2xx; a validator the scan did not send; or a conditional request that was not made, got no answer, or answered 401, 403, 407, 429 or a 5xx status, or 406, which refuses the scan's Accept header.

## Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

```bash
HOMEPAGE=https://example.com/
HEADERS=$(curl -sSL -o /dev/null -D - -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7' "$HOMEPAGE" | tr -d '\r')
printf '%s\n' "$HEADERS" | grep -iE '^(cache-control|etag|last-modified):'
ETAG=$(printf '%s\n' "$HEADERS" | sed -n 's/^[Ee][Tt][Aa][Gg]: *//p' | tail -1)
DATE=$(printf '%s\n' "$HEADERS" | sed -n 's/^[Ll][Aa][Ss][Tt]-[Mm][Oo][Dd][Ii][Ff][Ii][Ee][Dd]: *//p' | tail -1)
curl -sSL -o /dev/null -w 'status=%{http_code}\n' -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7' -H "If-None-Match: $ETAG" -H "If-Modified-Since: $DATE" "$HOMEPAGE"
```

## The fix

This is an example written for example.com. Where the scan can build the fix from your own site, the report shows that version instead.

```text
Cache-Control: <choose: a policy, for example no-cache or max-age=300>
ETag: "<choose: a value that changes when the page changes>"
```

> Most servers send an ETag or a Last-Modified header for a static page and answer If-None-Match with 304 Not Modified. A page rendered on each request needs the framework to set them.

## Standards referenced

- `RFC 9111`
- `RFC 9110`

## Related checks

| Check | Kind | Maturity | Same skill |
| --- | --- | --- | --- |
| `reachable` — Readable homepage | Scored check | Adopted | — |
| `sitemap-xml` — XML sitemap | Scored check | Adopted | — |
| `llms-txt` — /llms.txt published and well-structured | Scored check | Draft | — |
| `link-headers` — Discovery via Link response headers | Scored check | Adopted | — |
| `dns-aid` — DNS-AID discovery record | Unscored check | Experimental | — |
| `canonical` — Canonical URL | Scored check | Adopted | — |
| `indexing-directives` — Homepage indexable: no noindex directive | Scored check | Adopted | — |
| `not-found-status` — A real 404 for a missing page | Scored check | Adopted | — |