A real 404 for a missing page

Sources

• RFC 9110, 15.5.5: 404 Not Found, IETF
• Google: HTTP status codes and soft 404s, Google Search Central
• Bing Webmaster Guidelines, §9: 404 for missing pages, Microsoft Bing

What is measured

The scanner requests one path that does not exist on the measured origin: /aws-probe- followed by 12 random hexadecimal characters, new for every scan, with the Accept header of a page. It follows redirects on the same site only, and the request is kept out of the paths found, so a soft 404 never counts as a page the site serves. Pass = 404 or 410, after any redirect. Partial = a redirect to the homepage that answers 2xx, which Google reads as a soft 404; a redirect to another site, which is not followed; or another 4xx status. Fail = a 2xx answer at the path, or after a redirect to a page other than the homepage: a soft 404. Not measured = 401, 403, 407, 429 or a 5xx status, 406, which refuses the scan's Accept header, or no answer.

Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

              ORIGIN=https://example.com
MISSING="/aws-probe-$(head -c 6 /dev/urandom | od -An -tx1 | tr -d ' \n')"
curl -sS -o /dev/null -w 'status=%{http_code} location=%{redirect_url}\n' -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7' "$ORIGIN$MISSING"
            

The fix

This page shows no artifact for this check, because the change depends on something only the site owner can supply. The note below says what to do.

Standards referenced

• RFC 9110

Related checks