Homepage indexable: no noindex directive

Sources

• Bing Webmaster Guidelines, §10: NOINDEX and the other robots directives, Microsoft Bing
• Bing: robots meta tags and attributes that Bing supports, Microsoft Bing
• Google: block indexing with noindex, Google Search Central
• Google: robots meta tag specifications, Google Search Central

What is measured

The scanner reads the homepage response it already holds ($home in the probe map), so the check makes no request of its own. It reads the content of each <meta name="robots">, <meta name="bingbot"> and <meta name="googlebot"> in the document head, and the X-Robots-Tag response header. A header value that opens with a crawler name and a colon, such as googlebot: noindex, applies to that crawler until the next name: one for bingbot or googlebot is read, and one for another crawler is reported and not judged. Directives are split at commas, semicolons and white space, so noindex nofollow is read as two. Pass = no noindex and no none in any of them. Fail = noindex, or none, in any of them; the detail names each place it was found. Bing keeps a page with noindex out of Bing search, Copilot and grounding results (Bing Webmaster Guidelines, §10), and Google leaves it out of Search results. none is Google's value, which it reads as noindex, nofollow. Bing's list of the robots values it supports does not name none, so a none states Google's effect alone, and a none only for bingbot is partial. Repeated X-Robots-Tag headers arrive joined by commas, so a noindex that follows another directive after a crawler name may be a second header for every crawler; the detail says so. The fix is each value without noindex, none, nofollow and the defaults index, follow and all, or the tag removed when nothing is left: nofollow on the homepage stops crawlers following its links. Not measured = a homepage that answered a status other than 2xx, or did not answer, and a noindex on a homepage with fewer than 25 words of visible text, reachable's floor: that page may be an interstitial such as a bot challenge, which is served with 200 and often carries a noindex of its own. A short page with no noindex is read as served.

Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

              HOMEPAGE=https://example.com/
curl -sSL -o /dev/null -D - -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7' "$HOMEPAGE" | tr -d '\r' | grep -i '^x-robots-tag:'
curl -sSL -A 'AgenticWebScore/0.3 (+https://agenticwebscore.com/bot)' -H 'Accept-Language: en' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.7' "$HOMEPAGE" | perl -0777 -ne 'my $h = /^(.*?)<\/head>/si ? $1 : substr($_, 0, 8192); while ($h =~ m{<meta\b[^>]*>}gi) { my $t = $&; next unless $t =~ /\bname=["\x27]?(robots|bingbot|googlebot)["\x27]?[\s>\/]/i; my $n = lc $1; my ($c) = $t =~ /\bcontent=["\x27]([^"\x27]*)/i; print "meta $n: ", ($c // ""), "\n" }'
            

The fix

This page shows no artifact for this check, because the change depends on something only the site owner can supply. The note below says what to do.

Related checks