How to find AI crawlers in your server logs, and tell real ones from fakes

A documented AI crawler names itself in the User-Agent field of every request, so grep for its token finds it in an access log. That name is a claim, because any client can type it. To check it, compare the source address with the operator's published list, or run a reverse DNS lookup for Googlebot and Bingbot. Agents that send no name cannot be found this way.

What does a log line show about a crawler?

nginx and Apache both document a combined format. It holds the client address, the time, the request line, the status, the bytes sent, the referrer and the user agent. nginx calls it the predefined combined log format, and Apache's log files page gives the same fields.

Split a line on double quotes. The request is field 2, the status starts field 3, and the user agent is field 6. Every recipe below uses awk -F'"' for that reason. If your log is JSON, or comes from a CDN, the field names differ and the idea stays the same.

Behind a proxy or load balancer, the first field is the proxy's address, and address checks will fail. nginx's realip module replaces it with the address from a header such as X-Forwarded-For, for the proxies you trust. A CDN that answers from cache or blocks a request at its edge never forwards it, so an origin log undercounts.

The recipes below ran on a 25-line sample file. Its user agents follow the operators' documented strings, and ClaudeBot's is shortened to its token because Anthropic's page gives no full string. The genuine crawlers use addresses inside the operators' published ranges. Two requests are forged, from the documentation ranges 203.0.113.0/24 and 198.51.100.0/24.

How do you count requests per AI crawler?

Take the user agent field only, find the first token in it, and count:

TOKENS='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Amzn-User|Applebot|DuckAssistBot|MistralAI-User|Meta-ExternalAgent|Google-Agent|Googlebot|bingbot'
awk -F'"' -v re="$TOKENS" 'match($6, re) { print substr($6, RSTART, RLENGTH) }' access.log | sort | uniq -c | sort -rn

   8 GPTBot
   4 Googlebot
   3 ClaudeBot
   2 bingbot
   2 OAI-SearchBot
   1 PerplexityBot
   1 Perplexity-User
   1 Google-Agent
   1 ChatGPT-User

Do not run grep -io over the whole line. Some user agents end with a URL that repeats the token in lower case, such as openai.com/gptbot, so each request counts twice. On the sample file, that gave 8 GPTBot and 8 gptbot.

The pattern holds the tokens from AI crawler user agents that send a user agent, plus Google-Agent, Googlebot and bingbot. Google-Extended and Applebot-Extended are missing on purpose, because Google and Apple describe them as control tokens that do not crawl. What does Google-Extended block? explains Google's.

What are they fetching, and what did your server answer?

Add the status to see whether your own rules treat crawlers the way you meant. $TOKENS is the pattern from above:

awk -F'"' -v re="$TOKENS" 'match($6, re) { split($3, s, " "); if (s[1] >= 400) print substr($6, RSTART, RLENGTH), s[1] }' access.log | sort | uniq -c

   2 GPTBot 429

A crawler you welcome that meets 403 is the case to find. A crawler that meets 429 and keeps requesting is covered in Retry-After and 429 for AI crawlers.

Requests per minute show a steady crawl or a burst. This one counts one token, GPTBot, by the minute in the time field:

awk -F'"' '$6 ~ /GPTBot/ { split($1, a, " "); print substr(a[4], 2, 17) }' access.log | sort | uniq -c

   4 01/Nov/2026:10:15
   4 01/Nov/2026:10:24

Two more cuts are worth keeping. List the distinct addresses behind each token, which also gives you the pairs to verify below. Then see who fetched /robots.txt:

awk -F'"' -v re="$TOKENS" 'match($6, re) { split($1, a, " "); print substr($6, RSTART, RLENGTH), a[1] }' access.log | sort -u
awk -F'"' -v re="$TOKENS" '$2 ~ /^GET \/robots\.txt / && match($6, re) { print substr($6, RSTART, RLENGTH) }' access.log | sort | uniq -c

OpenAI says GPTBot and OAI-SearchBot may add a robots.txt marker to the user agent string when they fetch that file. In the sample, grep -c '; robots.txt;' access.log returns 1. The marker helps when your log omits paths.

How do you check that a request came from the crawler it names?

A user agent string is text the client chooses. Google's crawler page says so in a caution, and Google, Bing and Perplexity each describe a second check. There are 3 you can run yourself.

Match the address against the operator's list

OpenAI, Anthropic, Perplexity, Google and Bing publish their ranges as JSON. Each file holds a prefixes array of ipv4Prefix or ipv6Prefix entries. Read on 1 November 2026, these answered with that shape:

Match each user agent against its own list. Perplexity's page says each of its crawlers has its own ranges, and OpenAI publishes one file per crawler. The IP ranges section of AI crawler user agents covers the other operators, such as Apple, Amazon and DuckDuckGo. A small Python script does the matching:

import ipaddress, json, sys

with open(sys.argv[1]) as f:
    prefixes = json.load(f)["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]

for line in sys.stdin:
    ip = ipaddress.ip_address(line.strip())
    hit = any(ip.version == n.version and ip in n for n in nets)
    print(f"{str(ip):<16} {'in' if hit else 'NOT in'} {sys.argv[1]}")

Save it as inlist.py, download the list, and feed it the distinct addresses behind one token:

curl -sS https://openai.com/gptbot.json -o gptbot.json
awk -F'"' '$6 ~ /GPTBot/ { split($1, a, " "); print a[1] }' access.log | sort -u | python3 inlist.py gptbot.json

20.171.206.54    in gptbot.json
203.0.113.9      NOT in gptbot.json

The second address claimed to be GPTBot and is not in OpenAI's list. The same run against common-crawlers.json accepted 66.249.66.1 as Googlebot and rejected 198.51.100.23.

Lists change. Bing says to refresh its list daily, and Perplexity says to fetch its endpoints periodically. Fetch them on a schedule, not once.

Run a reverse DNS lookup for Googlebot and Bingbot

Google's steps have 3 parts:

1. Look up the name of the address.
2. Check that it ends in googlebot.com, google.com or googleusercontent.com.
3. Look up that name and confirm it returns the same address.

Bing's steps are the same, with a name that ends in search.msn.com.

The forward lookup is the step a forger cannot fake. The owner of an address controls its reverse record, and not Google's zone.

verify() {
  ip=$1; shift
  name=$(host "$ip" | awk '/domain name pointer/ {print $NF}' | sed 's/\.$//' | head -n 1)
  [ -n "$name" ] || { echo "$ip no PTR record: not verified"; return 1; }
  for suffix in "$@"; do
    case "$name" in
      *."$suffix") host "$name" | grep -q "address $ip\$" && { echo "$ip $name verified"; return 0; } ;;
    esac
  done
  echo "$ip $name not verified"; return 1
}
verify 66.249.66.1 googlebot.com google.com googleusercontent.com
verify 157.55.39.1 search.msn.com
verify 198.51.100.23 googlebot.com google.com googleusercontent.com

Run it once per distinct address, because a lookup per log line is slow. These are real lookups. 66.249.66.1 and 157.55.39.1 are the addresses Google and Bing use in their own examples, and 198.51.100.23 is a documentation address:

66.249.66.1 crawl-66-249-66-1.googlebot.com verified
157.55.39.1 msnbot-157-55-39-1.search.msn.com verified
198.51.100.23 no PTR record: not verified

Log the Web Bot Auth headers

A signed request carries Signature-Agent, Signature-Input and Signature. Google says a subset of Google-Agent requests sets Signature-Agent to g="https://agent.bot.goog". It adds that it is not yet signing every request.

Log the header first. In nginx, add "$http_signature_agent" after the user agent in your log_format. In Apache, add \"%{Signature-Agent}i\" to the LogFormat.

awk -F'"' '$8 != "-" { split($1, a, " "); print a[1], $8 }' access.log

2001:4860:c::5 g=\x22https://agent.bot.goog\x22

That output came from a 3-line sample in this format, where the header is field 8. nginx escapes the quotes as \x22.

The header is a lead, not a verdict. The IETF draft calls an unresolved Signature-Agent a claim rather than an identity. A verifier must fetch the key directory and check the signature before it attaches policy.

What is Web Bot Auth? covers that step. Google tells you to keep IP and DNS checks as a fallback.

What can a log not show?

Three kinds of traffic will not show up as a token in your origin log.

• Google-Extended. Google crawls with its existing user agents, so a Disallow for the token changes no log line.
• Agents that send no name. Search Engine Journal reports that Meta's Muse has no published user agent. It relays Amazon's statement that Muse does not identify itself. Muse drives a Chromium browser, so its session looks like an ordinary customer's visit.
• Anything your edge answered. A cache hit or a block at the CDN is in the CDN's log, not your origin's.

For the second, behaviour is what remains: the request rate, the paths requested and the address ranges. Wikimedia asked on 5 October 2026 that operators make their agents easy for non-profit site owners to identify. Personal agents with no user agent covers what a site can do about them.

What do you do with the result?

A token counted is not a policy. Decide per token whether you welcome it, slow it or refuse it, and write the refusals in robots.txt, as How to block AI crawlers explains. The AI crawler checker shows which robots.txt group governs each of the 18 tokens on your site.

To see how the scanner reads your robots.txt rules for these crawlers, scan your site.