Crawler and agent policy page

              DOMAIN="example.com"
LINK=$(curl -sS "https://$DOMAIN/" | grep -Eio 'href="[^"]*(bot|agent|crawler)[^"]*"' | head -1 | cut -d'"' -f2); LINK=${LINK#https://$DOMAIN}; curl -sSL "https://$DOMAIN${LINK:-/bot}" -o /tmp/bot-page.html -w 'HTTP %{http_code} %{content_type}\n'; grep -Eio 'verified|signature|key directory|user-agent|ip range|allow|deny|contact' /tmp/bot-page.html | sort -u
            

              <!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Crawler and agent policy</title>
</head>
<body>
  <h1>Bot and agent policy</h1>
  <h2>How we verify identity</h2>
  <p>Signed requests are checked against our key directory at <a href="/.well-known/http-message-signatures-directory">/.well-known/http-message-signatures-directory</a>; unsigned clients are identified by User-Agent and published IP ranges only.</p>
  <h2>Welcome</h2>
  <ul>
    <li>Search and answer engines that identify themselves and cite the source.</li>
    <li>Agents acting for a person on a specific task, at 1 request per second.</li>
  </ul>
  <h2>Refused</h2>
  <ul>
    <li>Training and dataset crawlers; see the AI usage policy at /ai-policy.</li>
    <li>Clients that ignore robots.txt, spoof a User-Agent or exceed the stated rate.</li>
  </ul>
  <h2>Contact</h2>
  <p><a href="mailto:agents@example.com">agents@example.com</a></p>
</body>
</html>