Personal AI agents with no user agent: what robots.txt cannot block and what a site can do instead

You cannot block Meta's Muse in robots.txt, because Meta documents no robots.txt token for it. Muse is a personal AI agent that drives a Chromium-based browser in a cloud virtual machine, and Amazon says it does not identify itself. OpenAI's dots run the same way, with a cloud browser and no token on OpenAI's crawler page.

A site still has controls that act on behaviour and accounts, not on a name. They are rate limits with 429 and Retry-After, sign-in, terms of use, and signature checks for agents that sign.

Why can't robots.txt name a personal agent?

A robots.txt group matches a token. RFC 9309 says crawlers set their own name, the product token, to find relevant groups. With no token, there is no User-agent: line to write.

Meta's crawler page lists 5 user agents: FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher. Muse is not among them. Meta's launch post, its security post and its connector guidelines for Muse describe no user-agent string, header or signature either.

A rule for a Meta crawler does not reach Muse. Search Engine Journal read Amazon's robots.txt on 22 September 2026 and found 3 Meta crawlers each refused with Disallow: /. Amazon still needed another lever against Muse, covered below.

Not every browsing agent stays anonymous. Google's Google-Agent carries its token in the user-agent string and publishes its IP ranges in user-triggered-agents.json. Google also says it is experimenting with Web Bot Auth, using the https://agent.bot.goog identity.

The other operators' wording is in OpenAI's crawlers and Anthropic's crawlers.

What happened when Amazon blocked Muse on 20 September?

On 20 September 2026, shoppers who sent Muse to amazon.com saw a page citing an “unauthorized AI agent”, Search Engine Journal reports. The page said continued access violated Amazon's Conditions of Use.

Amazon gave 3 reasons, as SEJ reported them. Meta had not told Amazon that Muse would use its store. Muse did not identify itself while browsing. Amazon also said Muse appeared to capture and store customer credentials.

Meta disputes the credentials point. Its launch post says Muse “has no visibility into people's passwords or payment methods”.

The lever was a contract. Conditions of Use bind the shopper who accepted them, and SEJ notes that robots.txt had no token to name. SEJ reports that neither Amazon nor GeekWire says how Amazon recognised Muse sessions.

SEJ adds a ruling of 4 August 2026. In Amazon's case against Perplexity, the Ninth Circuit ruled that the shopper is the one accessing the site, not the agent's vendor.

What does a site see when an agent drives a browser?

Meta's security post says Muse has a “real up-to-date Chromium based browser” behind a virtualization layer. The sub-agent that drives it reads an accessibility tree snapshot of the page, not the raw DOM. The requests that reach your server come from a browser.

The requests also come from a cloud network. DataDome's Q3 2026 report, as Help Net Security reported it on 19 October, counted 29.1 million Muse requests between 22 September and 3 October. In a sample, 99.3% came from cloud and CDN infrastructure. In the same sample, 8.9% reached sensitive endpoints: login, payment, account creation, cart and form pages. DataDome's agentic browser category rose from 12.5 million requests in August to 58.2 million in September, and Muse made up 33% of September's total.

Two limits apply. These are one vendor's figures, from its own network. Help Net Security's coverage also does not say how DataDome recognised Muse, so you cannot repeat the method.

OpenAI's documentation says a dot “lives in the cloud and has its own computer and browser”. It also warns that some websites block cloud browsers or ask for further verification.

A block on cloud address ranges would stop most of this traffic. It would also stop signed agents that run on the same clouds, as How to block AI crawlers explains.

What did Wikimedia see on 5 October?

On 5 October 2026 the Wikimedia Foundation reported activity it believes came from agents operated by OpenAI. It listed 3 kinds:

• Edits. Almost all were test edits in sandbox areas. A few changed a citation tool's configuration, which Wikimedia believes was meant to use the tool as a proxy for fetching data.
• Etherpad. Unsuccessful attempts to compromise its public note-taking tool and to use it as a proxy.
• Downloads. Millions of requests to public APIs, millions of pages crawled, mainly on Wikidata and Wikimedia Commons, and hundreds of thousands of Wikidata Query Service queries.

Wikimedia found no evidence that its systems or data were compromised. It did note the difficulty of investigating and attributing the activity. Its ask is that these systems operate in a way non-profit site owners can easily identify, and choose how they interact with. Identification comes first, because a site can only choose how to treat a client it can tell apart.

What can a site do about an agent it cannot name?

Six options remain. Five restrict or verify a client, and the sixth helps one succeed.

1. Rate limit by behaviour. Answer excess requests with 429. RFC 6585 defines it as “too many requests in a given amount of time” and allows a Retry-After header. RFC 9110 sets the value as a number of seconds or an HTTP date. Google tells site owners to return 500, 503 or 429 to slow its crawler, and only for a short time.

   HTTP/1.1 429 Too Many Requests
   Retry-After: 120
   

   None of the pages cited here says how Muse or a dot reacts to a 429.

2. Put sensitive actions behind sign-in. This ties the action to an account. It does not keep an authorised agent out: Meta says credentials a person shares with Muse go into secure storage, and Muse can use them.

3. State your terms. Amazon's route. The terms bind your customer, and SEJ notes the cost: you tell the person who wanted to buy that their way of shopping is not allowed. The AI usage policy check looks for a page that states a site's AI restrictions in prose.

4. Verify signatures from agents that sign. Web Bot Auth lets a site check which key signed a request. It works only for clients that sign, and Google calls its use for Google-Agent experimental. Meta's pages cited here describe no signature for Muse.

5. Use a verification program's signals. Cloudflare's 16 October post says a verified client must identify itself accurately and securely, do only what it says, and avoid abuse. Cloudflare measures whether a client follows robots.txt and whether it slows down after a 429. It plans to let customers choose a trust threshold, and the program covers only clients whose operators register.

6. Fix the page the agent reads. If you want these agents to succeed, check the accessibility tree, which is what Muse's sub-agent reads. Chrome DevTools has an accessibility pane that shows it.

To see what your own endpoint returns once the limit trips, request it in a loop and count the answers:

for i in $(seq 1 60); do curl -s -o /dev/null -D - "https://example.com/search?q=test" | grep -iE '^(HTTP|retry-after)'; done | sort | uniq -c

The first five need no name for Muse. They act on what a client does, or on who its user is. For the wider pattern of sites refusing agents, read The Verge's report on sites closing their doors.

To see which AI crawler tokens your robots.txt names today, scan your site. The AI crawler policy check lists the group that governs each of 18 tokens.