What does your robots.txt tell AI crawlers?

Crawlmouse reads your robots.txt for each crawler, then requests pages as each AI crawler, so you see what you asked for next to what happened.

Free · up to 500 pages · about two minutes · nothing to install · the result is kept 24 hours; one email keeps it longer

robots.txt is a request, not a wall.

A crawler follows the group of rules that names its token, such as "User-agent: GPTBot", and a group that names a crawler does not inherit the rules written under "User-agent: *".

The large operators, OpenAI, Anthropic, Google and Microsoft among them, document that their crawlers follow it. It asks; it does not refuse a request, and it does not remove what earlier crawls collected.

Training and answers are separate switches. Disallowing GPTBot or ClaudeBot asks them to leave your pages out of future training. Disallowing OAI-SearchBot, Claude-SearchBot or PerplexityBot concerns their search answers, and it is not a clean exit: OpenAI says a site that blocks OAI-SearchBot is not shown in ChatGPT search answers but can still appear as a navigational link, Anthropic says blocking Claude-SearchBot may reduce your visibility in its search results, and Perplexity recommends allowing PerplexityBot for your site to appear in its results.

Some fetches sit outside it: OpenAI says robots.txt rules may not apply to ChatGPT-User, which fetches a page when a person asks about it, and Perplexity says Perplexity-User generally ignores them. A change is not instant either: OpenAI says about 24 hours for its search crawler, and Perplexity up to 24 hours.

What robots.txt cannot show is a firewall, a bot manager or a CDN default refusing a crawler it allows. Cloudflare has blocked AI crawlers by default on newly onboarded domains since 1 July 2025. That is why the audit also requests pages as each AI crawler; Googlebot and Bingbot are shown by robots.txt only.

Two minutes from now you’ll know what each crawler sees.