How does Crawlmouse crawl my site, and how do I allow or block it?
Our crawler’s user agent is “Mozilla/5.0 (compatible; Crawlmouse/2.0; +https://crawlmouse.com/bot)”.
To test AI crawlers, an audit also requests a sample of pages as each crawler’s user agent with “Crawlmouse/2.0 (+https://crawlmouse.com/bot)” added, and once per sampled page as an ordinary browser, which carries no tag. When a page refuses a crawler, the audit may request it once more as Googlebot, tagged the same way, to see whether the refusal is about that crawler.
Our requests obey the robots.txt groups named Crawlmouse and CrawlmouseBot, and the * group when your file has neither. A request sent as a crawler also obeys that crawler’s own group, and a page that group disallows is not requested.
To keep Crawlmouse out, add this group to your robots.txt:
User-agent: Crawlmouse
Disallow: /The audit still reads robots.txt itself, to learn the rule. To let Crawlmouse in where a broader group keeps bots out, give it a group that allows it:
User-agent: Crawlmouse
Allow: /See also: Your audit says a crawler is blocked, but we never blocked it. Why?
Not answered here? Email support