Help

Your audit says a crawler is blocked, but we never blocked it. Why?

To test an AI crawler, the audit requests a sample of your pages with that crawler’s user agent, sent from our servers with our tag added: “Crawlmouse/2.0 (+https://crawlmouse.com/bot)”. It reports what each request got back.

The audit says blocked in two cases: “blocked by robots.txt”, when your robots.txt disallows that crawler, and “blocked by a rule”, when a page refused that crawler’s user agent while it served another crawler the audit sent to the same page. A crawler your robots.txt disallows is your choice, so it is left out of the AI readability score rather than counted against it.

When a page refuses everything that claims to be a crawler, the audit cannot tell a rule about that crawler from a rule about who is asking. It then says “challenged: the real crawler may pass” and leaves that page out of the crawler’s score.

A rule that refuses crawlers by user agent is often set in a CDN or a firewall in front of the site, rather than in the site itself.

Where the audit opens a fix for a blocked crawler, its steps end with a command under Verify that you can run from your own machine. It sends the request with the crawler’s own user agent. Our request differs from it: it carries our tag after the user agent and a From header, and it comes from our servers. A rule that matches the exact user-agent string, reads the From header, or looks at where a request comes from can answer the two differently.

See also: How does Crawlmouse crawl my site, and how do I allow or block it?

Not answered here? Email support