The Crawlmouse blog
Practical guides to internal linking, site structure, and the technical SEO that decides whether your pages get found.
Cloudflare Starts Blocking AI Crawlers by Default on September 15. Here's Who It Affects — and How to Check Your Site
· 9 min readOn September 15, 2026, Cloudflare changes what happens to AI crawlers on a large slice of the web — by default, without site owners touching anything. Here is what actually changes, who it affects, and how to check where your site stands — from inside the dashboard and from the outside, the way a crawler sees it.
Read more →Should You Block AI Crawlers? A robots.txt Guide for GPTBot, ClaudeBot & Co.
· 9 min readA quarter of top sites now block GPTBot, and Cloudflare blocks AI crawlers by default. Before you follow: training bots and AI search bots are different switches. A practical robots.txt guide with copy-paste rules for each stance.
Read more →How to Find Orphan Pages on Squarespace (Free, Nothing to Install)
· 8 min readSquarespace ships with a built-in orphan-page factory: the Not Linked section. Pages there stay public and indexable with zero internal links. Here is how to find every orphan on your site for free, and fix the ones that matter.
Read more →Screaming Frog vs Lumar (DeepCrawl): Desktop Tool vs Enterprise Platform
· 9 min readPeople still search "Screaming Frog vs DeepCrawl" — but DeepCrawl became Lumar, and the comparison changed. Desktop audits vs enterprise platform: an honest August 2026 breakdown of pricing, monitoring, CI/CD, and fit.
Read more →Sitebulb vs JetOctopus vs Oncrawl: Cloud SEO Crawlers Compared
· 10 min readOnce your site outgrows a desktop crawler, you end up comparing Sitebulb Cloud, JetOctopus, and Oncrawl. Here is an honest July 2026 comparison — pricing, log analysis, JavaScript crawling, and when none of the three is the right spend.
Read more →How to Find Orphan Pages on Webflow (Free, No App)
· 8 min readWebflow orphans pages in one sneaky way most people miss: limiting a CMS Collection List. Here is how to find every orphan on your Webflow site for free, and how to fix them.
Read more →Sitebulb vs Screaming Frog: Which SEO Crawler Should You Use?
· 10 min readScreaming Frog and Sitebulb are the two big technical-SEO crawlers, and they overlap heavily. Here is an honest comparison of features, pricing, and learning curve — and when you need neither.
Read more →Can AI Crawlers See Your JavaScript Site? (How to Check)
· 11 min readGooglebot renders JavaScript; the AI crawlers behind ChatGPT, Claude, and Perplexity don't. Here is why a JS site can rank on Google yet be blank to AI search, and how to check yours.
Read more →How to Find Orphan Pages on Shopify (Free, No App Needed)
· 8 min readShopify orphans pages faster than almost any platform — flat URLs, collection templates, thin collections. Here is how to find them all for free, and fix the ones worth keeping.
Read more →Discovered – Currently Not Indexed: Why It Happens and How to Fix It
· 9 min readA page that returns 200 and sits in your sitemap can still refuse to index. Here is what the not-indexed statuses mean, and the internal-linking fix that actually moves them.
Read more →Sitebulb Alternatives (2026): Free and Paid Options, Honestly Compared
· 10 min readSitebulb has no free tier and its pricing now runs $18–$245/month. Here are six real alternatives — free, paid, and open source — with current pricing and an honest note on when Sitebulb is still the right buy.
Read more →Screaming Frog Alternatives in 2026: Six Options, Compared Honestly
· 8 min readScreaming Frog is the default answer to "how do I crawl my site" — and "default" is not the same as "right for you." Six real alternatives matched to the actual jobs people hire a crawler for, and the honest cases where you should just pay the £199.
Read more →How to Find Orphan Pages in WordPress (Free, No Plugin Needed)
· 8 min readWordPress orphans pages faster than almost any platform — tag archives, page-builder pages, theme swaps. Here is how to find them all for free, with or without a plugin, and fix them.
Read more →How Deep Is Too Deep? Crawl Depth Limits and How to Fix Buried Pages
· 8 min readThere is no single magic number, but there is a clear answer for your site. Here is how deep is too deep by site size, why it matters, and how to fix the pages that slipped too far.
Read more →Crawl Depth & Site Architecture: Why Click-Depth Affects Rankings
· 9 min readA page buried eight clicks deep gets crawled less, earns less internal authority, and ranks worse than the same content three clicks from home. Here is why, and what to do.
Read more →How to Find Orphan Pages on Your Website (Free, No Install)
· 9 min readAn orphan page is one nothing links to — so crawlers rarely find it and the work that went in is wasted. Here is how to find every orphan on your site, free, and fix the ones worth keeping.
Read more →Internal Link Checker: How to Audit Your Site's Internal Links Free
· 8 min readYou do not need a paid crawler or a desktop install to check your internal links. Here is how to audit them free, what a good checker measures, and how to read the result.
Read more →