We fetched 96 websites as AI crawlers. The answer depended on who was asking.

In October we ran three rounds of live tests. Each fetched real websites with the published user-agent strings of AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot, and compared the answer with a plain browser fetch of the same page.^[1] Across the rounds we fetched 96 distinct websites. Round one fetched only the homepage of each; the later rounds went back to some of them and went deeper.^[1][2][3][4]

The headline: the answer a site gave depended on more than the crawler name in the request. The same crawler names, sent to the same five sites from two cloud networks a day apart, did not get the same answers. Our notes single out nike.com, en.wikipedia.org and neilpatel.com.^[1]

What we tested, and how

A web request carries a user-agent string, a line of text naming the software that sent it. We sent requests carrying the strings AI crawlers publish, and recorded what came back. The tests paired those requests with a control: the same page fetched with a plain Chrome browser string. The control is what makes a result readable. When the browser fetch is challenged too, a refused “GPTBot” request says nothing about GPTBot; it says the site challenges traffic from data centres.^[2]

Round one made 768 homepage fetches. It fetched each homepage whatever robots.txt said, and recorded robots.txt alongside.^[2][3] A re-run on 16 of the sites sent each AI crawler string twice, exactly as published and tagged with a Crawlmouse suffix, and added ChatGPT-User and Claude-SearchBot. Tagging changed no answer on the 16 sites tested.^[2][1]

Round two went deeper on fewer sites: it crawled up to 150 pages of each of the first 10, and up to 100 of each of the other 5, then fetched a sample of pages (the homepage, a page per template, the biggest hubs, the deepest and newest pages) under each crawler name. It sent at most 2 requests per second to any host. Its second part honoured robots.txt per crawler name; its first part, because of a bug, never read robots.txt at all.^[1][3]

What the labels mean

The results are recorded as verdict labels. We show them as written, with what our notes say they mean.

• allowed: the crawler-named request got the page. In round two, that meant a word count of at least 80% of the control’s.^[3]
• blocked_by_ua_rule: the control got a 200, and the crawler-named request got a 4xx, 451 or 999 status.^[2]
• disallowed_by_robots: robots.txt closed the page to that crawler name, so no request was sent under it.^[3]
• edge_blocks_automation: the browser control itself was challenged, so the result is reported for the site and not attributed to any crawler.^[2]
• challenged_ua is shown in the files by example rather than defined: on neilpatel.com, it was a 403 carrying Cloudflare’s cf-mitigated: challenge header.^[3]
• refused_unverified and redirected_away appear only in round three, and the files do not define them. Read from their names: the site refused the request, which did not come from a verified crawler; and the site answered with a redirect away from the page requested.^[4]

The main finding: same names, different networks, different answers

The five sites in the later rounds are among those that had treated crawler names differently in round one. Round two fetched them from a Supabase Edge Function on 26 October, sampling 20 pages per site. Round three fetched them from a Fly.io worker on 27 October with the same six crawler names; its counts are out of 20 pages per site, except on neilpatel.com, where it measured 1.^[1][3][4]

Our research notes single out nike.com, en.wikipedia.org and neilpatel.com as the sites that answered differently from the Supabase network and the Fly.io network.^[1] On nike.com, the most common answer changed for each crawler name: Claude-SearchBot and ClaudeBot went from allowed to refused_unverified, and OAI-SearchBot, ChatGPT-User, PerplexityBot and GPTBot went from blocked_by_ua_rule to refused_unverified. On en.wikipedia.org, ClaudeBot went from blocked_by_ua_rule to allowed. On neilpatel.com, each crawler name went from challenged_ua to edge_blocks_automation, the label for a challenge to the browser control itself, on the page measured.^[2][3][4]

On css-tricks.com, blocked_by_ua_rule stayed the most common answer for each crawler name in both rounds. On cnn.com, round two sent no crawler-named requests, so it has no live answer to compare, and the robots.txt counts themselves differ between the rounds (see the box below).^[3][4]

What this shows, in the words of our notes: a simulated request measures how a site treats the infrastructure the request comes from, and conclusions drawn from it have to be labelled as simulated. The networks differ in more than their addresses; our notes list egress IPs, TLS fingerprints and DNS.^[1][3]

Small-business homepages mostly answered every crawler name we sent; large sites more often refused automation outright

Round one split its sites into two sets of 48: large consumer, media and platform sites with enterprise bot management, and sites like the ones our customers run, from Shopify stores, agencies and SEO vendors to WordPress, Webflow and Squarespace sites, SaaS marketing sites and local Indiana organisations.^[2]

The gap is in the first row. Nearly half of the large sites challenged or blocked a plain browser request from a cloud network. On those sites a test from outside says nothing about any particular crawler, and our notes record them as not testable from outside rather than as blocked.^[2] In the small-business set, 42 of the 48 homepages were recorded as everything allowed.^[2]

Where the raw rows differ from that classification. The table above is the summary in our notes; the same file also has a raw row per site, and they do not agree everywhere. The row for large sites where an AI crawler name was refused counts 8 but names fewer sites than that. The summary counts 21 large sites with a challenged or blocked control; the raw rows show 20 whose control got an answer other than a 200. Among the small-business sites recorded as everything allowed, purdue.edu and pacers.com have raw rows with errors or different word counts across identities. The large set also included our own sites, crawlmouse.com and nahltech.com, whose raw rows show a 200 for every identity.^[2]

Rules cut both ways. bluehost.com challenged our Chrome control but served our requests under all four AI crawler names. bombas.com, hosted on Vercel, returned 429 to our Chrome and Googlebot requests but 200 to our requests named GPTBot and OAI-SearchBot.^[2]

Why a robots.txt check alone is not enough

robots.txt says what a site asks of crawlers. In our tests, it and the response our requests received disagreed in both directions.

css-tricks.com’s robots.txt explicitly allows GPTBot, ChatGPT-User and ClaudeBot. In round two, each of the six AI crawler names we sent got a 403 on 20 of 20 sampled pages, while Chrome and our own crawler got 200.^[3]

The reverse also happened. On 4 of the 48 large sites, robots.txt disallowed some of the AI crawler names we sent (bbc.com and amazon.com: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot; techcrunch.com: GPTBot and ClaudeBot; wired.com: ClaudeBot and PerplexityBot), and the homepage still came back to every identity, because round one fetched regardless of robots.txt.^[2]

On 21 of the 48 large sites, by our notes’ classification, the edge challenged or blocked even the browser control.^[2]

Sites also answered different crawler names differently. In the round one re-run, cnn.com returned 451 to our requests named GPTBot, ChatGPT-User, ClaudeBot and PerplexityBot, and 200 to those named OAI-SearchBot and Claude-SearchBot; en.wikipedia.org returned 403 to our request named ClaudeBot and 200 to the request named Claude-SearchBot.^[2]

Why a simulated request is not the real crawler

The real GPTBot comes from OpenAI’s published IP ranges and, on Cloudflare, is a verified bot. Ours sent the same user-agent string from an ordinary cloud IP.^[2]

In round one, by our notes’ count, a request carrying the Googlebot or Bingbot name was refused while the Chrome control got the page on 5 of the 48 large sites and 1 of the 48 small-business sites; walmart.com is in that count, though its raw row shows the same answer for every identity. Sites that verify Googlebot by reverse DNS refuse an impersonation. That is the site working as intended, and it says nothing about how the real Googlebot is treated.^[2]

The same holds for AI crawlers. A site that allows the real GPTBot and challenges ours shows up in our data as challenged; a site that blocks only verified bots shows up as allowed.^[3]

The answers that differed between the networks are the clearest case. Neither network is a crawler’s own, so neither round shows what GPTBot or ClaudeBot itself receives.^[1]

What to check on your own site

1. Read your robots.txt for each crawler name. Look for each name separately: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and PerplexityBot. Sites in our tests answered these names differently. Our robots.txt guide for AI crawlers covers writing the rules.^[2]
2. Check your CDN or firewall’s bot settings. In our tests the response did not always follow robots.txt: on css-tricks.com, the crawler names that robots.txt allows got a 403. Answers also varied by page: on neilpatel.com, a Cloudflare challenge covered 17 of 20 sampled pages and skipped three. If your site is behind Cloudflare, our post on its AI crawler settings shows where they live.^[3]
3. Look for the real crawlers in your own logs. Find hits from the crawler names in your server or CDN logs, and confirm them against the IP ranges the crawler’s operator publishes (our notes cite OpenAI’s for GPTBot), or by reverse DNS for Googlebot. A user-agent string alone is what our tests sent too.^[2]

Methodology and limits

Round two’s file has a limits section of its own. Round one’s limits below are drawn from its method and results, and round three’s from the README.^[1][2][3]

Round one, 25 October 2026

Homepages of 96 sites, each fetched as 8 identities, plus robots.txt, from a Supabase Edge Function: 768 homepage fetches.^[1][2]

• Homepages are not whole sites, and cloud IPs are not the crawlers’ own IPs.^[1]
• Where the browser control was itself challenged or blocked, per-crawler access was not testable from outside: 21 large sites and 4 small-business sites, by our notes’ count. The fourth of those small-business sites, iuhealth.org, failed on our side: our notes call it a DNS failure, and its raw row records an IPv6 egress failure for 7 identities.^[2]
• The summary table and the raw rows in the round’s file do not agree everywhere (see above).^[2]
• A refused Googlebot or Bingbot request is not evidence about the real Googlebot or Bingbot.^[2]
• It fetched each homepage regardless of robots.txt; a polite crawler would not have made some of these requests.^[2][3]
• Detecting a challenge from page text alone produced a false positive (a “captcha” string on a normal page); our notes adopted header-first detection afterwards.^[2]

Round two, 26 October 2026

First part: 10 sites, 308 sampled pages, 3,250 requests. Second part: five sites, a 20-page sample each. Both from a Supabase Edge Function.^[1][3]

• It was not the Fly.io worker: it ran on Supabase’s egress IPs and runtime, with no persistent connections, and Fly egress IPs, TLS fingerprints and DNS differ.^[3]
• Nothing was rendered; no JavaScript ran.^[3]
• Not the real crawlers: verified-bot and signed-bot treatment is untestable from a cloud IP.^[3]
• The first part’s sites were a limited mix: none treated crawler names differently, and none was behind a challenge for all automation.^[3]
• The sample was drawn only from pages our crawler had read. On a site in the first part, pages that returned a Cloudflare challenge during the crawl were never retested.^[3]
• A bug: the first part never read robots.txt, because robots.txt files served as plain text were discarded unread. Its statements about robots rules are unproven, and its requests may include paths robots.txt forbids.^[3][1]
• A first run of the second part had the same bug. It was discarded and rerun, so neilpatel.com, css-tricks.com and nike.com took double the load.^[3]
• On brooklinen.com, a limit on link parsing made the crawl’s discovery invalid; its sample, drawn from the sitemap, is valid.^[3]
• On en.wikipedia.org, a very large page exceeded the function’s resource limit and was skipped by hand, and the requests of an aborted first run are not counted.^[3]
• Pages that returned 404 to the control were first labelled edge_blocks_automation; the published counts correct this.^[3]
• On cnn.com, a stop rule (4 consecutive challenged pages) tripped twice and was overridden, a deviation from the test’s instructions, because the challenges were confined to the site’s /cnn-underscored section.^[3]
• In the second part, four of the five sites were testable live (cnn.com’s robots.txt disallowed every sampled page); 20 pages per site, not 50; and verdicts were compared by word count only.^[3]

Round three, 27 October 2026

The same five sites from a Fly.io worker with a dedicated cloud IP; verdict counts per site and crawler name.^[1][4]

• It saved counts per site, not each page’s response.^[1]
• A dedicated cloud IP is still not a crawler’s own.^[1]
• On neilpatel.com it measured 1 page.^[4]
• Its robots.txt counts differ from round two’s on en.wikipedia.org and cnn.com, and the files do not record why.^[4]
• Its file records no page addresses or request headers, and does not define the labels refused_unverified and redirected_away. It also has a summary row per site, marked pass: false, that no file explains; this article does not use those rows.^[4]

Across the rounds, the numbers are dated observations of the sites as they were on those days.^[1]

Data

The numbers in this article come from these files, in docs/research/2026-10-crawler-access/ of the Crawlmouse repository. A test checks each number against the file it cites.

1. README.md: what each round tested, when, from which network, and the limits the rounds share.
2. 01-homepage-probe-96-sites.md: the first round: method, results, raw tables per site and identity, and the tagged re-run.
3. 02-sample-probe.md: the second round, both parts: method, per-site tables, verdict counts, bugs found and limits.
4. fly-run-2/discriminators-2026-10-27T21-02-49-840Z.json: the third round: verdict counts per site and crawler name.