Should I block AI crawlers in robots.txt?

It depends on which crawler and what you want. Blocking training bots (GPTBot, ClaudeBot, CCBot) keeps your content out of future model training without affecting your visibility. Blocking search/retrieval bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes you from AI search answers — OpenAI states sites that block OAI-SearchBot will not appear in ChatGPT search results. Decide the two questions separately.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content for training OpenAI’s models. OAI-SearchBot fetches pages to surface in ChatGPT’s search answers. They honor separate robots.txt rules, so you can block training while staying visible in ChatGPT search — or vice versa. Conflating them is the most common mistake in AI-era robots.txt files, and blocking the wrong one silently removes you from an answer channel.

Does blocking AI crawlers in robots.txt actually work?

Mostly, with caveats. robots.txt is a voluntary convention: the major operators (OpenAI, Anthropic, Google, Microsoft) document compliance, while some crawlers — ByteDance’s Bytespider has been repeatedly reported as an offender — ignore it. Blocking also is not retroactive: content already collected in past crawls or in Common Crawl snapshots does not get removed. For enforcement rather than requests, CDN-level blocking (e.g. Cloudflare) actually refuses the connection.

Will blocking GPTBot hurt my Google rankings?

No. GPTBot is OpenAI’s bot and has nothing to do with Google Search. Googlebot is controlled separately, and Google’s AI training opt-out (Google-Extended) is a separate robots.txt token that does not affect Search either. What blocking GPTBot does affect: your content’s presence in future OpenAI model training.

What is Cloudflare doing about AI crawlers in 2026?

Cloudflare has blocked AI crawlers by default on newly onboarded domains since July 1, 2025, and announced that from September 15, 2026 it will also block “mixed-use” AI crawlers by default on pages carrying ads, alongside a “Pay Per Use” model that compensates publishers when content surfaces in AI answers. Practical takeaway: if your site is behind Cloudflare, check what is being blocked on your behalf — you may be blocking AI search bots you want.

How do I check which AI crawlers are visiting my site?

Grep your server or CDN logs for the user-agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot. If the search-focused bots are absent entirely, check robots.txt and your CDN’s bot settings before assuming they are not interested — an inherited block is the most common cause.

Should You Block AI Crawlers? A robots.txt Guide for GPTBot, ClaudeBot & Co.

August 18, 2026·9 min read

Quick answer

Don’t decide “AI crawlers: yes or no.” Decide two separate questions. Training bots (GPTBot, ClaudeBot, CCBot) collect content for future models — blocking them costs you no visibility. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages to cite in AI answers — blocking them removes you from those answers. Most sites that want AI visibility should allow the search bots, choose deliberately on the training bots, and — especially behind Cloudflare — verify nothing is being blocked on their behalf.

A quarter of the web’s biggest sites now block OpenAI’s training crawler — about 25% of the top 1,000 websites per Originality.ai’s tracking, up from 5% when GPTBot launched in 2023. Cloudflare blocks AI crawlers by default for new domains, and from September 15, 2026 extends default blocking to “mixed-use” AI crawlers on ad-supported pages. Blocking has become the fashionable default — and that’s exactly why it’s worth slowing down, because the single most common mistake in AI-era robots.txt files is blocking bots whose job is to send you visibility while trying to opt out of training. This guide separates the bots by what they actually do, gives you copy-paste rules for each stance, and covers what blocking can and cannot achieve. Facts verified August 2026.

Know what each bot does before you block it

The decision only makes sense per-purpose. Three categories cover the bots that matter:

Category	Bots (user-agents)	What blocking costs you
Training	GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider, CCBot, Google-Extended*	Nothing visible today; your content stays out of future model training
AI search / retrieval	OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot, Applebot	Presence in AI search answers & citations
User-triggered fetch	ChatGPT-User, Claude-User	The assistant can’t open your page when a user asks it to

*Google-Extended is a robots.txt control token rather than a crawler — it opts your content out of Gemini training without affecting Google Search. That’s the model to keep in mind everywhere: training and visibility are separate switches. OpenAI’s own documentation states that sites blocking OAI-SearchBot will not be shown in ChatGPT search results — while blocking GPTBot has no effect on ChatGPT search at all. (What these bots can read once you allow them is its own topic: most execute no JavaScript, which we cover in Can AI Crawlers See Your JavaScript Site?)

Copy-paste robots.txt for the three sensible stances

Stance 1 — visible everywhere, out of training. The most popular deliberate setup: appear in ChatGPT, Claude, and Perplexity answers, contribute nothing to model training.

# Allow AI search & user fetches
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Block AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Stance 2 — open to everything. Maximum AI visibility and training inclusion: simply have no rules targeting these bots. If your robots.txt doesn’t mention a bot, it’s allowed. Nothing to add — just make sure nothing upstream (CDN, firewall) blocks silently.

Stance 3 — closed to everything. Some publishers legitimately want out of the AI ecosystem entirely. Block every user-agent in the table above with Disallow: /. Understand the trade: you disappear from AI answers, which for a growing share of queries is where the audience is.

One nuance worth naming on CCBot: Common Crawl is an open dataset used by many research and commercial models, but also by search-adjacent and archival projects — blocking it is the broadest single opt-out and the least targeted one.

What blocking can and cannot do

robots.txt is a request, not a wall. The major operators — OpenAI, Anthropic, Google, Microsoft, Meta, Apple — document compliance. Some crawlers don’t play by the rules; ByteDance’s Bytespider has repeatedly been reported ignoring robots.txt. If you need enforcement rather than requests, block at the CDN/WAF level, where the connection is actually refused.
It’s not retroactive. Blocking today keeps future crawls out; it doesn’t remove what past crawls or existing Common Crawl snapshots already contain.
Check what your CDN decided for you. Cloudflare has blocked AI crawlers by default on new domains since July 1, 2025; from September 15, 2026 it also default-blocks mixed-use AI crawlers on pages carrying ads, and has moved from per-crawl payments to a “Pay Per Use” compensation model. These defaults are reasonable for publishers who never chose a stance — and quietly wrong for anyone who wants AI search visibility. If you’re behind Cloudflare or a similar WAF, audit its bot settings against the stance you actually chose.
Verify with logs, not vibes. Grep your access logs for the user-agents above. The asymmetry is real — Cloudflare’s 2026 Radar data measured AI crawlers fetching hundreds to thousands of pages per referral sent — so seeing heavy crawl traffic is normal; seeing zero visits from search bots you meant to allow means something upstream is blocking them.

Allowing a bot in is step one — being readable is step two

A robots.txt that welcomes OAI-SearchBot achieves nothing if what the bot receives is an empty JavaScript shell, or if it can’t discover your pages because your internal links only exist after scripts run. Most AI crawlers read raw server HTML and follow the links they find there — no rendering, no patience. Crawlmouse crawls your site exactly that way — raw HTML, no JavaScript execution — and grades whether your pages and internal links are actually reachable to a non-rendering bot, free and with nothing to install. Run it after you’ve set your robots.txt stance: allowed-but-unreadable is the failure mode nobody’s dashboard reports.

Crawlmouse is a free internal-linking grader built by Nahl Technologies.

Frequently asked questions

Should I block AI crawlers in robots.txt?: It depends on which crawler and what you want. Blocking training bots (GPTBot, ClaudeBot, CCBot) keeps your content out of future model training without affecting your visibility. Blocking search/retrieval bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes you from AI search answers — OpenAI states sites that block OAI-SearchBot will not appear in ChatGPT search results. Decide the two questions separately.
What is the difference between GPTBot and OAI-SearchBot?: GPTBot collects content for training OpenAI’s models. OAI-SearchBot fetches pages to surface in ChatGPT’s search answers. They honor separate robots.txt rules, so you can block training while staying visible in ChatGPT search — or vice versa. Conflating them is the most common mistake in AI-era robots.txt files, and blocking the wrong one silently removes you from an answer channel.
Does blocking AI crawlers in robots.txt actually work?: Mostly, with caveats. robots.txt is a voluntary convention: the major operators (OpenAI, Anthropic, Google, Microsoft) document compliance, while some crawlers — ByteDance’s Bytespider has been repeatedly reported as an offender — ignore it. Blocking also is not retroactive: content already collected in past crawls or in Common Crawl snapshots does not get removed. For enforcement rather than requests, CDN-level blocking (e.g. Cloudflare) actually refuses the connection.
Will blocking GPTBot hurt my Google rankings?: No. GPTBot is OpenAI’s bot and has nothing to do with Google Search. Googlebot is controlled separately, and Google’s AI training opt-out (Google-Extended) is a separate robots.txt token that does not affect Search either. What blocking GPTBot does affect: your content’s presence in future OpenAI model training.
What is Cloudflare doing about AI crawlers in 2026?: Cloudflare has blocked AI crawlers by default on newly onboarded domains since July 1, 2025, and announced that from September 15, 2026 it will also block “mixed-use” AI crawlers by default on pages carrying ads, alongside a “Pay Per Use” model that compensates publishers when content surfaces in AI answers. Practical takeaway: if your site is behind Cloudflare, check what is being blocked on your behalf — you may be blocking AI search bots you want.
How do I check which AI crawlers are visiting my site?: Grep your server or CDN logs for the user-agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot. If the search-focused bots are absent entirely, check robots.txt and your CDN’s bot settings before assuming they are not interested — an inherited block is the most common cause.