Cloudflare Starts Blocking AI Crawlers by Default on September 15. Here's Who It Affects — and How to Check Your Site

On September 15, 2026, Cloudflare changes what happens to AI crawlers on a large slice of the web — by default, without site owners touching anything. If your site runs through Cloudflare, or your clients’ sites do, this is worth ten minutes before the deadline. If you care about showing up in AI answers, it’s worth twenty.

Here is what actually changes, who it affects, and how to check where your site stands — from inside the dashboard and from the outside, the way a crawler sees it.

What changes on September 15

Cloudflare announced the change on July 1, 2026. The old one-click “Block AI bots” toggle is being replaced by three independent categories, each with its own policy:

• Search — crawlers that collect or index content to answer questions later. Think classic search indexing.
• Agent — automated fetches made in real time on a person’s behalf, such as an assistant opening your page to answer a question someone just asked.
• Training — crawlers collecting content to train or fine-tune AI models.

Starting September 15, the default settings change: Training and Agent crawlers are blocked by default on pages that display ads, while Search crawlers stay allowed. Per Cloudflare’s announcement, the new defaults apply to new domains joining Cloudflare, new sites added by existing customers, and existing free-plan customers who haven’t changed their settings. Site owners can opt out — or opt in harder — in the dashboard at any time.

Two details in the fine print matter more than the headline.

Detail one: the Googlebot trap

Some crawlers do more than one job. Googlebot, Bingbot, and Applebot all crawl both for search indexing and for AI purposes. Cloudflare’s new rule treats these multi-purpose crawlers under the strictest applicable policy: if your zone blocks Training, a crawler that does Search and Training gets blocked entirely — even though Search is allowed.

Read that again, because it’s the part that will surprise people in October: a site that blocks AI training can end up blocking Googlebot itself on the affected pages. Coverage from Search Engine Journal at the time of the announcement flagged exactly this scenario, and Cloudflare’s own documentation confirms the strictest-rule behavior. If you enabled the old “Block AI bots” toggle at some point and never revisited it, that legacy setting is folded into the new classification — a choice that felt safe in 2025 can behave differently after September 15.

Detail two: the Agent block, or how to vanish from AI answers by accident

Almost all coverage of this change is written for publishers who want to keep AI out. But flip it around. When someone asks an AI assistant a question and the assistant reads your page to compose its answer, that live fetch is Agent traffic — and Agent is blocked by default on ad pages under the new settings.

So picture a business that runs display ads, sits on a Cloudflare free plan, and has spent this year trying to show up in AI answers. On September 15, its zone quietly inherits the new defaults. The assistant that used to be able to open its pages now gets turned away at the network edge. No error on the site. No warning email from the AI company. The site simply stops being readable at the moment of the question — which is the moment that matters.

If AI visibility is something you want rather than something you’re defending against, the new defaults are a setting you need to make deliberately, not inherit.

Who is affected — and who isn’t

You’re in scope for the new defaults if all of these are true: the site is behind Cloudflare (orange-cloud, proxied), it’s a new domain, a new site on an existing account, or on the free plan with unchanged settings, and the pages in question display ads — Cloudflare detects ad units automatically.

Nothing changes automatically for: sites not on Cloudflare, sites whose DNS runs through Cloudflare but unproxied (grey-cloud — the edge never sees the traffic), paid-plan zones with settings already configured, and pages without ads. And one more thing worth stating plainly: this enforcement happens at Cloudflare’s network layer, before a request reaches your server. Unlike robots.txt, which is advisory and only binds crawlers polite enough to obey it, an edge block stops the request itself.

How to check your settings (inside)

Log into the Cloudflare dashboard, pick the zone, and go to Security → Settings → Configure AI bot policies (naming at the time of writing; Cloudflare has been renaming this area as the feature rolls out). You’ll see the three categories. For each one, the options are block on all pages, block only on pages with ads, or don’t block.

Make an explicit choice for all three and save it. A recorded choice is exempt from default flips — the September 15 change only moves zones that never chose. If your business depends on specific crawlers, set explicit allows for them rather than relying on category defaults, and check your logs in the week after the 15th.

How to check what crawlers actually experience (outside)

Here’s the part the dashboard can’t tell you: what your site serves and what a crawler experiences are two different layers, and they can disagree. Your robots.txt can say allow while an edge rule says block. Your pages can be perfectly crawlable and still be unreadable to AI systems for a completely different reason — because the content only exists after JavaScript runs, and AI crawlers don’t render JavaScript.

This isn’t a rare setup. Across the sites audited on Crawlmouse, roughly one in three sits behind a detected edge or CDN layer whose rules can override whatever robots.txt declares. For those sites, reading robots.txt tells you the site’s stated policy — not its behavior.

So the outside check has three parts:

1. Read your served robots.txt — the file your domain actually returns today, not the one in your repo. If you use Cloudflare’s managed robots.txt, directives are injected at the edge, and after September 15 you may find declarations there you didn’t write, including the extended Content Signals use parameter Cloudflare is rolling out.
2. Check per-crawler access — whether GPTBot, ClaudeBot, PerplexityBot and the rest are allowed, disallowed, or unmentioned, and whether the retrieval bots (the ones that fetch pages to answer live questions) are treated differently from the training ones. Our guide to blocking or allowing AI crawlers in robots.txt walks through every major bot token.
3. Look at what a crawler can actually read — a page that’s reachable but empty without JavaScript is invisible in practice, whatever the access rules say.

A free Crawlmouse audit runs all three checks across your whole site: the per-bot access matrix, a note when an edge/CDN layer is detected that could override robots.txt, and a page-by-page view of what’s readable without JavaScript. It takes about the time it took to read this section.

What to do, by situation

You want AI visibility and you run ads on a Cloudflare free plan: act before September 15. Make explicit choices in the AI bot policies panel — at minimum, decide deliberately whether Agent stays allowed. Then verify from the outside after the 15th.

You want AI visibility and you’re grey-cloud or not on Cloudflare: nothing changes for you on the 15th, but this is a good excuse to verify your actual crawler access anyway — most sites have never checked.

You want AI crawlers out: the new controls are genuinely better than the old toggle — you can now block Training while keeping Search, which the one-click block couldn’t do. Just mind the multi-purpose rule: blocking Training can take Googlebot with it on ad pages. If Google traffic matters to you, set an explicit allow for the crawlers you depend on and watch your logs after the deadline.

You manage client sites: this is a before-and-after moment. A crawler-access check on every client zone this week, and again the week after the 15th, will catch any zone that inherited defaults nobody chose.

Sources: Cloudflare’s July 1, 2026 announcement and developer documentation on the September 15 default changes; Search Engine Journal’s coverage of the multi-purpose crawler rule (July 2026); Help Net Security’s summary of the Content Signals extension (July 2, 2026). Policy details are as published at the time of writing — September 6, 2026 — and Cloudflare may adjust naming or scope; check your own dashboard for the current state of your zone.