Can AI Crawlers See Your JavaScript Site? (How to Check)

Here’s a problem most teams don’t know they have: your site can rank on the first page of Google and be completely invisible to ChatGPT at the same time. The reason is a quiet split in how crawlers work. Googlebot runs a full browser engine that executes your JavaScript before indexing the result. The AI crawlers — the bots behind ChatGPT, Claude, and Perplexity — don’t. They fetch your raw HTML, read whatever text is in it, and leave. If your content only appears after JavaScript runs, they see nothing. And this is no longer a niche concern: in June 2026, Cloudflare reported that automated traffic passed human traffic on the web for the first time — 57.5% of HTML requests on its network came from bots.

Which AI crawlers execute JavaScript? (verified July 2026)

The definitive public evidence remains the Vercel and MERJ study of AI crawler behavior, which analyzed hundreds of millions of fetches across Vercel’s network and found that none of the major AI crawlers rendered JavaScript. They download JavaScript files — GPTBot fetched JS on about 11.5% of requests, Claude on about 24% — but they never execute them. We re-verified each bot’s documented behavior for this update. Here’s the full picture:

The pattern is stark: the bots that render are the ones attached to decades-old search infrastructure. Everything built for the AI era reads raw HTML. There’s a logic to it — AI crawlers operate under tight timeouts at enormous scale, and executing JavaScript for every page would be hugely expensive in compute. Skipping it keeps them fast and cheap. This isn’t a temporary gap they’re rushing to close; for now, it’s a design choice.

OAI-SearchBot vs GPTBot: the mix-up that silently costs AI visibility

OpenAI runs multiple crawlers, and conflating them is one of the most common — and most invisible — mistakes in AI-era robots.txt files. GPTBot gathers content for model training. OAI-SearchBot fetches pages to show in ChatGPT’s search answers, and OpenAI’s own documentation is blunt about it: sites opted out of OAI-SearchBot will not be shown in ChatGPT search results. These are independent decisions. Blocking GPTBot to stay out of training data is a legitimate choice that costs nothing in ChatGPT search. Blocking OAI-SearchBot — often by accident, via a blanket disallow or an over-eager firewall rule — removes you from an answer channel entirely, with no error, no warning, and nothing in your analytics to tell you it happened.

If you want search visibility without contributing to training, the robots.txt is simply:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This matters more than it used to, because blocking is becoming the default posture of the web. Cloudflare has blocked AI crawlers by default on newly onboarded domains since July 1, 2025, and announced that from September 15, 2026 it will also block “mixed-use” AI crawlers by default on pages carrying ads, alongside a “Pay Per Use” model that compensates publishers when content surfaces in AI answers. Per Originality.ai’s tracking, about 25% of the top 1,000 websites now block GPTBot, up from 5% at its 2023 launch. If your site sits behind a CDN or WAF, it is worth checking what it decided on your behalf.

Does Google crawl JavaScript-generated content?

Yes — and it’s worth being precise about how, because “Google renders JavaScript” hides a two-phase process. In phase one, Googlebot fetches your raw HTML and can index what’s in it immediately. In phase two, the page enters a render queue for Google’s Web Rendering Service, an evergreen Chromium kept current with stable Chrome, which executes your JavaScript and indexes the full result. That second phase happens once resources allow — usually quickly, sometimes hours, occasionally days. (Google most recently updated its JavaScript documentation in December 2025, clarifying how it handles status codes, canonicals, and noindex in JavaScript-rendered environments.)

Two practical consequences. First, server-rendered content skips the queue entirely — it’s indexable from the first fetch. Second, and more importantly for this page: Google’s rendering maturity applies only to Google. Solving JavaScript SEO for Googlebot does nothing for the AI crawlers, because they never enter a rendering phase at all.

Why a page can rank on Google but be blank to ChatGPT

This is the part that’s easy to miss, because every signal you normally watch says things are fine. Googlebot renders your client-side app, indexes the result, and the page ranks. An AI crawler fetches the same URL, reads the raw HTML, finds a near-empty shell of <div> and script tags, and moves on. One URL, two readers, two entirely different outcomes. The pages most likely to be client-rendered in modern stacks — pricing tables, product grids, comparison pages, docs, FAQs — are exactly the pages AI engines most often draw on for answers.

The subtler failure modes bite even on mostly server-rendered sites: content that loads on scroll via Intersection Observer, tab and accordion panels fetched on click, product lists that render client-side after filtering, and internal links injected by JavaScript. Each of those is invisible to a non-rendering bot even when the rest of the page reads fine. JavaScript-injected internal links are the quiet one — they can make whole sections of a site unreachable to an AI crawler that discovers pages by following links in raw HTML.

Should you care? Honestly: proportionally. AI referral traffic is still small next to Google, and the crawl-to-referral exchange is lopsided — Cloudflare’s 2026 Radar data measured Anthropic crawling roughly 10,000 pages per referral sent, OpenAI around 900, versus about 5 for Googlebot. The case for fixing it isn’t today’s traffic share; it’s that the fix is usually one-time architectural work, the channel keeps growing, and the same raw-HTML completeness also serves Google’s fast first-phase indexing. You don’t want your best pages structurally unreadable to the fastest-growing discovery surface on the web.

How to check what AI crawlers see on your site

You don’t need special software for the basic test — you need to look at your raw HTML, the version that exists before any JavaScript executes. Four ways, from quickest to most thorough:

• View Page Source. Right-click a page and choose “View Page Source” — not “Inspect,” which shows the post-JavaScript DOM and will mislead you. Search the source for your real text, headings, and internal links. If they’re there, non-rendering bots can read them. If you find mostly an empty <div id="root"> and script tags, they can’t.
• Fetch it like a bot. From a terminal: curl -A "GPTBot" -s https://yoursite.com | less. That response body is literally what a non-rendering crawler receives. (Testing with a bot user-agent also reveals whether your CDN or firewall serves bots something different — or blocks them outright.)
• Disable JavaScript and reload. Turn off JS in your browser settings and refresh. Whatever disappears is what an AI crawler never had.
• Crawl the static HTML at site scale. The three tests above check one page at a time. Crawlmouse crawls your whole site the way a non-rendering bot does — raw server HTML, no JavaScript execution — and maps the internal-link graph it finds. If pages or links are missing from that crawl, or it flags the site as JavaScript-rendered, those pages and links are invisible to AI crawlers too. It won’t diagnose every content type — it’s an internal-linking grader, not a rendering debugger — but it shows you the site structure a raw-HTML reader actually sees, free and with nothing to install. (It’s the same crawl behind a full internal-link audit.)

One more check that costs thirty seconds: grep your server or CDN logs for GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot. If they’re absent entirely, your problem may be access — robots.txt, firewall, or CDN defaults — rather than rendering.

How to fix it

The fix is well-established and mostly a one-time architectural decision: get your important content and internal links into the initial HTML response, not loaded afterward by client-side JavaScript. In practice that means server-side rendering — Next.js for React, Nuxt for Vue, Angular Universal for Angular — or pre-rendering pages to static HTML. Content in the raw HTML before scripts run is readable by everyone: Google, Bing, and every AI crawler in the table above. It doesn’t have to be visible prose, either — server-rendered payloads and JSON-LD in the initial response are readable; content fetched after load is not.

And don’t forget the plumbing: none of this helps if crawlers can’t reach the page in the first place. Make sure your internal linking is solid and your pages are actually getting crawled and indexed — a page an AI crawler could read but never finds is just as invisible as one it finds but can’t read. And check your access layer: an inherited robots.txt rule or a CDN’s default AI-block can undo everything the architecture gets right.

Run the View Source test on your five most important pages today. If your content and links are in the raw HTML, you’re in good shape. If they’re not, you’ve just found the reason your best pages aren’t showing up in AI answers.