Why AI crawlers cannot read your JavaScript site

Googlebot renders JavaScript. It has for years, and that shaped a generation of frontend decisions.

AI crawlers mostly do not. GPTBot, ClaudeBot and PerplexityBot fetch your HTML and work with what arrives. There is no second rendering pass, no headless browser, no waiting for hydration.

If your content appears after JavaScript runs, it does not exist for them.

What this looks like in practice

A client-rendered single-page app serves something close to this:

<!doctype html>
<html>
  <head><title>Acme</title></head>
  <body>
    <div id="root"></div>
    <script src="/static/js/main.8f3a2b.js"></script>
  </body>
</html>

A browser turns that into a full page. A crawler sees a title and an empty div. Asked what Acme does, a model has your title tag and nothing else.

See what they see

You do not need special tooling. In your browser, use View page source — not the inspector. The inspector shows the DOM after JavaScript ran; view-source shows what was actually delivered.

Or from a terminal:

curl -s https://example.com/ | wc -c

Then read it. If your headline, your product description and your pricing are not in there, they are not in there for a crawler either.

A rough threshold: under about 1,500 characters of visible text in the delivered HTML, a page carries almost no meaning for a model. Most pages that fail this are not slightly short — they are near-empty.

What to fix, in order

1. Render the content that matters on the server. Next.js, Nuxt, SvelteKit, Astro and Remix all do this by default now. The work is usually removing client-only patterns, not adopting a framework.

2. Do not rely on hydration for text. Content that only appears after a client-side fetch is invisible. If a product description arrives from an API call in useEffect, move that fetch to the server.

3. Static-render what does not change. Marketing pages, docs and pricing have no reason to be dynamic. Prerendered HTML is both faster for people and complete for machines.

4. Ignore the script-size panic. A server-rendered page can still ship a large bundle. That is fine. What matters is whether the text is in the HTML before the bundle runs, not how big the bundle is. Serialised hydration state is data, not client-side rendering.

The awkward part

This is not a quick fix if your architecture went the other way. Moving a client-rendered app to server rendering is real work.

The reasonable middle path: server-render the pages a model would need to answer questions about you — homepage, pricing, product pages, docs — and leave the logged-in application client-side. Nobody's dashboard needs to be crawlable.

Want the character count for your own homepage? Run a free scan — five checks, under 60 seconds, no login.

A product by The Autopilot — https://the-autopilot.com