When someone asks ChatGPT, Claude or Perplexity about your business, the assistant often goes and reads your website. What it reads is not always what your visitors see. A person’s browser downloads your page and then runs its JavaScript, the code that can add text, prices, reviews and menus after the page arrives. Most AI crawlers skip that second step. They read the page exactly as your server sent it, and anything that only appears once scripts have run is not there for them.
For a lot of sites this makes no difference, because the words are in the page from the start. For others it means an assistant sees a heading, a cookie banner and not much else. The good news is that you can tell which kind of site you have in a few minutes, without any technical help.
What a crawler sees, and what your visitors see
A crawler is a program that fetches web pages automatically. Search engines use them to build their index, and AI companies use them to collect pages for training, for search results and for answering a user’s question in the moment.
When a visitor opens your page, their browser goes through more steps than a crawler does. The difference is where text can go missing.
HTML is the document your server sends: the text, the headings, the links, and references to other files like images and scripts. If a sentence is written into that document, every crawler can read it. If a script writes it in later, only a crawler that runs scripts will see it.
Which AI crawlers run JavaScript
The clearest public measurement comes from Vercel, a hosting company, and MERJ, a search consultancy. In December 2024 they published an analysis of AI crawler traffic across Vercel’s network. Their finding was blunt: “none of the major AI crawlers currently render JavaScript.” To render a page means to build it fully, scripts included, the way a browser does.
Two caveats. First, this was a measurement taken in late 2024, not a promise from the AI companies, and any of them could start rendering pages at any time. We have not seen a public re-test that found a change. Second, Google is the exception that matters. Its own documentation on how Google Search processes JavaScript describes a rendering step where “a headless Chromium renders the page and executes the JavaScript,” and the same study notes that Gemini uses that infrastructure. So a page can be fine in Google and empty in ChatGPT.
The crawlers do fetch script files. They just don’t run them. In the same study, JavaScript files made up a noticeable share of what the OpenAI and Anthropic crawlers downloaded.
If you want to control which of these crawlers may visit at all, that is a separate setting in your robots.txt file, which we covered in AI crawlers in robots.txt. Allowing a crawler in does not help if the words are not in the page when it arrives.
Where text goes missing on ordinary business sites
Most small-business sites are built on platforms that write the text into the HTML on the server. WordPress, for example, does this by default. Problems usually come from one section or one add-on rather than the whole site. These are the patterns we look for:
- A page builder or plugin that draws a section with scripts. Sliders, tabbed sections, “load more” lists and animated counters are common culprits. The rest of the page is fine and the one section with your key message is empty.
- Third-party widgets. Reviews, booking calendars, price lists and menus are often embedded from another company’s service and filled in by a script. A crawler sees a placeholder.
- Content fetched after the page loads. Some sites load product details, FAQs or locations from a separate data file once the page is open in a browser.
- Sites built as a JavaScript app. A site made with a framework such as React and not set up to render on the server can send an almost empty HTML document with a single empty container that scripts fill in later.
One thing that is not a problem: text inside a closed accordion or a hidden tab, as long as it is written into the HTML. A crawler reads it whether or not anyone clicks to open it. What matters is whether the words exist in the document, not whether they are visible on screen.
How to check your own pages in five minutes
You don’t need developer tools for the first step, and none of these steps change anything on your site.
1. View the source. In Chrome, Edge or Firefox on a computer, right-click the page and choose “View page source” (Ctrl+U on Windows). This shows the HTML exactly as your server sent it. Press Ctrl+F (Cmd+F) and search for a sentence that matters: your main headline, a price, the name of a service. If you find it, crawlers can read it. If you can see it on the page but not in the source, a script put it there.
2. Turn JavaScript off and reload. Chrome’s DevTools documentation gives the steps: open DevTools (F12), press Ctrl+Shift+P (Cmd+Shift+P on a Mac), type “javascript”, choose “Disable JavaScript” and reload. Whatever is left on screen is roughly what a fetch-and-read crawler gets. Closing DevTools switches scripts back on.
3. Run our free check. Alongside the speed results from Google’s PageSpeed tool, it fetches your page once, with no browser and no JavaScript, and reports how many words of readable text came back. If the page is close to empty, it says so, and where it can tell, it names the likely cause, such as an empty container a framework fills in or text sitting in a data block for scripts to render. A PageSpeed score cannot tell you this, because PageSpeed loads the page with scripts running.
Check more than the homepage. Your pricing, services and contact pages are the ones an assistant is most likely to quote.
What fixing it looks like
The fix is to get the words into the HTML your server sends. How that happens depends on what is drawing the text with scripts.
| What we found | Usual fix |
|---|---|
| One section of a WordPress page built by a slider or builder widget | Rebuild that section with standard blocks that output plain text |
| Reviews, menus or prices from a third-party widget | Write the key facts on the page as text, and keep the widget for the interactive part |
| A site built as a JavaScript app | Switch on server-side or static rendering in the framework |
Server-side rendering means your server builds the full HTML for each request. Static rendering means the pages are built ahead of time, which is how this site works. Google’s guidance recommends these approaches and calls the older trick of serving crawlers a separate pre-built copy “a workaround and not a long-term solution.”
There is a side benefit. Google’s documentation notes that server-side or pre-rendering “makes your website faster for users and crawlers.” Text that arrives in the HTML can appear on screen before any script has finished, which helps on slow phones as well as with assistants. If your site is on WordPress, our WordPress speed fix covers the theme and plugin side of this. If you want the words checked and fixed on every key page, with a written crawler policy and your prices marked up so assistants quote them correctly, that is the AI-ready setup.
Start with one page
Pick the page you would most want an assistant to read correctly, usually your pricing or services page, and run it through the free check. It takes a minute or two, and the section headed “What an AI assistant sees” tells you how many of your words made it into the HTML. If the number is healthy, you can cross this off. If it is close to zero, you know where to start.