Your JavaScript might deindex your site

Bring live Ahrefs data straight into ChatGPT & Claude

Want live Ahrefs data inside ChatGPT and Claude?
Plug real-time keyword metrics and backlink profiles straight into your AI prompts.
Query live SERP data without manual CSV exports or custom code.
Build efficient AI agents instantly.

Last week a ticket landed on my desk that looked boring.

Search Console was flagging product pages on the biggest electrical store in Poland we work with as “Excluded by noindex tag.” Thousands of SKUs. The kind of pages that pay the bills.

So I did what everyone does. I opened one of the URLs, hit Ctrl+U and searched for “robots.”

<meta name="robots" content="INDEX,FOLLOW">

Status 200. No X-Robots-Tag header. Nothing weird in robots.txt.

The page was telling me one thing and Google was telling me the opposite. And I’ve been doing this long enough to know that when those two disagree, Google is usually the one telling the truth. It just isn’t telling you why.

Here’s the thing most of us know but don’t really act on.

Google doesn’t just index the HTML your server sends. It later indexes the DOM after rendering. It fetches the HTML and tries to index it, then it runs the JavaScript in its Web Rendering Service, waits a bit, takes a snapshot, and indexes that in the 2nd wave of indexing.

On a static site, those are the same thing. On a modern JavaScript framework, they’re two different documents that usually look alike.

This store runs on Nuxt 3. Vue with server-side rendering, then client-side hydration. The server sends a fully rendered product page with INDEX, FOLLOW. Then the browser downloads the app, “wakes up” the page, and takes control of everything, including the head tags.

That last part matters more than people realise. Once the app has hydrated, your meta robots isn’t a line of HTML anymore. It’s runtime state. Any component on the page can change it. So can any error.

And I had a hunch it was an error.

Normal debugging goes like this. You load the page, it works, you shrug.

That’s the happy path, and the happy path is exactly where this bug doesn’t live. Googlebot isn’t a normal user. It fetches resources on its own schedule, caches them aggressively, sometimes renders an older copy of the HTML, and gets treated differently by firewalls and rate limiters.

So instead of asking “does the page work?” I asked, “what happens when one part of it doesn’t?”

I recorded every resource the product page loads. There were 161: JS chunks, CSS files, fonts, images, API calls, analytics beacons, cookie consent scripts.

Then I rendered the page 161 more times. Each time I made exactly one of those resources fail and checked what the meta robots looked like once the page settled.

The baseline render came back INDEX,FOLLOW. Good, the test setup worked.

Then the results started coming in.

58 of the 161 resources flipped the page to NOINDEX,NOFOLLOW.

Almost all of them were JavaScript chunks: the layout, the price module, the stock module, the add-to-cart button, the product carousel, the analytics wrapper. And then the ones that made me laugh out loud: tiny icon components. The home icon. The copy icon. The print icon.

If the file that draws a little printer icon failed to load, the whole product page told Google not to index it.

Once I knew where to look, the mechanism was simple. Annoyingly simple.

When something fails during hydration, the app doesn’t just skip that component. It throws, and the global error handler swaps the entire page for a client-side 500 error component.

That component does two things in its head config. It sets the title to a “500, an unexpected error occurred” message. And it sets meta robots to NOINDEX,NOFOLLOW.

On paper, that’s reasonable. You don’t want error pages indexed.

The problem is what doesn’t change. The HTTP status is still 200, because the server did its job. The URL is still the product URL. Only the rendered DOM is now an error page with a noindex on it.

And the rendered DOM is what Google indexes in the 2nd wave.

There was one more culprit that wasn’t a JS file. Out of 8 GraphQL queries on the page, one of them killed it on failure: the query fetching the lowest price before discount, the Omnibus-style price history every EU store now has to show.

I tested every way that request can go wrong. A 403. A 429. A 500. A plain network error. Even a 200 response with GraphQL errors in the body. Every single one turned the page into a noindexed 500. The only thing that didn’t was a request that hung forever, because then the error never fired.

So a small compliance widget that most shoppers never look at was effectively a kill switch for indexing.

The list of things that were safe to fail was interesting too. CSS, fonts, images, cookie consent, analytics beacons, the logged-in customer endpoint. Even the main entry script. If that one fails, the app never boots, so the server-rendered HTML with INDEX,FOLLOW just stays there untouched.

Think about that for a second. Failing to load the entire app was safer than failing to load one icon.

This is the part that explains why a bug like this survives for months.

The error page reloads itself after a few seconds. A shopper on a flaky connection might see a flash of “500” and then the product page comes back. Most wouldn’t even register it. No angry emails to customer support, no drop in conversion rate big enough to set off alarms.

Googlebot doesn’t wait around for the reload. It takes a snapshot of the rendered page at some point after load, and if that moment lands on the error state, the error state is what gets indexed.

So why would Googlebot hit failures more than real users? I’d point to two likely reasons.

The first is stale chunks after a deploy. Nuxt builds have hashed file names. Every deploy creates new ones and often deletes the old ones. Google’s renderer caches resources, and sometimes renders an older copy of the HTML that still points to last week’s chunk names. Those return 404s, hydration fails, and you get a noindexed 500 page.

The second is firewall and rate-limiting rules on the API. Bot protection that treats a crawler hammering a GraphQL endpoint as suspicious will happily serve it a 403 or 429. For a human, that’s one failed request. For this page it’s a noindex.

I checked robots.txt too, because of course I did. It wasn’t the culprit. The disallow rules that matched any of these requests only covered ones that were safe to fail.

You don’t need a custom setup for this. Chrome does it out of the box.

Open a product page and open DevTools. Press Ctrl+Shift+P (Cmd+Shift+P on a Mac) and run “Show Network request blocking.” Enable it and add a pattern for one resource, for example a single JS chunk or one API call. Reload.

Then go to the Elements tab and search for “robots.” Compare what you see with the raw HTML from Ctrl+U.

If those two don’t match, you have the same problem. It’s just waiting for the right failure to show up.

Do it for the boring stuff too. The icons, the tracking wrappers, the widgets nobody on the SEO side ever thinks about. Those were the ones that surprised me.

Then go to URL Inspection in Search Console, run a live test, and look at “Page resources” and the JavaScript console messages. That’s the closest you’ll get to seeing what Google actually failed to load.

The fixes I handed to the dev team weren’t complicated. They were just never on anyone’s radar.

None of these are SEO tasks in the traditional sense. They’re front-end architecture decisions. And I think that’s the real point of this whole story.

Picture the standard technical audit for this site.

A crawler runs, maybe with JavaScript rendering switched on. The pages load fine because the crawler is a well-behaved browser on a good connection, hitting freshly deployed chunks. Meta robots reads INDEX,FOLLOW. Green tick. On to the next section.

Nothing in that report would be wrong. It just wouldn’t be useful.

The bug only exists in the failure states. You only find it if you sit inside the stack, read the actual JavaScript, ask what happens when things break, and then go break them on purpose.

That’s why I’ve stopped believing in audits as the main deliverable for sites like this. Not because audits are useless, but because the problems that cost the most money on large JavaScript sites rarely show up so easily in audits.

For years I thought my job was to check that the page was right.

I’d argue now that the job is to find all the ways it can go wrong while still looking right.

If you’re running a big JavaScript storefront and you worry about rendering issues, reply “Rendering” and your domain. I will pick one site see what’s going on.