🔎 Focus: Pagination Crawling & Indexation
🔴 Impact: High
🟡 Difficulty: Medium
Sponsored by Ahrefs
Stop wondering if AI is talking about your brand. This post is brought to you by Ahrefs, the leading SEO & marketing intelligence platform.
With their new Brand Radar tool, you can finally monitor your visibility across major AI engines: from ChatGPT to Google’s AI Overviews. Secure your brand as the top choice in the era of AI.
A Fortune 100 website with 40 million pages… and only 3% indexed.
Back in 2021, I started working on a Fortune 100 project.
The largest project in my career. A huge client selling heavy machinery. The project was in a highly advanced stage in terms of SEO, but their websites were super broken from the technical SEO standpoint.
I focused on their website selling spare parts for their machines. This website has over 30 different language versions and over 1 million products.
Overall, when I sat down to the project, I saw a sitemap with over 40,000,000 pages to index. That's huge.
Me looking at a Fortune 100/
I checked the indexing report, and I saw that only 1.2 million pages were actually indexed - only around 3% of pages from the sitemap.
Then I checked the Crawl Stats report in Google Search Console and I found that Google was crawling from 8,000 to 10,000 HTML pages per day.
Yes, Google was visiting only around 0,02% of the site every day.
If we theorize that Google would crawl every single page only once, it would take Google almost 14 years (!) to crawl the entire site - literally half my life.
The site was heavily relying on product page traffic, so indexing more pages just made sense in their case. I started working on improving the crawling and indexation of the site.
How we indexed millions of pages:
We fixed JavaScript pagination - Pagination was added fully with JavaScript; there were no links to further paginated pages, so Google couldn't crawl further paginated pages and products.
We added crawlable pagination with proper <a href> linksWe fixed crawling issues - I found over 40 different crawling issues on the site. Some of them were creating literally over 200 copies of the entire site (literally, Google could crawl close to 10 billion pages on the site).
We blocked low-quality sections of the site and removed legacy pages that were wasting Google resourcesWe fixed rendering issues - the site was fully client-side-rendered. Crucial parts of the website content were added with JavaScript.
We moved crucial sections of pages to render to the server side to help Google actually find them in the clean HTML without the need for JS renderingWe improved average page quality - We added product images, internal linking sections, and fixed structured data.
Overall page quality improved, and Google started indexing them waaay more efficiently.We updated XML sitemaps - This client had over 30 language versions. We triggered a recrawl of the site with all the previous changes after we updated XML sitemaps when adding new language versions, together with new hreflangs in XML sitemaps.
This recrawl boosted the indexation and crawling significantly.
Results:
The number of indexed pages grew from 1,200,000 pages to over 9,000,000 pages.
The indexation grew from 3% of the site to around 22,5%. Still a long way to go but a nice improvement.
Last indexation improvements in 2023
The traffic has grown by 400% since we started this project - The traffic doubled YoY for 2 years in a row.
We went from around 3000 daily clicks to around 12,000 daily clicks
Traffic changes from the time of the indexation project
Revenue from SEO grew by 110%
Users were able to find spare parts directly from the Google Index and buy them way quicker.Crawling grew from Google crawling 8,000 pages every day to crawling up to 1,200,000 pages per day - 150x improved Crawling. Massive
This was the case that made me understand how powerful indexing for Google and SEO traffic.
Improving page quality and simplifying Google routes pointing to your pages and content is key when you want to improve indexation.
Reply “Indexation Audit” and your domain I’ll show you the exact indexation fixes needed on your site.
I use them for enterprise e-commerce clients every day.
How I analyze Technical SEO on Fortune 100 Stores.
Here I audited Lowe’s - huuuuuge US-based hardware store. Before you ask, yes, they also struggle with indexation.
More Fortune 100 brand analyses are coming. Should I livestream this?
Until next time 👋
oh that’s a human
Recommended Reads
🔎 Understand the JavaScript SEO basics
[Google Developers]
🔎 Overview of crawling and indexing topics
[Google Developers]
—