🔎 Focus: Site Crawling
🔴 Impact: High
🟢 Difficulty: Low
Sponsored by Ahrefs
Want to automate the grunt work?
Meet Agent A, the AI teammate with full access to your Ahrefs data. From fixing keyword cannibalization to shipping technical reports straight into Notion or Google Docs, it does the work you’d rather not.
We can’t catch a break
Since November 2025, we have been working with a large e-commerce client selling comics. We did a hell of a job:
Crawling and indexation cleaned up - 600k pages removed from Google database
Site structure is bigger with 3x the number of pages and cleaned up of low-quality pages
Site quality went through the roof, and we grew the traffic by 300-400% in 6 months
Solid Growth in 6 months. We’re cooking
We slowed down the cooperation so the client can focus on the fulfillment but we still monitor what is going on on the site. And then we found this - Google started crawling thousands of the 403 pages on the site since June 1st.
403 page crawling
Google was spending around 20-25% of its budget on 403 pages. It’s a huge quality blow, and it’s time to react. We also saw that this crawling issue could have impacted the performance already - we saw an impression drop literally on the same day.
Impression drop literally on the same day
The App that destroyed crawling
Client started using the Propel Replays app on Shopify for heatmaps. The app started creating thousands of such URLs for every page in the structure
/apps/propel-replays/api/public/shop?shop=brand-name.myshopify.com&pageUrl=https://brand-name.com/products/product-URL
This got even more interesting when I started investigating those URLs. They were returning status code 200, what?
Status code 200 for default user agents.
However, when I switched the user-agent to Googlebot, the status code changed to 403 - we found it.
403 status code for Googlebot user agent
We decided to react quickly, and we blocked those pages in robots.txt so Google is not visiting them. We’re waiting for the results and will let you know in another newsletter issue.
I hate it when such small issues literally destroy months of great SEO work. That’s why I like to be a 2nd or 3rd set of eyes for my clients.
The Quick Fixes Are Not That Easy To See
I audited over 1000 sites in my life, and every single one struggled with issues that are easy to fix, but hard to find. Those technical issues are what actually destroy website quality from the SEO perspective and impact your business.
Is your website perfect?
Reply to this email with “Tech Fix” and your domain. I will see what hard to spot issue I can find on your site.
Until next time 👋
oh that’s a human
—