🔎 Focus: Technical SEO
🔴 Impact: High
🟠 Difficulty: Dumb Simple
Sponsored by Ahrefs
Want to automate the grunt work?
Meet Agent A, the AI teammate with full access to your Ahrefs data. From fixing keyword cannibalization to shipping technical reports straight into Notion or Google Docs, it does the work you’d rather not.
Dear Tech SEO 👋
We’ll dive today on a dead simple fix that took minutes to fix. It was on the robots.txt
Too Dumb Simple To Check?
You know when you just get on a SEO paranoid mode that you forget to check stupid simple things 🤦
In this edition I will show you how a simple robots.txt disallow rule ruined and boosted my traffic performance…
Don’ forget to check simple things…
It will take you just 3 mins…
Ready? Let’s goooo
A Silent SEO Mistake = Silent Revenue
Bad content, bad links, bad branding?
Why care about those when you got THE TECHNICAL INQUISITION!
Enough MEMEs for today
I got this marketplace on my hands and the real problem was much more dangerous.
Someone accidentally told Google that inventory did not exist.
MILLIONS of highly specific industrial products not available… The rest of the site was great:
Great site design
Clean navigation
Fast pages
Yet organic traffic was completely stuck. I’m talking about bad bad not good:
Website with millions of pages and only 401 clicks/day
The issue?
One careless robots.txt rule was blocking parameter URLs. BUT the rules was set incorrectly:
Pagination blocked: Product further than page 1 were not visible
Folder blocked: all product URLs were under a subfolder
The robots.txt directives looked something like this:
# 1. The Pagination Assassin (Blocked page 2, 3, etc.)
Disallow: *?p
# 2. The Product Hitman (Blocked product directory)
Disallow: /p
The marketplace was hiding the exact SKU pages customers were searching for when they were ready to buy.
Part numbers
Database identifiers
Product variations
All invisible to search engines, AI bots and ultimately, invisible to potential customers
The Dumb Fix Was A Robots.txt Directive
We allowed crawl access to parameters tied to the product subfolder, remove the disallow to paginated landings and that was pretty much it.
That single shift changed the Pages report within a few days:
Google started crawling product pages aggressively.
Then rankings appeared for hyper-specific SKU searches.
By week 8, organic traffic doubled.
+100% growth.
100% boost of traffic with a simple fix
Growth happened:
not because of new content. (well, technically it is new content if it was not crawled before… 🤔)
not because of backlinks
BUT because the machines could finally see the inventory
“Simple” fix, strong impact: the power of Tech SEO.
Next steps:
Improve categorization tree
Upgrade product descriptions
Optimize PDPs conversion rates
Are you suspicious of a Tech issue holding you back?
Reply to this email with TECH SEO and your domain name. I'll take a look and see the technical issues you should be working on right now. (Note: first come - first served, thousands of people receive this newsletter every week and cannot check them all 🥵)
Until next time 👋
—