🔎 Focus: Technical SEO Strategy
🔴 Impact: High
🟠 Difficulty: Mid-High

Want to automate the grunt work?
Meet Agent A, the AI teammate with full access to your Ahrefs data. From fixing keyword cannibalization to shipping technical reports straight into Notion or Google Docs, it does the work you’d rather not.

Dear Tech SEO 👋

Today I want to share the most critical Tech SEO upgrades to work on from now until the end of the year.

1 - Audit robots.txt for AI crawlers: Block zero-ROI training bots (GPTBot, ClaudeBot) while explicitly allowing revenue-generating citation crawlers (OAI-SearchBot, PerplexityBot, Google-Extended).

2 - Tier 1 vs. Tier 2 AI Retrieval: AI search engines first attempt to answer user prompts using fast (<200ms) search engine API snippets (Tier 1)[cite: 1]. If snippet data is insufficient or the query is complex, they trigger a live agentic crawl to inspect your full page HTML (Tier 2)[cite: 1].

3 - Search engine API dependencies: LLMs query traditional search engine APIs first; if your pages aren't indexed in search engines, AI crawlers will not cite them.

4 - Sitemap & llms.txt Limitations: AI search crawlers fetch specific URLs on demand via search API results rather than crawling full sitemaps or reading llms.txt files like traditional spiders.

5 - Fix international hreflang setup: Align regional and language pairs correctly to prevent regional index cannibalization and ensure proper location targeting.

6 - Avoid and solve bot attacks: Stop dynamic parameter spikes from draining crawl budget using Cloudflare WAF and .htaccess 410 headers.

7 - Apply noindex, follow to paginated landings: Preserve crawl budget on ?page=2+ while leaving deep category and product links open for discovery.

🙋‍♂ Warning: this technique is not a common practice and requires a careful implementation and monitoring. Implement with care and with a Tech SEO specialist like my buddy Patryk Wawok.

8 - Build semantic category cluster networks: Link broad categories (/shoes) to long-tail intent hubs (/shoes/wide-feet) to align with LLM vector search and query fan-out.

9 - De-JavaScript core navigation: Ensure top navigation and mega-menus are visible in raw HTML source code so crawlers don't skip category trees.

10 - De-obfuscate product recommendation links: Force third-party "Related Products" widgets to output standard HTML links instead of lazy-loaded JS elements.

11 - Implement contextual navigation siloing: Avoid universal site menu bloat across every page to preserve thematic category authority.

12 - Shift to Server-Side Rendering (SSR): Deliver instant full HTML payloads so AI crawlers don't time out or miss critical page text.

13 - Hit the 2.5s LCP citation threshold: Pages meeting the 2.5s Core Web Vitals target are 1.47x more likely to be cited in AI search responses.

14 - Implement Markdown content negotiation: Enable Accept: text/markdown server responses to deliver lightweight text payloads directly to AI agents.

15 - Optimize mobile above-the-fold grids: Fit at least 2 products per row on mobile layouts to reduce Cumulative Layout Shift (CLS) and scroll depth.

16 - Enforce explicit @graph entity wiring: Connect internal schema nodes explicitly by linking #org to #product or #article.

17 - Sync JSON-LD with merchant ad feeds: Align landing page structured data directly with Google Merchant Center feeds to avoid product rejection.

18 - Clean up conflicting structured data scripts: Remove duplicate or contradictory JSON-LD templates running simultaneously on product pages.

19 - Optimize for Chrome's Accessibility Tree: Use native semantic tags (<button type="submit"> vs <span class="buy-btn">) so autonomous AI agents can parse and click buy buttons.

20 - Expose Universal Commerce Protocol (UCP) manifests: Implement /.well-known/ucp and /.well-known/mcp/server-card.json endpoints so agents can read store rules programmatically.

21 - Prepare Agent Payment Protocol (AP2) infrastructure: Configure cart and checkout paths for tokenized, machine-to-machine transactions.

22 - Maintain clean <a> vs. <button> semantics: Style <a> tags with CSS instead of nesting <button> tags inside links to preserve both crawlability and accessibility.

23 - Audit server logs for "absence of signal": Check if high-margin revenue pages receive zero bot visits; uncrawled pages cannot be indexed or cited.

24 - Verify the JavaScript rendering gap: Ensure JS assets load at the exact same frequency as raw HTML pages in server logs.

25 - Clear negative SEO parameter backlogs: Issue immediate 410 status codes on spam parameter URLs to protect overall domain authority.

That’s it for today, practical and simple.

Until next time 👋

Jairo

—