technical-seo

Technical SEO Audit — Phase 2

Executive Summary

Technical SEO is the infrastructure layer that determines whether all other optimization work is visible to Google. A technically broken site cannot rank regardless of content quality or link authority. In 2025, technical SEO has expanded to include three new critical areas: (1) INP (Interaction to Next Paint), which replaced FID as a Core Web Vital in March 2024 — target <200ms; (2) AI rendering readiness — Googlebot now renders JavaScript pages before indexing, meaning JS-dependent content must be verified as indexable; (3) GenAI citation readiness — structured data (Organization, FAQPage, HowTo) is the primary mechanism by which Google's AI Overviews and third-party LLMs extract and cite site content. A technically clean site is the prerequisite for AI visibility.

2025 technical SEO benchmarks:

• INP (Core Web Vital, replaced FID March 2024): <100ms Excellent; <200ms Good; >500ms Poor
• LCP: <2.5s Good; <4.0s Needs Improvement; >4.0s Poor
• CLS: <0.1 Good; <0.25 Needs Improvement; >0.25 Poor
• TTFB: <800ms Good; <1,800ms Needs Improvement; >1,800ms Poor
• Crawl budget (large sites): indexed pages should be within 20% of sitemap count
• HTTPS: required — any HTTP-accessible URL is a trust and ranking signal issue
• Canonical coverage: every page must have a canonical (self-referencing or pointing to preferred)
• Structured data: Organization + LocalBusiness schema on homepage; FAQPage on all service pages; HowTo where applicable
• Hreflang: required for any multi-language or multi-region site — errors cause ranking in wrong country
• Site speed: Google uses mobile-first indexing — test mobile CWV separately from desktop

Numbered Action Plan:

Immediate (Week 1, No Dev Needed)

1. Run site crawler — python3 scripts/site_crawler.py --url [URL] --max-pages 100 --output {DATA_DIR}/crawl/ --csv. Captures all pages, status codes, titles, meta descriptions, H1s, word counts, internal links. Effort: 30 min setup + crawl time.
2. Check GSC Coverage report — identify Excluded, Crawled Not Indexed, and Discovered Not Indexed pages. Flag any page type systematically excluded. Effort: 20 min.
3. Validate robots.txt — curl https://domain.com/robots.txt. Check for accidental Disallow: / rule. Verify sitemap URL is listed. Effort: 5 min. Priority: 25 (5×5).
4. Validate XML sitemap — check all sitemap URLs return 200, are in canonical form, and match GSC indexation count. Flag if sitemap includes redirect or noindex URLs. Effort: 15 min.
5. Test HTTPS enforcement — navigate to http://domain.com (no https) — should 301 redirect to HTTPS. Check SSL expiry date. Effort: 5 min.

Short-Term (Week 2–4, Dev Involvement)

1. Fix redirect chains — use curl -IL [URL] to trace chains. Any 301→301→200 chain = wasted PageRank. Flatten to direct 301. Effort: 30 min identification + 1–2 hrs dev. Priority: 20.
2. Audit canonical tags — check for: missing canonicals, self-conflicting canonicals, canonicals pointing to redirect URLs, paginated pages with wrong canonical. Effort: 1–2 hrs.
3. Fix duplicate content — www vs non-www, http vs https, trailing slash vs no trailing slash — all should 301 to one canonical version. Check: site:domain.com -www. Effort: 1 hr dev.
4. Implement missing structured data — add Organization + LocalBusiness schema to homepage, FAQPage to all service pages, HowTo to process pages. Use Google's Rich Results Test to validate. Effort: 2–4 hrs.
5. Test JavaScript rendering — check key content (services, contact info) via Google Search Console → URL Inspection → "View Tested Page" → rendered HTML. Any missing content = JS rendering issue. Effort: 30 min.

Step 1: Read Project Context & Run Crawl

Read {AUDIT_DIR}/intake-data.md for business name, URL, and project paths.

# Run site crawler for comprehensive technical data
python3 scripts/site_crawler.py \
  --url [WEBSITE_URL] \
  --max-pages 100 \
  --output {DATA_DIR}/crawl/ \
  --csv

# Validate URL + basic signals
python3 scripts/check_url.py --url [WEBSITE_URL] --full

Use crawler output (crawl-issues-*.json) to populate findings. Supplement with:

Tools for this phase: | Tool | Purpose | Cost | |------|---------|------| | Screaming Frog SEO Spider | Full site crawl — redirects, canonicals, status codes, schema, headings | Paid/Free (≤500 URLs) | | Google Search Console | Index coverage, manual actions, Core Web Vitals report, URL inspection | Free (requires access) | | Ahrefs Site Audit | Crawl health score, broken links, orphan pages, crawl depth issues | Paid | | SEMrush Site Audit | Technical health score, crawlability, HTTPS, Core Web Vitals | Paid | | Google Rich Results Test | Schema validation — confirms rich result eligibility per page | Free | | Schema Markup Validator (validator.schema.org) | Schema syntax validation without rich result eligibility check | Free | | Lighthouse | CWV, best practices, PWA check — per-page technical baseline | Free | | Wayback Machine | Historical snapshots — detect major content/structure changes | Free |

Section 1: Crawlability & Indexation

Robots.txt Analysis

Fetch [domain]/robots.txt → check:

• File exists and returns HTTP 200?
• Blocking critical resources (CSS, JS files needed for rendering)? → Use Google URL Inspection → View Rendered Page to verify
• Correct User-agent directives (Googlebot, GPTBot, PerplexityBot if desired)?
• Sitemap(s) declared (Sitemap: https://domain.com/sitemap.xml)?
• No accidental Disallow: / on production site?
• Rate limiting: Crawl-delay directive set? (optional — Google mostly ignores it)

AI Crawler Access Check (14 Crawlers)

AI crawlers are now as important as Googlebot for visibility. Over 35% of top 1,000 websites inadvertently block at least one major AI crawler (Originality.ai 2025). Check robots.txt for all 14 AI crawlers:

Tier 1 — AI Search (MUST ALLOW for AI visibility):

| Crawler | User-Agent | Operator | Check Result | |---------|-----------|----------|-------------| | GPTBot | GPTBot | OpenAI (ChatGPT Search, 900M+ users) | Allowed / Blocked / Not Mentioned | | OAI-SearchBot | OAI-SearchBot | OpenAI (search-only, no training) | | | ChatGPT-User | ChatGPT-User | OpenAI (user-initiated browsing) | | | ClaudeBot | ClaudeBot | Anthropic (Claude web search) | | | PerplexityBot | PerplexityBot | Perplexity (best AI referral traffic) | |

Tier 2 — AI Ecosystem (RECOMMEND ALLOW):

| Crawler | User-Agent | Operator | Notes | |---------|-----------|----------|-------| | Google-Extended | Google-Extended | Google — Gemini training. Blocking does NOT affect search ranking or AIO appearance. | | GoogleOther | GoogleOther | Google — research, experimental features | | Applebot-Extended | Applebot-Extended | Apple — Apple Intelligence (2B+ devices) | | Amazonbot | Amazonbot | Amazon — Alexa, Amazon AI | | FacebookBot | FacebookBot | Meta — Meta AI (3B+ app users) |

Tier 3 — Training Only (strategy-dependent):

| Crawler | User-Agent | Recommendation | |---------|-----------|---------------| | CCBot | CCBot | Context-dependent (Common Crawl training data) | | anthropic-ai | anthropic-ai | Context-dependent (Claude training, separate from ClaudeBot) | | Bytespider | Bytespider | BLOCK for most Western businesses (aggressive, low value) | | cohere-ai | cohere-ai | Context-dependent (enterprise AI training) |

Also check for:

• <meta name="robots" content="noai"> on key pages (blocks all AI use)
• X-Robots-Tag: noai HTTP headers
• Bot-specific meta tags: <meta name="GPTBot" content="noindex">
• JavaScript-rendered content (AI crawlers do NOT execute JS — content invisible to them)

AI Crawler Access Score: Tier 1 all allowed = 50 pts; Tier 2 all allowed = 25 pts; no blanket AI blocks = 15 pts; llms.txt present = 10 pts. Total: /100.

IndexNow Protocol Check

I