Programmatic SEO Competitor Spy

Before building your own programmatic SEO pages, know what your competitors are already doing. This skill crawls competitor sitemaps, detects which URL patterns are programmatic, estimates which ones actually drive traffic, assesses template quality, and finds the gaps they're missing.

Core principle: The best pSEO strategy starts with competitive intelligence. If 3 competitors have /vs/ pages and none have /for-{industry}/ pages, that's a signal. This skill turns competitor site structures into strategic intelligence.

When to Use

• "What programmatic SEO are competitors doing?"
• "Reverse-engineer competitor SEO pages"
• "Which pSEO patterns actually work in our space?"
• "Find pSEO gaps our competitors are missing"
• "Analyze competitor URL structure for SEO patterns"

Tool Enhancement (Optional)

This skill works with existing capabilities but is dramatically better with domain analytics data that shows actual traffic per URL pattern.

Agent Prompt to User

"I can analyze competitor site structures with our existing crawling tools. For the most accurate results — especially knowing which patterns actually drive traffic vs. just existing — I'd recommend connecting a domain analytics API."

Recommended: DataForSEO (pay-per-use, ~$0.02-0.05 per domain analysis)

• Sign up at dataforseo.com → get API login + password
• Set DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD env vars

Alternatives that also work:

• SEMrush API (if you have a subscription) → set SEMRUSH_API_KEY
• Ahrefs API (if you have a subscription) → set AHREFS_API_TOKEN
• SimilarWeb API (if you have access) → set SIMILARWEB_API_KEY

"Want to use one of these, or should I proceed with baseline mode? Baseline will detect patterns and assess template quality, but traffic estimates will be inferred rather than measured."

Mode Selection

• Enhanced mode — Domain analytics API provides organic keywords and estimated traffic per URL. Enables confident "this pattern drives X traffic" conclusions. Can show which specific pages within a pattern rank well.
• Baseline mode — Uses site-content-catalog for URL crawling, seo-domain-analyzer for domain-level metrics, web_search for spot-checking rankings. Pattern detection is equally good. Traffic attribution is directional, based on page count × domain authority × keyword indicators.

Phase 0: Intake

1. Competitors — 1-5 competitor URLs to analyze
2. Your product URL — So we can compare coverage
3. Category — What market/category are these competitors in?
4. Specific interest? — Looking for anything specific (vs/ pages, integrations, etc.) or full analysis?
5. Tool preference — Enhanced mode with domain analytics API, or baseline? (see Tool Enhancement above)

Phase 1: Sitemap Crawl

For each competitor, run site-content-catalog:

python3 skills/site-content-catalog/scripts/catalog_content.py \
  --url "<competitor_url>" \
  --output json

Collect:

• All URLs from sitemap and internal crawl
• Page titles
• URL structure/hierarchy
• Last modified dates (if available)

Phase 2: Pattern Detection

2A: URL Clustering

Group URLs by structural patterns using regex matching:

For each detected pattern:

• Page count — How many pages follow this pattern?
• Data axis — What varies per page? (competitor name, tool name, industry, etc.)
• URL consistency — Are URLs cleanly structured or messy?
• Coverage completeness — Have they covered the obvious variations?

2B: Programmatic vs. Editorial Classification

Not every URL pattern is programmatic. Classify each cluster:

• Programmatic — Clearly templated: consistent URL structure, similar page titles following a pattern, high page count, structured content
• Semi-programmatic — Template with heavy manual customization
• Editorial — Individually written, no template pattern
• Auto-generated — Thin/low-quality programmatic (tag pages, archive pages)

Focus analysis on Programmatic and Semi-programmatic clusters only.

Phase 3: Traffic & Performance Analysis

Enhanced Mode (DataForSEO / SEMrush / Ahrefs)

For each competitor domain, pull organic keywords data:

# DataForSEO example
POST /v3/dataforseo_labs/google/ranked_keywords/live
{
  "target": "competitor.com",
  "filters": [
    ["ranked_serp_element.serp_item.url", "contains", "/vs/"]
  ]
}

Per pattern cluster:

• Total ranking keywords across all pages in the pattern
• Estimated monthly organic traffic to the pattern
• Average ranking position for pages in this pattern
• Top-performing pages within the pattern (which variations rank best?)
• Keyword difficulty distribution (are they targeting easy or hard keywords?)

Baseline Mode

Estimate traffic indicators:

• Run seo-domain-analyzer for overall domain metrics
• Spot-check 3-5 pages per pattern via web_search — do they appear in top 10?
• Check if pages are indexed (site:competitor.com/vs/)
• Infer relative traffic: pattern page count × indexation rate × domain authority proxy
• Note: these are directional estimates, not exact numbers

Phase 4: Template Quality Assessment

For each programmatic pattern, fetch 3-5 sample pages via fetch_webpage:

# Pick pages from the pattern — one high-variation, one low-variation, one middle

Evaluate template quality on:

Quality tiers:

• 4-5 avg: Well-executed pSEO — hard to beat without significant differentiation
• 3-4 avg: Decent but improvable — opportunity to out-template them
• 1-3 avg: Thin/low-quality — easy to dominate with better content

Phase 5: Gap Analysis

5A: Pattern Coverage Matrix

5B: Variation Gaps

Within each pattern competitors use, find missing variations:

• Competitors they haven't written vs/ pages for
• Integrations they support but haven't built pages for
• Industries they serve but haven't targeted
• These are your quick-win entries

5C: Pattern White Space

Entire pattern types no competitor has built:

• Highest oppo