seo-llms-txt

llms.txt — Generate, Validate & Audit

Analyze, generate, or validate llms.txt files per the llms.txt specification (proposed by Jeremy Howard / Answer.AI). The llms.txt standard provides a Markdown file at /llms.txt that helps LLMs understand a website's content structure — similar to how robots.txt guides traditional crawlers.

Adoption: 950+ sites including Anthropic, Cloudflare, Docker, Vercel, Stripe, FastHTML, Cursor, Hugging Face, and many more.

Inputs

Three modes, auto-detected from the user's input:

| Input Pattern | Mode | Example | |---|---|---| | /seo llms-txt <url> | Audit | /seo llms-txt example.com | | /seo llms-txt generate <url> | Generate | /seo llms-txt generate example.com | | /seo llms-txt validate <url-or-path> | Validate | /seo llms-txt validate /path/to/llms.txt |

• If user provides a URL without sub-command → Audit mode (fetch and check existing file)
• If user says "generate" → Generate mode (crawl site, produce llms.txt)
• If user says "validate" → Validate mode (parse and check against spec)

llms.txt Spec Reference

@skills/seo/references/llms-txt-spec.md

Quick Spec Summary

The llms.txt file must be valid Markdown with this structure:

1. H1 heading (required) — Project/site name
2. Blockquote (optional) — Short site summary
3. H2 sections — Categorized link lists with descriptions
4. Link format: - [Name](url): Description (colon + space after URL)
5. ## Optional section — Links useful but not essential for understanding
6. Companion files (optional):
7. llms-full.txt — Comprehensive version with full content
8. Individual .md page mirrors for key pages

Execution: Audit Mode

When user provides a URL (no sub-command):

1. Fetch llms.txt: Use WebFetch to retrieve <url>/llms.txt
2. Run 8-point check:

| # | Check | Pass Criteria | |---|---|---| | 1 | File exists | HTTP 200 at /llms.txt | | 2 | Valid Markdown | Parseable as Markdown, no broken syntax | | 3 | Has H1 | Exactly one # heading | | 4 | Has blockquote summary | > line present after H1 (recommended) | | 5 | Has H2 sections | At least one ## section with links | | 6 | Link format correct | Links follow - [Name](url): description pattern | | 7 | Has ## Optional | Contains an Optional section (recommended) | | 8 | Companion llms-full.txt | llms-full.txt exists at same path (recommended) |

1. Link spot-check: WebFetch up to 5 linked URLs to verify they resolve (HTTP 200)
2. Generate score card and recommendations

Execution: Generate Mode

When user says "generate":

1. Crawl site structure:
2. WebFetch the homepage — extract title, meta description, nav links
3. WebFetch /sitemap.xml (or /sitemap_index.xml) — extract all URLs
4. If no sitemap, follow nav links from homepage (up to 50 pages)

1. Categorize pages into sections based on URL patterns and content:
2. ## Docs — /docs/, /documentation/, /guides/, /reference/
3. ## Blog — /blog/, /articles/, /posts/, /news/
4. ## API — /api/, /developers/, /reference/
5. ## Products — /products/, /features/, /pricing/, /solutions/
6. ## About — /about/, /team/, /company/, /contact/
7. ## Legal — /privacy/, /terms/, /legal/
8. ## Optional — Catch-all for remaining useful pages

1. Extract metadata per page:
2. Title (from <title> or <h1>)
3. Description (from meta[name=description] or first paragraph)
4. Truncate description to ~100 characters

1. Generate llms.txt:

```markdown # Site Name

> Brief site description from meta or homepage hero text.

## Docs

• Getting Started: Introduction and setup guide
• API Reference: Complete API documentation

## Blog

• Latest Post: Description of the post

## Optional

• About: Company background
• Contact: Get in touch

```

1. Generate llms-full.txt skeleton (optional — note to user):

`` Suggest that llms-full.txt should contain the full Markdown content of all key pages concatenated. Provide the structure but note that generating full content requires crawling every page. ``

1. Output the generated file — display it in a code block and offer to write to disk

Execution: Validate Mode

When user says "validate":

1. Load the file:
2. If URL → WebFetch <url>/llms.txt
3. If local path → Read the file

1. Run spec compliance checks:

| # | Rule | Severity | Check | |---|---|---|---| | 1 | Has exactly one H1 | FAIL | Count # lines (not ##) | | 2 | H1 is first heading | FAIL | First line starting with # must be H1 | | 3 | Valid Markdown | FAIL | No broken link syntax, unclosed brackets | | 4 | Links use correct format | WARN | - [Name](url): description pattern | | 5 | Has at least one H2 section | FAIL | At least one ## heading | | 6 | No empty sections | WARN | Every H2 should have at least one link | | 7 | Blockquote present | INFO | > after H1 is recommended | | 8 | ## Optional section exists | INFO | Recommended by spec | | 9 | No deep heading levels | WARN | Only H1 and H2 allowed (no H3+) | | 10 | Descriptions present | WARN | Links should have : description after URL |

1. Output pass/fail per rule with line numbers for failures

Output Format

Audit Mode Output

## llms.txt Audit: [domain]

**Score: X/8 checks passed**

| # | Check | Result | Details |
|---|---|---|---|
| 1 | File exists | PASS/FAIL | [status code or error] |
| 2 | Valid Markdown | PASS/FAIL | [issue if any] |
| ... | ... | ... | ... |

### Link Spot-Check
| URL | Status |
|---|---|
| [url] | 200 OK / 404 / timeout |

### Recommendations
- [Prioritized list of improvements]

### What is llms.txt?
> llms.txt is a proposed standard (llmstxt.org) that provides LLMs with
> a structured Markdown overview of your site. 950+ sites have adopted it.
> While AI platform adoption of the standard is still emerging, early
> adoption positions your site for AI search visibility — similar to how
> early schema.org adoption paid off for rich snippets.

Generate Mode Output

## Generated llms.txt for [domain]

[The generated llms.txt content in a code block]

### Generation Notes
- Pages crawled: [count]
- Sections created: [list]
- Pages excluded: [count and reason]

### Next Steps
1. Review and edit the generated file
2. Upload to your site root as `/llms.txt`
3. Consider creating `llms-full.txt` with full page content
4. Add `<link rel="help" href="/llms.txt">` to your HTML head (optional)

Validate Mode Output

## llms.txt Validation: [source]

**Result: X/10 rules passed** | Y warnings | Z info

| # | Rule | Result | Line | Details |
|---|---|---|---|---|
| 1 | Has H1 | PASS/FAIL | [line#] | [details] |
| ... | ... | ... | ... | ... |

### Issues to Fix
- [FAIL items with specific fix instructions]

### Recommendations
- [WARN items with suggestions]