api-docs-quality-report

API Docs Audit Skill

Crawls every API reference endpoint page on a documentation site, scores each against 5 quality checks, detects site-wide patterns, and outputs an interactive HTML report matching the style of previously-generated audits in this session.

STEP 0 — COLLECT INPUT

You need exactly one input: the docs URL.

If the user hasn't provided it, ask:

Which API docs URL should I audit?

If the URL is provided, strip trailing slashes and confirm it before proceeding.

Special case — "do the same for X": If the user says "do the same" or "same audit for X", they want the identical workflow and HTML report format applied to the new URL. Proceed directly to Step 1.

STEP 1 — DISCOVER ALL ENDPOINT PAGES

1a. Try llms.txt first

Fetch <docs_url>/llms.txt.

If found:

• Extract every line matching the pattern - [Page Title](URL): description
• Filter to API reference pages only — look for paths containing any of:

api-reference, api-reference/, /api/, /endpoints/, /rest/, /docs/, /reference/ or versioned paths (e.g., /v1/)

• Exclude: overview/index pages (no HTTP method implied), SDK docs, CLI docs,

Terraform docs, guides, quickstart pages

• Store as ENDPOINT_PAGES[] — list of {title, url, description}

If not found or returns 404:

• Fetch the docs homepage
• Look for a nav sidebar or sitemap link referencing API/endpoints
• Extract all API reference page URLs from the nav structure
• Note in the report that llms.txt was unavailable

1b. Identify categories

Group discovered pages by their URL path prefix or nav section:

• e.g. /api-reference/agents/ → Agents category
• e.g. /api-reference/executions/ → Executions category
• Pages at the root level of api-reference → Uncategorized or infer from title

Store CATEGORIES[] — list of {name, page_count, pages[]}.

Report discovery summary:

Found N endpoint pages across K categories. Fetching now...

STEP 2 — CHECK OPENAPI SPEC AVAILABILITY

Before fetching individual pages, check if a standalone OpenAPI spec exists.

Try these discovery methods in order (stop at first success):

1. Inspect the Link HTTP header (RFC 8288) on the homepage or docs root for rel="openapi" or rel="describedby" — this is the standard discovery mechanism for API specifications
2. <docs_url>/openapi.json
3. <docs_url>/openapi.yaml
4. <docs_url>/api/openapi.json
5. Any URL referenced in llms.txt under "## OpenAPI Specs"

Result:

• If accessible: SPEC_AVAILABLE = true, store the URL
• If 404 on all attempts and no Link header found: SPEC_AVAILABLE = false

A missing spec is itself a top-priority finding — record it as Issue #1.

STEP 3 — FETCH AND PARSE ALL ENDPOINT PAGES

Fetch each URL in ENDPOINT_PAGES[]. For each page, extract:

3a. Page-level prose description

The text that appears above the OpenAPI block (or between the # title and the first ## OpenAPI heading). This is the human-written description.

3b. OpenAPI block

Look for a fenced code block with language hint yaml or json that contains an OpenAPI spec. Extract:

• HTTP_METHOD — e.g. GET, POST, PUT, DELETE, PATCH
• HTTP_PATH — e.g. /api/v1/agents/{agent_id}/execute
• REQUEST_BODY_SCHEMA — the requestBody schema section (if any)
• PARAMETERS — path + query params (if any)
• RESPONSES — the full responses object
• RESPONSE_200_SCHEMA — the schema under responses.200 or 201

If no OpenAPI block found, mark SPEC_INLINE = false.

STEP 4 — SCORE EACH ENDPOINT (5 CHECKS)

For every endpoint, evaluate all 5 checks. Store results as {check_name: "pass"|"warn"|"fail", current_state: string, fix_guidance: string}.

Read references/scoring-rules.md for the exact scoring criteria per check.

Check 1 — Endpoint Description Check 2 — OpenAPI Spec Check 3 — Body Param Descriptions Check 4 — Response Codes Check 5 — Response Schema

STEP 5 — DETECT SITE-WIDE PATTERNS

After scoring all endpoints, look for patterns that repeat across many pages.

A pattern qualifies if it affects ≥ 60% of endpoints:

| Pattern to detect | How to identify | |---|---| | Universal missing error codes | All endpoints missing 401, 429, 500 etc. | | Universal empty response schemas | schema: {} on most endpoints | | Free-form body schemas | additionalProperties: true on most request bodies | | Inline spec only / broken canonical spec | All pages embed spec inline but canonical 404s | | One-liner descriptions everywhere | Descriptions under 10 words on most pages | | 422-only validation errors | Only FastAPI's 422 documented, not business errors |

For each detected pattern, write a single paragraph summarising:

• What the pattern is
• How many endpoints it affects
• Root cause (e.g. FastAPI auto-generation, missing middleware docs)
• Impact on developer experience

Store as SITE_WIDE_PATTERNS[].

STEP 6 — COMPILE TOP ISSUES

Rank the 10 most impactful issues across all findings. Prioritise by:

1. Issues affecting the most endpoints
2. Issues on the highest-traffic / most important endpoints
3. Security-sensitive gaps (auth endpoints, credential endpoints with empty schemas)
4. Deprecated endpoints with no migration path
5. Spec availability (404 canonical spec blocks all SDK tooling)

For each issue, write:

• A concise <strong>title</strong>
• 1-2 sentence description with specific endpoint names or counts
• Badge: FAIL or WARN

Store as TOP_ISSUES[] (max 10).

STEP 7 — BUILD SCORECARD

For each category, calculate:

• total — endpoint count
• pass_count — endpoints with all 5 checks passing
• warn_count — endpoints with at least one WARN and no FAIL
• fail_count — endpoints with at least one FAIL

Calculate overall totals.

Store as SCORECARD[].

STEP 8 — GENERATE HTML REPORT

Read references/html-template.md for the complete HTML structure, CSS variables, JavaScript, and all section markup.

Substitute all {{VARIABLE}} placeholders with real audit data, ensuring all values are HTML-escaped to prevent XSS.

Key variable map:

| Variable | Source | |---|---| | {{DOCS_URL}} | Input URL | | {{AUDIT_DATE}} | Today's date | | {{TOTAL_ENDPOINTS}} | Count of ENDPOINT_PAGES | | {{TOTAL_PASS}} | Sum of all pass_count across categories | | {{TOTAL_WARN}} | Sum of all warn_count | | {{TOTAL_FAIL}} | Sum of all fail_count | | {{SPEC_COUNT}} | 1 if SPEC_AVAILABLE, else 0 | | {{SCORECARD_ROWS}} | Generated <tr> rows from SCORECARD | | {{PATTERN_BOX}} | Pattern box HTML (if SITE_WIDE_PATTERNS found) | | {{TOP_ISSUES_HTML}} | Generated issue row HTML from TOP_ISSUES | | {{SIDEBAR_LINKS}} | Generated sidebar <div class="sbl"> links | | {{SECTIONS_HTML}} | All section HTML for each category |

For section HTML generation rules, see references/html-template.md → Section: SECTIONS_HTML.

Output the file to:

/mnt/user-data/outputs/<slug>-api-audit-report.html

Where <slug> is the domain name with dots replaced by hyphens (e.g. docs.kubiya.ai → kubiya-ai).

Then call present_files to deliver it.

ERROR HANDLING

| Situation | Action | |---|---| | llms.txt returns 404 | Crawl nav links from homepage; note in report | | Individual endpoint page returns 404 | Skip it; add to "inaccessible pages" count in footer | | Page has no OpenAPI block | Score Spec check as FAIL; still score other checks from prose | | Page has no prose description | Score Description as FAIL with note "No prose description" | | Endpoint page is robots.txt blocked | Note in report; do not fetch | | More than 200 endpoints found | Batch fetching — process in groups of 20, progress-report to user | | Canonical spec accessible | Note as positive finding; do not flag as issue |

WHAT NOT TO DO

• ❌ Never start generating HTML before all pages are fetched and scored
• ❌ Never skip an endpoint page — fetch every one listed in llms.txt
• ❌ Never