firecrawl-scraper

Firecrawl Web Scraper Skill

Status: Production Ready ✅ Last Updated: 2025-11-21 Official Docs: https://docs.firecrawl.dev API Version: v2.5

What is Firecrawl?

Firecrawl is a Web Data API for AI that turns entire websites into LLM-ready markdown or structured data. It handles:

• JavaScript rendering - Executes client-side JavaScript to capture dynamic content
• Anti-bot bypass - Gets past CAPTCHA and bot detection systems
• Format conversion - Outputs as markdown, JSON, or structured data
• Screenshot capture - Saves visual representations of pages
• Browser automation - Full headless browser capabilities

API Endpoints

1. /v2/scrape - Single Page Scraping

Scrapes a single webpage and returns clean, structured content.

Use Cases:

• Extract article content
• Get product details
• Scrape specific pages
• Convert HTML to markdown

Key Options:

• formats: ["markdown", "html", "screenshot"]
• onlyMainContent: true/false (removes nav, footer, ads)
• waitFor: milliseconds to wait before scraping
• actions: browser automation actions (click, scroll, etc.)

2. /v2/crawl - Full Site Crawling

Crawls all accessible pages from a starting URL.

Use Cases:

• Index entire documentation sites
• Archive website content
• Build knowledge bases
• Scrape multi-page content

Key Options:

• limit: max pages to crawl
• maxDepth: how many links deep to follow
• allowedDomains: restrict to specific domains
• excludePaths: skip certain URL patterns

3. /v2/map - URL Discovery

Maps all URLs on a website without scraping content.

Use Cases:

• Find sitemap
• Discover all pages
• Plan crawling strategy
• Audit website structure

4. /v2/extract - Structured Data Extraction

Uses AI to extract specific data fields from pages.

Use Cases:

• Extract product prices and names
• Parse contact information
• Build structured datasets
• Custom data schemas

Key Options:

• schema: Zod or JSON schema defining desired structure
• systemPrompt: guide AI extraction behavior

Authentication

Firecrawl requires an API key for all requests.

Get API Key

1. Sign up at https://www.firecrawl.dev
2. Go to dashboard → API Keys
3. Copy your API key (starts with fc-)

Store Securely

NEVER hardcode API keys in code!

# .env file
FIRECRAWL_API_KEY=fc-your-api-key-here

# .env.local (for local development)
FIRECRAWL_API_KEY=fc-your-api-key-here

SDK Quick Start

Python

pip install firecrawl-py  # v4.5.0+

from firecrawl import FirecrawlApp
import os

app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))
result = app.scrape_url("https://example.com", params={"formats": ["markdown"], "onlyMainContent": True})
print(result.get("markdown"))

TypeScript/Node.js

bun add @mendable/firecrawl-js  # v4.4.1+

import FirecrawlApp from '@mendable/firecrawl-js';

const app = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY });
const result = await app.scrapeUrl('https://example.com', { formats: ['markdown'], onlyMainContent: true });
console.log(result.markdown);

See: templates/ for crawl, extract, and advanced examples

Common Use Cases

| Use Case | Endpoint | Key Options | |----------|----------|-------------| | Documentation scraping | crawl_url() | limit: 500, allowedDomains | | Product data extraction | extract() | Zod schema + systemPrompt | | News article scraping | scrape_url() | onlyMainContent: true, removeBase64Images | | URL discovery | map() | Find all pages before crawling |

See: references/common-patterns.md for complete examples.

Error Handling

# Python
try:
    result = app.scrape_url("https://example.com")
except FirecrawlException as e:
    print(f"Firecrawl error: {e}")

// TypeScript
try {
  const result = await app.scrapeUrl('https://example.com');
} catch (error) {
  console.error('Error:', error.message);
}

Rate Limits & Best Practices

| Best Practice | Why | |---------------|-----| | Use onlyMainContent: true | Reduces credits, cleaner output | | Set reasonable limit | Avoid excessive costs | | Use map endpoint first | Plan crawling strategy | | Cache results | Avoid re-scraping | | Batch extract calls | More efficient for multiple URLs |

Credits: Free tier = 500/month, paid tiers higher.

Cloudflare Workers Integration

⚠️ SDK cannot run in Workers (Node.js dependencies). Use direct REST API:

const response = await fetch('https://api.firecrawl.dev/v2/scrape', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${env.FIRECRAWL_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ url, formats: ['markdown'], onlyMainContent: true })
});

See: references/common-patterns.md for complete Workers example with caching.

When to Use This Skill

| ✅ Use Firecrawl | ❌ Don't Use | |------------------|--------------| | Modern JS-rendered sites | Simple static HTML (use cheerio) | | Clean markdown for LLMs | Existing Puppeteer setup works | | RAG/chatbot content | Direct API available | | Structured data extraction | Budget constraints | | Bot protection bypass | |

Common Issues

| Issue | Cause | Fix | |-------|-------|-----| | "Invalid API Key" | Key not set | Check $FIRECRAWL_API_KEY starts with fc- | | "Rate limit exceeded" | Monthly credits used | Check dashboard, upgrade plan | | "Timeout error" | Page slow to load | Add waitFor: 10000 | | "Content is empty" | JS loads late | Add actions: [{type: "wait", milliseconds: 3000}] |

Advanced Features

| Feature | Usage | |---------|-------| | Browser actions | actions: [{type: "click", selector: "button"}] | | Custom headers | headers: {"User-Agent": "Custom Bot"} | | Webhooks | webhook: "https://your-domain.com/webhook" | | Screenshots | formats: ["screenshot"] |

See: references/endpoints.md for complete API reference.

When to Load References

| Reference | Load When... | |-----------|--------------| | endpoints.md | Need complete API endpoint documentation | | common-patterns.md | Cloudflare Workers, caching, batch processing, error handling |

Package Versions

| Package | Version | |---------|---------| | firecrawl-py | 4.5.0+ | | @mendable/firecrawl-js | 4.4.1+ | | API | v2 |

Note: Node.js SDK requires Node.js >=22.0.0, cannot run in Workers.

Official Docs: https://docs.firecrawl.dev | GitHub: https://github.com/mendableai/firecrawl

Token Savings: ~60% | Production Ready: ✅