firecrawl-scraper

Firecrawl Web Scraper Skill

Status: Production Ready ✅ Last Updated: 2025-10-24 Official Docs: https://docs.firecrawl.dev API Version: v2

What is Firecrawl?

Firecrawl is a Web Data API for AI that turns entire websites into LLM-ready markdown or structured data. It handles:

• JavaScript rendering - Executes client-side JavaScript to capture dynamic content
• Anti-bot bypass - Gets past CAPTCHA and bot detection systems
• Format conversion - Outputs as markdown, JSON, or structured data
• Screenshot capture - Saves visual representations of pages
• Browser automation - Full headless browser capabilities

API Endpoints

1. /v2/scrape - Single Page Scraping

Scrapes a single webpage and returns clean, structured content.

Use Cases:

• Extract article content
• Get product details
• Scrape specific pages
• Convert HTML to markdown

Key Options:

• formats: ["markdown", "html", "screenshot"]
• onlyMainContent: true/false (removes nav, footer, ads)
• waitFor: milliseconds to wait before scraping
• actions: browser automation actions (click, scroll, etc.)

2. /v2/crawl - Full Site Crawling

Crawls all accessible pages from a starting URL.

Use Cases:

• Index entire documentation sites
• Archive website content
• Build knowledge bases
• Scrape multi-page content

Key Options:

• limit: max pages to crawl
• maxDepth: how many links deep to follow
• allowedDomains: restrict to specific domains
• excludePaths: skip certain URL patterns

3. /v2/map - URL Discovery

Maps all URLs on a website without scraping content.

Use Cases:

• Find sitemap
• Discover all pages
• Plan crawling strategy
• Audit website structure

4. /v2/extract - Structured Data Extraction

Uses AI to extract specific data fields from pages.

Use Cases:

• Extract product prices and names
• Parse contact information
• Build structured datasets
• Custom data schemas

Key Options:

• schema: Zod or JSON schema defining desired structure
• systemPrompt: guide AI extraction behavior

Authentication

Firecrawl requires an API key for all requests.

Get API Key

1. Sign up at https://www.firecrawl.dev
2. Go to dashboard → API Keys
3. Copy your API key (starts with fc-)

Store Securely

NEVER hardcode API keys in code!

# .env file
FIRECRAWL_API_KEY=fc-your-api-key-here

# .env.local (for local development)
FIRECRAWL_API_KEY=fc-your-api-key-here

Python SDK Usage

Installation

pip install firecrawl-py

Latest Version: firecrawl-py v4.5.0+

Basic Scrape

import os
from firecrawl import FirecrawlApp

# Initialize client
app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))

# Scrape a single page
result = app.scrape_url(
    url="https://example.com/article",
    params={
        "formats": ["markdown", "html"],
        "onlyMainContent": True
    }
)

# Access markdown content
markdown = result.get("markdown")
print(markdown)

Crawl Multiple Pages

import os
from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))

# Start crawl
crawl_result = app.crawl_url(
    url="https://docs.example.com",
    params={
        "limit": 100,
        "scrapeOptions": {
            "formats": ["markdown"]
        }
    },
    poll_interval=5  # Check status every 5 seconds
)

# Process results
for page in crawl_result.get("data", []):
    url = page.get("url")
    markdown = page.get("markdown")
    print(f"Scraped: {url}")

Extract Structured Data

import os
from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))

# Define schema
schema = {
    "type": "object",
    "properties": {
        "company_name": {"type": "string"},
        "product_price": {"type": "number"},
        "availability": {"type": "string"}
    },
    "required": ["company_name", "product_price"]
}

# Extract data
result = app.extract(
    urls=["https://example.com/product"],
    params={
        "schema": schema,
        "systemPrompt": "Extract product information from the page"
    }
)

print(result)

TypeScript/Node.js SDK Usage

Installation

npm install @mendable/firecrawl-js
# or
pnpm add @mendable/firecrawl-js
# or use the unscoped package:
npm install firecrawl

Latest Version: @mendable/firecrawl-js v4.4.1+ (or firecrawl v4.4.1+)

Basic Scrape

import FirecrawlApp from '@mendable/firecrawl-js';

// Initialize client
const app = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY
});

// Scrape a single page
const result = await app.scrapeUrl('https://example.com/article', {
  formats: ['markdown', 'html'],
  onlyMainContent: true
});

// Access markdown content
const markdown = result.markdown;
console.log(markdown);

Crawl Multiple Pages

import FirecrawlApp from '@mendable/firecrawl-js';

const app = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY
});

// Start crawl
const crawlResult = await app.crawlUrl('https://docs.example.com', {
  limit: 100,
  scrapeOptions: {
    formats: ['markdown']
  }
});

// Process results
for (const page of crawlResult.data) {
  console.log(`Scraped: ${page.url}`);
  console.log(page.markdown);
}

Extract Structured Data with Zod

import FirecrawlApp from '@mendable/firecrawl-js';
import { z } from 'zod';

const app = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY
});

// Define schema with Zod
const schema = z.object({
  company_name: z.string(),
  product_price: z.number(),
  availability: z.string()
});

// Extract data
const result = await app.extract({
  urls: ['https://example.com/product'],
  schema: schema,
  systemPrompt: 'Extract product information from the page'
});

console.log(result);

Common Use Cases

1. Documentation Scraping

Scenario: Convert entire documentation site to markdown for RAG/chatbot

app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))

docs = app.crawl_url(
    url="https://docs.myapi.com",
    params={
        "limit": 500,
        "scrapeOptions": {
            "formats": ["markdown"],
            "onlyMainContent": True
        },
        "allowedDomains": ["docs.myapi.com"]
    }
)

# Save to files
for page in docs.get("data", []):
    filename = page["url"].replace("https://", "").replace("/", "_") + ".md"
    with open(f"docs/{filename}", "w") as f:
        f.write(page["markdown"])

2. Product Data Extraction

Scenario: Extract structured product data for e-commerce

const schema = z.object({
  title: z.string(),
  price: z.number(),
  description: z.string(),
  images: z.array(z.string()),
  in_stock: z.boolean()
});

const products = await app.extract({
  urls: productUrls,
  schema: schema,
  systemPrompt: 'Extract all product details including price and availability'
});

3. News Article Scraping

Scenario: Extract clean article content without ads/navigation

article = app.scrape_url(
    url="https://news.com/article",
    params={
        "formats": ["markdown"],
        "onlyMainContent": True,
        "removeBase64Images": True
    }
)

# Get clean markdown
content = article.get("markdown")

Error Handling

Python

from firecrawl import FirecrawlApp
from firecrawl.exceptions import FirecrawlException

app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))

try:
    result = app.scrape_url("https://example.com")
except FirecrawlException as e:
    print(f"Firecrawl error: {e}")
except Exception as e:
    print(f"Unexpected error: {e}")

TypeScript

import FirecrawlApp from '@mendable/firecrawl-js';

const app = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY
});

try {
  const result = await app.scrapeUrl('https://example.com');
}