← Back to the catalog Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extract", or needs content from many pages on the same site. Handles depth limits, path filtering, and concurrent extraction.
View on GitHub ↗ Copy repo URL Copy SKILL.md link /plugin marketplace add aiskillstore/marketplace The exact command may vary by repository. Check the README on GitHub.
For the skill author
Shows your skill is listed on Skillteca, generates a backlink and trackable traffic.
Markdown HTML
[](https://www.skillteca.com.br/skills/firecrawl-crawl?utm_source=badge&utm_medium=readme&utm_campaign=badge) Copy snippet Creates original algorithmic art using p5.js, seeded randomness, and interactive parameter exploration. Use this for requests involving generative art, flow fields, or particle systems, ensuring no copyright infringement.
Escrita e Conteúdo by anthropics
Guides users through a structured workflow for co-authoring documentation, proposals, technical specs, and similar content. It helps efficiently transfer context, refine content through iteration, and verify its effectiveness for readers.
Escrita e Conteúdo by anthropics
This skill enforces Sentry's blog writing standards across every post — whether you're helping an engineer write their first blog post or a marketer draft a product announcement.
Escrita e Conteúdo #github #git by sickn33
This skill should be used when the user asks to "create AGENTS.md", "update AGENTS.md", "maintain agent docs", "set up CLAUDE.md", or needs to keep agent instructions concise. Enforces research-backed best practices for minimal, high-signal agent documentation.
Escrita e Conteúdo #github #git by sickn33
Category alert
One short email with only the new Escrita e Conteúdo skills. 4 minutes of reading, no spam, unsubscribe with one click.
You confirm your email on the first send. No spam. Unsubscribe with one click.
firecrawl crawl
Bulk extract content from a website. Crawls pages following links up to a depth/limit.
When to use
You need content from many pages on a site (e.g., all /docs/)
You want to extract an entire site section
Step 4 in the workflow escalation pattern : search → scrape → map → crawl → interact
Quick start
# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json
# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json
# Check status of a running crawl
firecrawl crawl <job-id>
Options
Option Description --waitWait for crawl to complete before returning --progressShow progress while waiting --limit <n>Max pages to crawl --max-depth <n>Max link depth to follow --include-paths <paths>Only crawl URLs matching these paths --exclude-paths <paths>Skip URLs matching these paths --delay <ms>Delay between requests --max-concurrency <n>Max parallel crawl workers --prettyPretty print JSON output -o, --output <path>Output file path
Tips
Always use --wait when you need the results immediately. Without it, crawl returns a job ID for async polling.
Use --include-paths to scope the crawl — don't crawl an entire site when you only need one section.
Crawl consumes credits per page. Check firecrawl credit-usage before large crawls.
See also
Read full description↓
Comments · No comments No comments yet. Be the first. 