just-scrape

just-scrape CLI

Search, scrape, crawl, extract structured JSON, and monitor page changes using the just-scrape CLI.

Run just-scrape --help or just-scrape <command> --help for full option details.

If the task is to integrate ScrapeGraph AI into application code, add SGAI_API_KEY to a project, or choose endpoint usage in product code, inspect the project first and use the ScrapeGraph AI SDK/API docs directly instead of this CLI skill.

Prerequisites

Must be installed and authenticated. Check with just-scrape validate and just-scrape credits.

which just-scrape || npm install -g just-scrape@latest
just-scrape validate
just-scrape credits

• API key: Set SGAI_API_KEY, use a .env file, use ~/.scrapegraphai/config.json, or complete the interactive prompt.
• Credits: Remaining ScrapeGraph AI credits. Each operation consumes credits.

Before doing real work, verify the setup with one small request:

mkdir -p .just-scrape
just-scrape scrape "https://example.com" --json > .just-scrape/install-check.json

just-scrape search "query" --num-results 3 --json > .just-scrape/search-check.json

Workflow

Follow this escalation pattern:

1. Search - No specific URL yet. Find pages, answer questions, discover sources.
2. Scrape - Have a URL. Extract markdown, html, screenshots, links, images, summaries, or branding.
3. Extract - Need structured JSON from a known URL with an AI prompt and optional schema.
4. Crawl - Need bulk content from an entire site section.
5. Monitor - Need scheduled page-change tracking with optional webhook notifications.

| Need | Command | When | | --------------------------- | ---------- | ------------------------------------------ | | Find pages on a topic | search | No specific URL yet | | Get a page's content | scrape | Have a URL, need one or more page formats | | AI-powered data extraction | extract | Need structured data from a known URL | | Bulk extract a site section | crawl | Need many pages or docs sections | | Track changes over time | monitor | Need recurring scraping and webhooks | | Inspect prior requests | history | Need past request IDs, status, or payloads | | Check credit balance | credits | Need remaining API credits | | Validate API setup | validate | Need health check and API key validation |

For detailed command reference, run just-scrape <command> --help.

Scrape vs extract:

• Use scrape for raw page formats: markdown, html, screenshot, branding, links, images, summary.
• Use scrape -f json -p "<prompt>" or extract -p "<prompt>" for AI-structured output.
• Use extract when the task is only structured data. Use scrape when mixed formats are needed in one call.

Avoid redundant fetches:

• search -p can extract structured data from search results. Do not re-scrape those URLs unless results are incomplete.
• crawl already fetches per-page formats. Do not re-scrape every crawled URL unless a second pass is required.
• Check .just-scrape/ for existing data before fetching again.

Commands

Search

just-scrape search "query"
just-scrape search "query" --num-results 10
just-scrape search "query" -p "Extract provider names and prices"
just-scrape search "query" -p "Extract provider names and prices" --schema '<json-schema>'
just-scrape search "query" --format html
just-scrape search "query" --country us
just-scrape search "query" --time-range past_week

Time ranges: past_hour, past_24_hours, past_week, past_month, past_year.

Scrape

just-scrape scrape "<url>"
just-scrape scrape "<url>" -f markdown
just-scrape scrape "<url>" -f html
just-scrape scrape "<url>" -f markdown,html,links --json
just-scrape scrape "<url>" -f screenshot
just-scrape scrape "<url>" -f branding
just-scrape scrape "<url>" -f summary
just-scrape scrape "<url>" -f json -p "Extract all products"
just-scrape scrape "<url>" -f json -p "Extract all products" --schema '<json-schema>'
just-scrape scrape "<url>" --html-mode reader
just-scrape scrape "<url>" --mode js --stealth --scrolls 5
just-scrape scrape "<url>" --country DE

Formats: markdown, html, screenshot, branding, links, images, summary, json.

Extract

just-scrape extract "<url>" -p "Extract product names and prices"
just-scrape extract "<url>" -p "Extract headlines and dates" --schema '<json-schema>'
just-scrape extract "<url>" -p "Extract visible items" --scrolls 5
just-scrape extract "<url>" -p "Extract account stats" --cookies "{\"session\":\"$SESSION_COOKIE\"}" --stealth
just-scrape extract "<url>" -p "Extract table rows" --headers "{\"Authorization\":\"Bearer $API_TOKEN\"}"
just-scrape extract "<url>" -p "Extract article data" --html-mode reader
just-scrape extract "<url>" -p "Extract localized prices" --country DE

Use --schema for a strict output shape.

Crawl

just-scrape crawl "<url>"
just-scrape crawl "<url>" -f markdown,links
just-scrape crawl "<url>" --max-pages 50 --max-depth 3
just-scrape crawl "<url>" --max-links-per-page 20
just-scrape crawl "<url>" --allow-external
just-scrape crawl "<url>" --include-patterns '["^https://example\\.com/docs/.*"]'
just-scrape crawl "<url>" --exclude-patterns '[".*\\.pdf$"]'
just-scrape crawl "<url>" --mode js --stealth

Set --max-pages, --max-depth, and include/exclude patterns before broad crawls.

Monitor

just-scrape monitor create --url "<url>" --interval 1h --name "Pricing tracker" -f markdown
just-scrape monitor create --url "<url>" --interval "0 * * * *" --webhook-url "$WEBHOOK_URL"
just-scrape monitor list
just-scrape monitor get --id <cronId>
just-scrape monitor update --id <cronId> --interval 30m
just-scrape monitor activity --id <cronId> --limit 50
just-scrape monitor pause --id <cronId>
just-scrape monitor resume --id <cronId>
just-scrape monitor delete --id <cronId>

Intervals accept cron expressions or shorthands such as 30m, 1h, and 1d.

History

just-scrape history
just-scrape history scrape
just-scrape history extract --json
just-scrape history crawl --page-size 100 --json
just-scrape history scrape <request-id> --json

Services: scrape, extract, search, crawl, monitor.

Credits and Validate

just-scrape credits
just-scrape credits --json
just-scrape validate
just-scrape validate --json

When to Load References

• Searching the web or finding sources first -> use just-scrape search
• Scraping a known URL -> use just-scrape scrape
• AI-powered structured extraction from a known URL -> use just-scrape extract
• Bulk extraction from a docs section or site -> use just-scrape crawl
• Recurring page-change tracking -> use just-scrape monitor
• Install, auth, or setup problems -> run just-scrape validate and inspect SGAI_API_KEY
• Output handling and safe file-reading patterns -> use .just-scrape/ and incremental reads
• Integrating ScrapeGraph AI into an app, adding SGAI_API_KEY to .env, or choosing endpoint usage in product code -> use SDK/API docs, not this CLI flow

Output & Organization

Unless the user specifies to return in context, write results to .just-scrape/ with shell redirection. Add .just-scrape/ to .gitignore. Always quote URLs - shell interprets ? and & as special characters.

just-scrape search "react hooks" --json > .just-scrape/search-react-hooks.json
just-scrape scrape "<url>" --json > .just-scrape/page.json
just-scrape extract "<url>" -p "Extract title and author" --json > .just-scrape/extract-title-author.json

Naming conventions:

.just-scrape/search-{query}.json
.just-scrape/{site}-{path}-scrape.json
.just-scrape/{site}-{path}-extract.json
.just-scrape