data-acquisition

Data Acquisition

Overview

Safely acquire data from confirmed open/unauthenticated databases. All acquisition follows a read-only, rate-limited approach with safety caps. Raw dumps are stored in the dumps/ directory and registered in the acquisition tracker before any downstream processing.

Pre-Acquisition Checklist

Before dumping any database:

1. Confirm source is classified as open in the source tracker
2. Verify the active hunt profile authorizes the acquisition scope
3. Ensure sufficient disk space for the estimated dump size
4. Note the target IP/port and database name

Acquisition via dumper.js

node ${CLAUDE_PLUGIN_ROOT}/scripts/dumper.js probe <url> [--service auto|mongo|elastic|redis|couch|mysql|postgres]
node ${CLAUDE_PLUGIN_ROOT}/scripts/dumper.js dump <url> --service <type> --source-id <source-tracker-id> [--timeout N]

This is a full-dump tool — it acquires all accessible data from the target. Probe first to estimate size before committing to a full dump.

Post-Acquisition Registration

After a successful dump, register it in the acquisition tracker:

node ${CLAUDE_PLUGIN_ROOT}/scripts/acquisition-tracker.js add \
  --source-id <source-tracker-id> \
  --file dumps/<filename> \
  --record-count <N> \
  --format <jsonl|csv|bson|json>

Format Conversion

Convert raw dumps to normalized JSONL for downstream pipeline processing:

node ${CLAUDE_PLUGIN_ROOT}/scripts/pipeline.js convert \
  --input dumps/<raw-file> \
  --output dumps/<target>-normalized.jsonl \
  --format <source-format>

Supported input formats: BSON, JSON array, CSV (auto-detect headers), NDJSON.

Per-Database Acquisition Patterns

MongoDB: Use mongoexport for collection-level exports or mongosh for selective queries. Prefer JSONL output (--type json --jsonArray false).

Elasticsearch: Use scroll API for paginated extraction. elasticdump with --type data --limit 1000 handles pagination automatically.

Redis: Use KEYS * followed by GET for small keyspaces. For large instances, use SCAN with cursor to avoid blocking. Export as JSON key-value pairs.

CouchDB: GET /_all_docs?include_docs=true for full database. Use limit and skip parameters for pagination.

MySQL/PostgreSQL: mysqldump / pg_dump for schema+data. Use --no-create-info for data-only exports.

Additional Resources

• references/dumping-guide.md — Step-by-step database dumping for each supported type, safety limits, progress monitoring
• references/format-conversion.md — Format conversion reference: JSONL, CSV, JSON, SQL, BSON; encoding issues; data normalization