data-migration-validator

Data Migration Validator Skill

Validates data integrity throughout the migration process with comprehensive verification checks and reconciliation reporting.

Purpose

Enable data validation for:

• Row count validation
• Checksum verification
• Sample data comparison
• Referential integrity checking
• Business rule validation

Capabilities

1. Row Count Validation

• Compare source/target counts
• Track by table/partition
• Identify discrepancies
• Generate count reports

2. Checksum Verification

• Calculate table checksums
• Compare hash values
• Identify data drift
• Verify data consistency

3. Sample Data Comparison

• Random sample selection
• Field-by-field comparison
• Statistical sampling
• Confidence scoring

4. Referential Integrity Checking

• Verify foreign keys
• Check orphaned records
• Validate relationships
• Report violations

5. Business Rule Validation

• Apply custom rules
• Check data constraints
• Verify transformations
• Validate calculations

6. Reconciliation Reporting

• Generate audit reports
• Track discrepancies
• Document exceptions
• Provide sign-off reports

Tool Integrations

| Tool | Purpose | Integration Method | |------|---------|-------------------| | Great Expectations | Data validation | Library | | dbt tests | Transform validation | CLI | | Custom SQL | Database checks | CLI | | DataGrip | Manual verification | GUI | | Apache Griffin | Data quality | API |

Output Schema

{
  "validationId": "string",
  "timestamp": "ISO8601",
  "results": {
    "rowCounts": {
      "tables": [
        {
          "name": "string",
          "source": "number",
          "target": "number",
          "match": "boolean"
        }
      ]
    },
    "checksums": {
      "tables": [],
      "overall": "string"
    },
    "samples": {
      "checked": "number",
      "matched": "number",
      "discrepancies": []
    },
    "referentialIntegrity": {
      "valid": "boolean",
      "violations": []
    },
    "businessRules": {
      "passed": "number",
      "failed": "number",
      "failures": []
    }
  },
  "summary": {
    "status": "passed|failed|warning",
    "score": "number"
  }
}

Integration with Migration Processes

• database-schema-migration: Post-migration validation
• cloud-migration: Data validation

Related Skills

• schema-comparator: Pre-migration comparison
• etl-pipeline-builder: Migration execution

Related Agents

• data-integrity-validator: Orchestrates validation
• database-migration-orchestrator: Uses for verification