receipt-scanner-master

Receipt Scanner Master

Master the receipt scanning system that uses AI-powered OCR to extract structured data from receipt images and store them in the database.

What This Skill Does

This skill helps you:

1. Parse receipt images (JPG, PNG, WebP, PDF) into structured data
2. Debug OCR accuracy issues and extraction errors
3. Enhance the receipt parsing engine and prompts
4. Test receipt uploads through the web interface
5. Troubleshoot database integration issues
6. Validate extracted data against actual receipts
7. Improve categorization and line item extraction

System Architecture

Frontend Components

Receipt Scanner Component: /home/adamsl/planner/office-assistant/js/components/receipt-scanner.js

• Primary receipt scanning interface at http://localhost:8080/receipt-scanner.html
• Drag-and-drop or file upload for receipt images
• Parses receipts and displays line items in a table
• Each line item has a category-picker dropdown
• Items auto-save to database immediately when categorized
• Only categorized items are saved (uncategorized items ignored)
• No overall receipt-level category picker (removed)

Upload Component: /home/adamsl/planner/office-assistant/js/upload-component.js

• Alternative upload interface (bank statements)
• Displays recent downloads from the system
• Shows real-time processing feedback via terminal display
• Handles streaming responses from backend (Server-Sent Events)
• Auto-refreshes file list after successful imports

Backend Components

Receipt Parser: app/services/receipt_parser.py

• Validates file types and sizes
• Processes and compresses images
• Manages temporary and permanent file storage
• Coordinates with AI engine for extraction

Receipt Engine: app/services/receipt_engine.py

• Uses Google Gemini AI for OCR and extraction
• Implements strict accuracy validation rules
• Returns structured data via Pydantic models
• Tries models in order: gemini-2.5-flash (first), 2.0-flash, 2.5-pro, pro-latest
• Flash model used first to avoid pro quota limits
• Separate quotas for flash vs pro models

API Endpoints: app/api/receipt_endpoints.py

• /api/parse-receipt - Uploads and parses receipt image (returns temp data, doesn't save)
• /api/receipt-items - Auto-saves individual line items when categorized
• /api/save-receipt - Final save for categorized items (batch operation)
• /api/receipts/{expense_id} - Retrieves receipt metadata
• /api/receipts/file/{year}/{month}/{filename} - Serves stored receipt files

Data Models: app/models/receipt_models.py

• ReceiptExtractionResult - Complete receipt data structure
• ReceiptItem - Individual line items with categorization
• ReceiptTotals - Subtotal, tax, tip, discount, total
• ReceiptPartyInfo - Merchant details
• ReceiptMeta - Parsing metadata and model info
• PaymentMethod - Enum: CASH, CARD, BANK, OTHER

Database Integration

Tables:

• expenses - Main expense entries (amount, date, category, method)
• receipt_metadata - Parsing metadata (model, confidence, raw response)

Storage Structure:

app/data/receipts/
├── YYYY/
│   ├── MM/
│   │   ├── receipt_TIMESTAMP_filename.jpg
│   │   └── receipt_TIMESTAMP_filename.pdf
└── temp/
    └── temp_receipt_TIMESTAMP_filename.jpg

How to Use This Skill

Step 1: Test Receipt Parsing

Parse a receipt image to extract structured data:

# Start the API server if not running
python3 api_server.py

# Test with curl (from another terminal)
curl -X POST "http://localhost:8000/api/parse-receipt" \
  -F "file=@/path/to/receipt.jpg"

Expected Response:

{
  "parsed_data": {
    "transaction_date": "2025-01-15",
    "payment_method": "CARD",
    "party": {
      "merchant_name": "Walmart",
      "merchant_phone": null,
      "merchant_address": "123 Main St",
      "store_location": "Store #1234"
    },
    "items": [
      {
        "description": "MILK WHOLE GAL",
        "quantity": 1.0,
        "unit_price": 4.99,
        "line_total": 4.99
      }
    ],
    "totals": {
      "subtotal": 4.99,
      "tax_amount": 0.35,
      "tip_amount": 0.0,
      "discount_amount": 0.0,
      "total_amount": 5.34
    },
    "meta": {
      "currency": "USD",
      "receipt_number": "12345",
      "model_name": "gemini-2.5-pro"
    }
  },
  "temp_file_name": "temp_receipt_20250115T120000Z_receipt.jpg"
}

Step 2: Debug OCR Accuracy Issues

When OCR produces incorrect amounts or descriptions:

Common Issues:

1. Digit Confusion: 4↔9, 3↔8, 5↔6, 0↔8, 1↔7
2. Missing Items: Items not extracted from receipt
3. Wrong Totals: Extracted amounts don't match
4. Poor Image Quality: Blurry, dark, or low-resolution images

Debug Process:

1. Check the raw image quality:

``bash # View the receipt image open /path/to/receipt.jpg # or xdg-open /path/to/receipt.jpg ``

• Is text clearly readable?
• Is image properly oriented?
• Is there sufficient contrast?

1. Review the Gemini prompt in app/services/receipt_engine.py:96-173:
2. Look for the accuracy rules and verification steps
3. Check if new issue types need specific instructions
4. Verify digit confusion prevention rules are clear

1. Test with higher quality image:
2. Increase RECEIPT_IMAGE_MAX_WIDTH_PX in settings
3. Increase JPEG quality in receipt_parser.py:80,83

1. Add validation logic:
2. Check quantity × unit_price = line_total for each item
3. Verify sum(line_totals) ≈ subtotal
4. Compare subtotal + tax - discount = total

1. Examine the raw AI response:

``python # Add debug logging in receipt_engine.py:78 print(f"Raw Gemini Response: {json_response}") ``

Step 3: Enhance the Receipt Parser

To improve parsing accuracy and features:

Modify the Gemini Prompt (app/services/receipt_engine.py):

def _get_prompt(self) -> str:
    return """
    You are an expert at extracting structured data from receipt images with EXTREME ACCURACY.

    [Add new instructions here, such as:]

    **NEW RULE**: For grocery store receipts, items often have:
    - Short codes (e.g., "VEG", "DAIRY", "MEAT")
    - Weight-based pricing (price per lb/kg)
    - Multi-buy discounts (e.g., "2 for $5")

    **VALIDATION ENHANCEMENT**: Before returning JSON:
    1. Verify every item's math: quantity × unit_price = line_total
    2. Sum all line_totals and compare to subtotal
    3. Check: subtotal + tax - discount + tip = total_amount
    4. If any validation fails, RE-EXAMINE the receipt more carefully

    ... [rest of prompt]
    """

Improve Image Processing (app/services/receipt_parser.py):

async def _process_image(self, image_data: bytes, mime_type: str):
    if mime_type.startswith("image/"):
        img = Image.open(BytesIO(image_data))

        # Add preprocessing steps:
        # 1. Auto-rotate based on EXIF
        # 2. Increase contrast for faded receipts
        # 3. Sharpen slightly for better OCR
        # 4. Convert to grayscale if color isn't needed

Add Custom Validation (app/api/receipt_endpoints.py):

@router.post("/parse-receipt")
async def parse_receipt_endpoint(file: UploadFile = File(...)):
    parsed_data, temp_file_name = await parser.process_receipt(file)

    # Add validation here:
    validation_errors = validate_receipt_data(parsed_data)
    if validation_errors:
        return JSONResponse(
            status_code=422,
            content={
                "errors": validation_errors,
                "parsed_data": parsed_data,
                "temp_file_name": temp_file_name
            }
        )

    return ParseReceiptResponse(...)

Step 4: Test Through Web Interface

Test the complete workflow including UI:

1. Start the API server:

``bash cd /home/adamsl/planner/nonprofit_finance_db python3 api_server.py ``

1. Open the web interface:

```bash cd /home/ad