Building a Hybrid Invoice Parser with OCR, Rules and LLMs
Combining EasyOCR text extraction, fast rule-based regex parsers, and LLM fallback engines into a production FastAPI microservice for multi-format invoice processing.
Introduction & Technical Context
Document parsing in enterprise workflows involves dealing with invoices in diverse formats—scanned PDFs, image uploads, Excel files, and CSV spreadsheets—each with unpredictable layouts and varying scan qualities. To handle this variability reliably and cost-effectively, I designed a hybrid parsing architecture combining EasyOCR, deterministic regex rule engines, and LLM fallback parsing wrapped in a containerized FastAPI service.
1. The Fallback Cascade Architecture
Pure LLM parsing across all documents is computationally expensive and slow. Pure rule-based OCR fails when invoice layouts change unpredictably. The hybrid solution implements a tiered execution strategy: 1) Direct file reader for structured formats, 2) EasyOCR + Regex parser for standard key-value invoice layouts, 3) LLM fallback engine for complex, multi-line, or highly unstructured scanned documents.
2. Schema Validation with Pydantic & FastAPI
Raw OCR outputs are chaotic. Every extracted payload must pass through Pydantic data models to enforce data typing (dates, floating-point totals, currency codes) before returning JSON payloads to API clients.
from fastapi import FastAPI, UploadFile, File
from pydantic import BaseModel
from typing import List
class InvoiceItem(BaseModel):
description: str
quantity: float
unit_price: float
total: float
class InvoicePayload(BaseModel):
vendor_name: str
invoice_number: str
date: str
total_amount: float
items: List[InvoiceItem]
@app.post("/api/v1/parse-invoice", response_model=InvoicePayload)
async def parse_invoice(file: UploadFile = File(...)):
# 1. Direct file reader / EasyOCR regex pass
result = regex_ocr_engine.parse(file)
if not result.is_complete:
# 2. LLM fallback engine for complex scans
result = llm_fallback_engine.parse(file)
return InvoicePayload(**result.dict())3. Docker Containerization & Production Deployment
The system was packaged into a Docker container ensuring system-level dependencies for OpenCV and EasyOCR execute consistently across cloud deployment targets.
Key Engineering Takeaways
- Deterministic rules and OCR handle 80% of standard document layouts at low latency.
- LLMs serve as powerful fallback engines for unstructured edge cases.
- Strict Pydantic validation guarantees consistent downstream JSON contracts regardless of parsing path.
Sujan K S — AI/ML Engineer