Receipt to Expense turns a receipt photo into an expense entry, and splits the bill with friends.

We fine-tuned Google’s open Gemma 4 E2B vision-language model with LoRA (Unsloth) on the 626 labelled SROIE receipts. It reads the photo directly, with no OCR step, and returns the merchant, date, address and total as JSON.

On 100 held-out receipts, compared with the same base model and settings:

• 10x more receipts read perfectly (7% → 70%)
• 3/4 fewer field errors (70% → 92% of fields right)
• 2x better than a hand-engineered prompt (37% → 76% of receipts fully right; McNemar p < 1e-10)

(The best of 5 configurations, chosen on the same test set. Our pre-planned first model already scored 66% vs the prompt’s 37%.)

What we learned:

• Doubling Gemma 4’s visual token budget (560 → 1120) beat using the 2x larger E4B model (76% vs 69%).
• A pixel-level check caught the training pipeline silently shrinking receipts to 512 px. We fixed it, so training now sees exactly what the app sees.

We also ran:

• a prompted baseline;
• out-of-distribution tests (WildReceipt, CORD);
• a seen-vs-new-shop analysis;
• a learning curve;
• LoRA rank, epoch and visual-budget ablations;
• a retrieval “vendor memory”;
• zero-shot line items.

The app (Gradio):

• Snap or upload a receipt.
• Review the fields, with a check against the plain model that flags what to double-check.
• Add it to your expenses, or split the bill equally or by item. Tax is shared in proportion, and everyone settles up in the fewest payments.

The adapters and a full model card are on Hugging Face.

technologies

Gemma 4 (E2B; E4B in experiments) · Unsloth · LoRA / PEFT · Hugging Face Transformers · TRL · PyTorch · Hugging Face Hub · Hugging Face Datasets · Gradio · Google Colab · Python · pandas · SciPy · matplotlib

The only AI model used is Gemma 4 E2B, self-hosted. No external AI API is called.