Multi-modal ingestion — local models, structured out

Hard problem

Half the evidence a crypto ledger needs never touched a chain — funds frozen on an exchange, a bank statement, a screenshot, a screen recording. No amount of on-chain reconstruction reaches that money, and sending the material to a third-party API means handing over somebody’s complete financial history.

Approach

Ingest it locally. MLX Whisper for audio, Qwen3-VL for vision and document layout, OCR for the rest, every model running on-device. Each artifact is content-hashed so a re-import is a no-op, and every extracted figure keeps a pointer back to the page, frame or timestamp it came from.

When FTX collapsed the proof lived off-chain: funds frozen on an exchange that never touched a wallet. No amount of on-chain reconstruction reaches that money — the only record is a statement, a screenshot, a screen recording. This pipeline is what turns those into entries a ledger can book, and it runs entirely on local models, which matters when the input is somebody's complete financial history.

The stages

Ingest

Exchange statements, bank PDFs, screenshots, and screen recordings — the records that never touched a chain. Each artifact is content-hashed on the way in, so re-importing the same source is a no-op rather than a duplicate.

Transcribe

Audio and video through MLX Whisper, running on-device. Timestamps are retained per segment, so a figure lifted from a recording can point back at the second it was spoken.

Read

Qwen3-VL for vision and document layout, OCR for the rest. Statements and screenshots become text with their structure intact — tables stay tables, columns stay columns.

Index

Everything lands in a full-text searchable corpus (SQLite FTS5, BM25-ranked) covering 100GB+ of source, with the path and page of every hit preserved.

Structure

The model reads the corpus and proposes ledger entries; the deterministic engine decides whether they stand. Every extracted figure carries a pointer back to the page, frame, or timestamp it came from.

Why local

The input is a person's complete financial record — account numbers, balances, addresses, and in the reconciliation work, somebody else's. Sending that to a hosted model is a disclosure decision, not a technical one, and the answer is no. Running Whisper and Qwen3-VL on-device removes the question: nothing leaves the machine, there is no retention policy to read, and the cost of reprocessing a 100GB corpus is electricity rather than per-token billing. It also makes the pipeline reproducible — the same inputs and the same model weights give the same output a year later.

What happens to these entries once they exist — and why a model is allowed to propose them but never to write them — is the harness architecture.