Overview
A receiving clerk at a dispensary has a stack of invoices and a form with 16 fields to fill from them. Trakie lets them scan a QR code on the desktop, take photos of the invoice and the product labels on a phone, and watch the form fill itself. A vision model reads the photos, the clerk reviews every product, and nothing is confirmed until the required fields are filled.
The invoice and label photos in the repo are sample photos, and session state lives in memory.
Key Features
- Phone to desktop pairing. The extension opens a tab with a QR code, and the phone and the tab join one Socket.IO room by a session token, so only the paired phone can send photos.
- Reading a whole invoice. The model extracts every line item, matches label photos to them by batch number and gives each of the 16 fields a confidence of high, medium or needs review, which the form shows as colour.
- Review before anything is kept. Product name, brand, category, vendor and quantity block the confirm until they are filled, and prices must be numbers.
- Direct to storage uploads. With a bucket configured, the phone asks for a presigned URL and uploads the photo straight to S3 compatible storage. The server fetches it, and deletes the session's objects when the result or an error is out, when the desktop leaves or when the session expires.
- Resumable video uploads. A video goes up in 8 MiB multipart parts and only the missing parts get new URLs after a dropped connection. The phone page still sends photos only, so this was tested through the same Socket.IO events the phone would use.
- A local model option. The same server runs against a vision model on Ollama, which is how the demo recording was made.
Technical Achievements
The presigned URL signs the file's content type and length and expires after 300 seconds, so the store itself refuses another type or a larger file. The server accepts a key only as exactly sessions/<token>/<n>.jpg for that session, refuses a repeat report of the same key and answers only the paired phone.
Videos go up as resumable multipart uploads in 8 MiB parts. The server keeps the upload id, and asking again for the same slot signs only the parts the store does not hold, so a dropped connection costs one part. The parts are joined only when every one is there at its exact size with the store's checksum, and an unfinished upload is aborted when the session ends. It ran on MinIO with a 17 MiB file through three parts, byte for byte the same, and 19 more guards are removed one at a time, each caught.
The photo upload tests: 146 tests run against a real MinIO, with skipped tests forbidden in CI: 127 unit tests over a generated table of bad tokens, indexes, sizes and key shapes, 10 storage tests, 7 end to end over Socket.IO and 2 with the server as a child process. A script removes 19 guards one at a time in a temporary copy and requires the suite to fail each time, and all 19 are caught. A headless Chromium run of the real phone page uploaded two photos and showed the error text when I forced a 403.
What I Learned
Two things went wrong that I did not expect. The minio/minio image had been removed from Docker Hub, so the tests and CI pin a maintained fork by tag. And the mutation script read every mutant as a survivor in CI, because Node 22 prints test results in a different format when it is not on a terminal. The reporter is now pinned.
On the model side, a 9B local model with thinking left on spent its whole token budget reasoning and returned nothing. With thinking off, a two photo run takes about 5 seconds, but the model marks every field medium confidence and invented an expiry date for a product that had no label photo. The review step is the reason the form exists.
Limits
The upload path is tested on MinIO, an S3-compatible store. MinIO answers every origin, so the bucket CORS rule is the one setting to apply on the real bucket. The extraction accuracy of the default o4-mini setup was not re-measured for this page.
Tech Stack
Extension: Chrome Manifest V3, vanilla JavaScript Server: Node.js, Express, Socket.IO, OpenAI SDK, AWS SDK v3 for presigned uploads Models: OpenAI o4-mini by default, any OpenAI compatible vision model, Qwen on Ollama locally Testing: node:test with generated table tests, a real MinIO, headless Chromium, a mutation script CI: GitHub Actions on Node 22