Annotation Maker — Local CV Workbench

Overview

Annotation Maker is the whole detection pipeline — Import → Extract → Annotate → Dataset → Train → Evaluate → Infer → Export — running as one FastAPI process on a single machine. It was built because the alternative for a small team is stitching together a hosted annotation tool, a cloud bucket and a rented GPU, then paying for all three and handing over the footage.

The interesting engineering is in the job model. Frame extraction, dataset builds, training and inference are launched by a supervisor inside the API but execute as separate OS processes with all state in SQLite. That makes cancellation real — a training loop cannot be interrupted from another thread — keeps a CUDA out-of-memory error from taking the API down with it, and lets an interrupted job resume after a restart. There is no message broker to install.

Two rules hold the data model honest. Dataset versions are immutable: a version copies annotation geometry at snapshot time rather than referencing live rows, so editing a box later cannot silently change what a model was trained on. And predictions are never ground truth — model output is stored separately from annotations, and promoting a prediction is an explicit action.

GPU use is explicit rather than best-effort: asking for CUDA when CUDA is unavailable fails immediately with a diagnostic instead of quietly falling back to CPU, because a training run that drops to CPU wastes hours before anyone notices.

The problem

Building a detection model meant renting three services — a hosted annotation tool, cloud storage and a GPU — and uploading the source footage to all of them. For work on private or client-owned video that is both a recurring cost and a disclosure problem, and the round trip between labelling and training made iteration slow.

Discovery

Traced where time actually went on a real detection project and found it was not labelling — it was the hand-offs: exporting a dataset, uploading it, configuring a training job, then pulling results back to decide what to label next.

The solution

Collapsed the entire loop into one local application. A FastAPI process serves the built React SPA and owns a SQLite database of metadata and paths; large assets stay on disk, dataset versions hardlink frames instead of copying them, and training runs against the local GPU. Labelling, training and evaluation are three clicks apart, so the next labelling decision is informed by the last run.

Technical stack

System architecture

1. 1

   FastAPI serves both the REST API and the built React/Vite SPA — one process runs the whole application

2. 2

   SQLite (WAL) holds metadata and paths only; video, frames, weights and exports live on the filesystem

3. 3

   Job supervisor inside the API launches each long task as a separate OS process, with all state in SQLite

4. 4

   Ultralytics YOLO on PyTorch with an explicit CUDA device policy — no silent CPU fallback

5. 5

   Immutable dataset versions snapshot annotation geometry and hardlink frames into versioned directories

6. 6

   Alembic migrations over a workspace directory that is separate from the code checkout

Engineering challenges

Core features

• Video import with FFmpeg probing and in-place referencing (no copy)
• Frame extraction at a chosen stride, with thumbnails and a scrub timeline
• Annotation canvas with class management, review queue and keyboard flow
• Model-assisted pre-labelling, with every prediction requiring an explicit accept
• Immutable dataset versions — geometry snapshotted, frames hardlinked not copied
• Video-aware train/validation splits so near-identical frames cannot leak across
• YOLO training on a local GPU with live loss and mAP curves
• Evaluation with confusion matrix and side-by-side ground-truth vs prediction review
• Inference on new footage plus model export
• Background jobs as separate OS processes — real cancellation, crash isolation, resume

Results & impact

The full detect-model loop — import, label, version, train, evaluate, infer — runs on one machine with no cloud dependency and no per-seat or per-GPU-hour cost. Footage never leaves the device, which makes the tool usable on client-owned and private video that could not be uploaded at all.

Screenshots

What I learned

• Process isolation, not threads, is what makes cancellation and crash recovery real for GPU work.
• Immutability at the dataset boundary is the cheapest way to make ML experiments reproducible.
• Failing loudly on a missing GPU beats a fallback that silently wastes hours.