Click a cake in a photograph and only the cake wobbles; the plate under it stays put. The model that draws that boundary is SAM2, Meta’s Segment Anything Model 2, running entirely on your own machine.
Getting a segmentation model to work is the easy half. Getting it to feel instant, on three different hardware stacks, without freezing the window while it thinks — that is the part worth writing down.
What SAM2 is, and the one fact that shapes everything
SAM2 is a promptable segmentation model. Rather than being trained to recognise a fixed list of categories, it takes a prompt — a point, a box — and returns the object there. You point; it works out where the thing begins and ends.
Architecturally it comes in three pieces, and the split matters more to an application developer than the model quality does:
flowchart LR
IMG[Input image] --> IE["Image encoder<br/>(ViT backbone)"]
PRMPT["Prompt<br/>(point / box / mask)"] --> PE[Prompt encoder]
IE --> MD[Mask decoder]
PE --> MD
MD --> MASK["Mask candidates<br/>(ranked by confidence)"]The image encoder is enormous and runs once per image, producing an embedding of roughly 16 MB. The prompt encoder and mask decoder are tiny and run once per click, taking about 50 ms. All the cost is on one side of that line, and it is paid before the user has clicked anything.
That asymmetry is the entire design problem. Do it naively — encode when the user clicks — and every first click on a new photo stalls the app for seconds.
Three stacks, not one
The tidy story would be that ONNX makes the model portable and only the execution provider changes per platform. That is not what happened here. WobblePic ships three different segmentation paths, because each platform pushed back differently:
| Platform | Runtime | Where it runs | Encode |
|---|---|---|---|
| Windows | ONNX Runtime + DirectML | GPU — AMD, NVIDIA or Intel | — |
| macOS (Apple Silicon) | Apple’s official CoreML SAM2.1 build | CPU + GPU | ~310 ms |
| macOS (Intel) | ONNX Runtime | CPU | ~3 s |
Windows is the straightforward one. DirectML sits on DirectX 12, so a single build accelerates on any vendor’s GPU — no CUDA, no per-vendor packaging. The model ships as a split pair, sam2.1_encoder.onnx and sam2.1_decoder.onnx, which is what allows the two halves to live in different processes.
Apple Silicon does not use the ONNX model at all. It runs Apple’s own CoreML conversion of SAM2.1 (apple/coreml-sam2.1-baseplus), which arrives as a .mlpackage, gets compiled once to .mlmodelc on first run, and afterwards loads from cache in about 0.3 s.
Two expectations died here. The first: the Neural Engine does not run this model. SAM2’s Hiera backbone at 1024×1024 is not ANE-compatible, so it executes on CPU + GPU — and at ~310 ms, that is fine. The second is more instructive: the same CoreML path on an Intel Mac takes about 26 seconds to encode. The same code, the same format, the same OS — and a result so far outside interactive range that Intel Macs get the ONNX CPU path instead, at ~3 s with a runtime cache that brings model load down to ~1.4 s.
The lesson is not that CoreML is bad. It is that “the framework is cross-platform” says nothing about whether the performance crosses with it.
flowchart LR
IMG[Image] --> ONNX["ONNX Runtime<br/>(inference engine)"]
ONNX --> DML["DirectML<br/>(execution provider)"]
DML --> GPU["GPU<br/>(NVIDIA / AMD / Intel)"]
GPU --> MASK[Segmentation mask]Keeping the window responsive
Since all the cost is in the encoder, the encoder is the only thing that needs to be got out of the way. WobblePic splits the model across processes:
- The encoder runs in a worker process. It starts encoding the moment an image is displayed, and it also encodes the neighbouring images in the folder before you navigate to them — forward-first when you are paging forward, reversed when you are paging back.
- The decoder runs synchronously in the main process. At ~50 ms per click there is nothing to gain from making it asynchronous, and plenty to lose in complexity.
The embedding then has to cross the process boundary, and 16 MB through a multiprocessing.Queue means pickling 16 MB on one side and unpickling it on the other, every time. Instead the embeddings live in a SharedMemory ring cache and the queue carries only metadata — which slot, which file. The bytes are never copied.
flowchart LR
subgraph WORKER["Worker process"]
ENC["Image encoder<br/>runs ahead of your click"]
end
subgraph MAIN["Main process"]
DEC["Prompt encoder + mask decoder<br/>~50 ms, on click"]
WGT["Mask becomes a<br/>per-vertex weight"]
SIM["Only that region deforms"]
end
CLICK["Your click"] --> DEC
ENC -->|"16 MB embedding<br/>SharedMemory, no copy"| DEC
DEC --> WGT --> SIMA few consequences fall out of this design:
- The model loads in the background at startup. The image appears immediately, with a “Loading SAM2 model…” overlay while the worker gets ready.
- Clicking before encoding finishes still works — you get a whole-image wobble instead of a segmented one. Degrading to the simpler behaviour is better than blocking, and by the time most people have aimed at something, the embedding has arrived.
- Stale work is dropped. Page quickly through a folder and the worker drains its queue, keeps only the newest current-image request, and throws away preloads for images you have already left.
- Preloading disables itself under memory pressure, rather than competing with the app it is meant to accelerate.
From click to mask
The prompts map onto SAM2’s own vocabulary:
- Click outside the current mask — a point prompt
- Drag a rectangle — a box prompt, with a translucent overlay while you drag
- Shift + click/drag — add to the current segment
- Alt + click/drag — remove from it
The decoder returns three candidate masks with predicted IoU scores, and WobblePic takes the highest-scoring one. That is the model’s own confidence talking, not a heuristic about where you clicked.
Those candidates come back as 256×256 logits, which is far below the resolution of any real photo. Thresholding first and scaling afterwards would give you a staircase along every boundary, so the order is reversed: upscale the logits bilinearly to full resolution, then threshold. The boundary lands between the original grid points and comes out smooth.
The threshold itself is exposed as the Selection Range slider, mapping 0 → +2.0, 50 → 0.0, 100 → −2.0. A lower threshold accepts weaker logits and grows the selection, which is the fastest way to recover when a mask clips the edge of the thing you wanted.
What the mask actually does
It is tempting to assume the segmented object becomes its own mesh. It does not — there is exactly one mesh, a fixed grid over the whole image, and the mask becomes a per-vertex weight on it. Vertices outside the object are never displaced by a drag, and any that hold leftover displacement are snapped back and held at rest. The background you see is a second render pass over the same vertex buffer, drawn from the undeformed rest positions.
So “only the cake wobbles” is not two simulations. It is one simulation and a weight deciding who is allowed to move. (How WobblePic’s Physics Simulation Works covers that side in detail.)
Because the mask is data rather than geometry, some things become nearly free. The selection is drawn 10% brighter and the background 10% darker, so you can see what you have. Ctrl + drag moves the segmented region and Ctrl + wheel scales it, both implemented as texture-coordinate offsets in the shader — the background never moves, and the segment keeps its original pixel resolution because nothing is being resampled into a new buffer.
The privacy part, briefly
All of this runs on your machine. There is no upload, no API key, and no account — the model weights ship with the app, which is most of its ~600 MB install size. The runtime does not need PyTorch either: conversion to ONNX and CoreML happens ahead of time, and only the inference runtimes ship.
Local execution is usually pitched as a privacy feature, and it is. It is also the reason clicking feels the way it does: a round trip to a server would cost more than the 50 ms decode no matter how fast the server was.
Improving SAM2 Mask Quality picks up where this leaves off — what to do when the mask SAM2 hands back is not quite the one you wanted.