micah brown/ comp
back to lab
lab · writeup·Apr 22, 2026

SAM2 as a roto baseline

Two weeks of using Meta's SAM2 segmentation as the starting layer for hand-cleanup roto on hero shots. Where it saves time, where it costs time, and where I'd never let it near a frame.

rotomlnukeexperiment

The premise

Hand roto on hero performers is the most expensive cleanup line item on most of the shots I touch. The pitch for ML roto has always been "draw the matte once, refine forever." SAM2's video segmentation is the first time I've seen the pitch hold up well enough that I wanted to actually time it against my own hand-roto on a real shot.

Setup

I took a 96-frame plate from a previous show — a medium-tracked two-shot, soft key from camera left, performer wearing a layered coat over a textured sweater. The kind of edge that punishes any single-pass roto. Three runs:

  • Hand only. Spline in Silhouette, refined per-frame for the coat collar and the hair edge.
  • SAM2 only. One prompt-click per performer per shot, no manual cleanup. Exported as alpha sequence.
  • SAM2 + cleanup. SAM2 base alpha pulled into Nuke, used as a starting matte for spline correction in the spots where it broke.

What I found

The headline: SAM2 + cleanup beat hand-only by about 38% wall-clock. Most of that win came on the sweater texture, where the boundary is soft and forgiving and SAM2 had no opinion about getting it perfect. That's the same place I'd normally be tweaking shape tangents for an hour.

The cost: SAM2 has strong opinions about garment boundaries that match its training prior, and it's confidently wrong at exactly the moments you can't notice in still frames. Specifically, on a single dropped key frame the alpha jumped inward by ~2 pixels on the coat collar — invisible in playback at 24fps, but it caused a chatter that I only caught when I A/B'd the comp at frame-step.

Where it doesn't go

Hair edges. Don't. SAM2 will give you a matte that looks plausible in a thumbnail and falls apart the moment you composite it against any background that isn't a near-match to the original plate. For now, hair stays manual.

The next experiment

I want to try a hybrid: use SAM2 as a per-frame guide layer rather than the alpha itself — a confidence-weighted shape that pulls spline tangents toward where SAM2 thinks the boundary lives, without ever letting it commit. That's a comp-side problem more than an ML problem, and it lives well inside Nuke's expression and Python APIs.

Writeup of that is the next thing on this page when it lands.