# Vision

> On-device detection, segmentation, pose and mattes as reactive signals — pin layers to subjects, cut them out, reframe, blur, and analyze clips.

framefields runs ONNX vision models on the rendered frames and exposes the results as signals that layers and effects consume like any other. All models are Apache-2.0, pinned to an immutable revision, verified by SHA-256, and downloaded lazily on first use.

## Turn it on

```ts
// Whole composition → a reactive bundle
const vision = comp.withVision({
  enableDetection: true,      // default
  enableSegmentation: true,
  enablePose: true,
  variant: "s",               // "t" | "s" | "m"
  confidence: 0.35,           // default 0.3
  classes: ["person"],        // COCO filter
});

// One layer → render it through a mode
Layer.video("assets/footage/dancer.mp4").withVision({ mode: "matte", enableSegmentation: true });
```

| Option | Model | Gives you |
| --- | --- | --- |
| `enableDetection` | RTMDet-Ins | COCO-80 boxes, tracked over time → `vision.objects` |
| `enableSegmentation` | RTMDet-Ins (same pass) | soft per-instance masks → `vision.masks`, `vision.segmentation` |
| `enablePose` | RTMO | 17 COCO keypoints per person → `track.pose`, `vision.poseLandmarks` |
| `enableMatte` | MediaPipe Selfie Segmenter | fast person alpha → `vision.segmentation.matte` |

Layer modes: `passthrough`, `mask`, `matte`, `crop`, `skeleton`, `boxes`, `tracking`. For cutouts, `matteSource: "instance"` (default; any class, merges attached parts like a dress or instrument) or `"selfie"` (people filling the frame only, fastest).

`classes` picks which instance is the subject, not which parts merge into it. COCO has no clothing class, so a flowing dress may come back as a "kite" or "umbrella" and still joins a `classes: ["person"]` cutout (2.2+).

## On-demand clip vision (`comp.vision`)

For physics, particle deflection, and interactive layout, `comp.vision(source, options)` analyzes the raw clip frames **on demand** before drawing a frame. Unlike render-time `withVision` (which analyzes a layer's previous rendered frame with a one-frame delay), clip vision decodes the source directly and makes poses and silhouettes available synchronously to the current frame:

```ts
const dancer = comp.vision("assets/footage/dancer.mp4", {
  pose: true,       // 17 COCO keypoints (default true)
  mask: true,       // soft silhouette including dress/held objects (default true)
  maskCell: 8,      // grid cell size in source px (default 8)
  smoothing: 2,     // keypoint temporal averaging in frames (default 2)
});

// Map into composition coordinates
const person = dancer.person.place({ width: 1920, height: 1080, fit: "cover" });

// 1. Read coordinates at any frame
const hand = person.point("rightWrist", 45); // { x, y, visibility }

// 2. Reactive signals for layers
const badge = Layer.box({ width: 80, height: 80, background: "#ff5a1f" });
badge.animate(LayerAnimation.create()
  .signal("x", person.signals("nose").x)
  .signal("y", person.signals("nose").y));

// 3. Silhouette alpha & gradient (for physics deflection & particles)
const sample = person.sample(45, 960, 540); // { alpha, gx, gy }
```

### Caching to disk

Results are cached on disk in `~/.cache/framefields/vision` (or `$FRAMEFIELDS_VISION_CACHE_DIR`) keyed by file size, modification time, and options. A second run, or a final render after a browser preview, loads instantly from cache without re-running model inference.

### Integration with physics and particles

- **Live physics worlds:** When used with `world.live(comp)`, the simulation advances frame-by-frame alongside vision preparation, so preview starts immediately without waiting to analyze the whole video first.
- **GPU particles:** Pass `push: { person, strength, sweep }` to `Layer.particles` to have the performer push particles away from their moving silhouette.
- **Initial layout:** Call `await dancer.ready(0)` up front if you need the initial pose at frame 0 to position elements before the animation starts.

## Pin, drive, cut out

```ts
// Pin a label to the main subject
Layer.text("SUBJECT 01", { fontSize: 40, fill: "#f8fafc" }).pinToObject(vision.objects.primary, {
  anchor: "topCenter", offsetY: -48, smoothFrames: 5, hideWhenLost: true,
});

// Drive an effect from mask coverage
Layer.box({ width: 1920, height: 1080, background: "#000" })
  .withEffect(new Vignette({ strength: vision.segmentation.subject.coverage, radius: 0.9 }));

// Editorial one-liners
comp.addSubjectSandwich({ source: "assets/footage/dancer.mp4", behind: [headline], feather: 4, fit: "cover" }); // text behind subject
comp.addSmartFraming({ source: "assets/footage/action.mp4", target: vision.objects.primary, targetAspect: 9 / 16, damping: 0.15 });
comp.addSubjectOutline(vision.segmentation.subject, { source: "assets/footage/character.mp4", color: "#ff5a1f", width: 6, blur: 12 });
layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });                               // anonymize
```

Anchors: `center`, `topCenter`, `bottomCenter`, `topLeft`, `topRight`, `bottomLeft`, `bottomRight`, `centerLeft`, `centerRight`. `pinToLandmark(coord, { offsetX, offsetY })` pins to a pose keypoint.

## Signals you get

| Group | Highlights |
| --- | --- |
| `objects` | `primary`, `get(trackId)`, `byCategory(cat, rank)`, `count`, `hasCategory` |
| `objects.*.bounds` | `x`, `y`, `width`, `height`, `screenX/Y/Width/Height`, `area`, `aspectRatio` |
| `objects.*.kinematics` | `vx`, `vy`, `speed`, `acceleration`, `headingDeg` |
| `objects.*.pose` | 17 keypoints, `hasPose`, `wristSpeed`, `handRaised`, `bodyTiltAngle` |
| `masks` | `subject`, `get(trackId)`, `count`; per mask `area`, `coverage`, `solidity` |
| `segmentation` | `subject`, `humanSilhouette`, `instanceMasks`, `matte.coverage` |
| `classes` | per class `count`, `maxConfidence`, `present` |

`vision.summary(frame)` returns a deterministic, serializable snapshot — safe inside a frame hook.

## Analyze before authoring

```ts
const report = await comp.analyzeVisionSequence("assets/footage/street.mp4", { tasks: ["detect", "pose"], categories: ["person"] });
report.tracks; // [{ trackId, category, frames: [start, end], centerPath, … }]
```

A zod-serializable report of what is in the clip, without ffmpeg — use it to decide where to place things.

## Timing: the one-frame delay

Vision reads each layer's **previous** rendered frame. Frame 0 has no results, and masks trail the plate by one frame (invisible at speed).

- `renderVideo` renders in order and handles this.
- For stills, render at least two **consecutive** frames with the same `HeadlessMediaRenderer` and keep the last.
- **Don't verify vision with `renderFrameGrid`**: jumping between frames cuts out pixels from another frame and shows a ghosted double.

## GPU or CPU

On framefields 2.2+, models run on the GPU through WebGPU (`onnxruntime-web`, installed with framefields), on the same device that draws the frames, so frames never leave the GPU. They fall back to the CPU (`onnxruntime-node`) when the adapter is a software one or WebGPU fails. The log tells you which:

```
{"backend":"webgpu","sharedDevice":true,"msg":"[vision] Inference on the GPU (WebGPU)"}
```

| Env | Effect |
| --- | --- |
| `FRAMEFIELDS_VISION_PROVIDER` | `auto` (default), `webgpu`, `cpu`, or `cuda` (Linux x64; needs CUDA 12 and cuDNN 9). A forced backend that fails throws instead of falling back |
| `FRAMEFIELDS_VISION_THREADS` | CPU threads per model (default on Linux: the CPUs the process may use) |

Measured on the GPU, three vision layers, 60 frames at 1280×720: about 17 s on an M1 Pro, about 7 s on an RTX 3070.

## Choosing settings

| Task (on the CPU, per ~2K frame) | `t` | `s` | `m` |
| --- | --- | --- | --- |
| detect + segment (one pass) | 150–300 ms | 200–340 ms | 420–700 ms |
| pose | ~50 ms | ~120 ms | ~280 ms |
| matte | ~20 ms | ~20 ms | ~20 ms |

- Start with `"s"`; use `"m"` for fast motion or busy backgrounds.
- Raise `confidence` to ~0.5 on stylized footage; restrict `classes` to stop the subject switching.
- Garment drops out of a cutout → `variant: "m"` or `keyBackground: true`. Halo on dark backgrounds → lower `featherRadius` or raise `maskThreshold`.

## Models offline

Models cache in `$FRAMEFIELDS_MODELS_DIR` (default `~/.cache/framefields/models`). For CI or air-gapped renders, preload once or point `FRAMEFIELDS_MODELS_BASE_URL` at a mirror.

```ts
import { VisionRunner } from "@framefields/vision";

const runner = VisionRunner.create({ variant: "s" }); // zero I/O until used
await runner.preload(["detect", "pose"]);             // downloads + verifies SHA-256 (~380 MB for all models)
runner.close();
```