Vision

framefields runs ONNX vision models on the rendered frames and exposes the results as signals that layers and effects consume like any other. All models are Apache-2.0, pinned to an immutable revision, verified by SHA-256, and downloaded lazily on first use.

Turn it on

// Whole composition → a reactive bundle
const vision = comp.withVision({
  enableDetection: true,      // default
  enableSegmentation: true,
  enablePose: true,
  variant: "s",               // "t" | "s" | "m"
  confidence: 0.35,           // default 0.3
  classes: ["person"],        // COCO filter
});

// One layer → render it through a mode
Layer.video("assets/footage/dancer.mp4").withVision({ mode: "matte", enableSegmentation: true });

Option	Model	Gives you
enableDetection	RTMDet-Ins	COCO-80 boxes, tracked over time → vision.objects
enableSegmentation	RTMDet-Ins (same pass)	soft per-instance masks → vision.masks, vision.segmentation
enablePose	RTMO	17 COCO keypoints per person → track.pose, vision.poseLandmarks
enableMatte	MediaPipe Selfie Segmenter	fast person alpha → vision.segmentation.matte

Layer modes: passthrough, mask, matte, crop, skeleton, boxes, tracking. For cutouts, matteSource: "instance" (default; any class, merges attached parts like a dress or instrument) or "selfie" (people filling the frame only, fastest).

classes picks which instance is the subject, not which parts merge into it. COCO has no clothing class, so a flowing dress may come back as a “kite” or “umbrella” and still joins a classes: ["person"] cutout (2.2+).

On-demand clip vision (comp.vision)

For physics, particle deflection, and interactive layout, comp.vision(source, options) analyzes the raw clip frames on demand before drawing a frame. Unlike render-time withVision (which analyzes a layer’s previous rendered frame with a one-frame delay), clip vision decodes the source directly and makes poses and silhouettes available synchronously to the current frame:

const dancer = comp.vision("assets/footage/dancer.mp4", {
  pose: true,       // 17 COCO keypoints (default true)
  mask: true,       // soft silhouette including dress/held objects (default true)
  maskCell: 8,      // grid cell size in source px (default 8)
  smoothing: 2,     // keypoint temporal averaging in frames (default 2)
});

// Map into composition coordinates
const person = dancer.person.place({ width: 1920, height: 1080, fit: "cover" });

// 1. Read coordinates at any frame
const hand = person.point("rightWrist", 45); // { x, y, visibility }

// 2. Reactive signals for layers
const badge = Layer.box({ width: 80, height: 80, background: "#ff5a1f" });
badge.animate(LayerAnimation.create()
  .signal("x", person.signals("nose").x)
  .signal("y", person.signals("nose").y));

// 3. Silhouette alpha & gradient (for physics deflection & particles)
const sample = person.sample(45, 960, 540); // { alpha, gx, gy }

Caching to disk

Results are cached on disk in ~/.cache/framefields/vision (or $FRAMEFIELDS_VISION_CACHE_DIR) keyed by file size, modification time, and options. A second run, or a final render after a browser preview, loads instantly from cache without re-running model inference.

Integration with physics and particles

• Live physics worlds: When used with world.live(comp), the simulation advances frame-by-frame alongside vision preparation, so preview starts immediately without waiting to analyze the whole video first.
• GPU particles: Pass push: { person, strength, sweep } to Layer.particles to have the performer push particles away from their moving silhouette.
• Initial layout: Call await dancer.ready(0) up front if you need the initial pose at frame 0 to position elements before the animation starts.

Pin, drive, cut out

// Pin a label to the main subject
Layer.text("SUBJECT 01", { fontSize: 40, fill: "#f8fafc" }).pinToObject(vision.objects.primary, {
  anchor: "topCenter", offsetY: -48, smoothFrames: 5, hideWhenLost: true,
});

// Drive an effect from mask coverage
Layer.box({ width: 1920, height: 1080, background: "#000" })
  .withEffect(new Vignette({ strength: vision.segmentation.subject.coverage, radius: 0.9 }));

// Editorial one-liners
comp.addSubjectSandwich({ source: "assets/footage/dancer.mp4", behind: [headline], feather: 4, fit: "cover" }); // text behind subject
comp.addSmartFraming({ source: "assets/footage/action.mp4", target: vision.objects.primary, targetAspect: 9 / 16, damping: 0.15 });
comp.addSubjectOutline(vision.segmentation.subject, { source: "assets/footage/character.mp4", color: "#ff5a1f", width: 6, blur: 12 });
layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });                               // anonymize

Anchors: center, topCenter, bottomCenter, topLeft, topRight, bottomLeft, bottomRight, centerLeft, centerRight. pinToLandmark(coord, { offsetX, offsetY }) pins to a pose keypoint.

Signals you get

Group	Highlights
objects	primary, get(trackId), byCategory(cat, rank), count, hasCategory
objects.*.bounds	x, y, width, height, screenX/Y/Width/Height, area, aspectRatio
objects.*.kinematics	vx, vy, speed, acceleration, headingDeg
objects.*.pose	17 keypoints, hasPose, wristSpeed, handRaised, bodyTiltAngle
masks	subject, get(trackId), count; per mask area, coverage, solidity
segmentation	subject, humanSilhouette, instanceMasks, matte.coverage
classes	per class count, maxConfidence, present

vision.summary(frame) returns a deterministic, serializable snapshot — safe inside a frame hook.

Analyze before authoring

const report = await comp.analyzeVisionSequence("assets/footage/street.mp4", { tasks: ["detect", "pose"], categories: ["person"] });
report.tracks; // [{ trackId, category, frames: [start, end], centerPath, … }]

A zod-serializable report of what is in the clip, without ffmpeg — use it to decide where to place things.

Timing: the one-frame delay

Vision reads each layer’s previous rendered frame. Frame 0 has no results, and masks trail the plate by one frame (invisible at speed).

• renderVideo renders in order and handles this.
• For stills, render at least two consecutive frames with the same HeadlessMediaRenderer and keep the last.
• Don’t verify vision with renderFrameGrid: jumping between frames cuts out pixels from another frame and shows a ghosted double.

GPU or CPU

On framefields 2.2+, models run on the GPU through WebGPU (onnxruntime-web, installed with framefields), on the same device that draws the frames, so frames never leave the GPU. They fall back to the CPU (onnxruntime-node) when the adapter is a software one or WebGPU fails. The log tells you which:

{"backend":"webgpu","sharedDevice":true,"msg":"[vision] Inference on the GPU (WebGPU)"}

Env	Effect
FRAMEFIELDS_VISION_PROVIDER	auto (default), webgpu, cpu, or cuda (Linux x64; needs CUDA 12 and cuDNN 9). A forced backend that fails throws instead of falling back
FRAMEFIELDS_VISION_THREADS	CPU threads per model (default on Linux: the CPUs the process may use)

Measured on the GPU, three vision layers, 60 frames at 1280×720: about 17 s on an M1 Pro, about 7 s on an RTX 3070.

Choosing settings

Task (on the CPU, per ~2K frame)	t	s	m
detect + segment (one pass)	150–300 ms	200–340 ms	420–700 ms
pose	~50 ms	~120 ms	~280 ms
matte	~20 ms	~20 ms	~20 ms

• Start with "s"; use "m" for fast motion or busy backgrounds.
• Raise confidence to ~0.5 on stylized footage; restrict classes to stop the subject switching.
• Garment drops out of a cutout → variant: "m" or keyBackground: true. Halo on dark backgrounds → lower featherRadius or raise maskThreshold.

Models offline

Models cache in $FRAMEFIELDS_MODELS_DIR (default ~/.cache/framefields/models). For CI or air-gapped renders, preload once or point FRAMEFIELDS_MODELS_BASE_URL at a mirror.

import { VisionRunner } from "@framefields/vision";

const runner = VisionRunner.create({ variant: "s" }); // zero I/O until used
await runner.preload(["detect", "pose"]);             // downloads + verifies SHA-256 (~380 MB for all models)
runner.close();