The main data flow

Passtape separates description from execution:

MediaGraph
    ↓ prepare(request)
PreparedPlan
    ↓ start output or request a frame
Runtime services
    ↓
Native decode → prepared operations → display or encode

MediaGraph is immutable after construction. PreparedPlan contains the fixed programs, formats, resource requirements, and diagnostics for one graph and execution request. Frame execution does not walk the original graph.

If you have not worked on a media engine before, read Media engine background first. It explains the domain constraints this design responds to — frame sizes, stateful decoders, color metadata, and loudness measurement — in a few minutes. A glossary at the end of this document defines the recurring terms.

Graph model

Every edge references a named output port with a declared media type. Operations declare their complete input, output, and parameter schema. This lets graph construction reject invalid wiring before a backend is involved.

Video, audio, masks, text, and tensors can travel through ordinary graph outputs. Operations with multiple outputs execute their demanded ports together, while unused ports are removed during preparation.

Graph time and source time are separate. Trimming selects the visible graph range; retiming controls how that range maps back to the source. Keyframed parameters are evaluated in graph time.

Preparation

Preparation is the compiler-like stage. It starts from demanded outputs, validates descriptors, selects backend implementations, resolves formats, inserts required conversions, and produces pass tapes for execution.

The request contributes facts that do not belong in the graph: output size, cadence, destination, codec requirements, cache budgets, concurrency limits, and whether hardware paths are required or preferred.

Preparation may fuse an operation with an immediately following conversion when the backend declares that combined implementation. It can also keep nested temporal programs intact when an operation needs neighboring frames.

Native surfaces and zero-copy

The platform-neutral engine describes surface requirements without exposing Metal, Core Video, D3D11, or Vulkan in the graph interface. A platform backend owns the corresponding native resources.

On macOS, VideoToolbox decodes into Core Video pixel buffers. Metal creates texture views over that storage, and intermediate frames come from reusable Core Video pools. Encoding accepts prepared pixel buffers without requiring Passtape to copy frame pixels through CPU memory.

Zero-copy is a property of the complete prepared path, not an operation in isolation. Format conversion, resizing, CPU-buffer output, or an incompatible codec may introduce a new surface or explicit readback. Preparation reports the selected path and any permitted fallback.

Execution

Video execution consumes precompiled pass tapes. It records the operations for one frame into at most one Metal command buffer on the current macOS backend. A source-only frame that already matches its destination can avoid GPU command submission entirely.

Audio executes as bounded blocks of planar floating-point samples on the CPU. Decode, processing, and encoding reuse allocated storage where possible. Pitch-preserving retiming delegates to the native macOS time-pitch implementation while the surrounding pipeline remains block based.

Sequential output sessions overlap decoding, GPU work, and encoding with bounded queues. Interactive requests instead prioritize latency and wait for the requested result before returning.

Runtime ownership

Native media frameworks often require mutable state to stay on an owning thread. Passtape’s concurrent runtime therefore uses worker-owned audio and video lanes rather than placing non-thread-safe native objects behind a shared mutex.

Applications send work through a thread-safe interface. Each worker retains ownership of its decoders, caches, command queues, and output state. This makes concurrency an execution concern rather than something every graph caller must reconstruct.

Extending Passtape

A semantic operation descriptor belongs to the platform-neutral engine. Its implementation belongs to a backend. The same operation identity can therefore have Metal, Vulkan, D3D11, or CPU implementations without changing saved graphs.

Runtime-generated effects follow the same split. An application registers the descriptor and native implementation in a candidate runtime generation, prepares the affected graph, then publishes the ready runtime and plan together. Existing playback can continue on the previous generation while compilation happens.

This separation is the central design rule: graph interfaces express media intent; platform backends own native execution details.

Detailed internals

Background

• Media engine background

Planning and execution

• Runtime and output sessions
• Operations
• Video segments
• Pass tapes
• Surface planning
• Frame execution

Media paths

• Audio blocks
• Loudness normalization
• Video color metadata

Caching and interactive use

• Caching and reuse
• Decoded frame cache
• Interactive video frame requests

Runtime behavior and backends

• Output cancellation and progress
• Runtime statistics
• macOS backend
• Windows and Linux zero-copy prototype

Reference

• Operation catalog
• Glossary

Passtape’s design follows from a few facts about media that hold on every platform. This chapter explains them for readers who have not worked on a media engine before. The rest of this document assumes them.

Decoded video is enormous

Compressed video is small; decoded video is not. One decoded 1080p BGRA frame is about 8 MB (1920 × 1080 pixels × 4 bytes). A 4K frame is about 33 MB. At 30 frames per second, decoded 4K video is roughly one gigabyte of pixels every second.

Copying that much data — between CPU and GPU memory, or even within GPU memory — competes with the actual rendering work for memory bandwidth. This is why:

• Passtape keeps pixels on the GPU from decode to display or encode whenever the prepared path allows it (zero-copy);
• intermediate frames live in reused, pooled surfaces — GPU-resident pixel storage such as a Metal texture or a Core Video pixel buffer — rather than fresh allocations;
• every cache in the engine has a byte budget, because even a few dozen frames is a large amount of memory.

Video decoders are stateful

A video codec stores most frames as differences from nearby frames. Only occasional keyframes are complete images; the frames between them cannot be reconstructed without decoding their predecessors. A decoder is therefore a stateful, mostly forward-moving machine:

• decoding a frame usually requires having decoded the frames before it;
• seeking backward, or far forward, means reopening near an earlier keyframe and decoding forward to the requested frame;
• decoded frames are worth caching, because producing one again is expensive.

This is why Passtape sources use serial decode workers that favor nearby forward requests, why interactive sessions retain decoded frames, and why sequential export simply decodes in order.

Native media frameworks pick their threads

Hardware decoders, encoders, and the OS frameworks around them (VideoToolbox, Media Foundation, VA-API) generally require their mutable state to stay on one owning thread. Wrapping such an object in a shared lock does not make it safe. Passtape therefore gives each worker its own decoders, caches, and command queues — worker-owned lanes — and applications talk to the runtime through a thread-safe interface.

Color is metadata, not pixels

Two frames can contain identical bytes and represent different colors. What the stored numbers mean is defined by color metadata: which red, green, and blue the format refers to, how stored values map to physical light, how RGB relates to the YCbCr encoding most codecs use, and whether the full numeric range is used. Getting it wrong does not crash — it produces a subtly wrong image: washed-out blacks, a green tint, wrong brightness. A media engine must carry color metadata with every frame and convert between interpretations explicitly. Video color metadata describes how Passtape does this.

Audio runs on its own clock

Video is a sequence of frames, tens per second. Audio is a stream of samples, typically 48,000 sample frames per second. Audio is also small: one second of 48 kHz stereo f32 PCM (uncompressed audio samples) is under 400 KB, against the gigabyte-scale second of 4K video above. The two media types therefore get different pipelines: video renders individual frames on the GPU, while audio processes blocks of samples on the CPU. They meet only at the output, where a container interleaves them by timestamp.

Interactive and sequential output want opposite things

An export wants throughput. Decoding, GPU work, and encoding run at the same time on different frames, connected by bounded queues. When a queue fills — the encoder cannot keep up, the disk is slow — the stages before it must pause. This is backpressure: a full downstream stage pausing upstream work so memory use stays bounded.

Interactive playback and scrubbing want latency. Show the frame the user asked for as soon as possible, and abandon work for frames they no longer care about.

Passtape prepares one plan and lets the output session choose which behavior applies.

Loudness is measured, not guessed

How loud audio sounds is not its peak sample value. Delivery platforms specify perceived loudness in LUFS (Loudness Units relative to Full Scale), measured by the ITU BS.1770 algorithm; many streaming services normalize programmes to around −14 LUFS. True peak, in dBTP, estimates the analog peak between samples by oversampling. Hitting a loudness target is therefore a measurement pass followed by a gain adjustment — see Loudness normalization.

MediaRuntime owns the services reused across graph preparations and output requests. Its generic video and audio providers supply decoded media, while the matching backends execute prepared programs.

Every output follows the same path:

MediaGraph
  -> ExecutionRequest
  -> MediaRuntime::prepare
  -> PreparedPlan
  -> video, audio, or audio/video output session
  -> output target

Building a graph

Graph construction is request-neutral: the graph describes media intent and knows nothing about output size, codecs, or destinations. The preferred API is the imperative builder:

let mut graph = MediaGraph::builder("transcode");
let input = graph.add_video_source("input", range, source)?;
let output = graph.add_video_output("output", input)?;
let graph = graph.build();

Built-in operations use named options that keep their wiring explicit:

let amount = graph.add_scalar_parameter("amount", 0.5)?;
let mixed = graph.add_operation(video_mix::Options {
    id: "mix",
    active_range: range,
    first,
    second,
    amount,
})?;

The graph names the semantic operation. Request-specific preparation selects the implementation supplied by the active backend. The operation catalog lists each operation’s inputs and parameters.

Requesting an output

A RequestedVideoOutput connects a logical graph output to a concrete target. The target advertises its format and cadence (its frame timing) before preparation, then opens its sink after those settings have been resolved.

let target = RequestedVideoOutput::new(output.node_id(), target);
let plan = runtime.prepare(graph, &target.execution_request())?;
runtime.start_video_output(plan, target)?.run()?;

When the final path is not known until execution, keep a path-independent specification in the requested output:

let requested = RequestedVideoOutput::new(output.node_id(), specification);
let plan = runtime.prepare(graph, &requested.execution_request())?;
let requested = requested.bind_target(|specification| specification.at(staging_path));
runtime.start_video_output(plan, requested)?.run()?;

bind_target consumes the same specification used during preparation. This prevents application code from accidentally rebuilding different codec, format, or container settings when it attaches an atomic staging path.

Output dimensions are part of the target request. They do not add an operation to the graph because they describe how the final logical output is delivered:

let request = VideoOutputRequest {
    size: OutputSize::Dimensions(OutputDimensions {
        width: 1_280,
        height: 720,
    }),
    ..VideoOutputRequest::default()
};
let target = target.with_request(request);

Preparation resolves the dimensions and verifies that the backend can produce them. Frame execution applies the resize when the output is materialized.

VideoOutputDestination::GpuSurface keeps pixels in backend storage for display or encoding. VideoOutputDestination::CpuBuffer explicitly requests a tightly packed CpuVideoFrame for CPU consumers such as thumbnails or pixel analysis. CPU output must be allowed by the request’s preparation policy.

Running a session

run advances through the prepared output range, renders each frame, responds to sink backpressure, and finalizes the destination. Interactive callers can render explicit times directly or use a latest-request frame scheduler.

File backends can build their video sink with EncodedVideoPipeline. It keeps rendering, compression, and container writing as separate stages, with a fixed packet queue between the encoder and muxer (the component that writes encoded packets into the container file). If the container stops accepting packets, that bound propagates backpressure to frame rendering instead of letting compressed data accumulate without limit.

Video, audio, and media sessions also provide run_with_control. It accepts a thread-safe cancellation token and reports exact work accepted by the destination. See output control.

Sequential run calls discard decoded history. Interactive requests can opt into several levels of reuse — decoded frames, prepared programs, intermediate results, and rendered outputs. See caching and reuse.

Audio sessions

Audio uses the same request, preparation, target, and session pattern through RequestedAudioOutput and AudioOutputSession. Calling with_audio consumes a video runtime and returns a runtime whose type includes AudioServices. Only that configured runtime exposes start_audio_output, so missing audio services are caught at compile time. The session executes prepared audio blocks on their sample timeline.

Combined media output

A destination containing any non-empty set of streams uses RequestedMediaOutput and MediaOutputSession:

let outputs = MediaOutputs::audio_video(
    video.node_id(),
    video_request,
    audio.node_id(),
    audio_request,
)
.with_text_tracks([
    TextTrackOutput::new(english_captions.node_id())
        .with_language("en-US")?
        .enabled_by_default(true),
    TextTrackOutput::new(spanish_captions.node_id()).with_language("es")?,
])?;
let target = RequestedMediaOutput::new(outputs, target);
let plan = runtime.prepare(graph, &target.execution_request())?;
runtime.start_media_output(plan, target)?.run()?;

The selected graph outputs are prepared together. Text graph outputs compile into timed cue tracks; they are destination data, not rendered video, so they stay selectable in containers that support captions. Video-only and audio-only destinations use MediaOutputs::video and MediaOutputs::audio.

The earliest start becomes time zero in the destination, so an output that starts later keeps that delay. Video keeps its frame cadence and audio keeps its sample timeline. The session submits the earliest available timestamp first, but lets the other stream proceed when that input applies backpressure. It keeps at most one pending frame and one pending audio block, and marks each stream complete as soon as its timeline ends.

MediaOutputSession is available on a media runtime configured with both video and audio services, even when only one stream is selected. Callers with a lane-only runtime or a lane-specific destination should use VideoOutputSession or AudioOutputSession instead.

The selected graph outputs own timeline completeness: delayed starts and gaps must materialize black video or silence before they reach a file destination. Destinations may therefore require monotonically contiguous samples without inventing media for missing graph intervals.

Observing the runtime

MediaRuntime::stats returns cumulative preparation, source, rendering, submission, backpressure, output-run, and decoded-media-cache statistics. Duration collection is opt-in so normal frame and audio-block execution avoids clock reads. See runtime statistics.

Operations transform media inside a graph. Each operation has a stable ID, ordered inputs and parameters, and one or more statically named outputs. Preparation selects an implementation from the active backend.

Built-in operations expose named options through ops:

let transformed = graph.add_operation(video_transform_2d::Options {
    id: "transform",
    active_range: range,
    input,
    translation_x,
    translation_y,
    scale_x,
    scale_y,
    rotation_degrees,
    anchor_x,
    anchor_y,
    crop_left,
    crop_top,
    crop_right,
    crop_bottom,
})?;

This keeps the graph builder small as operations are added. The complete operation catalog — each operation’s inputs, outputs, parameters, and edge-case behavior — lives in the main documentation; this chapter explains the machinery they all share.

Descriptors

Plugins and data-driven graphs can use OperationOptions with an OperationDescriptor directly. The descriptor is the complete contract, so the node cannot repeat or disagree with its operation ID, inputs, or outputs.

Multi-output operations declare their ports once and return an OperationOutputs collection. Names are resolved while building the graph; runtime edges carry compact OutputRef values:

let descriptor = OperationDescriptor::new(
    OpRef::new("example.video.split", 1),
    "Split video",
    &[MediaType::Video],
    OutputDescriptor::new("image", MediaType::Video),
)
.with_output_schema(&[
    OutputDescriptor::new("image", MediaType::Video),
    OutputDescriptor::new("matte", MediaType::Mask),
]);

let outputs = graph.add_operation(OperationOptions {
    id: "split",
    active_range: range,
    descriptor: &descriptor,
    inputs: &[input],
    params: &[],
})?;
let image = outputs.get("image").expect("declared output");
let matte = outputs.get("matte").expect("declared output");

The graph snapshot retains the complete named schema and validation compares it with the registered descriptor, so reordering same-media ports cannot silently change routing. Only demanded ports — the ports a downstream output actually needs — are scheduled. When several ports from one operation are needed, the backend receives them together in one execute_passes call; its pass and output slices are aligned, and each pass identifies its port through VideoPass::output_port_id or AudioPass::output_port_id. Audio and visual ports cannot be mixed on one operation because they use different clocks and execution pipelines.

Parameters and keyframes

Typed parameters may be constant or keyframed in graph timeline time. Scalar, two-dimensional point, and RGB values are supported. A keyframe controls the span after it with hold, linear, preset easing, or a custom cubic Bézier timing curve:

let curve = ScalarKeyframeCurve::new([
    Keyframe::new(range.start(), 0.0, KeyframeInterpolation::Linear),
    Keyframe::new(range.end(), 1.0, KeyframeInterpolation::Hold),
])?;
let amount = graph.add_keyframed_parameter("amount", curve)?;

Values before the first keyframe use its value, and values after the last use the last value. Preparation stores the immutable curve once; frame execution evaluates it at the pass time without traversing the graph.

Scalar spans also support KeyframeInterpolation::ValueBezier(CubicBezierValue). Unlike CubicBezier timing easing, its vertical controls are absolute parameter values. It can therefore leave a value and return to it, or overshoot either endpoint. For example, controls [0.25, 10.0] and [0.75, 10.0] between values 2.0 and 2.0 reach 8.0 halfway through the span. Horizontal controls stay in [0, 1], measured from the start of the span, and all coordinates must be finite. Value handles apply to scalar curves; point and RGB curves retain timing easing.

The curve constructor prepares value-segment polynomial coefficients once. Random-access sampling finds the span by binary search; sequential audio sampling keeps its span cursor and Bézier solver hint. Neither path allocates per sample. KeyframeInterpolation::sample_scalar exposes the same scalar evaluator for an editor’s local value readout without constructing a whole curve. Exact keyframe times return the authored values. Range validation includes the curve’s interior extrema, and parameter fingerprints include every handle coordinate.

Decibel gain streams evaluate both keys and controls in decibels before mapping the result to linear amplitude. Preview and export consume these same streams.

Temporal operations

An operation that needs neighboring frames declares their exact time offsets:

let descriptor = descriptor
    .try_with_temporal_offsets(&[previous_frame, next_frame])
    .expect("temporal offsets must be representable");

Preparation derives the enclosing TemporalWindow and compiles those samples into the video pass tape. The output frame rate does not change which samples the operation receives. Ordinary operations declare no offsets. Temporal audio operations are rejected until audio execution can supply neighboring blocks.

Cache policy

Custom operations are not eligible for rendered-output reuse by default. An operation can opt in with with_cache_policy(OperationCachePolicy::Deterministic). This is a correctness promise: inputs, parameters, requested time, source content revisions, and versioned implementation identity must contain every output-affecting dependency. Operations that use hidden mutable state, randomness, external resources, or unrepresented environment state must keep the default Never policy.

What is not an operation

Output resizing is not a graph operation. It is an output-target setting because it changes how a logical output is delivered without changing the graph’s media transformation topology.

Fused conversions

A backend can declare that an operation also performs selected immediately following conversions. Preparation uses this only when the operation result feeds that conversion and nothing else. The combined pass removes the surface between them while preserving the graph operation and converted format.

Passtape prepares stable timeline ranges rather than individual frames:

[0s, 5s)   clip A
[5s, 7s)   clip A + clip B + transition
[7s, 10s)  clip B

This produces three segments because the active sources and operations change at 5 and 7 seconds. Every frame within a segment reuses the same execution structure; only timestamps and parameter values vary.

Where segments are used

MediaGraph
  -> semantic analysis finds stable topology ranges
SemanticPlan
  -> preparation selects requested ranges and compiles segment programs
PreparedPlan
  -> frame execution finds the segment for a requested time

Semantic analysis first creates backend-neutral segments. Preparation filters them to the requested outputs and creates a VideoSegmentProgram. Frame execution uses the program to find active sources and its pass tape.

Program shape

pub struct VideoSegmentProgram {
    id: u32,
    semantic_id: Arc<str>,
    time_range: TimeRange,
    active_source_indices: Box<[SourceIndex]>,
    pass_tape_index: VideoPassTapeIndex,
    temporal_window: TemporalWindow,
    cache_key: Option<ProgramCacheKey>,
}

• id identifies the program within this prepared plan.
• semantic_id identifies the matching global semantic span across preparations. It remains equal while the graph ID, active topology, and span boundaries remain equal; it is opaque and not persistent.
• time_range is half-open: it includes its start and excludes its end.
• active_source_indices identifies the sources needed in the range.
• pass_tape_index selects the work to execute for each frame.
• temporal_window records how much neighboring source time the operations need before and after each frame.
• cache_key makes deterministic rendered work eligible for reuse. It includes relevant source content revisions, operations and parameters, selected pass implementations, output settings, timeline range, and backend. It is absent when a source or operation has not promised deterministic output.

The cache key is an in-process reuse fingerprint, not a media checksum or a persistent file identifier.

Operations declare exact neighboring offsets. Windows summarize their furthest offsets and add along an operation chain. For example, two effects that each need the previous frame require input from two frames earlier. Parallel branches use the larger enclosing window.

Preparation propagates the exact offsets through upstream operations. The pass tape contains the resulting source requests and pass jobs at relative times. Frame execution adds the requested output time, acquires those decoded frames, and follows the fixed tape.

When an offset reaches outside the selected stable-topology segment, execution repeats the current frame for that sample. This keeps the first and last output frames renderable without executing a tape against different topology.

The type is defined in src/prepared_plan/program.rs.

A VideoPassTape is the ordered recipe a backend executor follows for every frame in a video segment.

Preparation creates one tape for each prepared segment. The segment owns the timeline range and stores an index into the prepared plan’s tape table. Tape times are relative to the requested output frame, so one immutable tape works throughout the segment.

It is similar to a list of GPU commands, but it is not raw GPU work yet. Each pass names the selected backend implementation, its inputs, its parameters, and where its result goes. A backend executor turns that information into GPU or CPU work.

Tape shape

Each tape contains three kinds of records:

pub struct VideoPassTape {
    source_slots: Box<[VideoSourceSlot]>,
    passes: Box<[VideoPass]>,
    output_slots: Box<[VideoOutputSlot]>,
    // Per value: the format and capabilities its storage must have.
    surface_descriptor_indices: Box<[VideoSurfaceDescriptorIndex]>,
    // Per value: the reusable allocation slot for an operation result.
    // Decoded source values have no slot.
    surface_slot_indices: Box<[Option<VideoSurfaceSlotIndex>]>,
    // Per slot: the descriptor used when allocating that reusable surface.
    surface_slot_descriptor_indices: Box<[VideoSurfaceDescriptorIndex]>,
}

pub struct VideoSourceSlot {
    source_index: SourceIndex,
    value_index: VideoValueIndex,
    time_offset: RationalTime,
}

pub struct VideoPass {
    kind: VideoPassKind,
    implementation_id: Arc<str>,
    time_offset: RationalTime,
    input_values: Box<[VideoValueIndex]>,
    temporal_input_values: Box<[VideoTemporalInput]>,
    output_value: VideoValueIndex,
    parameter_stream_indices: Box<[ParameterStreamIndex]>,
    recyclable_values: Box<[VideoValueIndex]>,
}

pub enum VideoPassKind {
    Operation(OperationIndex),
    Conversion,
}

pub struct VideoOutputSlot {
    output_index: OutputIndex,
    value_index: VideoValueIndex,
}

• Source slots identify decoded frames and their times relative to the requested output frame.
• Passes transform those frames in dependency order. VideoPassKind distinguishes graph operations from format conversions inserted by preparation. An operation pass keeps semantic input and parameter order. A conversion pass has one input and no parameter streams. Temporal passes also identify neighboring values for each semantic input; backends can consume them in one scan through VideoPassInputs::temporal_samples. Each pass records intermediate values whose surfaces become reusable after that pass.
• Output slots identify the results sent to requested graph outputs.

The runtime can decode the source slots, execute the passes from first to last, and materialize the output slots. It does not need to traverse MediaGraph or sort operations for every frame. See frame execution for how a backend runs this recipe.

How records connect

Source slots and passes each produce a logical video-frame result. Passtape calls each result a value. A value can therefore be a decoded source frame or an intermediate frame produced by an operation.

Records refer to values using dense VideoValueIndex numbers:

source A -> value 0
source B -> value 1
mix(value 0, value 1) -> value 2
requested output -> value 2

The three surface arrays describe storage without allocating it. Each value position holds a surface-descriptor index; the descriptor describes the value’s format and required capabilities, not an allocated texture or buffer. Operation results additionally map to prepared surface slots — reusable allocations planned so that frame execution never searches for free storage at runtime — and each slot has its own descriptor covering every value that will use it. Surface planning explains how the descriptors and slots are computed.

The contracts are defined in src/prepared_plan/video_pass.rs and compiled in src/prepared_plan/prepare.rs.

A video pass tape describes frame results as logical values. Surface planning describes the storage each value will require when the tape runs.

Passtape does not allocate GPU resources during preparation. It creates a table of VideoSurfaceDescriptor records instead:

pub struct VideoSurfaceDescriptor {
    format: VideoFormat,
    required_capabilities: VideoSurfaceCapabilities,
}

• format contains the required dimensions, pixel format, and color metadata.
• required_capabilities records the access a compatible surface must support.

The current capabilities are:

A value can require several capabilities. For example, a decoded source consumed by a mix operation requires both DECODE_WRITE and SHADER_READ.

Each VideoPassTape stores one VideoSurfaceDescriptorIndex for every dense VideoValueIndex. Compatible values share entries in the prepared plan’s descriptor table:

value 0 -> descriptor 0: BGRA 1920x1080, decode write + shader read
value 1 -> descriptor 0: BGRA 1920x1080, decode write + shader read
value 2 -> descriptor 1: BGRA 1920x1080, render write

The runtime will later acquire a surface whose actual capabilities contain all of the descriptor’s required capabilities. The descriptor does not choose a concrete texture, pool entry, queue transition, or API object.

Output format selection

A video output request names one preferred pixel format and may list ordered fallbacks. Preparation selects the first format supported by the backend and the required graph conversion. The selected format is stored in VideoOutputPlan, so execution never negotiates formats per frame.

An encoder target can therefore prefer its native input while accepting a more general surface when the backend cannot prepare that native layout. Callers that require an exact format leave the fallback list empty.

Working set

PreparedPlan::video_surface_working_set records the maximum number of each descriptor used by one frame’s operation results and output transforms. This is deterministic because the pass tapes and output materialization steps are fixed during preparation.

Preparation records the last pass that reads each intermediate value and assigns operation results to reusable surface slots. A slot is reassigned only after its previous value’s final read. For example, a linear chain of three passes needs two intermediate slots rather than three:

pass 1 -> slot 0
pass 2 -> slot 1  (then slot 0 becomes available)
pass 3 -> slot 0

Each slot’s descriptor combines the capabilities required by every value that uses it. Frame execution therefore follows fixed slot indices and does not search for reusable surfaces.

Decoded source surfaces are owned by the source provider and are never reused as operation outputs. Surfaces retained by requested outputs are also excluded from reuse so a later pass cannot overwrite a displayed or encoded frame.

A backend can use the table to prewarm its surface pools before frame execution. It is a minimum working set, not a hard allocation limit. Decoder reference frames belong to the decoder, and a display or encoder may retain submitted frames until its bounded backpressure allows more work.

Frame execution describes when those surfaces are acquired and who owns their handles.

The contracts are defined in src/prepared_plan/video_surface.rs and compiled in src/prepared_plan/prepare.rs.

Frame execution happens after graph analysis and request-specific preparation. It is the per-frame part of the engine:

once:       MediaGraph -> MediaRuntime::prepare -> PreparedPlan
per frame:  VideoOutputSession -> decoded sources -> VideoFrameExecutor -> target

For a requested time, the runtime’s VideoFrameRenderer selects the prepared video segment. It asks each VideoSourceFrameProvider for the required source frames, passing each source identity, source-local time, decoder constraints, and surface descriptor. A temporal pass tape contains the exact neighboring offsets declared by its operations. At a segment boundary, an unavailable neighbor repeats the current frame. The provider returns each decoded surface and the exact source-local range that surface represents.

The provider does not choose the graph’s working format. Preparation inserts pixel or color conversion passes before operations that need them. A requested output resize is resolved during preparation and applied when that output is materialized.

Interactive sessions may reuse those frames through the decoded frame cache. Sequential output sends ordinary decoded frames directly to execution and does not retain a history. Temporal execution keeps only its prepared decode working set so adjacent output frames do not repeatedly seek and decode the same neighborhood. This working set is separate from the caller’s optional cache budget, and preparation rejects a temporal schedule whose estimated decoded surfaces exceed the engine limit.

The renderer supplies the resolved surfaces to VideoFrameExecutor, which runs the center and neighboring pass jobs in prepared order and materializes only the requested center-time outputs. Materialization can retain the final surface as-is or resize it to dimensions resolved from the output request. The executor gives the backend one frame boundary around this work, allowing a GPU backend to record all passes and output conversions before submitting them together.

This stage runs for every requested frame, but it does not repeat preparation: it never traverses MediaGraph, chooses implementations, or sorts operations. An output session adds cadence, target selection, backpressure, and finalization around this per-frame path.

Runtime traits

Rust uses a trait to define behavior that different types can implement.

• VideoSourceFrameProvider supplies decoded source surfaces; its concrete implementation owns media lookup, demuxing, and decoding.
• VideoFrameBackend acquires intermediate surfaces, executes prepared passes, materializes outputs, and completes the recorded frame.

For example, a macOS implementation can decode into Core Video surfaces and execute passes with Metal while the prepared plan remains platform-neutral.

Ownership

The executor retains source handles, holds intermediate handles while the frame runs, and returns the materialized outputs to the caller. The backend owns the actual allocation and pooling. These lifetimes keep pixels on the GPU and stop a surface from being reused while it is still needed.

The prepared plan and executor backend must have the same stable backend ID. This prevents a pass tape compiled for one backend from being run by another.

The contracts are defined in src/video_frame.rs and src/video_frame/render.rs. The surrounding runtime and session interface is defined in src/runtime.rs.

Audio is processed as decoded PCM samples. It uses its own sample clock: a 48 kHz output contains 48,000 sample frames per second.

Passtape prepares one AudioBlockProgram for each timeline span where the active audio work stays the same:

pub struct AudioBlockProgram {
    time_range: TimeRange,
    sample_range: AudioSampleRange,
    pass_tape_index: AudioPassTapeIndex,
    format: AudioFormat,
}

Each program points to an ordered pass tape:

decode 44.1 kHz stereo
  -> resample to 48 kHz
  -> run graph operations
  -> send to output

The source provider only decodes the file’s native PCM format. Audio formats identify every speaker and its storage order, so stereo is not confused with two unrelated channels. Preparation inserts explicit resampling or channel conversion passes when needed. The backend executes those passes and the graph operations in order.

The current audio operations use the requested output format as their working format, so differently formatted sources are converted before processing.

AudioOutputSession divides the programs into fixed-size blocks, executes them, and sends the results to a file, device, or other destination. The preparation policy selects the block size.

See src/prepared_plan/audio_pass.rs and src/audio_block.rs.

Loudness normalization is an explicit two-pass graph workflow: measure the programme’s loudness first, then apply one gain that brings it to the target. Media engine background explains why loudness is measured rather than derived from peaks.

Prepare the unnormalized audio output in its delivery format, then scan that prepared output:

let analysis = runtime.analyze_audio_loudness(plan, output_node)?;
let settings = AudioLoudnessSettings::new(-14.0, -1.0)?;
let normalization = analysis.normalization(settings);

The settings request an integrated loudness of −14 LUFS with a true-peak ceiling of −1 dBTP.

Analysis implements ITU-R BS.1770, the standard loudness measurement: K-weighting (a frequency curve approximating how loudly humans perceive each frequency), speaker-position weighting, overlapping 400 ms measurement blocks, and absolute and relative gates that exclude silence and very quiet passages from the average. True peak is measured with four-times oversampling to estimate peaks that fall between samples. LFE channels do not contribute to programme loudness.

Rebuild the graph with the resolved normalization before its audio output:

let normalized = graph.add_operation(ops::audio_normalize::Options {
    id: "delivery-normalization",
    input: programme_audio,
    normalization,
})?;

media.audio.normalize applies one constant gain to every channel. The gain is the smaller of the amount needed to reach the loudness target and the amount allowed by the true-peak ceiling. It does not clip or limit samples, so a peak-constrained programme can remain quieter than the requested loudness. The ceiling applies to the graph PCM measured before encoding. Lossy codecs can introduce new reconstructed peaks; validate the encoded deliverable separately when its decoded peak is part of the delivery contract.

Analysis is never part of ordinary graph preparation. Callers decide when to scan and can keep the resulting graph separate from the editable programme. The resolved gain retains the analyzed graph identity, graph version, and timeline range. The typed operation rejects it when rebuilding another graph revision and uses the measured range automatically. Preserve the graph identity and version for a delivery-only rebuild; increment the version and analyze again after changing programme content. Analyze again when changing delivery format.

Long-running EBU Tech 3341 vectors and the release throughput matrix are explicitly invoked:

cargo test -p passtape-engine ebu_tech_3341 -- --ignored
cargo test --release -p passtape-engine loudness_analysis_benchmark_matrix \
  -- --ignored --nocapture

VideoFormat carries the information needed to interpret pixel values:

pub struct ColorMetadata {
    pub primaries: ColorPrimaries,
    pub transfer: TransferFunction,
    pub matrix: YcbcrMatrix,
    pub range: ColorRange,
}

• primaries identify which physical red, green, and blue the format refers to. A mismatch shifts saturation: colors look too vivid or too muted.
• transfer defines the relationship between linear light and stored values. A mismatch renders the image too dark or too bright.
• matrix describes RGB and YCbCr conversion. A mismatch tints the whole frame.
• range distinguishes video range (which reserves the codes near 0 and 255) from full range. A mismatch washes out blacks or crushes shadows.

None of these mistakes fail loudly — the pixels remain “valid,” just wrong — which is why the engine carries the metadata explicitly instead of guessing.

Sources provide this metadata. A missing declaration remains Unspecified instead of being labeled as a known color space. Format-preserving operations, prepared values, surface descriptors, and outputs keep the metadata with the frame. OutputColor::Source preserves it; OutputColor::Metadata requires an explicit output interpretation.

Passtape does not silently reinterpret pixels. For an operation with several video inputs, the first input selects the working color metadata. If the backend implementation requires a pixel format, that requirement changes only the pixel storage; it does not replace the working color metadata.

Preparation rejects inputs that cannot be reconciled deterministically. When complete input and working metadata differ, preparation inserts a conversion pass before the operation. The same happens before an output whose requested format differs. Identical conversions of one value are shared. The prepared plan reports FormatConversionInserted when conversion passes are present.

Pixel storage follows the same rule. A backend advertises exact conversion pairs and an operation may require a specific pixel format. Preparation inserts only conversions the backend explicitly supports; otherwise it rejects the plan. Scaling support is also declared per pixel format. Changing pixel layout requires complete color metadata because YCbCr matrix and range affect how the stored values are encoded.

Scaling preserves the pixel format and color metadata of its input. Backends declare support separately for packed RGB and planar YCbCr because those layouts require different sampling and output-storage paths.

For RGB storage such as BGRA or half-float RGBA, matrix and range are retained for a later YCbCr conversion but do not change the RGB arithmetic. Primaries and transfer function define the pixel transform between RGB formats.

Editing is repetitive: the same graph is prepared after every small change, and the same frames are rendered after every scrub. The runtime therefore reuses work at several levels. Every level follows the same rules:

• an entry is keyed by everything that can change its result, so an older entry can never satisfy a request it does not exactly match;
• stores are byte- or entry-bounded and evict entries in least-recently-used order;
• reuse of media is opt-in per request, because retention costs memory.

The levels, from cheapest to most complete:

semantic analysis     reuse graph validation and topology
prepared programs     reuse compiled pass instructions
decoded media         reuse decoder output
intermediate values   reuse unchanged upstream pass results
rendered outputs      reuse complete frames and audio blocks

Semantic analysis

The runtime retains semantic analysis for the most recently prepared immutable graph snapshot. Preparing another output request from a clone of that snapshot reuses validation and topology analysis; request-specific lowering still selects formats and execution programs independently. Successfully analyzing a different snapshot replaces the cached analysis, even when the new snapshot has the same graph ID or version.

Prepared programs

Compiled video and audio pass instructions are retained separately. When a new graph snapshot keeps the same graph ID, routing, formats, backend implementations, and output requirements, preparation reuses those immutable instructions and only reconnects them to the new plan’s tables. Parameter values do not invalidate a prepared program because passes read the new plan’s parameter streams at execution time.

The runtime retains at most 256 prepared programs by default. Applications can change this entry limit with set_prepared_program_cache_capacity, inspect it with prepared_program_cache_stats, or release it with clear_prepared_program_cache.

Decoded media

Sequential run calls discard decoded history. Explicit video requests can retain decoded frames with with_decoded_frame_retention; explicit audio-block requests can use with_decoded_block_retention. Each runtime-owned store has its own byte budget and reuses only deterministic media that exactly matches the requested decoder settings and time or sample range. Callers can inspect or clear each store through the matching runtime methods. The decoded frame cache describes the video policy in detail.

Intermediate values

Explicit requests can retain deterministic intermediate pass results with with_intermediate_value_retention(IntermediateValueRetention::interactive()). This is useful after a graph edit: unchanged upstream work can be reused while changed downstream passes run again. Keys include the complete upstream source, operation, parameter, format, implementation, and time state. Video and audio have separate byte-bounded stores, and sequential run calls bypass and clear them.

Rendered outputs

The runtime can retain completed deterministic video frames and audio blocks. Explicit requests opt in with with_requested_rendered_output_retention(RenderedOutputRetention::interactive()), and the latest-frame scheduler enables this policy by default. Complete sequential output does not fill the cache.

Changing source content, parameters, implementations, output settings, or backend produces a different key, so an older result cannot satisfy the new request. Video and audio use independent stores. The interactive policy applies its 256 MiB budget to each enabled store, in addition to decoded-frame retention. Discard bypasses retained output without clearing older entries; callers can release those entries through the runtime’s rendered-output clear methods.

stats().rendered_outputs() reports combined activity and residency; the runtime also exposes separate video and audio cache snapshots.

Decoding a video frame is expensive. When an interactive caller requests the same part of a source again, Passtape can keep the decoded frame and reuse it instead of decoding it again.

For each requested source time:

1. The runtime looks for a decoded frame that covers that time.
2. If it finds one, it reuses the frame’s surface.
3. Otherwise, the source provider decodes the frame and the runtime caches it.

A decoded frame includes the exact source-time range it covers. This matters because several requested times can refer to the same video frame.

Only sources marked SourceCachePolicy::Deterministic enter this cache. That policy promises that identity, content revision, source time, and decode settings fully determine the returned media. Live, procedural, or externally mutable sources should keep the default Never policy.

If deterministic media changes without receiving a new source identity, set a new VideoSource::with_content_revision value. Previously decoded frames then have a different cache identity and cannot be reused.

The cache is disabled by default. Interactive sessions enable it by selecting a retention policy:

let session = session.with_decoded_frame_retention(
    DecodedVideoFrameRetention::interactive(),
);

VideoFrameRequestScheduler uses the interactive policy by default.

The cache has a byte limit. When it reaches that limit, it removes the decoded frames that have gone unused for the longest time.

Temporal operations use a separate, preparation-bounded working set for the neighboring frames needed during execution. They do not raise this cache’s caller-selected byte limit.

The contracts are defined in src/video_frame/decoded.rs.

Interactive output should render what the user wants now, not every frame requested on the way there. VideoFrameRequestScheduler therefore keeps only the newest pending graph time.

let scheduler = VideoFrameRequestScheduler::for_plan(session.plan());

// This handle can also be cloned into the UI thread.
scheduler.request_frame(timeline_time);

// The output owner supplies destination timing separately.
session.render_latest_request(&scheduler, presentation_time)?;

// Run look-ahead decoding only when no current request needs service.
session.prefetch_next_frames(&scheduler);

The output thread calls render_latest_request. If another request arrives while a decoder or GPU call is running, Passtape lets that synchronous call finish and discards the result at the next safe boundary. A request arriving while destination submission is already underway cannot undo that submission. The next request is then serviced before the method returns. The session also stops retrying a backpressured frame once that frame becomes obsolete. Drive the output session from a worker thread because decoding and backend execution are synchronous.

After submitting a frame, the worker can call prefetch_next_frames when idle to decode a small number of later source frames into the runtime’s byte-bounded decoded-frame store. A new request makes older prefetch work obsolete. Prefetch does not run video operations. It is best-effort: a failure is recorded in runtime statistics but does not turn an already displayed frame into an error.

Temporal operations already acquire their exact neighboring frames during normal rendering, using their separate preparation-bounded working set. Frame-request prefetch skips those segments.

The contracts are defined in src/runtime/frame_request.rs.

Long-running output sessions can be cancelled and can report accepted work:

let cancellation = CancellationToken::new();
let cancel_handle = cancellation.clone();

// Give `cancel_handle` to the UI or coordinator. It can stop this session with:
// cancel_handle.cancel();

session.run_with_control(&cancellation, |progress| {
    if let Some(video) = progress.video_frames() {
        println!("{} / {} frames", video.completed(), video.total());
    }
})?;

CancellationToken is cheap to clone and safe to send to another thread. All clones share one signal. Cancellation cannot be reset, so each output session should use a new token.

OutputProgress reports video in frames and audio in sample frames. The totals come from the prepared timelines, and completed counts advance only after the destination accepts the media. Backpressure retries do not increase progress.

The progress callback runs on the output thread. It should return quickly and send any slower UI or logging work elsewhere.

Cancellation is checked between decoded frames or audio blocks and while a destination applies backpressure. Work already inside a synchronous decoder or backend call returns before cancellation is observed. The session then asks the destination to abandon its partial output and returns a Cancelled error. Destinations that can finalize asynchronously may also observe cancellation while finalization is in progress.

The contracts are defined in src/runtime/control.rs, and the session methods are defined in src/runtime.rs, src/runtime/audio_output.rs, and src/runtime/media_output.rs.

MediaRuntime keeps cumulative statistics for the work it performs. Callers take snapshots between synchronous runtime operations:

let stats = runtime.stats();

println!("rendered {} video frames", stats.video().frames_rendered());
println!("submitted {} audio blocks", stats.audio().blocks_submitted());

The snapshot separates the main stages of the engine:

• preparation counts analysis-cache hits and misses plus the outcomes of request-specific lowering (turning the analyzed graph into backend execution programs);
• video counts source requests, rendered frames, accepted frames, and backpressure;
• frame-request counts started requests, obsolete renders, and nearby-frame prefetch;
• audio counts source requests, rendered and accepted blocks, sample frames, and backpressure;
• output runs combine completed, cancelled, and failed video, audio, and audio/video calls to run or run_with_control;
• prepared-program statistics report reused and rebuilt pass instructions, evictions, and current residency;
• decoded-video and decoded-audio statistics report cache activity and current residency.

Counters are always active and use saturating arithmetic. Duration collection is disabled by default because reading a clock for every frame or audio block adds work to hot paths. It can be enabled when profiling:

runtime.reset_stats();
runtime.set_timing_collection_enabled(true);

Reset before enabling timing and leave it enabled for the complete measurement window when calculating averages from counters and durations. Counters remain active while timing is disabled, so enabling timing after work has already run makes durations cover only part of the counter window.

Video and audio source statistics are a subset of render statistics. Source durations measure provider calls, while render durations include source acquisition and backend execution. A provider failure therefore increments both source and render failures. A decoded-video or decoded-audio cache hit avoids the matching provider and does not increment source_requests.

Output run duration covers a complete run or run_with_control call, including destination backpressure and finalization. Manually driven sessions using render_frame, render_range, or finish do not contribute to output-run statistics.

reset_stats clears counters and durations without releasing the cached semantic plan, prepared programs, or decoded media. Prepared-program and decoded-media hit, miss, and eviction counters reset while their entries and capacities remain unchanged. It also preserves whether timing collection is enabled.

The public snapshot types are defined in src/runtime_stats.rs.

PROTOTYPE — throw this code away after it answers the question.

Question

Can Windows and Linux execute a GPU video operation between hardware decode and hardware encode without Passtape having to move frame pixels through CPU memory?

This probe uses FFmpeg only as disposable orchestration around the operating system GPU APIs. It is not a proposal to make FFmpeg a Passtape dependency.

The probe establishes an API-level zero-copy candidate when:

• decoding produces hardware frames;
• the effect consumes and produces hardware frames;
• encoding directly accepts those hardware frames; and
• the filter graph contains no hwdownload, hwupload, or software scaler.

That does not prove that a driver performs no internal GPU copy. Use PIX on Windows or a vendor GPU profiler on Linux to inspect resource history and copy engines while this probe runs.

Windows

Requirements:

• Windows 10 or later;
• a hardware H.264 decoder and encoder;
• a recent FFmpeg build containing d3d11va, scale_d3d11, and h264_mf;
• Python 3.11 or later.

Run from the repository root:

python crates/passtape/prototypes/platform-zero-copy/probe.py windows C:\path\to\input.mp4

The probe verifies the expected Ryzen 5 5600G host, discovers the installed display adapters, then selects the GeForce RTX 3060 by PCI vendor ID (0x10de). It requires exactly one NVIDIA adapter so that vendor selection is unambiguous, and fails rather than silently using the AMD iGPU or another machine. The candidate path is:

D3D11VA decode -> ID3D11Texture2D -> scale_d3d11 -> ID3D11Texture2D -> Media Foundation H.264 encode

Capture the process in PIX and verify that Media Foundation uses the same RTX 3060, that the frame does not visit a CPU-visible upload/readback heap, and that no unexpected copy pass occurs at API boundaries.

Linux

Requirements:

• a VA-API-capable GPU and driver;
• Vulkan support for the same physical GPU;
• a recent FFmpeg build containing vaapi, vulkan, scale_vulkan, and h264_vaapi;
• Python 3.11 or later.

Run from the repository root:

python3 crates/passtape/prototypes/platform-zero-copy/probe.py linux /path/to/input.mp4

The probe verifies the expected Ryzen 5 5600G host, discovers every DRM render node, and selects its integrated AMD GPU by PCI vendor ID (0x1002). Automatic selection requires exactly one AMD render node; an explicit override is checked against the discovered device. It sets LIBVA_DRIVER_NAME=radeonsi, verifies H.264 encode support with vainfo, and derives the Vulkan device from that exact VA-API device. This prevents an accidental AMD-to-NVIDIA frame transfer.

Override automatic selection when necessary:

python3 crates/passtape/prototypes/platform-zero-copy/probe.py linux \
  /path/to/input.mp4 --render-node /dev/dri/renderD129

The candidate path is:

VA-API decode -> DMA-BUF mapping -> Vulkan scale -> DMA-BUF mapping -> VA-API H.264 encode

The current Linux probe intentionally rejects the NVIDIA adapter. NVIDIA’s Linux path requires a separate NVDEC/Vulkan/NVENC prototype because its VA-API compatibility driver does not support encoding.

Use the profiler for the installed driver (Intel GPA, Radeon GPU Profiler, or Nsight Systems) to check copy-engine activity and Vulkan image transitions.

Output

The probe prints one JSON report containing the discovered CPU, GPU devices, selected device, and media capabilities. verdict has one of these values:

• api-level-zero-copy-candidate: the entire command completed with hardware frames and no explicit software transfer;
• unsupported: the installed FFmpeg build or GPU lacks a required capability;
• failed: the path exists but failed while processing the supplied media.

Use --frames to change the bounded run length. Use --dry-run on any platform to inspect the generated command without executing it.

WGSL compilation probe

The pipeline probes deliberately use native FFmpeg filters first, separating surface interoperability from shader-language translation. Compile the included WGSL operation into the two native shader formats with:

cargo run --manifest-path \
  crates/passtape/prototypes/platform-zero-copy/kernel-compiler/Cargo.toml -- \
  crates/passtape/prototypes/platform-zero-copy/invert.wgsl \
  /tmp/passtape-platform-kernels

This produces:

• invert.hlsl, suitable for a Direct3D compute pipeline after compilation to DXBC/DXIL; and
• invert.spv, suitable for a Vulkan compute pipeline.

The compiler is MIT/Apache-2.0 licensed Naga. It does not introduce a GPU runtime or surface abstraction, so choosing WGSL as source does not itself add a frame copy. This translation-only probe uses ordinary RGBA storage textures; a production backend must separately validate how native multi-plane NV12 surfaces are viewed or adapted for these bindings without a copy.

Short definitions of the recurring terms in this document. Passtape-specific terms link to the chapter that explains them.

• Backend — the platform-specific half of the engine. It implements the execution contracts (decode, passes, output) with native APIs such as Metal and VideoToolbox, while the graph stays platform-neutral.
• Backpressure — a full downstream stage (encoder, container, display) pausing upstream work so queues and memory stay bounded.
• Cadence — the frame timing of an output: which timestamps it expects and at what rate.
• Container — the file format (MOV, MP4, M4A, …) that holds encoded video, audio, and text tracks. Writing packets into it is muxing; reading them out is demuxing.
• Demanded output — a graph output that the execution request actually asks for. Preparation starts from demanded outputs and drops unused work.
• Graph time vs. source time — graph time is the timeline position of the composition; source time is the position inside one media file. Trimming selects the visible graph range; retiming maps it back to source time.
• Keyframe (codec) — a compressed frame that is a complete image. Frames between keyframes are stored as differences and require prior decoding. Unrelated to a keyframe (parameter), a timed value on an animation curve.
• Lowering — the request-specific part of preparation: turning the analyzed graph into concrete formats and backend execution programs.
• LUFS / dBTP — the units of measured loudness and true peak. See Loudness normalization.
• Pass — one prepared unit of execution: a graph operation or an inserted conversion, with its inputs, parameters, and result.
• Pass tape — the ordered list of passes a backend follows for every frame in a segment. See Pass tapes.
• PCM — uncompressed audio samples. Planar storage keeps each channel in its own array; interleaved storage alternates channels sample by sample.
• Preparation — the compiler-like stage between graph and execution: validation, implementation selection, format resolution, and pass-tape compilation. See How Passtape works.
• Premultiplied alpha — color channels stored already multiplied by their alpha value, so compositing blends with a single multiply-add.
• Sample frame — one audio sample across all channels; 48 kHz audio has 48,000 sample frames per second.
• Segment — a timeline range where the active sources and operations stay the same. See Video segments.
• Surface — GPU-resident pixel storage, such as a Metal texture or a Core Video pixel buffer. See Surface planning.
• Value — one logical frame result inside a pass tape: a decoded source frame or an intermediate produced by a pass.
• Working format — the pixel format and color interpretation an operation computes in; preparation inserts conversions to and from it.
• YCbCr / chroma subsampling — the luma-plus-color-difference encoding most codecs use. 4:2:0 and 4:2:2 store color at reduced resolution because eyes resolve brightness more finely than color.
• Zero-copy — moving a frame from decode through effects to encode or display without copying its pixels through CPU memory.