The main data flow

Passtape separates description from execution:

MediaGraph
    ↓ prepare(request)
PreparedPlan
    ↓ start output or request a frame
Runtime services
    ↓
Native decode → prepared operations → display or encode

MediaGraph is immutable after construction. PreparedPlan contains the fixed programs, formats, resource requirements, and diagnostics for one graph and execution request. Frame execution does not walk the original graph.

Graph model

Every edge references a named output port with a declared media type. Operations declare their complete input, output, and parameter schema. This lets graph construction reject invalid wiring before a backend is involved.

Video, audio, masks, text, and tensors can travel through ordinary graph outputs. Operations with multiple outputs execute their demanded ports together, while unused ports are removed during preparation.

Graph time and source time are separate. Trimming selects the visible graph range; retiming controls how that range maps back to the source. Keyframed parameters are evaluated in graph time.

Preparation

Preparation is the compiler-like stage. It starts from demanded outputs, validates descriptors, selects backend implementations, resolves formats, inserts required conversions, and produces pass tapes for execution.

The request contributes facts that do not belong in the graph: output size, cadence, destination, codec requirements, cache budgets, concurrency limits, and whether hardware paths are required or preferred.

Preparation may fuse an operation with an immediately following conversion when the backend declares that combined implementation. It can also keep nested temporal programs intact when an operation needs neighboring frames.

Native surfaces and zero-copy

The platform-neutral engine describes surface requirements without exposing Metal, Core Video, D3D11, or Vulkan in the graph interface. A platform backend owns the corresponding native resources.

On macOS, VideoToolbox decodes into Core Video pixel buffers. Metal creates texture views over that storage, and intermediate frames come from reusable Core Video pools. Encoding accepts prepared pixel buffers without requiring Passtape to copy frame pixels through CPU memory.

Zero-copy is a property of the complete prepared path, not an operation in isolation. Format conversion, resizing, CPU-buffer output, or an incompatible codec may introduce a new surface or explicit readback. Preparation reports the selected path and any permitted fallback.

Execution

Video execution consumes precompiled pass tapes. It records the operations for one frame into at most one Metal command buffer on the current macOS backend. A source-only frame that already matches its destination can avoid GPU command submission entirely.

Audio executes as bounded blocks of planar floating-point samples on the CPU. Decode, processing, and encoding reuse allocated storage where possible. Pitch-preserving retiming delegates to the native macOS time-pitch implementation while the surrounding pipeline remains block based.

Sequential output sessions overlap decoding, GPU work, and encoding with bounded queues. Interactive requests instead prioritize latency and wait for the requested result before returning.

Runtime ownership

Native media frameworks often require mutable state to stay on an owning thread. Passtape’s concurrent runtime therefore uses worker-owned audio and video lanes rather than placing non-thread-safe native objects behind a shared mutex.

Applications send work through a thread-safe interface. Each worker retains ownership of its decoders, caches, command queues, and output state. This makes concurrency an execution concern rather than something every graph caller must reconstruct.

Extending Passtape

A semantic operation descriptor belongs to the platform-neutral engine. Its implementation belongs to a backend. The same operation identity can therefore have Metal, Vulkan, D3D11, or CPU implementations without changing saved graphs.

Runtime-generated effects follow the same split. An application registers the descriptor and native implementation in a candidate runtime generation, prepares the affected graph, then publishes the ready runtime and plan together. Existing playback can continue on the previous generation while compilation happens.

This separation is the central design rule: graph interfaces express media intent; platform backends own native execution details.

Detailed internals

Planning and execution

• Runtime and output sessions
• Operations
• Video segments
• Pass tapes
• Surface planning
• Frame execution

Media paths

• Audio blocks
• Loudness normalization
• Video color metadata
• Decoded frame cache
• Interactive video frame requests

Runtime behavior and backends

• Output cancellation and progress
• Runtime statistics
• macOS backend
• Windows and Linux zero-copy prototype

MediaRuntime owns the services reused across graph preparations and output requests. Its generic video and audio providers supply decoded media, while the matching backends execute prepared programs.

The runtime retains semantic analysis for the most recently prepared immutable graph snapshot. Preparing another output request from a clone of that snapshot reuses validation and topology analysis. Request-specific lowering still selects formats and execution programs independently. Successfully analyzing a different snapshot replaces the cached analysis even when it has the same graph ID or version.

Compiled video and audio pass instructions are retained separately. When a new graph snapshot keeps the same graph ID, routing, formats, backend implementations, and output requirements, preparation reuses those immutable instructions and only reconnects them to the new plan’s tables. Parameter values do not invalidate a prepared program because passes read the new plan’s parameter streams at execution time.

Explicit requests can also retain deterministic intermediate pass results with with_intermediate_value_retention(IntermediateValueRetention::interactive()). This is useful after a graph edit: unchanged upstream work can be reused while changed downstream passes run again. Keys include the complete upstream source, operation, parameter, format, implementation, and time state. Video and audio have separate byte-bounded stores, and sequential run calls bypass and clear them.

The runtime can also retain completed deterministic video frames and audio blocks. Explicit requests opt in with with_requested_rendered_output_retention(RenderedOutputRetention::interactive()), and the latest-frame scheduler enables this policy by default. Complete sequential output does not fill the cache.

Entries are bounded by estimated media bytes and evicted in least-recently-used order. Changing source content, parameters, implementations, output settings, or backend produces a different key, so an older result cannot satisfy the new request. stats().rendered_outputs() reports combined activity and residency; the runtime also exposes separate video and audio cache snapshots.

Video and audio use independent stores. The interactive policy applies its 256 MiB budget to each enabled store, in addition to decoded-frame retention. Discard bypasses retained output without clearing older entries; callers can release those entries through the runtime’s rendered-output clear methods.

The runtime retains at most 256 prepared programs by default. Applications can change this entry limit with set_prepared_program_cache_capacity, inspect it with prepared_program_cache_stats, or release it with clear_prepared_program_cache.

MediaGraph
  -> ExecutionRequest
  -> MediaRuntime::prepare
  -> PreparedPlan
  -> video, audio, or audio/video output session
  -> output target

Graph construction remains request-neutral. The preferred API is the imperative builder:

let mut graph = MediaGraph::builder("transcode");
let input = graph.add_video_source("input", range, source)?;
let output = graph.add_video_output("output", input)?;
let graph = graph.build();

Built-in operations use named options that keep their wiring explicit:

let amount = graph.add_scalar_parameter("amount", 0.5)?;
let mixed = graph.add_operation(video_mix::Options {
    id: "mix",
    active_range: range,
    first,
    second,
    amount,
})?;

The graph names the semantic operation. Request-specific preparation selects the implementation supplied by the active backend. The operations guide lists their inputs and parameters.

A RequestedVideoOutput connects a logical graph output to a concrete target. The target advertises its format and cadence before preparation, then opens its sink after those settings have been resolved.

let target = RequestedVideoOutput::new(output.node_id(), target);
let plan = runtime.prepare(graph, &target.execution_request())?;
runtime.start_video_output(plan, target)?.run()?;

When the final path is not known until execution, keep a path-independent specification in the requested output:

let requested = RequestedVideoOutput::new(output.node_id(), specification);
let plan = runtime.prepare(graph, &requested.execution_request())?;
let requested = requested.bind_target(|specification| specification.at(staging_path));
runtime.start_video_output(plan, requested)?.run()?;

bind_target consumes the same specification used during preparation. This prevents application code from accidentally rebuilding different codec, format, or container settings when it attaches an atomic staging path.

Output dimensions are part of the target request. They do not add an operation to the graph because they describe how the final logical output is delivered:

let request = VideoOutputRequest {
    size: OutputSize::Dimensions(OutputDimensions {
        width: 1_280,
        height: 720,
    }),
    ..VideoOutputRequest::default()
};
let target = target.with_request(request);

Preparation resolves the dimensions and verifies that the backend can produce them. Frame execution applies the resize when the output is materialized.

VideoOutputDestination::GpuSurface keeps pixels in backend storage for display or encoding. VideoOutputDestination::CpuBuffer explicitly requests a tightly packed CpuVideoFrame for CPU consumers such as thumbnails or pixel analysis. CPU output must be allowed by the request’s preparation policy.

run advances through the prepared output range, renders each frame, responds to sink backpressure, and finalizes the destination. Interactive callers can render explicit times directly or use a latest-request frame scheduler.

File backends can build their video sink with EncodedVideoPipeline. It keeps rendering, compression, and container writing as separate stages, with a fixed packet queue between the encoder and muxer. If the container stops accepting packets, that bound propagates backpressure to frame rendering instead of letting compressed data accumulate without limit.

Sequential run calls discard decoded history. Explicit video requests can retain decoded frames with with_decoded_frame_retention; explicit audio-block requests can use with_decoded_block_retention. Each runtime-owned store has its own byte budget and reuses only deterministic media that exactly matches the requested decoder settings and time or sample range. Callers can inspect or clear each store through the matching runtime methods. See the decoded frame cache for the video policy.

Audio uses the same request, preparation, target, and session pattern through RequestedAudioOutput and AudioOutputSession. Calling with_audio consumes a video runtime and returns a runtime whose type includes AudioServices. Only that configured runtime exposes start_audio_output, so missing audio services are caught at compile time. The session executes prepared audio blocks on their sample timeline.

A destination containing any non-empty set of streams uses RequestedMediaOutput and MediaOutputSession:

let outputs = MediaOutputs::audio_video(
    video.node_id(),
    video_request,
    audio.node_id(),
    audio_request,
)
.with_text_tracks([
    TextTrackOutput::new(english_captions.node_id())
        .with_language("en-US")?
        .enabled_by_default(true),
    TextTrackOutput::new(spanish_captions.node_id()).with_language("es")?,
])?;
let target = RequestedMediaOutput::new(outputs, target);
let plan = runtime.prepare(graph, &target.execution_request())?;
runtime.start_media_output(plan, target)?.run()?;

The selected graph outputs are prepared together. Text graph outputs compile into timed cue tracks; they are destination data, not rendered video, so they stay selectable in containers that support captions. Video-only and audio-only destinations use MediaOutputs::video and MediaOutputs::audio. The earliest start becomes time zero in the destination, so an output that starts later keeps that delay. Video keeps its frame cadence and audio keeps its sample timeline. The session submits the earliest available timestamp first, but lets the other stream proceed when that input applies backpressure. It keeps at most one pending frame and one pending audio block, and marks each stream complete as soon as its timeline ends.

MediaOutputSession is available on a media runtime configured with both video and audio services, even when only one stream is selected. Callers with a lane-only runtime or a lane-specific destination should use VideoOutputSession or AudioOutputSession instead.

The selected graph outputs own timeline completeness: delayed starts and gaps must materialize black video or silence before they reach a file destination. Destinations may therefore require monotonically contiguous samples without inventing media for missing graph intervals.

Video, audio, and media sessions also provide run_with_control. It accepts a thread-safe cancellation token and reports exact work accepted by the destination. See output control.

MediaRuntime::stats returns cumulative preparation, source, rendering, submission, backpressure, output-run, and decoded-media-cache statistics. Duration collection is opt-in so normal frame and audio-block execution avoids clock reads. See runtime statistics.

Operations transform media inside a graph. Each operation has a stable ID, ordered inputs and parameters, and one or more statically named outputs. Preparation selects an implementation from the active backend.

Built-in operations expose named options through ops:

let transformed = graph.add_operation(video_transform_2d::Options {
    id: "transform",
    active_range: range,
    input,
    translation_x,
    translation_y,
    scale_x,
    scale_y,
    rotation_degrees,
    anchor_x,
    anchor_y,
    crop_left,
    crop_top,
    crop_right,
    crop_bottom,
})?;

This keeps the graph builder small as operations are added. Plugins and data-driven graphs can use OperationOptions with an OperationDescriptor directly. The descriptor is the complete contract, so the node cannot repeat or disagree with its operation ID, inputs, or outputs.

Multi-output operations declare their ports once and return an OperationOutputs collection. Names are resolved while building the graph; runtime edges carry compact OutputRef values:

let descriptor = OperationDescriptor::new(
    OpRef::new("example.video.split", 1),
    "Split video",
    &[MediaType::Video],
    OutputDescriptor::new("image", MediaType::Video),
)
.with_output_schema(&[
    OutputDescriptor::new("image", MediaType::Video),
    OutputDescriptor::new("matte", MediaType::Mask),
]);

let outputs = graph.add_operation(OperationOptions {
    id: "split",
    active_range: range,
    descriptor: &descriptor,
    inputs: &[input],
    params: &[],
})?;
let image = outputs.get("image").expect("declared output");
let matte = outputs.get("matte").expect("declared output");

The graph snapshot retains the complete named schema and validation compares it with the registered descriptor, so reordering same-media ports cannot silently change routing. Only demanded ports are scheduled. When several ports from one operation are needed, the backend receives them together in one execute_passes call; its pass and output slices are aligned, and each pass identifies its port through VideoPass::output_port_id or AudioPass::output_port_id. Audio and visual ports cannot be mixed on one operation because they use different clocks and execution pipelines.

Typed parameters may be constant or keyframed in graph timeline time. Scalar and two-dimensional point values are currently supported. A keyframe controls the span after it with hold, linear, preset easing, or a custom cubic Bézier timing curve:

let curve = ScalarKeyframeCurve::new([
    Keyframe::new(range.start(), 0.0, KeyframeInterpolation::Linear),
    Keyframe::new(range.end(), 1.0, KeyframeInterpolation::Hold),
])?;
let amount = graph.add_keyframed_parameter("amount", curve)?;

Values before the first keyframe use its value, and values after the last use the last value. Preparation stores the immutable curve once; frame execution evaluates it at the pass time without traversing the graph.

Video corner pin

media.video.corner_pin projectively maps a video frame into four normalized destination points ordered clockwise from top-left. Each corner is a Point2 parameter, so tracking output can animate the perspective mapping without rebuilding the graph. Coordinates may extend outside [0, 1]; pixels outside the mapped source are transparent.

An operation that needs neighboring frames declares their exact time offsets:

let descriptor = descriptor
    .try_with_temporal_offsets(&[previous_frame, next_frame])
    .expect("temporal offsets must be representable");

Preparation derives the enclosing TemporalWindow and compiles those samples into the video pass tape. The output frame rate does not change which samples the operation receives. Ordinary operations declare no offsets. Temporal audio operations are rejected until audio execution can supply neighboring blocks.

Custom operations are not eligible for rendered-output reuse by default. An operation can opt in with with_cache_policy(OperationCachePolicy::Deterministic). This is a correctness promise: inputs, parameters, requested time, source content revisions, and versioned implementation identity must contain every output-affecting dependency. Operations that use hidden mutable state, randomness, external resources, or unrepresented environment state must keep the default Never policy.

Output resizing is not a graph operation. It is an output-target setting because it changes how a logical output is delivered without changing the graph’s media transformation topology.

A backend can declare that an operation also performs selected immediately following conversions. Preparation uses this only when the operation result feeds that conversion and nothing else. The combined pass removes the surface between them while preserving the graph operation and converted format.

2D video transform

media.video.transform_2d transforms one video within its existing frame. Scale and rotation use the frame center, then translation moves the result.

The operation preserves frame dimensions and makes pixels outside the source transparent. A zero scale produces a transparent frame. Sampling is bilinear; strong minification can alias because the operation does not prefilter or build image pyramids.

Gaussian video blur

media.video.blur.gaussian softens one video input by averaging nearby pixels with Gaussian weights.

let radius = graph.add_scalar_parameter("radius", 12.0)?;
let blurred = graph.add_operation(video_blur_gaussian::Options {
    id: "blur",
    active_range: range,
    input,
    radius,
})?;

A radius of 0 preserves the input. Values outside the documented range are rejected during graph analysis. The operation preserves the prepared video format.

Video color adjust

media.video.color_adjust changes exposure, contrast, and saturation in one operation.

The adjustment is defined in linear RGB. Preparation converts the input to a supported linear working format and converts the result as needed. The input must declare complete color metadata so saturation has a defined luminance.

HDR tone mapping

media.video.tone_map compresses HDR linear luminance into a declared target display range while preserving RGB channel ratios.

let source_peak_nits = graph.add_scalar_parameter("source-peak", 1_000.0)?;
let target_peak_nits = graph.add_scalar_parameter("target-peak", 100.0)?;
let mapped = graph.add_operation(video_tone_map::Options {
    id: "tone-map",
    active_range: range,
    input,
    source_peak_nits,
    target_peak_nits,
})?;

Both peaks are constants because they describe the source programme and target display rather than a frame-by-frame effect. Preparation decodes PQ or HLG into linear half-float RGB before this operation and converts the result into the output’s requested transfer function afterward. Linear value 1.0 represents 100 nits.

Video alpha over

media.video.alpha_over places a foreground video over a background using the foreground alpha at each pixel.

let composited = graph.add_operation(video_alpha_over::Options {
    id: "alpha-over",
    active_range: range,
    background,
    foreground,
})?;

A transparent foreground reveals the background; an opaque foreground replaces it. Preparation converts both inputs to the same BGRA color format and premultiplied-alpha representation before the operation runs.

Video mix

media.video.mix blends two video inputs.

The inputs are ordered: the first input is selected at an amount of 0, and the second is selected at an amount of 1.

let amount = graph.add_scalar_parameter("amount", 0.5)?;
let mixed = graph.add_operation(video_mix::Options {
    id: "mix",
    active_range: range,
    first,
    second,
    amount,
})?;

Amounts outside the range from 0 to 1 are clamped to the nearest endpoint. Both inputs use the format selected during preparation.

Audio mix

media.audio.mix sums two audio inputs sample by sample.

let mixed = graph.add_operation(audio_mix::Options {
    id: "mix",
    active_range: range,
    first,
    second,
})?;

Both inputs must have the same sample rate, channel layout, and sample storage. Planar floating-point samples are added without clipping, preserving values outside the range from -1 to 1 for later processing or output conversion.

Audio loudness normalization

media.audio.normalize applies the constant gain resolved by an explicit complete-program loudness analysis. It supports every prepared planar-f32 speaker layout and preserves relative channel levels.

The operation does not perform analysis itself. See Loudness normalization for the two-pass data flow.

Passtape prepares stable timeline ranges rather than individual frames:

[0s, 5s)   clip A
[5s, 7s)   clip A + clip B + transition
[7s, 10s)  clip B

This produces three segments because the active sources and operations change at 5 and 7 seconds. Every frame within a segment reuses the same execution structure; only timestamps and parameter values vary.

Where segments are used

MediaGraph
  -> semantic analysis finds stable topology ranges
SemanticPlan
  -> preparation selects requested ranges and compiles segment programs
PreparedPlan
  -> frame execution finds the segment for a requested time

Semantic analysis first creates backend-neutral segments. Preparation filters them to the requested outputs and creates a VideoSegmentProgram. Frame execution uses the program to find active sources and its pass tape.

Program shape

pub struct VideoSegmentProgram {
    id: u32,
    semantic_id: Arc<str>,
    time_range: TimeRange,
    active_source_indices: Box<[SourceIndex]>,
    pass_tape_index: VideoPassTapeIndex,
    temporal_window: TemporalWindow,
    cache_key: Option<ProgramCacheKey>,
}

• id identifies the program within this prepared plan.
• semantic_id identifies the matching global semantic span across preparations. It remains equal while the graph ID, active topology, and span boundaries remain equal; it is opaque and not persistent.
• time_range is half-open: it includes its start and excludes its end.
• active_source_indices identifies the sources needed in the range.
• pass_tape_index selects the work to execute for each frame.
• temporal_window records how much neighboring source time the operations need before and after each frame.
• cache_key makes deterministic rendered work eligible for reuse. It includes relevant source content revisions, operations and parameters, selected pass implementations, output settings, timeline range, and backend. It is absent when a source or operation has not promised deterministic output.

The cache key is an in-process reuse fingerprint, not a media checksum or a persistent file identifier.

Operations declare exact neighboring offsets. Windows summarize their furthest offsets and add along an operation chain. For example, two effects that each need the previous frame require input from two frames earlier. Parallel branches use the larger enclosing window.

Preparation propagates the exact offsets through upstream operations. The pass tape contains the resulting source requests and pass jobs at relative times. Frame execution adds the requested output time, acquires those decoded frames, and follows the fixed tape.

When an offset reaches outside the selected stable-topology segment, execution repeats the current frame for that sample. This keeps the first and last output frames renderable without executing a tape against different topology.

The type is defined in src/prepared_plan/program.rs.

A VideoPassTape is the ordered recipe a backend executor follows for every frame in a video segment.

Preparation creates one tape for each prepared segment. The segment owns the timeline range and stores an index into the prepared plan’s tape table. Tape times are relative to the requested output frame, so one immutable tape works throughout the segment.

It is similar to a list of GPU commands, but it is not raw GPU work yet. Each pass names the selected backend implementation, its inputs, its parameters, and where its result goes. A backend executor turns that information into GPU or CPU work.

Tape shape

Each tape contains three kinds of records:

pub struct VideoPassTape {
    source_slots: Box<[VideoSourceSlot]>,
    passes: Box<[VideoPass]>,
    output_slots: Box<[VideoOutputSlot]>,
    surface_descriptor_indices: Box<[VideoSurfaceDescriptorIndex]>,
    surface_slot_indices: Box<[Option<VideoSurfaceSlotIndex>]>,
    surface_slot_descriptor_indices: Box<[VideoSurfaceDescriptorIndex]>,
}

pub struct VideoSourceSlot {
    source_index: SourceIndex,
    value_index: VideoValueIndex,
    time_offset: RationalTime,
}

pub struct VideoPass {
    kind: VideoPassKind,
    implementation_id: Arc<str>,
    time_offset: RationalTime,
    input_values: Box<[VideoValueIndex]>,
    temporal_input_values: Box<[VideoTemporalInput]>,
    output_value: VideoValueIndex,
    parameter_stream_indices: Box<[ParameterStreamIndex]>,
    recyclable_values: Box<[VideoValueIndex]>,
}

pub enum VideoPassKind {
    Operation(OperationIndex),
    Conversion,
}

pub struct VideoOutputSlot {
    output_index: OutputIndex,
    value_index: VideoValueIndex,
}

• Source slots identify decoded frames and their times relative to the requested output frame.
• Passes transform those frames in dependency order. VideoPassKind distinguishes graph operations from format conversions inserted by preparation. An operation pass keeps semantic input and parameter order. A conversion pass has one input and no parameter streams. Temporal passes also identify neighboring values for each semantic input; backends can consume them in one scan through VideoPassInputs::temporal_samples. Each pass records intermediate values whose surfaces become reusable after that pass.
• Output slots identify the results sent to requested graph outputs.

The runtime can decode the source slots, execute the passes from first to last, and materialize the output slots. It does not need to traverse MediaGraph or sort operations for every frame. See frame execution for how a backend runs this recipe.

How records connect

Source slots and passes each produce a logical video-frame result. Passtape calls each result a value. A value can therefore be a decoded source frame or an intermediate frame produced by an operation.

Records refer to values using dense VideoValueIndex numbers:

source A -> value 0
source B -> value 1
mix(value 0, value 1) -> value 2
requested output -> value 2

The tape also stores one surface-descriptor index at each value position. The descriptor describes the value’s format and required capabilities; it does not name an allocated texture or buffer. Operation results also map to prepared surface slots, allowing frame execution to reuse storage without searching at runtime. See surface planning.

The contracts are defined in src/prepared_plan/video_pass.rs and compiled in src/prepared_plan/prepare.rs.

A video pass tape describes frame results as logical values. Surface planning describes the storage each value will require when the tape runs.

Passtape does not allocate GPU resources during preparation. It creates a table of VideoSurfaceDescriptor records instead:

pub struct VideoSurfaceDescriptor {
    format: VideoFormat,
    required_capabilities: VideoSurfaceCapabilities,
}

• format contains the required dimensions, pixel format, and color metadata.
• required_capabilities records the access a compatible surface must support.

The current capabilities are:

A value can require several capabilities. For example, a decoded source consumed by a mix operation requires both DECODE_WRITE and SHADER_READ.

Each VideoPassTape stores one VideoSurfaceDescriptorIndex for every dense VideoValueIndex. Compatible values share entries in the prepared plan’s descriptor table:

value 0 -> descriptor 0: BGRA 1920x1080, decode write + shader read
value 1 -> descriptor 0: BGRA 1920x1080, decode write + shader read
value 2 -> descriptor 1: BGRA 1920x1080, render write

The runtime will later acquire a surface whose actual capabilities contain all of the descriptor’s required capabilities. The descriptor does not choose a concrete texture, pool entry, queue transition, or API object.

Output format selection

A video output request names one preferred pixel format and may list ordered fallbacks. Preparation selects the first format supported by the backend and the required graph conversion. The selected format is stored in VideoOutputPlan, so execution never negotiates formats per frame.

An encoder target can therefore prefer its native input while accepting a more general surface when the backend cannot prepare that native layout. Callers that require an exact format leave the fallback list empty.

Working set

PreparedPlan::video_surface_working_set records the maximum number of each descriptor used by one frame’s operation results and output transforms. This is deterministic because the pass tapes and output materialization steps are fixed during preparation.

Preparation records the last pass that reads each intermediate value and assigns operation results to reusable surface slots. A slot is reassigned only after its previous value’s final read. For example, a linear chain of three passes needs two intermediate slots rather than three:

pass 1 -> slot 0
pass 2 -> slot 1  (then slot 0 becomes available)
pass 3 -> slot 0

Each slot’s descriptor combines the capabilities required by every value that uses it. Frame execution therefore follows fixed slot indices and does not search for reusable surfaces.

Decoded source surfaces are owned by the source provider and are never reused as operation outputs. Surfaces retained by requested outputs are also excluded from reuse so a later pass cannot overwrite a displayed or encoded frame.

A backend can use the table to prewarm its surface pools before frame execution. It is a minimum working set, not a hard allocation limit. Decoder reference frames belong to the decoder, and a display or encoder may retain submitted frames until its bounded backpressure allows more work.

Frame execution describes when those surfaces are acquired and who owns their handles.

The contracts are defined in src/prepared_plan/video_surface.rs and compiled in src/prepared_plan/prepare.rs.

Frame execution happens after graph analysis and request-specific preparation. It is the per-frame part of the engine:

once:       MediaGraph -> MediaRuntime::prepare -> PreparedPlan
per frame:  VideoOutputSession -> decoded sources -> VideoFrameExecutor -> target

For a requested time, the runtime’s VideoFrameRenderer selects the prepared video segment. It asks each VideoSourceFrameProvider for the required source frames, passing each source identity, source-local time, decoder constraints, and surface descriptor. A temporal pass tape contains the exact neighboring offsets declared by its operations. At a segment boundary, an unavailable neighbor repeats the current frame. The provider returns each decoded surface and the exact source-local range that surface represents.

The provider does not choose the graph’s working format. Preparation inserts pixel or color conversion passes before operations that need them. A requested output resize is resolved during preparation and applied when that output is materialized.

Interactive sessions may reuse those frames through the decoded frame cache. Sequential output sends ordinary decoded frames directly to execution and does not retain a history. Temporal execution keeps only its prepared decode working set so adjacent output frames do not repeatedly seek and decode the same neighborhood. This working set is separate from the caller’s optional cache budget, and preparation rejects a temporal schedule whose estimated decoded surfaces exceed the engine limit.

The renderer supplies the resolved surfaces to VideoFrameExecutor, which runs the center and neighboring pass jobs in prepared order and materializes only the requested center-time outputs. Materialization can retain the final surface as-is or resize it to dimensions resolved from the output request. The executor gives the backend one frame boundary around this work, allowing a GPU backend to record all passes and output conversions before submitting them together.

This stage runs for every requested frame, but it does not repeat preparation: it never traverses MediaGraph, chooses implementations, or sorts operations. An output session adds cadence, target selection, backpressure, and finalization around this per-frame path.

Runtime traits

Rust uses a trait to define behavior that different types can implement.

• VideoSourceFrameProvider supplies decoded source surfaces; its concrete implementation owns media lookup, demuxing, and decoding.
• VideoFrameBackend acquires intermediate surfaces, executes prepared passes, materializes outputs, and completes the recorded frame.

For example, a macOS implementation can decode into Core Video surfaces and execute passes with Metal while the prepared plan remains platform-neutral.

Ownership

The executor retains source handles, holds intermediate handles while the frame runs, and returns the materialized outputs to the caller. The backend owns the actual allocation and pooling. These lifetimes keep pixels on the GPU and stop a surface from being reused while it is still needed.

The prepared plan and executor backend must have the same stable backend ID. This prevents a pass tape compiled for one backend from being run by another.

The contracts are defined in src/video_frame.rs and src/video_frame/render.rs. The surrounding runtime and session interface is defined in src/runtime.rs.

Audio is processed as decoded PCM samples. It uses its own sample clock: a 48 kHz output contains 48,000 sample frames per second.

Passtape prepares one AudioBlockProgram for each timeline span where the active audio work stays the same:

pub struct AudioBlockProgram {
    time_range: TimeRange,
    sample_range: AudioSampleRange,
    pass_tape_index: AudioPassTapeIndex,
    format: AudioFormat,
}

Each program points to an ordered pass tape:

decode 44.1 kHz stereo
  -> resample to 48 kHz
  -> run graph operations
  -> send to output

The source provider only decodes the file’s native PCM format. Audio formats identify every speaker and its storage order, so stereo is not confused with two unrelated channels. Preparation inserts explicit resampling or channel conversion passes when needed. The backend executes those passes and the graph operations in order.

The current audio operations use the requested output format as their working format, so differently formatted sources are converted before processing.

AudioOutputSession divides the programs into fixed-size blocks, executes them, and sends the results to a file, device, or other destination. The preparation policy selects the block size.

See src/prepared_plan/audio_pass.rs and src/audio_block.rs.

Loudness normalization is an explicit two-pass graph workflow.

Prepare the unnormalized audio output in its delivery format, then scan that prepared output:

let analysis = runtime.analyze_audio_loudness(plan, output_node)?;
let settings = AudioLoudnessSettings::new(-14.0, -1.0)?;
let normalization = analysis.normalization(settings);

Analysis applies BS.1770 K-weighting, speaker-position weighting, overlapping 400 ms blocks, the absolute and relative gates, and four-times oversampled true-peak measurement. LFE channels do not contribute to programme loudness.

Rebuild the graph with the resolved normalization before its audio output:

let normalized = graph.add_operation(ops::audio_normalize::Options {
    id: "delivery-normalization",
    input: programme_audio,
    normalization,
})?;

media.audio.normalize applies one constant gain to every channel. The gain is the smaller of the amount needed to reach the loudness target and the amount allowed by the true-peak ceiling. It does not clip or limit samples, so a peak-constrained programme can remain quieter than the requested loudness. The ceiling applies to the graph PCM measured before encoding. Lossy codecs can introduce new reconstructed peaks; validate the encoded deliverable separately when its decoded peak is part of the delivery contract.

Analysis is never part of ordinary graph preparation. Callers decide when to scan and can keep the resulting graph separate from the editable programme. The resolved gain retains the analyzed graph identity, graph version, and timeline range. The typed operation rejects it when rebuilding another graph revision and uses the measured range automatically. Preserve the graph identity and version for a delivery-only rebuild; increment the version and analyze again after changing programme content. Analyze again when changing delivery format.

Long-running EBU Tech 3341 vectors and the release throughput matrix are explicitly invoked:

cargo test -p passtape-engine ebu_tech_3341 -- --ignored
cargo test --release -p passtape-engine loudness_analysis_benchmark_matrix \
  -- --ignored --nocapture

VideoFormat carries the information needed to interpret pixel values:

pub struct ColorMetadata {
    pub primaries: ColorPrimaries,
    pub transfer: TransferFunction,
    pub matrix: YcbcrMatrix,
    pub range: ColorRange,
}

• primaries identify the red, green, and blue chromaticities.
• transfer defines the relationship between linear light and stored values.
• matrix describes RGB and YCbCr conversion.
• range distinguishes video range from full range.

Sources provide this metadata. A missing declaration remains Unspecified instead of being labeled as a known color space. Format-preserving operations, prepared values, surface descriptors, and outputs keep the metadata with the frame. OutputColor::Source preserves it; OutputColor::Metadata requires an explicit output interpretation.

Passtape does not silently reinterpret pixels. For an operation with several video inputs, the first input selects the working color metadata. If the backend implementation requires a pixel format, that requirement changes only the pixel storage; it does not replace the working color metadata.

Preparation rejects inputs that cannot be reconciled deterministically. When complete input and working metadata differ, preparation inserts a conversion pass before the operation. The same happens before an output whose requested format differs. Identical conversions of one value are shared. The prepared plan reports FormatConversionInserted when conversion passes are present.

Pixel storage follows the same rule. A backend advertises exact conversion pairs and an operation may require a specific pixel format. Preparation inserts only conversions the backend explicitly supports; otherwise it rejects the plan. Scaling support is also declared per pixel format. Changing pixel layout requires complete color metadata because YCbCr matrix and range affect how the stored values are encoded.

Scaling preserves the pixel format and color metadata of its input. Backends declare support separately for packed RGB and planar YCbCr because those layouts require different sampling and output-storage paths.

For RGB storage such as BGRA or half-float RGBA, matrix and range are retained for a later YCbCr conversion but do not change the RGB arithmetic. Primaries and transfer function define the pixel transform between RGB formats.

Decoding a video frame is expensive. When an interactive caller requests the same part of a source again, Passtape can keep the decoded frame and reuse it instead of decoding it again.

For each requested source time:

1. The runtime looks for a decoded frame that covers that time.
2. If it finds one, it reuses the frame’s surface.
3. Otherwise, the source provider decodes the frame and the runtime caches it.

A decoded frame includes the exact source-time range it covers. This matters because several requested times can refer to the same video frame.

Only sources marked SourceCachePolicy::Deterministic enter this cache. That policy promises that identity, content revision, source time, and decode settings fully determine the returned media. Live, procedural, or externally mutable sources should keep the default Never policy.

If deterministic media changes without receiving a new source identity, set a new VideoSource::with_content_revision value. Previously decoded frames then have a different cache identity and cannot be reused.

The cache is disabled by default. Interactive sessions enable it by selecting a retention policy:

let session = session.with_decoded_frame_retention(
    DecodedVideoFrameRetention::interactive(),
);

VideoFrameRequestScheduler uses the interactive policy by default.

The cache has a byte limit. When it reaches that limit, it removes the decoded frames that have gone unused for the longest time.

Temporal operations use a separate, preparation-bounded working set for the neighboring frames needed during execution. They do not raise this cache’s caller-selected byte limit.

The contracts are defined in src/video_frame/decoded.rs.

Interactive output should render what the user wants now, not every frame requested on the way there. VideoFrameRequestScheduler therefore keeps only the newest pending graph time.

let scheduler = VideoFrameRequestScheduler::for_plan(session.plan());

// This handle can also be cloned into the UI thread.
scheduler.request_frame(timeline_time);

// The output owner supplies destination timing separately.
session.render_latest_request(&scheduler, presentation_time)?;

// Run look-ahead decoding only when no current request needs service.
session.prefetch_next_frames(&scheduler);

The output thread calls render_latest_request. If another request arrives while a decoder or GPU call is running, Passtape lets that synchronous call finish and discards the result at the next safe boundary. A request arriving while destination submission is already underway cannot undo that submission. The next request is then serviced before the method returns. The session also stops retrying a backpressured frame once that frame becomes obsolete. Drive the output session from a worker thread because decoding and backend execution are synchronous.

After submitting a frame, the worker can call prefetch_next_frames when idle to decode a small number of later source frames into the runtime’s byte-bounded decoded-frame store. A new request makes older prefetch work obsolete. Prefetch does not run video operations. It is best-effort: a failure is recorded in runtime statistics but does not turn an already displayed frame into an error.

Temporal operations already acquire their exact neighboring frames during normal rendering, using their separate preparation-bounded working set. Frame-request prefetch skips those segments.

The contracts are defined in src/runtime/frame_request.rs.

Long-running output sessions can be cancelled and can report accepted work:

let cancellation = CancellationToken::new();
let cancel_handle = cancellation.clone();

// Give `cancel_handle` to the UI or coordinator. It can stop this session with:
// cancel_handle.cancel();

session.run_with_control(&cancellation, |progress| {
    if let Some(video) = progress.video_frames() {
        println!("{} / {} frames", video.completed(), video.total());
    }
})?;

CancellationToken is cheap to clone and safe to send to another thread. All clones share one signal. Cancellation cannot be reset, so each output session should use a new token.

OutputProgress reports video in frames and audio in sample frames. The totals come from the prepared timelines, and completed counts advance only after the destination accepts the media. Backpressure retries do not increase progress.

The progress callback runs on the output thread. It should return quickly and send any slower UI or logging work elsewhere.

Cancellation is checked between decoded frames or audio blocks and while a destination applies backpressure. Work already inside a synchronous decoder or backend call returns before cancellation is observed. The session then asks the destination to abandon its partial output and returns a Cancelled error. Destinations that can finalize asynchronously may also observe cancellation while finalization is in progress.

The contracts are defined in src/runtime/control.rs, and the session methods are defined in src/runtime.rs, src/runtime/audio_output.rs, and src/runtime/media_output.rs.

MediaRuntime keeps cumulative statistics for the work it performs. Callers take snapshots between synchronous runtime operations:

let stats = runtime.stats();

println!("rendered {} video frames", stats.video().frames_rendered());
println!("submitted {} audio blocks", stats.audio().blocks_submitted());

The snapshot separates the main stages of the engine:

• preparation counts analysis-cache hits and misses plus request-specific lowering outcomes;
• video counts source requests, rendered frames, accepted frames, and backpressure;
• frame-request counts started requests, obsolete renders, and nearby-frame prefetch;
• audio counts source requests, rendered and accepted blocks, sample frames, and backpressure;
• output runs combine completed, cancelled, and failed video, audio, and audio/video calls to run or run_with_control;
• prepared-program statistics report reused and rebuilt pass instructions, evictions, and current residency;
• decoded-video and decoded-audio statistics report cache activity and current residency.

Counters are always active and use saturating arithmetic. Duration collection is disabled by default because reading a clock for every frame or audio block adds work to hot paths. It can be enabled when profiling:

runtime.reset_stats();
runtime.set_timing_collection_enabled(true);

Reset before enabling timing and leave it enabled for the complete measurement window when calculating averages from counters and durations. Counters remain active while timing is disabled, so enabling timing after work has already run makes durations cover only part of the counter window.

Video and audio source statistics are a subset of render statistics. Source durations measure provider calls, while render durations include source acquisition and backend execution. A provider failure therefore increments both source and render failures. A decoded-video or decoded-audio cache hit avoids the matching provider and does not increment source_requests.

Output run duration covers a complete run or run_with_control call, including destination backpressure and finalization. Manually driven sessions using render_frame, render_range, or finish do not contribute to output-run statistics.

reset_stats clears counters and durations without releasing the cached semantic plan, prepared programs, or decoded media. Prepared-program and decoded-media hit, miss, and eviction counters reset while their entries and capacities remain unchanged. It also preserves whether timing collection is enabled.

The public snapshot types are defined in src/runtime_stats.rs.

PROTOTYPE — throw this code away after it answers the question.

Question

Can Windows and Linux execute a GPU video operation between hardware decode and hardware encode without Passtape having to move frame pixels through CPU memory?

This probe uses FFmpeg only as disposable orchestration around the operating system GPU APIs. It is not a proposal to make FFmpeg a Passtape dependency.

The probe establishes an API-level zero-copy candidate when:

• decoding produces hardware frames;
• the effect consumes and produces hardware frames;
• encoding directly accepts those hardware frames; and
• the filter graph contains no hwdownload, hwupload, or software scaler.

That does not prove that a driver performs no internal GPU copy. Use PIX on Windows or a vendor GPU profiler on Linux to inspect resource history and copy engines while this probe runs.

Windows

Requirements:

• Windows 10 or later;
• a hardware H.264 decoder and encoder;
• a recent FFmpeg build containing d3d11va, scale_d3d11, and h264_mf;
• Python 3.11 or later.

Run from the repository root:

python crates/passtape/prototypes/platform-zero-copy/probe.py windows C:\path\to\input.mp4

The probe verifies the expected Ryzen 5 5600G host, discovers the installed display adapters, then selects the GeForce RTX 3060 by PCI vendor ID (0x10de). It requires exactly one NVIDIA adapter so that vendor selection is unambiguous, and fails rather than silently using the AMD iGPU or another machine. The candidate path is:

D3D11VA decode -> ID3D11Texture2D -> scale_d3d11 -> ID3D11Texture2D -> Media Foundation H.264 encode

Capture the process in PIX and verify that Media Foundation uses the same RTX 3060, that the frame does not visit a CPU-visible upload/readback heap, and that no unexpected copy pass occurs at API boundaries.

Linux

Requirements:

• a VA-API-capable GPU and driver;
• Vulkan support for the same physical GPU;
• a recent FFmpeg build containing vaapi, vulkan, scale_vulkan, and h264_vaapi;
• Python 3.11 or later.

Run from the repository root:

python3 crates/passtape/prototypes/platform-zero-copy/probe.py linux /path/to/input.mp4

The probe verifies the expected Ryzen 5 5600G host, discovers every DRM render node, and selects its integrated AMD GPU by PCI vendor ID (0x1002). Automatic selection requires exactly one AMD render node; an explicit override is checked against the discovered device. It sets LIBVA_DRIVER_NAME=radeonsi, verifies H.264 encode support with vainfo, and derives the Vulkan device from that exact VA-API device. This prevents an accidental AMD-to-NVIDIA frame transfer.

Override automatic selection when necessary:

python3 crates/passtape/prototypes/platform-zero-copy/probe.py linux \
  /path/to/input.mp4 --render-node /dev/dri/renderD129

The candidate path is:

VA-API decode -> DMA-BUF mapping -> Vulkan scale -> DMA-BUF mapping -> VA-API H.264 encode

The current Linux probe intentionally rejects the NVIDIA adapter. NVIDIA’s Linux path requires a separate NVDEC/Vulkan/NVENC prototype because its VA-API compatibility driver does not support encoding.

Use the profiler for the installed driver (Intel GPA, Radeon GPU Profiler, or Nsight Systems) to check copy-engine activity and Vulkan image transitions.

Output

The probe prints one JSON report containing the discovered CPU, GPU devices, selected device, and media capabilities. verdict has one of these values:

• api-level-zero-copy-candidate: the entire command completed with hardware frames and no explicit software transfer;
• unsupported: the installed FFmpeg build or GPU lacks a required capability;
• failed: the path exists but failed while processing the supplied media.

Use --frames to change the bounded run length. Use --dry-run on any platform to inspect the generated command without executing it.

WGSL compilation probe

The pipeline probes deliberately use native FFmpeg filters first, separating surface interoperability from shader-language translation. Compile the included WGSL operation into the two native shader formats with:

cargo run --manifest-path \
  crates/passtape/prototypes/platform-zero-copy/kernel-compiler/Cargo.toml -- \
  crates/passtape/prototypes/platform-zero-copy/invert.wgsl \
  /tmp/passtape-platform-kernels

This produces:

• invert.hlsl, suitable for a Direct3D compute pipeline after compilation to DXBC/DXIL; and
• invert.spv, suitable for a Vulkan compute pipeline.

The compiler is MIT/Apache-2.0 licensed Naga. It does not introduce a GPU runtime or surface abstraction, so choosing WGSL as source does not itself add a frame copy. This translation-only probe uses ordinary RGBA storage textures; a production backend must separately validate how native multi-plane NV12 surfaces are viewed or adapted for these bindings without a copy.