passtape-macos connects Passtape’s backend-neutral execution contracts to macOS media surfaces.

video file -> MacVideoSourceProvider
    -> MacVideoSurface (retained CVPixelBuffer)
    -> prepared Metal passes through MacVideoBackend
    -> MacMediaOutput (GPU surface, packed CPU frame, or tensor)
    -> VideoOutputSession
    -> MacVideoFile -> H.264, HEVC, or ProRes

audio file -> MacAudioSourceProvider
    -> MacAudioBlock (shared planar f32 samples)
    -> prepared CPU work through MacAudioBackend
    -> AudioOutputSession
    -> MacAudioFile -> AAC, Apple Lossless, FLAC, Opus, or PCM

video and/or audio -> MediaOutputSession
    -> MacMediaFile -> encoded video and/or audio

new_runtime() configures both paths. Video and audio sources are registered independently because a graph may use either stream or assign different source identities and timeline ranges to them.

MacVideoSourceProvider::add_file opens a video and registers it under the identity used by a graph’s VideoSource. AVFoundation reads compressed samples from the container, and a reusable VideoToolbox session decodes them into Metal-compatible Core Video surfaces. The returned descriptor copies the identity and decoded format into the graph without storing a path or decoder object there. VideoToolbox prefers dedicated decoder hardware and may use its codec fallback when that hardware does not support the stream. add_file_stream selects a zero-based video track when a container holds more than one; MacAudioSourceProvider provides the equivalent audio-track API.

MacVideoSourceProvider::add_image_file decodes a still image once with ImageIO, applies its stored orientation, and keeps the resulting Metal-compatible surface. An ImageSource reuses those pixels for every frame in its graph node’s active timeline range.

add_bgra_file is the alpha-preserving alternative for foreground media. Ordinary files prefer native bi-planar 4:2:0 YUV surfaces and fall back to BGRA when a decoder does not declare how its chroma samples are positioned. Alpha foregrounds decode to straight-alpha BGRA, and preparation converts them as operations require.

The provider reuses its active compressed reader and VideoToolbox decoder for nearby forward requests. On a distant or backward request, it reopens them near the requested source time and decodes forward. If that preroll does not include the preceding frame, it restarts at the beginning to preserve frame selection.

Each registered file owns one serial decode worker. After nearby forward requests establish sequential playback, that worker retains at most three future frames while the calling thread renders and submits the current frame. The VideoToolbox decoder separately retains a bounded set of codec reference, reordering, and callback frames. VideoToolbox compression and movie writing also have their own bounded queues, so these stages can overlap without an unbounded frame backlog. A request still waits synchronously when its frame has not finished decoding, and file registration waits for initial inspection.

Source descriptors do not assume that files stay unchanged. Call deterministic_descriptor(content_revision) when the application controls the file and updates the revision whenever its contents change; this enables decoded-frame and rendered-output reuse.

MacVideoSurface owns each Core Video pixel buffer through Apple’s reference counted handle, so moving a frame through Passtape does not copy its pixels. Metal uses texture views over that storage and allocates intermediate frames from reusable Core Video pools.

AVFoundation color tags are stored in each source’s VideoFormat and preserved through prepared surfaces and format-preserving operations. Missing color characteristics remain unspecified; the missing full-range flag means video range according to Core Media. Encoders write the prepared tags. PQ and HLG are preserved by HEVC Main 10 and ProRes output.

When complete color metadata or pixel storage differs between inputs or outputs, preparation adds an explicit conversion pass. Built-in video operations declare the working representation they need: most use BGRA, while color adjustment and HDR tone mapping use linear half-float RGB. Source-only encoded output stays in its bi-planar YUV storage and avoids unnecessary conversion.

The conversion shaders support 8- and 10-bit 4:2:0 YUV, 10-bit 4:2:2 YUV, BGRA, and half-float RGBA. YUV surfaces retain their Core Video chroma location. PQ and HLG values pass through unchanged when their metadata is unchanged. Their transfer functions can also be decoded into linear light for an explicit media.video.tone_map operation before SDR output conversion.

Gaussian blur uses Metal Performance Shaders. Constant radii keep their exact value; animated radii use prebuilt half-pixel steps so rendering never creates a filter on the frame path. A zero radius uses the existing Metal copy path.

2D transforms use a bilinear Metal pass. They preserve the frame size and sample transparent pixels beyond the source bounds.

Alpha compositing runs as a Metal pass over premultiplied BGRA inputs. The foreground alpha selects how much of the background remains at each pixel.

Before an output session starts, the backend reads the prepared surface working set and prewarms each Core Video pool for the session’s bounded in-flight frames. The bound accounts for frame count and estimated surface bytes, including surfaces created only for output resizing. AVFoundation separately owns the codec’s reference surfaces.

When run_with_control is cancelled, the macOS sink tells AVAssetWriter to abandon the incomplete file. Cancellation is also observed while the writer is finalizing.

Output materialization can resize each supported pixel format with bilinear Metal kernels. YUV resizing samples its luma and chroma planes directly and accounts for 4:2:0 or 4:2:2 subsampling and chroma location. The resized surface comes from the same reusable Core Video pool system and stays on the GPU path before display or encoding.

Passtape’s custom Metal kernels live in src/shaders and are compiled into one library when the backend initializes. Gaussian blur uses Apple’s optimized Metal Performance Shaders implementation instead of a custom sampling loop.

For each rendered frame, the backend records its operations and output resizing into at most one Metal command buffer. Sequential output sessions retain a bounded FIFO of submitted frames and wait for the oldest frame before handing it to the destination. This overlaps Metal execution with later decoding and command recording without allowing surface residency to grow without limit. Direct and interactive frame requests still wait before returning. A source-only frame that needs no GPU work creates no command buffer.

Supported graph operations and their parameters are listed in the operations guide. The macOS backend executes video operations with Metal while retaining frame storage in Core Video surfaces.

Application-defined Metal effects

MacRuntimeConfiguration pairs a platform-neutral OperationDescriptor with a MetalEffectImplementation before creating a runtime generation. The graph continues to store only the semantic operation identity, inputs, outputs, and parameters; Metal remains confined to this crate.

let descriptor = OperationDescriptor::new(
    OpRef::new("editor.effect.agent-created", 1),
    "Agent-created Effect",
    &[MediaType::Video],
    OutputDescriptor::new("output", MediaType::Video),
)
.with_parameter_schema(&[
    ParameterDescriptor::new("amount", ParameterType::Scalar),
]);

let metal = MetalEffectImplementation::from_source(
    descriptor.op_ref().clone(),
    shader_source,
    "agent_created_effect",
);
let mut configuration = MacRuntimeConfiguration::new();
configuration.register_effect(descriptor, metal)?;
let pending = configuration.prepare_concurrent_generation(graph, &request)?;

// Native preparation runs on the candidate's video and audio owners while the
// currently published generation remains available for playback.
let next_generation = pending.wait()?;
let generations = MacRuntimeGenerationSlot::new(next_generation);

Custom kernels receive required video inputs at consecutive texture indices, followed by their output texture. Buffer zero contains one float4 per parameter in descriptor order: scalar values use .x, and points use .xy. Textures are straight-alpha linear-light RGBA16Float. Candidate preparation compiles only effects used by its graph, caches identical pipelines across generations from that configuration on the same Metal device, and preserves Metal’s compiler diagnostic in MacRuntimeGenerationError.

Each call that creates a runtime freezes a generation-specific registry and backend identity. An editor may continue using its existing generation while an agent adds an effect to the configuration, creates and validates a later generation, reprepares the graph, and then publishes the ready runtime and plan together with MacRuntimeGenerationSlot::publish. Readers that already loaded the previous generation can finish with it; previously created runtimes are not mutated.

A request with VideoOutputDestination::CpuBuffer returns a MacMediaOutput containing a CpuVideoFrame; callers access it with as_cpu_frame or into_cpu_frame. The backend waits for Metal, copies only the pixel bytes from each Core Video row, and removes platform-specific row padding. This path is useful for thumbnails and pixel analysis; encoded-file targets keep surfaces on the GPU and do not perform the copy.

MacVideoFile supports H.264, HEVC Main/Main 10, HEVC with alpha, and the ProRes 422 and 4444 families. Main 10 and ProRes 422 use native 10-bit YUV surfaces; alpha codecs use an alpha-preserving RGB surface. H.264 and ordinary HEVC support MOV and MP4. HEVC with alpha and ProRes use MOV. MacH264Mov remains a convenience preset. VideoOutputSession handles frame submission, backpressure, and finalization. Finalization blocks, so the session should run on a worker thread.

MacVideoFileSpecification holds the codec, container, and preparation request without a path. Calling at(path) creates the execution target. Audio and combined outputs provide matching specification types, which lets an application prepare once and choose an atomic staging path only when execution starts.

VideoToolbox compresses rendered Core Video surfaces asynchronously. AVAssetWriter only places the compressed samples into the MOV or MP4 container. Passtape’s generic encoded-video pipeline moves packets between those two components in order. Both its packet queue and VideoToolbox’s input window are bounded; when either fills, the output session pauses rendering until space is available.

MacVideoFileEncoder is the lower-level sink for callers that need to drive prepared frames themselves.

Audio

MacAudioSourceProvider uses AVFoundation to decode native planar f32 PCM. Nearby requests reuse the decoder; distant or backward requests reopen it near the requested time. When a file declares its speaker order, Passtape preserves that order. Missing or unrecognized layout metadata remains discrete rather than being guessed from channel count alone.

Preparation inserts CPU passes when the sample rate, speaker layout, or channel storage order must change. Common mono, stereo, 5.1, and 7.1 layouts are supported. Downmixing omits LFE and routes center and surround channels at -3 dB; upmixing leaves speakers absent from the source silent. MacAudioBackend executes these conversions and audio graph operations. A matching source-only block can pass through without copying its samples.

MacAudioFile writes AAC, Apple Lossless, FLAC, Opus, or linear PCM. Supported audio containers are M4A, CAF, WAV, AIFF, native FLAC, MOV, and MP4, subject to the selected codec. AAC and Apple Lossless support 5.1, and linear PCM supports 7.1. FLAC and Opus output currently support mono and stereo. MacAacM4a remains the AAC convenience target.

AudioToolbox compresses audio before container writing. PCM is explicitly interleaved and packetized. MOV, MP4, and M4A use AVAssetWriter; CAF, WAV, AIFF, and FLAC use Core Audio file services. Codec priming and final padding are stored as container metadata rather than becoming audible samples.

MacAacM4a is an audio output target. Its PCM format becomes part of the preparation request. It accepts mono, stereo, or 5.1 planar f32 blocks from AudioOutputSession and writes AAC in an .m4a file.

Combined movies

MacMediaFileSpecification connects a non-empty selection of prepared video and/or audio outputs to one MacMediaFile. Keeping the specification attached to the destination ensures the encoder consumes the exact formats requested during preparation. VideoToolbox compresses video, while AVAssetWriter muxes the selected tracks. MOV and MP4 support depends on the chosen codecs. MacH264AacMov remains a convenience preset. MediaOutputSession feeds each selected track according to its own backpressure and timeline.

MOV and MP4 outputs can also carry multiple selectable WebVTT caption tracks. Attach text graph outputs with RequestedMediaOutput::with_text_tracks; each track can declare a BCP 47 language and at most one track can be enabled by default. Text cues are compiled during preparation and muxed without entering the per-frame render loop.

MacAudioVideoFileEncoder exposes the same two-track writer for callers that drive prepared media themselves. Each writer input is marked complete when its stream ends, even when the other stream is longer.