Architecture

Updated: 2026-10-03 · v0.10.16

System Context

VideoToolbox Remote connects FFmpeg, stock Linux VA-API applications, a Plex packet integration and an experimental OBS plugin to a macOS VideoToolbox daemon. Client-side files, audio, subtitles and delivery stay local; supported video work runs on the Mac.

Frame producers can use the encoder directly: capture NV12/P010 on a PC, send the frames to the Mac and receive encoded H.264/HEVC packets. This separates where video originates from where it is encoded. See capture and live video for cameras, desktop capture, recording systems and application inputs.

flowchart LR
    Client["Patched FFmpeg"] -->|"Frames or packets / TCP"| Server["vtremoted on macOS"]
    Plex["Plex packet integration"] -->|"Compressed packets / TCP"| Server
    VA["Linux VA-API / OBS"] -->|"Raw frames / TCP"| Server
    Server --> VT["VideoToolbox decode / encode"]
    VT --> HW["Supported Mac media hardware"]

1. Components

Client (FFmpeg)

Server (vtremoted, macOS)

Linux VA-API driver

Plex packet integration

OBS plugin (experimental)

2. Data Flow (Encode)

  1. Handshake: Message HELLO exchange.
  2. Config: Client sends CONFIGURE, Server creates VTCompressionSession.
  3. Stream:
    • In: FRAME (pixels, optional side data)
    • Out: PACKET (H.264/HEVC, optional side data)
  4. Completion: Client sends FLUSH, receives delayed output until DONE, then closes. A timeout or fatal error is not successful completion.

3. Data Flow (Decode)

  1. Handshake: Message HELLO exchange.
  2. Config: Client sends CONFIGURE, Server creates VTDecompressionSession.
  3. Stream:
    • In: PACKET (Annex B, optional side data)
    • Out: FRAME (software planes or negotiated VideoToolbox output)
  4. Completion: Client sends FLUSH and receives delayed frames until DONE before closing.

4. Data Flow (Transcode)

  1. Handshake: Message HELLO exchange.
  2. Config: Client sends CONFIGURE with mode=transcode.
  3. Stream:
    • In: PACKET (Annex B, optional side data)
    • Out: PACKET (Annex B, optional side data)
  4. Completion: Client sends FLUSH and receives delayed packets until DONE before closing.

5. Capability-Gated Media Surfaces

The protocol advertises optional capabilities so newer clients can keep working with older servers for the original software-frame paths while failing newer requests during configure. Capability-gated surfaces include:

Hardware-frame ingest across a network is represented as an explicit upload path: local VideoToolbox frames are mapped into the negotiated wire pixel format before the server creates its own CVPixelBuffer. Handles such as IOSurface or CVPixelBuffer references are not treated as cross-host zero-copy objects.

6. Repository Layout

7. Performance Defaults

Defaults applied when the client does not override settings:

Property Default Purpose
ExpectedFrameRate from client Helps VT optimize encode pipeline
PrioritizeEncodingSpeedOverQuality unset Uses VideoToolbox default unless explicitly set
RealTime false Maximize throughput over latency
MaximizePowerEfficiency false Maximize speed over power
MaxFrameDelayCount from -bf Enable/limit frame reordering

[!NOTE] Remote decode defaults to async with a reorder depth of 2. The reorder buffer sorts by PTS and clamps only when PTS would regress.