VideoToolbox Remote Protocol

Status: Stable v1 Wire Format: Annex B (mandatory) Endianness: Big Endian (Network Byte Order)

1. Overview

The protocol uses a single TCP connection per session. It is stateful, starting with a handshake (HELLO), configuration (CONFIGURE), and then a stream of frames/packets.

Communication Modes

Mode Input Output Description
Encode FRAME PACKET Raw frames in, compressed NALs out.
Decode PACKET FRAME Compressed NALs in, raw frames out.
Transcode PACKET PACKET Compressed NALs in, compressed NALs out.

Sequence Flow (Encode)

sequenceDiagram
    participant C as FFmpeg (Client)
    participant S as vtremoted (Server)
    participant VT as VideoToolbox (Hardware)

    Note over C,S: Handshake
    C->>S: HELLO (token, codec, client_info)
    S-->>C: HELLO_ACK (status, caps)

    Note over C,S: Configuration
    C->>S: CONFIGURE (width, height, fmt)
    S->>VT: VTCompressionSessionCreate
    VT-->>S: Session Ready
    S-->>C: CONFIGURE_ACK (extradata)

    Note over C,S: Streaming (Encode Mode)
    loop Frames
        C->>S: FRAME (raw NV12 planes)
        S->>VT: VTCompressionSessionEncodeFrame
        VT-->>S: Callback (CMSampleBuffer)
        S-->>C: PACKET (Annex B encoded)
    end

    Note over C,S: Teardown
    C->>S: FLUSH
    S-->>C: Remaining PACKETs
    S-->>C: DONE

2. Transport & Framing

Header Structure

Offset Type Name Value
0 uint32 magic 0x56545231 (“VTR1”)
4 uint16 version 1
6 uint16 type Enum ID (see below)
8 uint32 length Payload size in bytes (excluding header)

Length and Geometry Limits

3. Message Types

ID Name Direction Payload Description
1 HELLO C → S Initial handshake with auth token.
2 HELLO_ACK S → C Server acceptance/rejection.
3 CONFIGURE C → S Stream parameters.
4 CONFIGURE_ACK S → C Finalized config & codec extradata.
5 FRAME Bidirectional Raw image data (planes).
6 PACKET Bidirectional Encoded bitstream (Annex B).
7 FLUSH C → S Request to drain pipeline.
8 DONE S → C Pipeline drained signal.
9 ERROR Bidirectional Fatal error info.
10 PING Bidirectional Keepalive.
11 PONG Bidirectional Keepalive response.
12 PACKET_ACK S → C Transcode-mode input packet credit.

4. Message Payloads

Handshake

HELLO (Type 1)

HELLO_ACK (Type 2)

Configuration

CONFIGURE (Type 3)

[!NOTE] options is a map of codec settings. Unknown keys are ignored. For mode=transcode, out_codec, out_width, out_height, scale_mode, and optional client feature requests such as packet_ack.v1=1 are passed here.

For accurate transcode seeking, transcode.presentation_window.v1 adds optional presentation_start_pts (inclusive) and presentation_end_pts (exclusive). Both are signed 64-bit timestamps in the configured timebase. The window applies to original decoded presentation timestamps after reordering and before encoding. Packets outside the window still reach the decoder to supply reference pictures; discarded frames never enter the encoder. The first retained frame starts the encoded stream with a keyframe. Invalid bounds and missing decoded PTS are errors. Windowed clients must also negotiate packet_ack.v1 so preroll can consume input credits without producing encoded packets.

transcode.hdr_to_sdr.v1 enables explicit hdr_to_sdr=1 in transcode mode, with HEVC input and NV12/P010 output. hdr_source_transfer is 16 (PQ) or 18 (HLG); the client validates 10-bit 4:2:0 BT.2020 input before configuring. Optional hdr_source_peak supplies the static peak luminance in nits, greater than zero and at most 10,000. Frame content-light metadata takes precedence; otherwise the static peak is used, defaulting to 10,000 nits for PQ and 1,000 for HLG. This capability is advertised on macOS 15+.

The daemon preserves P010 precision through decode and 32-bit floating-point precision through native headroom mapping and gamut conversion. Output uses limited-range BT.709 (color_range=1, colorspace=1, color_primaries=1, color_trc=1). Conflicting explicit output tags are rejected. HDR/ICC packet side-data types 20, 22, 28, 29, 31 and 41 are removed; unrelated records such as captions remain. The client also removes these records from output codec parameters and packets.

Streaming

FRAME (Type 5) Raw frame planes.

PACKET (Type 6)

With max_b_frames=0, encoded packets complete frames in submission order. The server can normalize duplicate or regressing PTS; clients associate completions by submission order and use the returned timestamps for output.

H.264 decode sessions accept in-band parameter-set changes. For HEVC, start a new session with updated extradata when parameter sets change. Encoded Annex B NAL units.

PACKET_ACK (Type 12) Empty payload. Servers that advertise packet_ack.v1 may send this only after a transcode-mode client also requests packet_ack.v1=1 in CONFIGURE. The ACK is emitted after the server accepts an input PACKET; clients use it for in-flight input credit instead of assuming every input packet produces one output packet.

5. Security & Error Handling

Clients must drain all delayed frames/packets after FLUSH until DONE. DONE is a completion signal; end-of-file, timeout or a fatal error is not a successful substitute. Message read/write deadlines cover the complete operation rather than resetting for each partial transfer. The transport is plain TCP; see security.

Error Codes

Code Meaning
1 Auth Failure
2 Server Busy
3 Unsupported Config
4 Bad Request
5 Internal Error