Realtime API

Low-latency speech and text over a WebSocket. This client covers the WebSocket event transport and ephemeral client-secret minting; audio is exchanged as base64 PCM inside events. WebRTC media capture and SIP telephony are out of scope. OpenAI only.

Bounds

realtime_connect takes a config::Union{Nothing,RequestConfig}, resolves it the usual four ways, and captures it on the session's config field:

The open phase is bounded by connect_timeout — a peer that accepts the TCP connection but never completes the upgrade throws UniLMTimeout(:connect, …) rather than blocking.
realtime_receive is bounded by the session's stream_idle_timeout and throws UniLMTimeout(:stream_idle, …) on a breach. Unblocking a parked read means closing the socket, and HTTP.jl's WebSocket close allows the peer up to ~5 s to acknowledge, so the breach surfaces within [limit, limit + ~5 s].
The session's lifetime is deliberately unbounded. Once your handler runs, how long it stays connected is your decision — a Realtime session is meant to sit idle waiting for input.

Realtime throws rather than returning a typed result value, and realtime_connect makes a single attempt; max_attempts does not apply. A minted client secret is a live credential, so RealtimeSecretSuccess redacts it when displayed — read .value programmatically. See Timeouts & Retries.

Session and Result Types

UniLM.RealtimeSession — Type

A live Realtime WebSocket session. Created by realtime_connect, which resolves a RequestConfig and captures it on config — realtime_receive reads its stream_idle_timeout from there. The two-argument constructor inherits the ambient config.

source

UniLM.RealtimeSecretSuccess — Type

Successful mint_realtime_secret result; value is the ephemeral client secret and raw the unparsed JSON response.

source

UniLM.RealtimeFailure — Type

Realtime API error result: HTTP status and the raw response body.

source

UniLM.RealtimeCallError — Type

Local/transport error from a Realtime API call (the request never completed).

source

Transport Functions

UniLM.realtime_connect — Function

realtime_connect(handler; model="gpt-realtime-2", service=OPENAIServiceEndpoint, config=nothing)

Open a Realtime WebSocket and run handler(session::RealtimeSession). Inside the handler use realtime_send to send event dicts and realtime_receive to read server events. The socket closes when handler returns.

Pass config::Union{Nothing,RequestConfig} to override the timeout budget for this session. ONLY the open phase is bounded, by connect_timeout: a peer that accepts the TCP connection but never completes the upgrade throws UniLMTimeout(:connect, …) instead of blocking forever. Once handler is running the session's lifetime is the caller's to decide, so no bound applies to it. The resolved config travels on the session, so realtime_receive inherits stream_idle_timeout. handler runs on an internal task (that is what makes the open phase separable); dynamically scoped values propagate into it.

source

UniLM.realtime_send — Function

realtime_send(session, event::AbstractDict)

source

UniLM.realtime_receive — Function

realtime_receive(session) -> Dict

Block for the next server event, bounded by the session's stream_idle_timeout. A parked read cannot be polled out of, so a breach closes the socket — the universal unblocker — and throws UniLMTimeout(:stream_idle, …).

source

Event Builders

UniLM.realtime_event — Function

realtime_event(type; kwargs...) — a generic client event dict.

source

UniLM.session_update — Function

session_update(session) — a `session.update` event.

source

UniLM.input_audio_append — Function

input_audio_append(audio_b64) — append base64 PCM to the input audio buffer.

source

UniLM.response_create — Function

response_create(; response=nothing) — request a model response.

source

Client Secret

UniLM.mint_realtime_secret — Function

mint_realtime_secret(; session=nothing, service=OPENAIServiceEndpoint)

Create an ephemeral client secret for client-side Realtime connections (POST /v1/realtime/client_secrets). session is an optional session-config dict. Returns RealtimeSecretSuccess (.value), RealtimeFailure, or RealtimeCallError.

Pass config::Union{Nothing,RequestConfig} to override the timeout budget for this call (a single bounded attempt; max_attempts does not apply).

source

Usage

# Mint an ephemeral client secret for a client-side (browser) connection
secret = mint_realtime_secret()
secret isa RealtimeSecretSuccess && println("client secret: ", secret.value)

# Open a WebSocket session, configure it, stream audio, and read events
realtime_connect(model="gpt-realtime-2") do session
    realtime_send(session, session_update(Dict("modalities" => ["text", "audio"])))
    realtime_send(session, input_audio_append(audio_b64))   # audio_b64 :: base64 PCM
    realtime_send(session, response_create())
    event = realtime_receive(session)                        # blocks for the next server event
    println(event["type"])
end