Streaming

Both APIs support real-time streaming of generated tokens, so you can display partial results as they arrive.

Chat Completions Streaming

Set stream=true and provide a callback:

chat = Chat(model="gpt-5.4-mini", stream=true)
push!(chat, Message(Val(:system), "You are a poet."))
push!(chat, Message(Val(:user), "Write a very short 2-line poem about coding."))
task = chatrequest!(chat, callback=function(chunk, close)
    if chunk isa String
        print(chunk)
    elseif chunk isa Message
        println("\n--- done ---")
    end
end)
result = fetch(task)
if result isa LLMSuccess
    println(result.message.content)
else
    println("Request failed — see result for details")
end

Request failed — see result for details

The Chat Completions callback fires in a fixed sequence: with a String argument as text arrives — each argument is newly-generated text forwarded verbatim (several wire deltas may be coalesced into one callback), so multibyte characters are never split across chunk boundaries — and then exactly once at end-of-stream with the fully assembled Message, whose content equals the concatenation of every forwarded String.

Stopping a Stream Early

The callback receives a Ref{Bool} that you can set to true to stop streaming:

task = chatrequest!(chat, callback=function(chunk, close)
    if chunk isa String
        print(chunk)
        if contains(chunk, "bad word")
            close[] = true  # stop the stream
        end
    end
end)

Streamed Tool Calls

When the model streams tool calls, pass on_tool_call to be notified as each call completes. It fires exactly once per tool call, in call order, receiving a fully assembled ToolCall whose arguments are already parsed (a zero-argument call arrives as an empty Dict). The text callback and on_tool_call are independent, so a single request can stream assistant text and surface tool calls as they finish:

# weather_tool defined as in the Tool Calling guide
chat = Chat(model="gpt-5.2", tools=[weather_tool], stream=true)
push!(chat, Message(Val(:system), "Use the tools you are given."))
push!(chat, Message(Val(:user), "What's the weather in Paris and Tokyo?"))

task = chatrequest!(chat;
    callback = (chunk, close) -> chunk isa String && print(chunk),
    on_tool_call = tc -> println("\ntool call: ", tc.func.name, " ", tc.func.arguments),
)
result = fetch(task)

on_tool_call is supported on the chatrequest! streaming path for every chat provider (OpenAI-wire, native Anthropic, native Gemini); the Responses-API respond path does not surface it. It is a notification hook — the final assembled Message still carries every tool call (alongside any assistant text), so code that does not set on_tool_call loses nothing and can read result.message.tool_calls after fetch.

Responses API Streaming

The Responses API provides an even cleaner streaming interface using Julia's do-block syntax:

task = respond("Write a haiku about Julia programming.", model="gpt-5.4-mini") do chunk, close
    if chunk isa String
        print(chunk)
    elseif chunk isa UniLM.ResponseObject
        println("\nDone! Status: ", chunk.status)
    end
end
result = fetch(task)
if result isa ResponseSuccess
    println(output_text(result))
else
    println("Request failed — ", output_text(result))
end

Request failed — Error: KeyError: key "OPENAI_API_KEY" not found

The do-block form automatically sets stream=true.

With Explicit Configuration

r = Respond(
    input="Explain quantum computing step by step",
    model="gpt-5.2",
    stream=true,
    max_output_tokens=2000,
)
println("Stream enabled: ", r.stream)
println("Request preview:")
println(JSON.json(r))

Stream enabled: true
Request preview:
{"input":"Explain quantum computing step by step","max_output_tokens":2000,"model":"gpt-5.2","stream":true}

Streaming Across Providers

Streaming is not OpenAI-only. The native Anthropic (ANTHROPICServiceEndpoint) and native Gemini (GEMINIServiceEndpoint) backends stream with the same callback / do-block API shown above — only the service (and model) change:

# Native Anthropic streaming (Chat Completions)
chat = Chat(service=ANTHROPICServiceEndpoint, stream=true)
push!(chat, Message(Val(:system), "You are a poet."))
push!(chat, Message(Val(:user), "Two lines about the sea."))
task = chatrequest!(chat, callback=(chunk, close) -> chunk isa String && print(chunk))
fetch(task)

# Native Gemini streaming (Chat Completions)
chat = Chat(service=GEMINIServiceEndpoint, stream=true)
push!(chat, Message(Val(:system), "You are a poet."))
push!(chat, Message(Val(:user), "Two lines about the mountains."))
task = chatrequest!(chat, callback=(chunk, close) -> chunk isa String && print(chunk))
fetch(task)

Streaming preserves provider-native fidelity too: for Anthropic streams, the content blocks (including thinking blocks and their signatures) are re-assembled verbatim from the SSE deltas and attached to the final Message.provider_content, so a streamed tool-calling turn round-trips exactly like a non-streamed one.

Providers on the OpenAI-compatible Chat Completions standard (DeepSeek, Ollama, vLLM, LM Studio, …) stream through the same stream=true + callback path.

Dropped SSE Payloads

An SSE data: payload the parser cannot read is dropped rather than allowed to abort the turn, and each drop is counted for that stream. The count rides the result as sse_dropped — on LLMSuccess, LLMFailure, ResponseSuccess and ResponseFailure alike — and a single warning naming the count, model and surface fires once the stream finalizes, not per line. sse_dropped == 0 means a clean stream, and is what a non-streamed call always reports. A non-zero count means the turn was assembled from an incomplete wire: a provider-side or transport anomaly, never routine operation. On a truncated stream it is often the reason no message could be built at all, so it is worth reading on a failure result as well as on a success.

Notes

Streaming runs on a separate Julia thread via Threads.@spawn. Make sure Julia is started with multiple threads (julia -t auto).
The returned Task can be fetched to get the final result.
The close Ref{Bool} can be set to true from the callback to terminate the stream early.
On completion, the Chat Completions callback receives a Message; the Responses API callback receives a ResponseObject.
Streamed usage: set stream_options=Dict("include_usage" => true) to capture token usage — it lands on the result's .usage once the stream completes. Empty-choices chunks, : keep-alive comment lines, and provider preambles (e.g. Azure content-filter results) are all tolerated without affecting the stream.
A provider error mid-stream on an otherwise-200 response (e.g. an Anthropic overloaded_error) surfaces as an LLMFailure/LLMCallError, never a truncated LLMSuccess — the else branch in the examples above catches it.
One Chat per in-flight call. A Chat is unsynchronized mutable state, so do not share one across concurrent streams, and do not push! to it while its stream task is still running. Use fork to fan out — see Concurrency.