Service Endpoints

ServiceEndpoint

Abstract supertype for LLM service backends. Subtypes control URL routing and authentication.

OpenAI-compatible backends subtype OpenAIWireEndpoint (itself a subtype of ServiceEndpoint) to inherit the chat request/response encoding and SSE handling; backends with a native wire (Anthropic, Gemini) subtype ServiceEndpoint directly and implement the wire seam (encode_request/decode_response/handle_sse_event!) themselves.

Built-in subtypes:

• OPENAIServiceEndpoint — OpenAI API (default)
• AZUREServiceEndpoint — Azure OpenAI Service
• GEMINIOpenAIServiceEndpoint — Google Gemini via OpenAI-compatible endpoint
• GEMINIServiceEndpoint — Google Gemini native generateContent API
• ANTHROPICServiceEndpoint — Anthropic (Claude) native Messages API
• GenericOpenAIEndpoint — any OpenAI-compatible provider (Ollama, Mistral, vLLM, etc.)

OpenAIWireEndpoint <: ServiceEndpoint

Abstract supertype for backends that speak the OpenAI-compatible chat wire. Subtypes inherit the OpenAI Chat Completions request/response encoding (encode_request/decode_response) and the default SSE stream handling (handle_sse_event!) for free, so a new OpenAI-compatible provider defines only get_url and auth_header (plus, optionally, the _api_base_url pattern that routes the Responses/agentic surface).

Backends that speak a genuinely different wire (Anthropic's Messages API, Gemini's native generateContent) subtype ServiceEndpoint directly and additionally implement encode_request, decode_response, and handle_sse_event!. A bare ServiceEndpoint subtype that omits those methods fails with a MethodError at call time rather than silently emitting OpenAI-shaped requests to a foreign API.

Built-in OpenAI-wire subtypes: OPENAIServiceEndpoint, AZUREServiceEndpoint, GEMINIOpenAIServiceEndpoint, GenericOpenAIEndpoint, DeepSeekEndpoint.

OpenAI API service endpoint (default). Requires OPENAI_API_KEY env variable.

Azure OpenAI Service endpoint. Requires AZURE_OPENAI_BASE_URL, AZURE_OPENAI_API_KEY, and AZURE_OPENAI_API_VERSION env variables.

Native Google Gemini generateContent API (x-goog-api-key; model in URL). Requires GEMINI_API_KEY.

Google Gemini endpoint (OpenAI-compatible). Requires GEMINI_API_KEY env variable.

Anthropic (Claude) native Messages API endpoint. Requires the ANTHROPIC_API_KEY env variable. Native wire format (content blocks, top-level system, user/assistant roles) — NOT OpenAI-compatible.

GenericOpenAIEndpoint <: OpenAIWireEndpoint

Configurable endpoint for any OpenAI-compatible API provider. Supports Chat Completions, Embeddings, and (where the provider implements it) the Responses API.

Fields

• base_url::String: Base URL without trailing slash (e.g., "http://localhost:11434")
• api_key::String: API key for Bearer auth. Use "" for local servers with no auth.

Example

# Ollama (local)
chat = Chat(service=GenericOpenAIEndpoint("http://localhost:11434", ""), model="llama3.1")

# Mistral
chat = Chat(service=GenericOpenAIEndpoint("https://api.mistral.ai", ENV["MISTRAL_API_KEY"]),
            model="mistral-large-latest")

ServiceEndpointSpec

Type alias accepting both marker types (OPENAIServiceEndpoint) and instances (GenericOpenAIEndpoint(...)). Used as the type of service fields.

OllamaEndpoint(; base_url="http://localhost:11434") -> GenericOpenAIEndpoint

Pre-configured endpoint for Ollama local server.

MistralEndpoint(; api_key=ENV["MISTRAL_API_KEY"]) -> GenericOpenAIEndpoint

Pre-configured endpoint for Mistral AI API.

DeepSeekEndpoint <: OpenAIWireEndpoint

Pre-configured endpoint for DeepSeek API. Supports chat completions, tool calling, FIM completion, and prefix completion.

FIM and prefix completion use the beta base URL (https://api.deepseek.com/beta).

add_azure_deploy_name!(model::String, deploy_name::String)

Register an Azure OpenAI deployment for a given model name.

Example

add_azure_deploy_name!("gpt-5.2", "my-gpt52-deployment")