Ultravox

Executive Summary

Ultravox has two related products: an open model that understands spoken input, and a hosted Realtime service for building voice agents. The service connects the model to speech output, tools, browser/mobile clients and telephony. Its documented architecture feeds audio directly into reasoning rather than using a transcript as the model's input.^[1]

That makes it a useful member of the agentic voice APIs comparison, especially for developers who want audio understanding with configurable application behavior. It does not mean the downloadable model is a complete phone agent: the repository explicitly describes text output, with speech generation a separate component.^[2]

Open Model and Hosted Service

The open model uses an audio encoder and a trained adapter to connect speech with a language-model backbone. The v0.7 model card describes GLM-4.6 plus Whisper's encoder, with a frozen language model and trained audio components. It reports vendor-run audio reasoning and recognition evaluations, not a production call-success benchmark.^[3]

The v0.7 configuration references zai-org/GLM-4.6 as its text model.^[4] Consequently, the small parameter count displayed for the uploaded audio component should not be treated as the memory requirement of the complete system. Choose the actual backbone and serving configuration before estimating hardware costs.

The repository's code license is MIT.^[5] The v0.7 model card also labels its release MIT, while other available variants use different underlying models.^[3] In particular, a deployment using Meta's Llama 3.3 must account for that backbone's Community License and conditions; an MIT audio component does not replace them.^[6]

Self-hosting gives a team responsibility for inference, speech output, transport and operations. The hosted service supplies those surrounding facilities as paid API access. This report did not run the weights or verify a minimum GPU configuration.

Speech Output and Turn Taking

Realtime offers built-in voices, voice cloning and external speech providers.^[7] Its external-provider guide distinguishes dedicated streaming integrations from a generic HTTP option. Generic integrations buffer text and estimate transcript timing when precise timing data is unavailable. That can affect the relationship between generated text, audible speech and interruption handling.^[8]

Ultravox exposes separate voice-activity settings for end-of-turn delay, minimum turn length and interruption sensitivity. The guide explains their tradeoffs: making the system react sooner can also make it interrupt a thoughtful pause or react to noise.^[9] These settings are not measurements of end-to-end response latency.

A useful voice test includes quiet speech, short confirmations, overlapping talk and the application's actual names or identifiers. Listen to what the caller heard, rather than judging only the final text transcript.

Tools and Conversation State

These are the three documented implementation paths. HTTP tools work across call media; ordinary client-tool handling is most natural with WebRTC or WebSocket clients.^[10]

Ordinary tool execution pauses the agent's responses until a result arrives. The default timeout is 2.5 seconds, configurable up to 40 seconds. Read-only, side-effect-free tools can be marked precomputable so they run while the agent speaks; interruption can discard the result and leave no invocation in conversation history. A payment or account mutation is therefore a poor candidate for speculative execution.^[11]

Call stages can replace the system prompt, tools, voice and conversation history. Unchanged settings inherit from the previous stage. The underlying model and call medium cannot be changed through that mechanism.^[12] Stages help narrow what a particular part of the conversation can do, but application authorization must still enforce the intended boundary.

Background threads provide another option for slow work. They fork conversation context and communicate through messages; only the main conversation receives or produces audio. Side threads are canceled when the call ends or a stage changes.^[13] A background result that must survive those events needs durable application ownership rather than an assumption that the thread will finish.

Human Handoff Is a Workflow

The SIP transfer guide distinguishes REFER, which removes the agent's SIP stack, from bridging, which keeps infrastructure in the audio path and can continue incurring SIP costs. A failed SIP cold transfer returns the caller to the agent. Warm transfer lets the human hear context and accept or reject the call.^[14]

One important exception is documented explicitly: the built-in cold-transfer tool used with Twilio ends the Ultravox call immediately, whether the transfer succeeds or not.^[14] Test the chosen provider's failure path; “supports transfer” is not enough to establish that an unanswered destination leaves the caller in a recoverable conversation.

Pricing and Capacity

The public page checked September 16, 2026 lists these US-dollar rates:^[15]

The pricing page's annual-rate heading matters: confirm the checkout commitment rather than assuming the displayed Pro figure is a cancel-any-month invoice. Free minutes should also not be assumed to renew monthly without confirmation.

The billing FAQ rounds call time up in six-second increments. It explains that billable duration and recording duration can differ, including processing at the end of a call.^[16] For example, 1,000 billed minutes at $0.05 cost $50 before a subscription, SIP, carrier or external-provider charges. This arithmetic is not the cost of 1,000 completed customer tasks.

Paid plans' lack of a hard call cap is not unlimited reserved capacity. The concurrency guide says infrastructure load can still produce a 429 response and instructs clients to honor Retry-After. That guide also retains references to a Scale tier absent from the checked pricing page; confirm priority guarantees in current account terms instead of inferring them from the older tier description.^[17]

Data Handling and Operations

Ultravox's FAQ says call data stays available until deleted. API deletion removes messages, recordings, stages and summaries while retaining the call ID and billing information. It also explains that visible transcripts are generated separately by ASR for inspection; they are not necessarily the exact input the audio model understood.^[16] Investigate a disputed response using the audio and relevant tool state, not only the transcript.

The privacy policy says voice data and audio inputs are not used to train or fine-tune underlying models. It also describes collection of call and account data and use of service providers.^[18] This is narrower than a promise of zero retention. External TTS integrations introduce their own processing path and provider credentials.^[8]

Webhook consumers should implement the documented timestamp and HMAC-SHA256 checks, including handling multiple signatures during key rotation.^[19] Separately, business endpoints should make repeated requests safe and record the actual result of a state change. An agent's spoken claim is not the transaction record.

A Worked Evaluation

Consider an appointment assistant that can search availability, reserve a slot and transfer a difficult case. A useful evaluation would:

1. Use a read-only availability lookup while the caller speaks, then interrupt with a different date and verify obsolete results are ignored.
2. Require confirmation before reserving. Inject a timeout after the backend commits and verify the application can recover the existing reservation without creating another.
3. Put a slow information lookup on a side thread, change stages and verify the application handles its cancellation.
4. Try both an answered and rejected human handoff using the actual telephony provider.
5. Compare billed duration, recorded audio, transcript and backend actions for the same test calls.

These are proposed acceptance tests, not hands-on results. They expose the costs and failure cases hidden by a successful conversational demo.

Practitioner Evidence and Fit

In a March 30, 2025 GitHub report, rojithaDev described three-to-four-second delays after short confirmations when using neural endpointing, despite preferring it to the previous behavior. The issue remained open when checked.^[20] This is a dated individual experience, not a measured current failure rate or proof that every deployment still behaves that way. It does justify keeping short utterances in an evaluation set.

Ultravox is a candidate when a team values direct audio input, configurable tools and the option to inspect or self-host model components. A team choosing it should budget for application state, provider integration and operational testing. Compare the hosted route with Grok Voice API or Hume EVI, and the self-hosted architecture with PersonaPlex, while keeping their different output and deployment boundaries explicit.

---

Research by Ry Walker Research • methodology