Grok Voice Agent: How to Build a Voice Agent with Grok Voice 3.0

A Grok voice agent is a program that talks with people in real time using xAI's Grok Voice 3.0 model: it hears the caller, reasons, calls your tools or searches the web, and answers in one of the Grok voices, with sub-second latency and the ability to be interrupted. xAI runs its own support lines this way (Tesla and Starlink, about 20,000 calls a day, roughly 70% resolved without a human), and the same API is open to anyone with a key.

This guide covers the pieces you need and the order to put them together. It is written from xAI's public documentation as of 16 October 2026; the official voice docs are the source of truth for request shapes.

What you are building

Every Grok voice agent is a Realtime session: a WebSocket connection to xAI carrying audio both ways. You configure the session with a model, a voice, instructions, and optionally tools. The model handles turn-taking, interruptions and speech on its own; your code only has to feed audio in, play audio out, and answer tool calls.

Three ways to connect a user to that session:

1. Server to server: your backend opens the WebSocket with your API key. Good for phone bots and server-driven apps.
2. Browser or mobile client: the client connects directly, but with a short-lived ephemeral client secret instead of your API key.
3. Phone: a provisioned or bring-your-own number routes calls into the agent over SIP.

Step 1: pick the model and reasoning mode

Use "model": "grok-voice-latest" to always get the newest version (today that is grok-voice-3.0; grok-voice-2.0 still works but is deprecated). Set "reasoning": { "effort": "high" } (the default) for multi-step tasks, or "none" for simple flows where latency matters more than deliberation.

Step 2: pick a voice

The voice is any built-in id (ara, eve, leo, rex, sal, altair, …, zenith) or a custom voice id. Since the agent and the text to speech endpoint share voices, the fastest way to choose is to type a few lines of your agent's script into the free text to speech tool and listen to candidates. Our suggested starting points for support agents are on each voice page.

Step 3 (browser clients): mint an ephemeral client secret

Never ship your API key to a browser. From your server:

curl -s https://api.x.ai/v1/realtime/client_secrets \
  -H "Authorization: Bearer $XAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "expires_after": { "seconds": 300 },
    "session": { "model": "grok-voice-latest", "reasoning": { "effort": "high" } }
  }'

The response is { "value": "xai-realtime-client-secret-…", "expires_at": 1750000000 }. The client uses value as its Bearer token (or in the sec-websocket-protocol header with the xai-client-secret. prefix) to open the WebSocket. Secrets expire after at most 3600 seconds (default 600), and the session config you attach is applied when the connection opens, so the browser cannot change the model or instructions.

Step 4 (phone): provision a number

curl -s https://api.x.ai/v2/phone-numbers \
  -H "Authorization: Bearer $XAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "origin": "xai_provisioned", "name": "Support line", "area_code": "415", "agent_id": "agent_abc123" }'

• origin: "xai_provisioned" gets you a number from xAI (US area code optional).
• origin: "byo_trunk" uses your own number: pass phone_number in E.164 form and, optionally, SIP digest credentials and allowed source CIDRs; xAI returns the sip_host (sip.voice.x.ai) your PBX or carrier should route to.
• Route calls to an agent_id (an agent configured in the console) or to a webhook URL that receives signed realtime.call.incoming events, so your own code can decide how to answer. The webhook signing secret is returned once at creation.

Step 5: give it tools

Tools are what turn a chatty model into an agent that resolves tickets. In the session config you declare functions (look up an order, reschedule a delivery, hand off to a human); when the model decides to call one, your code runs it and sends the result back as text. Each text input into the session is billed at $0.004, on top of the per-minute rate. Grok Voice 3.0's score on τ-Voice (71.6%, a benchmark for tool-using voice agents) is xAI's own measure of how reliably it does this.

Pricing summary

Try the calculator with your expected minutes.

No-code option

xAI's console has an agent builder (console.x.ai → Voice → Agents) where you set the voice, instructions and tools in a form and get an agent_id to attach to a phone number. It is the quickest way to get a working line before writing any WebSocket code.

Enterprise notes

From xAI's Voice API page: SOC 2 Type II, HIPAA eligibility with a BAA, GDPR data-processing agreements and EU residency options, zero data retention, custom concurrency limits, SAML SSO and audit logs. Contact xAI sales for those; this site is independent and cannot arrange them.