Skip to main content
Build voice agents with swappable providers, phone integration, function calling, and production deployment with built-in metrics. If you completed the Quickstart, you already have a working voice agent. This page explains the two ways to build one, and when to move beyond the default setup. There are two common approaches. The first uses a realtime model, a single API that handles listening and speaking end-to-end, such as OpenAI Realtime, Gemini Live, or Qwen OMNI. The second is a custom pipeline where you pick separate STT, LLM, and TTS providers and wire them together.
ModeBest forYou choose
Realtime ModelsFastest path, lowest latencyOne provider for speech in and out
Custom PipelineFull control over each stageSTT, LLM, and TTS independently

​
Realtime Mode

Realtime models handle the full voice loop (speech-to-text, reasoning, and text-to-speech) over a single WebRTC or WebSocket connection. No separate STT or TTS plugins needed. The Quickstart uses gemini.Realtime(). To swap providers, change one line in agent.py: 
llm=openai.Realtime()   # OpenAI
llm=gemini.Realtime()   # Gemini
llm=qwen.Realtime()     # Qwen OMNI
See the integration pages for provider-specific setup: OpenAI Realtime, Gemini Live, Qwen OMNI.

​
Custom Pipeline Mode

Use a custom pipeline when you want to mix providers. For example, Deepgram for transcription, Gemini for reasoning, and Inworld for voice output. You also get control over turn detection (when the agent starts and stops listening).

Copy this prompt into Claude Code, Cursor, Windsurf, or any coding agent to scaffold a custom pipeline.

Open in Cursor
Or follow the steps below manually.

​
1. Add plugins

The Quickstart scaffolds with Gemini Realtime. Add STT and TTS plugins on top: 
uv add "vision-agents[deepgram,inworld]"
Add these keys to your .env: 
DEEPGRAM_API_KEY=your_deepgram_api_key
INWORLD_API_KEY=your_inworld_api_key

​
2. Wire up the pipeline

Replace gemini.Realtime() with separate stt, llm, and tts components in agent.py: 
from dotenv import load_dotenv

from vision_agents.core import Agent, AgentLauncher, User, Runner
from vision_agents.plugins import getstream, gemini, deepgram, inworld

load_dotenv()


async def create_agent(**kwargs) -> Agent:
    return Agent(
        edge=getstream.Edge(),
        agent_user=User(name="Assistant", id="agent"),
        instructions="You're a helpful voice assistant.",
        llm=gemini.LLM(),
        stt=deepgram.STT(eager_turn_detection=True),
        tts=inworld.TTS(),
    )


async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        await agent.simple_response("Greet the user")
        await agent.finish()


if __name__ == "__main__":
    Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()
Audio flows: user speaks → STT transcribes → LLM generates a response → TTS speaks it back.

​
3. Mix and match providers

ComponentOptions
LLMGemini, OpenAI, OpenRouter, Anthropic, Grok, HuggingFace
STTDeepgram, ElevenLabs, Fast-Whisper, Fish, Wizper
TTSInworld, ElevenLabs, Cartesia, Deepgram, Grok, Pocket, AWS Polly
Turn DetectionDeepgram (built-in), ElevenLabs (built-in), Smart Turn, Vogent

​
Function Calling & MCP

Custom pipelines use an LLM component (gemini.LLM(), not gemini.Realtime()), which supports tool registration: 
@llm.register_function(description="Get weather for a location")
async def get_weather(location: str) -> dict:
    return {"temperature": "22C", "condition": "Sunny"}
Functions are automatically converted to the right format for each LLM provider. For MCP servers, external tools, and advanced patterns, see the Function Calling & MCP guide.

​
What’s Next

Phone Integration

Connect agents to inbound and outbound phone calls

RAG Support

Add knowledge bases with Gemini FileSearch or TurboPuffer

Docker Deployment

Docker setup and environment configuration

Built-in HTTP Server

Console mode and HTTP server for running agents

​
Examples