> ## Documentation Index
> Fetch the complete documentation index at: https://visionagents.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Voice Agents

> Build voice agents with realtime models or custom STT/LLM/TTS pipelines. Phone integration, function calling, and production deployment.

Build voice agents with swappable providers, [phone integration](https://visionagents.ai/guides/calling.md), [function calling](https://visionagents.ai/guides/mcp-tool-calling.md), and [production deployment](https://visionagents.ai/guides/deployment.md) with [built-in metrics](https://visionagents.ai/core/telemetry.md).

If you completed the [Quickstart](https://visionagents.ai/introduction/quickstart.md), you already have a working voice agent. This page explains the two ways to build one, and when to move beyond the default setup.

There are two common approaches. The first uses a **realtime model**, a single API that handles listening and speaking end-to-end, such as [OpenAI Realtime](https://visionagents.ai/integrations/realtime/openai.md), [Gemini Live](https://visionagents.ai/integrations/realtime/gemini.md), or [Qwen OMNI](https://visionagents.ai/integrations/realtime/qwen.md). The second is a **[custom pipeline](#custom-pipeline-mode)** where you pick separate STT, LLM, and TTS providers and wire them together.

| Mode | Best for | You choose |
| - | - | - |
| **Realtime Models** | Fastest path, lowest latency | One provider for speech in and out |
| **Custom Pipeline** | Full control over each stage | STT, LLM, and TTS independently |

## Realtime Mode

Realtime models handle the full voice loop (speech-to-text, reasoning, and text-to-speech) over a single WebRTC or WebSocket connection. No separate STT or TTS plugins needed.

The [Quickstart](https://visionagents.ai/introduction/quickstart.md) uses `gemini.Realtime()`. To swap providers, change one line in `agent.py`:

```python theme={null}
llm=openai.Realtime()   # OpenAI
llm=gemini.Realtime()   # Gemini
llm=qwen.Realtime()     # Qwen OMNI
```

See the integration pages for provider-specific setup: [OpenAI Realtime](https://visionagents.ai/integrations/realtime/openai.md), [Gemini Live](https://visionagents.ai/integrations/realtime/gemini.md), [Qwen OMNI](https://visionagents.ai/integrations/realtime/qwen.md).

## Custom Pipeline Mode

Use a custom pipeline when you want to mix providers. For example, Deepgram for transcription, Gemini for reasoning, and Inworld for voice output. You also get control over [turn detection](https://visionagents.ai/integrations/turn-detection/smart-turn.md) (when the agent starts and stops listening).

Copy this prompt into Claude Code, Cursor, Windsurf, or any coding agent to scaffold a custom pipeline.

Create a Python project for a Vision Agents custom voice pipeline using uv and Python 3.12.

&#x20; Steps:
&#x20; 1\. Scaffold: uvx vision-agents init my-voice-agent && cd my-voice-agent
&#x20; 2\. Add plugins: uv add "vision-agents\[deepgram,inworld]"
&#x20; 3\. Copy .env.example to .env and fill in: STREAM\_API\_KEY, STREAM\_API\_SECRET (from getstream.io), GOOGLE\_API\_KEY (from aistudio.google.com), DEEPGRAM\_API\_KEY (from deepgram.com), INWORLD\_API\_KEY (from studio.inworld.ai)
&#x20; 4\. Edit agent.py (created by init) to use a custom pipeline:

&#x20; from dotenv import load\_dotenv
&#x20; from vision\_agents.core import Agent, AgentLauncher, User, Runner
&#x20; from vision\_agents.plugins import getstream, gemini, deepgram, inworld

&#x20; load\_dotenv()

&#x20; def setup\_llm():
&#x20;   llm = gemini.LLM()

&#x20;   @llm.register\_function(description="Get current weather for a location")
&#x20;   async def get\_weather(location: str) -> dict:
&#x20;       return \{"temperature": "22C", "condition": "Sunny", "location": location}

&#x20;   return llm

&#x20; async def create\_agent(\*\*kwargs) -> Agent:
&#x20;   return Agent(
&#x20;       edge=getstream.Edge(),
&#x20;       agent\_user=User(name="Assistant", id="agent"),
&#x20;       instructions="You're a helpful voice assistant with access to tools. Be concise.",
&#x20;       llm=setup\_llm(),
&#x20;       stt=deepgram.STT(eager\_turn\_detection=True),
&#x20;       tts=inworld.TTS(),
&#x20;   )

&#x20; async def join\_call(agent: Agent, call\_type: str, call\_id: str, \*\*kwargs) -> None:
&#x20;   call = await agent.create\_call(call\_type, call\_id)
&#x20;   async with agent.join(call):
&#x20;       await agent.simple\_response("Greet the user and let them know you can check the weather")
&#x20;       await agent.finish()

&#x20; if \_\_name\_\_ == "\_\_main\_\_":
&#x20;   Runner(AgentLauncher(create\_agent=create\_agent, join\_call=join\_call)).cli()

&#x20; 5\. Run with: uv run agent.py run

&#x20; Reference docs: [https://visionagents.ai](https://visionagents.ai)
&#x20; MCP server: [https://visionagents.ai/mcp](https://visionagents.ai/mcp)
&#x20; Skill.md: [https://visionagents.ai/skill.md](https://visionagents.ai/skill.md)

Or follow the steps below manually.

### 1. Add plugins

The Quickstart scaffolds with Gemini Realtime. Add STT and TTS plugins on top:

```bash theme={null}
uv add "vision-agents[deepgram,inworld]"
```

Add these keys to your `.env`:

```bash theme={null}
DEEPGRAM_API_KEY=your_deepgram_api_key
INWORLD_API_KEY=your_inworld_api_key
```

### 2. Wire up the pipeline

Replace `gemini.Realtime()` with separate `stt`, `llm`, and `tts` components in `agent.py`:

```python theme={null}
from dotenv import load_dotenv

from vision_agents.core import Agent, AgentLauncher, User, Runner
from vision_agents.plugins import getstream, gemini, deepgram, inworld

load_dotenv()


async def create_agent(**kwargs) -> Agent:
    return Agent(
        edge=getstream.Edge(),
        agent_user=User(name="Assistant", id="agent"),
        instructions="You're a helpful voice assistant.",
        llm=gemini.LLM(),
        stt=deepgram.STT(eager_turn_detection=True),
        tts=inworld.TTS(),
    )


async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        await agent.simple_response("Greet the user")
        await agent.finish()


if __name__ == "__main__":
    Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()
```

Audio flows: user speaks → **STT** transcribes → **LLM** generates a response → **TTS** speaks it back.

### 3. Mix and match providers

| Component | Options |
| - | - |
| **LLM** | Gemini, OpenAI, OpenRouter, Anthropic, Grok, HuggingFace |
| **STT** | Deepgram, ElevenLabs, Fast-Whisper, Fish, Wizper |
| **TTS** | Inworld, ElevenLabs, Cartesia, Deepgram, Grok, Pocket, AWS Polly |
| **Turn Detection** | Deepgram (built-in), ElevenLabs (built-in), Smart Turn, Vogent |

## Function Calling & MCP

Custom pipelines use an LLM component (`gemini.LLM()`, not `gemini.Realtime()`), which supports tool registration:

```python theme={null}
@llm.register_function(description="Get weather for a location")
async def get_weather(location: str) -> dict:
    return {"temperature": "22C", "condition": "Sunny"}
```

Functions are automatically converted to the right format for each LLM provider. For MCP servers, external tools, and advanced patterns, see the [Function Calling & MCP guide](https://visionagents.ai/guides/mcp-tool-calling.md).

## What's Next

> [card] [Phone Integration](https://visionagents.ai/guides/calling.md)
>
> Connect agents to inbound and outbound phone calls

> [card] [RAG Support](https://visionagents.ai/guides/rag.md)
>
> Add knowledge bases with Gemini FileSearch or TurboPuffer

> [card] [Docker Deployment](https://visionagents.ai/guides/deployment.md)
>
> Docker setup and environment configuration

> [card] [Built-in HTTP Server](https://visionagents.ai/guides/http-server.md)
>
> Console mode and HTTP server for running agents

## Examples

* [Simple Agent](https://visionagents.ai/examples/simple-agent.md): Minimal voice agent with Deepgram STT + Inworld TTS + Gemini LLM
* [Twilio Phone Agent](https://visionagents.ai/examples/twilio-phone-agent.md): Step-by-step inbound and outbound calls
* [Phone & RAG](https://visionagents.ai/examples/phone-and-rag.md): Add knowledge retrieval to phone agents


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.