Agent that joins a session, connects to AI providers through swappable plugins, and responds in real time. The framework handles call lifecycle, audio/video routing, turn-taking, and deployment. You focus on instructions, provider choices and connecting to your existing knowledge bases.
What can you build?
Teams ship voice support bots, video coaches that watch a camera, phone agents with knowledge bases, and multimodal assistants that see and hear. Common starting points:- Simple voice agent: STT, LLM, and TTS in a custom pipeline
- Phone support with RAG: inbound calls backed by your docs
- Golf coach: YOLO pose detection plus realtime voice feedback
How it fits together
A client app (web, iOS, Android, or local camera) connects through an edge transport to your agent process. The agent calls AI plugins (LLM, STT, TTS, vision), keeps chat memory for conversation context, and connects to knowledge and tools (RAG, function calling, MCP). Edge transport (pick one, swap in one line):
Your users join the same session as the agent via the transport’s client SDK (or the Quickstart browser demo while prototyping). The agent runs server-side in Python.
AI plugins: realtime speech models, or separate STT + LLM + TTS, plus YOLO/VLM processors and avatars. See Integrations.
Chat memory: with Stream Video, transcripts and context persist to Stream Chat automatically. Use in-memory storage for local dev.
Knowledge and tools: TurboPuffer, Gemini FileSearch,
@llm.register_function, and MCP servers.
Getting started
Run your first agent
Follow the Quickstart to scaffold and talk to an agent in your browser (~5 min).
Extend voice or video
Read Voice Agents for custom pipelines and function calling, or Video Agents for VLMs and YOLO processors.
Try an example
Clone a recipe from Examples: phone, RAG, golf coach, and more.
Deploy
Ship with Deploy to production: Docker, Kubernetes, and metrics.