Skip to main content

View Phone & RAG Example on GitHub

Check out the complete Phone & RAG example in our GitHub repository
Builds on the Twilio Phone Agent example — complete ngrok and Twilio webhook setup there first. This tutorial adds RAG so your phone agent can answer questions from a knowledge base using Gemini FileSearch, TurboPuffer, or Qdrant.
Vision Agents uses Stream Video for real-time WebRTC transport by default. External WebRTC transports are supported as well. Most AI providers offer free tiers to get started.

​
What You Will Build

  • Answer inbound phone calls with an AI agent that retrieves knowledge in real time
  • Make outbound calls programmatically for tasks like booking reservations
  • Swap RAG backends with a single environment variable
  • Handle bidirectional audio via Twilio Media Streams over WebSocket

​
Prerequisites

Complete the Twilio Phone Agent setup first (.env, ngrok, Twilio webhook). Add these variables depending on your RAG backend:

​
Add RAG to your phone agent

Choose a RAG backend

Set RAG_BACKEND when running the inbound example:See the RAG guide for backend details.

Add knowledge files

The example ships with sample docs in examples/03_phone_and_rag_example/knowledge/ covering Stream Chat, Video, and Feeds APIs. Add your own .md files to this directory or replace them with product documentation.

Run inbound with RAG

From the example directory:
Swap RAG_BACKEND=turbopuffer or RAG_BACKEND=qdrant to try other backends.

Test with a phone call

Call your Twilio number and ask questions the knowledge base can answer, for example:
  • “What APIs does Stream offer for chat?”
  • “Tell me about Stream Video features.”
  • “How does Stream Feeds work?”
The agent retrieves relevant chunks before responding.

​
How RAG connects to the phone agent

The inbound script (inbound_phone_and_rag_example.py) wires RAG into the agent based on RAG_BACKEND:
  • Gemini — uses gemini.GeminiFilesearchRAG with File Search tools on the LLM
  • TurboPuffer / Qdrant — registers a search function the LLM calls during conversation
Phone plumbing is unchanged from the Twilio Phone Agent: Twilio webhooks start the media stream, attach_phone_to_call bridges audio to Stream, and the agent runs inside the Stream call. For outbound calls with RAG, extend outbound_phone_example.py using the same RAG initialization pattern from the inbound script.

​
Next Steps

RAG Guide

Learn more about RAG backends and knowledge retrieval

Twilio Integration

Plugin API reference and components

HTTP Server Guide

Deploy your phone bot with the built-in HTTP server

Deploying Overview

Move from local dev to production