Benchmarking LLMs for Voice Agent Use Cases

Industry leaders from Daily, Ultravox, and Coval discuss the biggest challenges in voice AI development: creating benchmarks that accurately measure LLM performance for real-world voice agents. In this conversation, we explore: - Why instruction following is the hardest benchmark to create for voice AI - The intelligence vs. latency trade-off facing production voice agents - Why year-old models (GPT-4o, Gemini 1.5 Flash) still dominate despite newer, smarter options. - How open-weight models are closing the gap with proprietary models - Ultravox's breakthrough in speech-to-speech model performance - The difference between foundation model evals and task-specific testing - What's still missing from voice AI benchmarks (backchanneling, prosody, naturalness) Featured speakers: Kwin Kramer, Co-founder of Daily (infrastructure & Pipecat framework) https://www.linkedin.com/in/kwkramer/ https://x.com/kwindla https://daily.co Zach Koch, CEO of Ultravox AI (real-time speech models) https://www.linkedin.com/in/zachkoch/ https://x.com/zachk https://www.ultravox.ai/ @UltravoxAI Brooke Hopkins, Founder of Coval (simulation & evaluation for voice agents) https://www.linkedin.com/in/bnhop/ https://x.com/bnicholehopkins https://www.coval.dev/ @coval_dev 🔗 Read the full benchmark results and methodology: https://www.daily.co/blog/benchmarking-llms-for-voice-agent-use-cases 🔗 Benchmark source code (open source): https://github.com/kwindla/aiewf-eval #voiceai #llm #benchmarking #aiagents #opensource #ultravox #voiceagents #pipecat