This talk examines how conversational voice agents are built in industry, tracing the evolution from cascaded pipelines (ASR —> LLM —> TTS) to thinker-talker and end-to-end architectures. We discuss key design decisions including audio representations, turn-taking, and full-duplex interaction, and address the practical challenges that separate research from production: training data at scale, evaluation beyond offline metrics, robustness under real-world conditions, echo cancellation, noise handling, and latency budgeting.
Sefik Emre Eskimez is a Research Engineer at Sesame and a former Principal Researcher at Microsoft, with over 10 years of experience in deep learning and machine learning. His research focuses on speech and audio processing, including text-to-speech synthesis, speech enhancement, generative models, and conversational voice AI. At Microsoft, he led the development and deployment of Voice Isolation (Personalized Speech Enhancement) for Microsoft Teams, a real-time system that extracts a target speaker's voice from noisy audio and is now used by millions of users worldwide. He also contributed to multiple state-of-the-art TTS systems, including E2 TTS and SpeechX. At Sesame, he works on next-generation conversational voice AI, including the Conversational Speech Model (CSM) and turn-taking research aimed at making voice interactions more natural and human-like. Sefik Emre Eskimez holds a Ph.D. in Electrical and Computer Engineering from the University of Rochester, where his dissertation focused on generating emotionally expressive talking faces from speech. He has published many papers in top venues including ICASSP, Interspeech, and IEEE/ACM Transactions on Audio, Speech, and Language Processing, and holds multiple U.S. patents.