Human conversation operates on an intuitive, lightning-fast cadence. Sociolinguistic research demonstrates that natural pauses between human speakers average just 200–300 milliseconds. If an AI voice agent takes 1.5 seconds to reply, the illusion breaks: users talk over the bot, conversations stall, and conversion drops to zero.

To power Closer—our voice-native AI sales executive—AxonFlow AI engineered a sub-250ms voice loop utilizing the Agora Conversational AI engine, low-latency WebSocket audio chunking, and bidirectional Voice Activity Detection (VAD).

1. The Four Latency Sinks in Voice AI

Achieving conversational speed requires shaving milliseconds across every leg of the round-trip pipeline:

  • Audio Transport (WebRTC / Agora RTC): Standard HTTP POST uploads add 300ms overhead. We utilize UDP-based Agora RTC channels delivering raw 16kHz PCM audio in 20ms frames.
  • Streaming Speech-to-Text (STT): Waiting for the user to finish an entire paragraph before transcribing destroys latency. Closer utilizes streaming Deepgram Nova-2 with endpointing detection.
  • First Token Time to Generation (TTFT): The LLM must output its first text token within 80ms of audio completion.
  • Streaming Text-to-Speech (TTS): ElevenLabs and Cartesia audio buffers stream the moment the first sentence clause is completed, playing audio while the LLM generates subsequent phrases.

2. Full-Duplex Interruption & Turn-Taking Architecture

Real prospects interrupt. If a customer says "Wait, how much does it cost?", the bot must immediately cease speaking, discard unplayed audio buffers, and pivot:

[Customer Speaks into Mic]
            │
     (20ms PCM Frames via UDP)
            │
            ▼
 [Client-side WebAssembly Silero VAD]
            │
     (Speech Detected) ───► [INSTANT BARGE-IN SIGNAL] ───► Cancel In-Flight TTS
            │
            ▼
 [Streaming Transcriber (Deepgram)]
            │
    (Partial Transcripts)
            │
            ▼
 [Groq / Gemini 2.0 Flash Stream]
            │
     (Sentence Slices)
            │
            ▼
 [Cartesia Sonic Low-Latency TTS]
            │
    (Audio Stream to Speaker)

3. WebRTC Audio Streaming Client Implementation

Below is a production TypeScript snippet configuring Agora RTC low-latency audio transmission with active noise suppression and acoustic echo cancellation:

import AgoraRTC, { IAgoraRTCClient, IMicrophoneAudioTrack } from "agora-rtc-sdk-ng";

export class LowLatencyVoiceSession {
  private client: IAgoraRTCClient;
  private localAudioTrack: IMicrophoneAudioTrack | null = null;

  constructor(appId: string) {
    this.client = AgoraRTC.createClient({ mode: "rtc", codec: "vp8" });
  }

  async start(channelName: string, token: string) {
    await this.client.join(appId, channelName, token, null);
    
    // Capture microphone with AEC and ANS enabled
    this.localAudioTrack = await AgoraRTC.createMicrophoneAudioTrack({
      AEC: true,  // Acoustic Echo Cancellation
      ANS: true,  // Automatic Noise Suppression
      AGC: true,  // Auto Gain Control
      encoderConfig: "speech_low_quality" // Optimized for voice speech, minimal bitrate
    });

    await this.client.publish([this.localAudioTrack]);
  }
}

4. Production Deployment Metrics

  • 240ms Total Glass-to-Ear Latency: Indistinguishable from natural human conversational turn-taking.
  • Multi-Lingual Fluency: Real-time dialect adaptation across English, Spanish, Hindi, and Arabic.
  • 3.4x Qualification Rate: Inbound leads engaged in voice dialogues converted into scheduled demos at 340% higher rates than static forms.
← Back to Engineering Blog Schedule Architecture Review →