← Back to Blog

When you're building voice AI, latency isn't just a technical metric—it's the difference between a natural conversation and an awkward one. Every millisecond of delay makes the experience feel less human.

At MATIS, we've obsessed over latency from day one. Here's how we got our end-to-end response time under 500ms, and what we learned along the way.

The Anatomy of Voice AI Latency

Before we could optimize, we had to understand where time was being spent. A typical voice AI response involves multiple steps:

1. Audio capture and streaming — Getting the audio from the user's phone to our servers
2. Speech-to-text (STT) — Converting audio to text
3. Natural language understanding — Figuring out intent and extracting entities
4. Response generation — Creating the appropriate response
5. Text-to-speech (TTS) — Converting text back to audio
6. Audio delivery — Streaming the response back to the user

In our early versions, this pipeline took 1.5-2 seconds. That's an eternity in conversation time.

Optimization 1: Edge Computing

The biggest wins came from moving computation closer to users. We deployed inference servers in 12 regions globally, ensuring that most users are within 50ms of a processing node.

This alone cut our network latency by 60-70% for users outside North America.

Optimization 2: Streaming Everything

Traditional voice AI waits for the user to finish speaking, then processes, then responds. We stream at every stage:

• STT begins transcribing while the user is still speaking
• NLU starts processing partial transcripts
• Response generation begins before the final transcript is ready
• TTS streams audio back before the full response is generated

This overlapping approach means we often start speaking within 300ms of the user finishing their sentence.

Optimization 3: Speculative Processing

Here's where it gets interesting. Based on conversation context, we predict likely user intents and pre-compute responses before the user even finishes speaking.

When we're right (about 40% of the time), response is nearly instant. When we're wrong, we fall back to the standard pipeline with no penalty.

Optimization 4: Model Selection

Not every query needs our most powerful models. We use a tiered approach:

• Simple queries (greetings, yes/no questions) → lightweight models
• Standard queries → balanced models
• Complex queries → full-power models

A classifier routes each query to the appropriate tier in under 10ms.

The Results

After six months of optimization work, our numbers looked like this:

• P50 latency: 380ms
• P95 latency: 520ms
• P99 latency: 750ms

More importantly, user satisfaction scores increased by 34%. Conversations felt natural. Customers stopped noticing they were talking to AI.

The best technology is invisible. When latency disappears, the conversation flows.

What's Next

We're not done. Our next target is sub-300ms P50 latency, which requires innovations in model architecture that we're actively researching. We're also exploring on-device processing for the first stage of the pipeline, which could shave off another 50-100ms.

The pursuit of lower latency never ends—but that's what makes this work exciting.