Function calling, latency, echo, interruption, cost, and the three levels a voice product can exist at. Most projects fail for the same handful of reasons, and none of them are model quality. This is the reference for avoiding them — written from measured numbers, including the ones that came out badly.
SpotBot — a workout coach. Same product at all three levels, so you can see what each one actually buys. The code is illustrative; the shapes are real.
What each level is, what it costs you to build, and the one thing it cannot do. Ninety seconds, then the levers worth knowing about before you pick.
If you read nothing else. Deep dives follow.
Push to talk. The user finishes, you transcribe, you answer. Turns strictly alternate, which makes almost every hard problem disappear.
A persistent socket, audio streaming both ways, and either party able to cut the other off. This is a different product from level 1, not a faster one.
Independent lanes running at once — speaking, thinking, acting, logging — spun up for a task and torn down after. The AI stops being a participant and starts running the session.
Before you write a line, name the one thing that must work. Everything at a given level is in service of that one thing, and a feature from a higher level bolted onto a lower one always breaks in the same place.
| Level | The core | You have it when | Symptom of skipping it |
|---|---|---|---|
| 1 · Cascaded | audio in works | you can capture, endpoint and transcribe a sentence reliably, every time | it "works on my machine," fails on a phone in a noisy room |
| 2 · Live | turn-taking works | either party can interrupt and the other yields cleanly | talking over each other; the model finishes a sentence nobody wanted |
| 3 · Superharness | channels reconcile | the model can speak, think, act and log at once without contradicting itself | it says one thing and does another; tool results arrive after they mattered |
Everything that separates a voice product that works from one that demos. Each links to its deep dive. If you are auditing an existing build, this is the checklist.
Each level end to end — how it works, what breaks, and the problems that only appear once you are at that level. Colour-coded throughout: cyan, violet, amber.
Push-to-talk, walkie-talkie turns. The oldest shape and still the correct one for most products. This is where CoachPal and PurlPal started, and it is a perfectly good place to stay. → REST tier reference
You do not send audio continuously. You accumulate frames while speech is present, and you commit the buffer when silence has lasted long enough to mean "they're done." Get the silence threshold wrong in either direction and the product feels broken — too short and you cut people off mid-thought, too long and it feels dead.
Use a neural VAD, not an energy-based one. The old GMM detectors
(webrtcvad and friends) trigger on typing, coughs, doors and background
conversation. Silero is small enough to run on anything, ships as a single
file, and is the sane default in 2026. Starting here saves you a week of tuning noise gates.
# spotbot/listen.py — accumulate while speaking, commit on silence
import torch
vad, _ = torch.hub.load('snakers4/silero-vad', 'silero_vad') # neural, ~1MB
MIN_SPEECH_MS = 300 # reject coughs, door slams, a stray "uh"
def listen_once(stream, hush):
"""hush = how long a pause must mean 'done'. NOT a constant you ship."""
buf, silent_ms, speech_ms = [], 0, 0
for frame in stream: # 512 samples @ 16kHz = 32ms
speaking = vad(torch.from_numpy(frame), 16000).item() > 0.5
if speaking:
buf.append(frame); speech_ms += 32; silent_ms = 0
elif buf:
buf.append(frame); silent_ms += 32 # keep the trailing silence
if silent_ms >= hush.current(): # adaptive, not hardcoded
return b"".join(buf) if speech_ms >= MIN_SPEECH_MS else None
return None
// spotbot/listen.ts — accumulate while speaking, commit on silence
import { SileroVad } from "@ricky0123/vad-web"; // neural, runs in the browser
const MIN_SPEECH_MS = 300; // reject coughs, door slams, a stray "uh"
export async function listenOnce(stream: AsyncIterable<Float32Array>, hush: Hush) {
const buf: Float32Array[] = [];
let silentMs = 0, speechMs = 0;
for await (const frame of stream) { // 512 samples @ 16kHz = 32ms
const speaking = (await vad.process(frame)).isSpeech > 0.5;
if (speaking) { buf.push(frame); speechMs += 32; silentMs = 0; }
else if (buf.length) {
buf.push(frame); silentMs += 32; // keep the trailing silence
if (silentMs >= hush.current()) // adaptive, not hardcoded
return speechMs >= MIN_SPEECH_MS ? concat(buf) : null;
}
}
return null;
}
// spotbot/listen.dart — accumulate while speaking, commit on silence
import 'package:onnxruntime/onnxruntime.dart';
const minSpeechMs = 300; // reject coughs, door slams, a stray "uh"
Future<List<int>?> listenOnce(Stream<Float32List> stream, Hush hush) async {
final buf = <Float32List>[];
var silentMs = 0, speechMs = 0;
await for (final frame in stream) { // 512 samples @ 16kHz = 32ms
final speaking = (await vad.run(frame)) > 0.5;
if (speaking) { buf.add(frame); speechMs += 32; silentMs = 0; }
else if (buf.isNotEmpty) {
buf.add(frame); silentMs += 32; // keep the trailing silence
if (silentMs >= hush.current) { // adaptive, not hardcoded
return speechMs >= minSpeechMs ? pcm16(buf) : null;
}
}
}
return null;
}
// spotbot/listen.rs — accumulate while speaking, commit on silence
use voice_activity_detector::VoiceActivityDetector; // silero, ort-backed
const MIN_SPEECH_MS: u32 = 300; // reject coughs, door slams, a stray "uh"
pub fn listen_once(
stream: &mut impl Iterator<Item = Vec<f32>>,
vad: &mut VoiceActivityDetector,
hush: &Hush,
) -> Option<Vec<i16>> {
let (mut buf, mut silent_ms, mut speech_ms) = (Vec::new(), 0u32, 0u32);
for frame in stream { // 512 samples @ 16kHz = 32ms
let speaking = vad.predict(frame.clone()) > 0.5;
if speaking {
buf.push(frame); speech_ms += 32; silent_ms = 0;
} else if !buf.is_empty() {
buf.push(frame); silent_ms += 32; // keep the trailing silence
if silent_ms >= hush.current() { // adaptive, not hardcoded
return (speech_ms >= MIN_SPEECH_MS).then(|| to_pcm16(&buf));
}
}
}
None
}
// spotbot/listen.go — accumulate while speaking, commit on silence
package spotbot
const minSpeechMs = 300 // reject coughs, door slams, a stray "uh"
func ListenOnce(stream <-chan []float32, vad *Silero, hush *Hush) []int16 {
var buf [][]float32
silentMs, speechMs := 0, 0
for frame := range stream { // 512 samples @ 16kHz = 32ms
speaking := vad.Predict(frame) > 0.5
if speaking {
buf = append(buf, frame); speechMs += 32; silentMs = 0
} else if len(buf) > 0 {
buf = append(buf, frame); silentMs += 32 // keep trailing silence
if silentMs >= hush.Current() { // adaptive, not hardcoded
if speechMs >= minSpeechMs { return toPCM16(buf) }
return nil
}
}
}
return nil
}
The payoff for cascading is that you never open a WebSocket. Audio goes in as part of an
ordinary request, alongside text and images. Any REST client can do it, it is trivially
debuggable with curl, and it retries cleanly.
# spotbot/turn.py — audio in, coaching out, one round trip
import base64, requests
def coach_turn(audio_bytes, set_log):
r = requests.post(
"https://api.hawktalk.ai/v1/chat/completions",
headers={"Authorization": f"Bearer {KEY}"},
json={
"model": "auto", # route per utterance
"messages": [{"role": "user", "content": [
{"type": "input_audio", "input_audio": {
"format": "pcm16", "data": base64.b64encode(audio_bytes).decode()}},
{"type": "text", "text": f"Sets so far: {set_log}. Coach the next one."},
]}],
"modalities": ["text", "audio"], # speech back in the same call
"audio": {"voice": "am_puck"},
}, timeout=30)
d = r.json()["choices"][0]["message"]
return d["content"], base64.b64decode(d["audio"]["data"])
// spotbot/turn.ts — audio in, coaching out, one round trip
export async function coachTurn(audio: Uint8Array, setLog: string) {
const r = await fetch("https://api.hawktalk.ai/v1/chat/completions", {
method: "POST",
headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
body: JSON.stringify({
model: "auto", // route per utterance
messages: [{ role: "user", content: [
{ type: "input_audio", input_audio: { format: "pcm16", data: b64(audio) } },
{ type: "text", text: `Sets so far: ${setLog}. Coach the next one.` },
]}],
modalities: ["text", "audio"], // speech back in the same call
audio: { voice: "am_puck" },
}),
});
const m = (await r.json()).choices[0].message;
return { text: m.content, audio: b64d(m.audio.data) };
}
// spotbot/turn.dart — audio in, coaching out, one round trip
import 'dart:convert';
import 'package:http/http.dart' as http;
Future<(String, List<int>)> coachTurn(List<int> audio, String setLog) async {
final r = await http.post(
Uri.parse('https://api.hawktalk.ai/v1/chat/completions'),
headers: {'Authorization': 'Bearer $KEY', 'Content-Type': 'application/json'},
body: jsonEncode({
'model': 'auto', // route per utterance
'messages': [{'role': 'user', 'content': [
{'type': 'input_audio',
'input_audio': {'format': 'pcm16', 'data': base64Encode(audio)}},
{'type': 'text', 'text': 'Sets so far: $setLog. Coach the next one.'},
]}],
'modalities': ['text', 'audio'], // speech back in the same call
'audio': {'voice': 'am_puck'},
}),
);
final m = jsonDecode(r.body)['choices'][0]['message'];
return (m['content'] as String, base64Decode(m['audio']['data']));
}
// spotbot/turn.rs — audio in, coaching out, one round trip
use base64::{Engine, engine::general_purpose::STANDARD as B64};
use serde_json::json;
pub async fn coach_turn(audio: &[u8], set_log: &str)
-> anyhow::Result<(String, Vec<u8>)>
{
let body = json!({
"model": "auto", // route per utterance
"messages": [{ "role": "user", "content": [
{ "type": "input_audio",
"input_audio": { "format": "pcm16", "data": B64.encode(audio) } },
{ "type": "text", "text": format!("Sets so far: {set_log}. Coach the next one.") },
]}],
"modalities": ["text", "audio"], // speech back in the same call
"audio": { "voice": "am_puck" },
});
let v: serde_json::Value = reqwest::Client::new()
.post("https://api.hawktalk.ai/v1/chat/completions")
.bearer_auth(std::env::var("HAWKTALK_API_KEY")?)
.json(&body).send().await?.json().await?;
let m = &v["choices"][0]["message"];
Ok((m["content"].as_str().unwrap_or_default().to_string(),
B64.decode(m["audio"]["data"].as_str().unwrap_or_default())?))
}
// spotbot/turn.go — audio in, coaching out, one round trip
package spotbot
import ("bytes"; "encoding/base64"; "encoding/json"; "net/http")
func CoachTurn(audio []byte, setLog string) (string, []byte, error) {
body, _ := json.Marshal(map[string]any{
"model": "auto", // route per utterance
"messages": []any{map[string]any{"role": "user", "content": []any{
map[string]any{"type": "input_audio", "input_audio": map[string]any{
"format": "pcm16", "data": base64.StdEncoding.EncodeToString(audio)}},
map[string]any{"type": "text", "text": "Sets so far: " + setLog + ". Coach the next one."},
}}},
"modalities": []string{"text", "audio"}, // speech back in the same call
"audio": map[string]any{"voice": "am_puck"},
})
req, _ := http.NewRequest("POST",
"https://api.hawktalk.ai/v1/chat/completions", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil { return "", nil, err }
defer res.Body.Close()
var out struct{ Choices []struct{ Message struct {
Content string `json:"content"`
Audio struct{ Data string `json:"data"` } `json:"audio"`
} `json:"message"` } `json:"choices"` }
json.NewDecoder(res.Body).Decode(&out)
m := out.Choices[0].Message
raw, err := base64.StdEncoding.DecodeString(m.Audio.Data)
return m.Content, raw, err
}
The single most common way level 1 projects waste a month.
When your AI speaks through a speaker, the microphone hears it. Left alone, the model transcribes its own voice and answers itself. There are two ways out, and the industry default is the worse one.
Muting the mic during playback is what most products do. It works, it is three lines, and it is a trap: a muted mic collects nothing. You lose the barge-in, you lose the affect signal, you lose the utterance the user started before you finished talking, and you lose every second of audio you would otherwise have had to learn from. You have solved echo by going deaf.
Ducking is the right answer: subtract the speaker's output from the mic signal and leave the mic open. You know exactly what you sent to the speaker, so you know exactly what to remove. The mic never stops collecting, the user can interrupt at any moment, and the audio you keep is clean. Our own implementation measures 11.7 dB ERLE with 0.93 correlation preserved on the user's voice during double-talk — the user survives the subtraction, which is the whole point.
The three lines below are what the lazy option looks like, so you can recognise it in a codebase — not a recommendation:
# the lazy option: mute the mic. three lines, zero DSP, total data loss
def say(audio):
mic.pause() # you are now deaf for the duration
speaker.play(audio) # blocking
mic.resume() # small guard delay for the room reverb tail
// the lazy option: mute the mic. three lines, zero DSP, total data loss
async function say(audio: AudioBuffer) {
mic.pause(); // you are now deaf for the duration
await speaker.play(audio);
setTimeout(() => mic.resume(), 120); // guard delay for the reverb tail
}
// the lazy option: mute the mic. three lines, zero DSP, total data loss
Future<void> say(Uint8List audio) async {
await mic.pause(); // you are now deaf for the duration
await speaker.play(audio);
await Future.delayed(const Duration(milliseconds: 120)); // reverb tail
await mic.resume();
}
// the lazy option: mute the mic. three lines, zero DSP, total data loss
async fn say(audio: &[u8]) {
mic.pause(); // you are now deaf for the duration
speaker.play(audio).await;
tokio::time::sleep(Duration::from_millis(120)).await; // reverb tail
mic.resume();
}
// the lazy option: mute the mic. three lines, zero DSP, total data loss
func Say(audio []byte) {
mic.Pause() // you are now deaf for the duration
speaker.Play(audio) // blocking
time.Sleep(120 * time.Millisecond) // guard delay for the reverb tail
mic.Resume()
}
The most common question at this point, and the answer is worth understanding rather than memorising.
A VAD answers "is this speech?" It cannot answer "is this speech from the user?" Your bot's TTS coming back through the microphone is speech — clean, well-formed, exactly what a VAD is built to detect. The VAD is working perfectly and giving you the wrong answer.
So the instinct is right: gate the VAD on what you know you are playing. That works, up to a point, and here is the ladder from cheapest to most correct:
| Approach | Cost | Where it breaks |
|---|---|---|
| Mute the mic while speaking | three lines | no barge-in, and no data. The wall you hit first. |
| Playback-gated VAD — raise the threshold while audio is out | an hour | a loud room or a loud speaker still trips it; and a quiet user gets ignored |
| Speaker verification — accept only the enrolled voice | a small model | rejects your own TTS and the TV cleanly, but the audio is still mixed |
| Ducking — subtract the known playback | real DSP work, or one platform flag | needs a sample-aligned loopback reference, so it lives on the client |
The case that kills every gate-only approach is double-talk — the user speaking while the bot is speaking, which is the entire point of barge-in. Gating trades a false positive for a false negative: instead of the bot hearing itself, the bot ignores the user. That is the worse failure, because the user knows they were ignored.
And the subtraction has to run on the client. It works by removing a reference signal — the exact audio you sent to the speaker — from the microphone input, sample-aligned. That alignment only exists on the device that owns both the speaker and the mic. Ship the mic audio to a server first and network jitter has already destroyed the timing relationship you needed. If you take one thing from this section: ducking is a client concern, always.
audioprocessing module ships AEC, noise suppression and auto gain, and browsers
give it to you for free via getUserMedia({ echoCancellation: true }). Native apps
get the platform one (VoiceProcessingIO on iOS, AcousticEchoCanceler
on Android). Writing your own is a project; the platform's is a flag. Reach for a custom one
only when you have measured the platform's and found it wanting.A persistent socket, audio streaming both directions, server-side endpointing,
and interruption. Gemini Live, GPT Realtime, and wss /v1/realtime all live here. →
WebSocket tier reference
The core is communication back and forth. Not speed — turn-taking. The model can begin answering before you have finished forming your thought, and you can cut it off without the session falling apart. That is a different product from level 1, not a faster one.
# spotbot/live.py — streaming both ways, with barge-in
import json, websockets
async def live(mic, speaker):
async with websockets.connect(
"wss://api.hawktalk.ai/v1/realtime?model=auto",
extra_headers={"Authorization": f"Bearer {KEY}"}) as ws:
async def send_audio():
async for frame in mic: # continuous — server endpoints
await ws.send(json.dumps({
"type": "input_audio_buffer.append", "audio": b64(frame)}))
asyncio.create_task(send_audio())
async for raw in ws:
m = json.loads(raw)
if m["type"] == "response.audio.delta":
speaker.enqueue(b64d(m["delta"]))
elif m["type"] == "input_audio_buffer.speech_started":
speaker.flush() # THE line. drop queued audio now.
await ws.send(json.dumps({"type": "response.cancel"}))
elif m["type"] == "response.text.delta":
caption(m["delta"])
// spotbot/live.ts — streaming both ways, with barge-in
const ws = new WebSocket("wss://api.hawktalk.ai/v1/realtime?model=auto",
["bearer", KEY]);
mic.onframe = f => ws.send(JSON.stringify({ // continuous — server endpoints
type: "input_audio_buffer.append", audio: b64(f),
}));
ws.onmessage = e => {
const m = JSON.parse(e.data);
switch (m.type) {
case "response.audio.delta":
speaker.enqueue(b64d(m.delta)); break;
case "input_audio_buffer.speech_started":
speaker.flush(); // THE line. drop queued audio now.
ws.send(JSON.stringify({ type: "response.cancel" }));
break;
case "response.text.delta":
caption(m.delta); break;
}
};
// spotbot/live.dart — streaming both ways, with barge-in
final ws = WebSocketChannel.connect(
Uri.parse('wss://api.hawktalk.ai/v1/realtime?model=auto'),
protocols: ['bearer', key],
);
mic.frames.listen((f) => ws.sink.add(jsonEncode({ // continuous
'type': 'input_audio_buffer.append', 'audio': base64Encode(f),
})));
ws.stream.listen((raw) {
final m = jsonDecode(raw);
switch (m['type']) {
case 'response.audio.delta':
speaker.enqueue(base64Decode(m['delta']));
case 'input_audio_buffer.speech_started':
speaker.flush(); // THE line. drop queued audio now.
ws.sink.add(jsonEncode({'type': 'response.cancel'}));
case 'response.text.delta':
caption(m['delta']);
}
});
// spotbot/live.rs — streaming both ways, with barge-in
use futures::{SinkExt, StreamExt};
use tokio_tungstenite::tungstenite::Message;
let (mut ws, _) = connect_async(req).await?; // ?model=auto
let (mut tx, mut rx) = ws.split();
tokio::spawn(async move { // continuous mic pump
while let Some(f) = mic.next().await {
let msg = json!({"type": "input_audio_buffer.append", "audio": b64(&f)});
tx.send(Message::Text(msg.to_string())).await.ok();
}
});
while let Some(Ok(Message::Text(raw))) = rx.next().await {
let m: serde_json::Value = serde_json::from_str(&raw)?;
match m["type"].as_str() {
Some("response.audio.delta") => speaker.enqueue(b64d(&m["delta"])?),
Some("input_audio_buffer.speech_started") => {
speaker.flush(); // THE line. drop queued audio.
cancel_tx.send(json!({"type": "response.cancel"})).await?;
}
Some("response.text.delta") => caption(&m["delta"]),
_ => {}
}
}
// spotbot/live.go — streaming both ways, with barge-in
conn, _, err := websocket.DefaultDialer.Dial(
"wss://api.hawktalk.ai/v1/realtime?model=auto", hdr)
if err != nil { return err }
defer conn.Close()
go func() { // continuous mic pump
for f := range mic {
conn.WriteJSON(map[string]any{
"type": "input_audio_buffer.append", "audio": b64(f)})
}
}()
for {
var m map[string]any
if err := conn.ReadJSON(&m); err != nil { return err }
switch m["type"] {
case "response.audio.delta":
speaker.Enqueue(b64d(m["delta"].(string)))
case "input_audio_buffer.speech_started":
speaker.Flush() // THE line. drop queued audio now.
conn.WriteJSON(map[string]any{"type": "response.cancel"})
case "response.text.delta":
Caption(m["delta"].(string))
}
}
speaker.flush() on speech_started. Cancelling generation server-side
is easy; throwing away audio you already buffered locally is what makes
interruption feel instant. If you only cancel and let the buffer drain, the AI keeps talking
for a second after being interrupted and the illusion dies. Our barge-in measures a
~0ms server decision with 12/12 hermetic checks green — and the client-side
flush is the other half of that number.Now ducking is mandatory, because you are listening while speaking by definition — and per the previous section, the subtraction runs on the client or not at all.
The transport question everyone hits here and almost nobody explains. Both are used in production by major providers, so this is a real choice, not a right answer.
| WebSocket | WebRTC | |
|---|---|---|
| Under the hood | TCP | UDP |
| Packet loss | Head-of-line blocking — one lost packet stalls every packet behind it, so a 2% loss rate becomes audible stutter | Loss is concealed and playback continues |
| Audio format | whatever you encode yourself | Opus, negotiated, with packet-loss concealment built in |
| Ducking / echo | you arrange it | Free — AEC, noise suppression and gain control come with the stack |
| Jitter buffer | you write it | included |
| Effort to first working call | an afternoon | a week, plus signalling and TURN servers |
Raw 16kHz PCM16 mono is 256 kbps. That is fine on wifi and rude on a mobile data plan. Every code sample here streams raw PCM because it is legible, not because it is right.
Encode to Opus before it leaves the device — 16–32 kbps for speech, roughly a tenth the bytes, with no meaningful quality cost for this purpose and packet-loss concealment as a bonus. It is one library call and it is the difference between an app people use on the train and one they only use at home.
One model, one stream of consciousness, one thing at a time. It cannot think privately while speaking. It cannot run a slow tool without either stalling the conversation or lying about what it knows. It cannot revise a plan in the background. Everything it does must fit in the single lane it talks through — and the moment your product needs two things happening at once, you have hit the ceiling.
"Barge-in" is one word covering four distinct events that need four distinct responses. Products that treat them as one are the ones that either talk over you or stop every time a door closes.
| Event | Example | Correct response | Getting it wrong |
|---|---|---|---|
| Backchannel | "mhm", "right", "yeah", a laugh | Keep talking. This is the listener signalling attention, not requesting the floor. | Stopping makes the AI seem skittish and breaks its own sentence for no reason. |
| True interruption | "wait—", "no, I meant", a question | Yield immediately. Flush the local buffer, cancel generation, listen. | Talking over the user is the single most disliked behaviour in voice AI. |
| Environmental | the TV, a bystander, a colleague across the room | Ignore entirely. Never yields, never enters the transcript. | The agent answers the television. Common and very obvious in a demo. |
| Self-echo | the AI's own voice via the speaker | Never reaches the decision. Cancelled in the signal, not judged. | A permanent self-interruption loop — it hears itself, stops, hears itself stop. |
| Signal | Separates | Cost |
|---|---|---|
| Duration | backchannel from interruption — under ~400ms with no continuation is almost always a backchannel | free, and gets you a long way |
| Speaker verification | the enrolled user from the TV, a bystander, and the AI's own voice | a small model, and the highest-value one here |
| Echo cancellation | self-echo from everything else, in the signal before any decision is made | platform AEC, one flag |
| Semantic content | "mhm" from "wait, no" — needs the words | a transcription hop, so it arrives late; use it to correct, not to gate |
HawkTalkLive. The AI stops being a participant in a conversation and starts being the thing running the session. → HawkTalkLive reference
The core is many channels at once, reconciled. Not one stream carrying everything in sequence — independent lanes that run concurrently and are made consistent with each other. The harness is ephemeral: spun up for a task, given its channels, torn down when done.
| Channel | Carries | SpotBot uses it for |
|---|---|---|
| audio | speech in and out, visemes, VAD | coaching you through the rep |
| text | token deltas | the on-screen caption and set counter |
| tools | function-call arguments | logging the set, adjusting tomorrow's plan |
| thinking | private reasoning, never spoken | deciding your form is degrading before it says so |
| async | slow work that lands late | a 40s deep analysis of the whole session |
# spotbot/harness.py — one session, five lanes, running at once
harness = hawktalk.live.session(
model="auto",
channels=["audio", "text", "tools", "thinking", "async"],
ephemeral=True,
)
@harness.on("thinking") # private. the user never hears this.
def _(ev):
if ev.confidence < 0.6:
harness.audio.hedge() # soften delivery, don't stop talking
@harness.on("tools") # fires WHILE it is still speaking
def _(call):
if call.name == "log_set":
db.write(call.args)
harness.tools.result(call.id, {"ok": True})
@harness.on("async") # lands 40s later, mid-conversation
def _(report):
harness.audio.interject(f"Your left side is lagging — {report.detail}")
harness.start() # audio flows immediately; the rest arrive when they arrive
// spotbot/harness.ts — one session, five lanes, running at once
const harness = hawktalk.live.session({
model: "auto",
channels: ["audio", "text", "tools", "thinking", "async"],
ephemeral: true,
});
harness.on("thinking", ev => { // private. the user never hears this.
if (ev.confidence < 0.6) harness.audio.hedge();
});
harness.on("tools", async call => { // fires WHILE it is still speaking
if (call.name === "log_set") {
await db.write(call.args);
harness.tools.result(call.id, { ok: true });
}
});
harness.on("async", report => { // lands 40s later, mid-conversation
harness.audio.interject(`Your left side is lagging — ${report.detail}`);
});
harness.start(); // audio flows immediately; the rest arrive when they arrive
// spotbot/harness.dart — one session, five lanes, running at once
final harness = HawkTalkLive.session(
model: 'auto',
channels: const ['audio', 'text', 'tools', 'thinking', 'async'],
ephemeral: true,
);
harness.on('thinking', (ev) { // private. the user never hears this.
if (ev.confidence < 0.6) harness.audio.hedge();
});
harness.on('tools', (call) async { // fires WHILE it is still speaking
if (call.name == 'log_set') {
await db.write(call.args);
harness.tools.result(call.id, {'ok': true});
}
});
harness.on('async', (report) { // lands 40s later, mid-conversation
harness.audio.interject('Your left side is lagging — ${report.detail}');
});
await harness.start(); // audio flows immediately; the rest arrive later
// spotbot/harness.rs — one session, five lanes, running at once
let harness = hawktalk::live::session(SessionCfg {
model: "auto",
channels: &["audio", "text", "tools", "thinking", "async"],
ephemeral: true,
}).await?;
harness.on_thinking(|ev| { // private. the user never hears this.
if ev.confidence < 0.6 { harness.audio().hedge(); }
});
harness.on_tools(|call| async move { // fires WHILE it is still speaking
if call.name == "log_set" {
db.write(&call.args).await?;
harness.tools().result(&call.id, json!({"ok": true})).await?;
}
Ok(())
});
harness.on_async(|report| { // lands 40s later, mid-conversation
harness.audio().interject(&format!("Your left side is lagging — {}", report.detail));
});
harness.start().await?; // audio flows immediately; the rest arrive later
// spotbot/harness.go — one session, five lanes, running at once
harness, err := live.Session(live.Config{
Model: "auto",
Channels: []string{"audio", "text", "tools", "thinking", "async"},
Ephemeral: true,
})
if err != nil { return err }
harness.OnThinking(func(ev live.Thinking) { // private. user never hears this.
if ev.Confidence < 0.6 { harness.Audio().Hedge() }
})
harness.OnTools(func(call live.ToolCall) { // fires WHILE it is still speaking
if call.Name == "log_set" {
db.Write(call.Args)
harness.Tools().Result(call.ID, map[string]any{"ok": true})
}
})
harness.OnAsync(func(r live.Report) { // lands 40s later, mid-conversation
harness.Audio().Interject("Your left side is lagging — " + r.Detail)
})
harness.Start() // audio flows immediately; the rest arrive when they arrive
Function calling, the latency budget, cost, evaluation, and the graveyard of things that were measured and did not pay. Cross-cutting — these bite at every level.
Everything else on this page — the voice, the latency, the channels — is delivery. Function calling is the part where the model stops describing the world and starts changing it. It is the building block of agentic work, and every agent you will ever ship is a stack of tool calls that either held or did not.
Which is why it is first. A voice agent that sounds wonderful and calls the wrong function is worse than one that sounds robotic and calls the right one — the first will be trusted with something that matters.
The pattern is everywhere: prompt the model to emit something like
<tool>get_weather(city="Denver")</tool>, then pull it back out with a
pattern match. It demos beautifully. It fails in production for reasons that have nothing to
do with model quality:
| Failure | Why it happens |
|---|---|
| A stray quote or newline eats the call | The model was never constrained to your format. It was asked. Asking is not a contract. |
| Markdown fences appear from nowhere | Instruction-tuned models like formatting. Your regex does not. |
| Parallel calls collapse into one | Free text has no structure for "two of these." You end up inventing a delimiter and then defending it. |
| It gets worse with unusual phrasing | Failure correlates with the user being interesting, which is exactly when the call mattered. |
| You cannot test it | You are testing your pattern against outputs you have already seen — not against the space of outputs that exist. |
The coach's whole job is log_set(exercise, reps, weight). Get the weight
wrong and the training plan is wrong — silently, for weeks. That call is the
product; the voice around it is how the product is delivered. Which is why this
section comes before the levels: SpotBot with perfect audio and a broken
log_set is worthless, and SpotBot with robotic audio and a correct one is a
training log.
In text, a broken tool call is survivable. The user sees something odd, scrolls back, retries. You log the raw output and fix it tomorrow. The interaction has a record and the record is repairable.
Voice has none of that. It is ephemeral by construction, and that changes the cost of every failure:
| In text | In voice |
|---|---|
| The user sees the malformed output and knows something broke | The user hears silence, or a confident wrong answer, with no indication anything failed |
| Scrollback is the log — you can both point at it | The user's memory is the only record, and it is not a log |
| A silent retry costs nothing | A retry is a 600ms hole in a conversation with no latency budget left to spend |
| "Try rephrasing that" is a mild ask | "Try rephrasing that" is the product telling the user it is broken |
| The failure is visible and therefore fixable | The failure is invisible and therefore permanent |
A voice turn happens once and is gone. There is no scrollback to inspect, no message to edit, no place to surface an error that does not also destroy the conversation. The contract has to hold on the first attempt, every attempt, because there is no second one that the user will not feel.
# WRONG — a wish and a pattern match
m = re.search(r'<tool>(\w+)\((.*?)\)</tool>', reply)
if m:
name, args = m.group(1), parse_args_somehow(m.group(2)) # good luck
# RIGHT — a typed object the API is contractually obliged to return
for call in resp.choices[0].message.tool_calls: # no parsing. no salvage path.
result = TOOLS[call.function.name](**json.loads(call.function.arguments))
session.tools.result(call.id, result)
// WRONG — a wish and a pattern match
const m = /<tool>(\w+)\((.*?)\)<\/tool>/.exec(reply);
if (m) { const [, name, raw] = m; parseArgsSomehow(raw); } // good luck
// RIGHT — a typed object the API is contractually obliged to return
for (const call of resp.choices[0].message.tool_calls ?? []) {
const result = await TOOLS[call.function.name](JSON.parse(call.function.arguments));
session.tools.result(call.id, result);
}
// WRONG — a wish and a pattern match
final m = RegExp(r'<tool>(\w+)\((.*?)\)</tool>').firstMatch(reply);
if (m != null) parseArgsSomehow(m.group(2)!); // good luck
// RIGHT — a typed object the API is contractually obliged to return
for (final call in resp.choices.first.message.toolCalls ?? []) {
final result = await tools[call.function.name]!(jsonDecode(call.function.arguments));
session.tools.result(call.id, result);
}
// WRONG — a wish and a pattern match
let re = Regex::new(r"<tool>(\w+)\((.*?)\)</tool>")?;
if let Some(c) = re.captures(&reply) { parse_args_somehow(&c[2]); } // good luck
// RIGHT — a typed object the API is contractually obliged to return
for call in resp.choices[0].message.tool_calls.iter().flatten() {
let args: serde_json::Value = serde_json::from_str(&call.function.arguments)?;
let result = dispatch(&call.function.name, args).await?;
session.tools_result(&call.id, result).await?;
}
// WRONG — a wish and a pattern match
m := regexp.MustCompile(`<tool>(\w+)\((.*?)\)</tool>`).FindStringSubmatch(reply)
if m != nil { parseArgsSomehow(m[2]) } // good luck
// RIGHT — a typed object the API is contractually obliged to return
for _, call := range resp.Choices[0].Message.ToolCalls {
var args map[string]any
json.Unmarshal([]byte(call.Function.Arguments), &args)
result, err := Tools[call.Function.Name](args)
if err != nil { return err }
session.ToolResult(call.ID, result)
}
| Layer | What it guarantees |
|---|---|
1 · Native tool_calls |
The transport returns a typed object with a name, typed arguments and an id. There is nothing to parse, so there is nothing to parse wrong. |
| 2 · Grammar-constrained decode | A GBNF grammar makes an invalid token unreachable at each slot. The model cannot emit malformed JSON because malformed JSON is not in the sampling space. |
| 3 · Negative-case training | The model must decline to call anything when no tool applies. This is the failure that costs real money, and it is the one prompting never fixes. |
The gap to frontier here is a format gap, not a brains gap. That is worth internalising, because it means reliability is an engineering choice rather than a model-budget problem. Constrain the decode, take the typed object, and the whole class of failure disappears — on a small model, on-device, at 40ms.
You can usually diagnose a voice product in thirty seconds of using it, without seeing a line of its code. These are the tells, what each one actually means, and what the fix is.
| The smell | What it actually is | The fix |
|---|---|---|
| Long silence, then a whole paragraph at once | A thinking model sitting on the voice chain. The user is waiting through chain-of-thought that was never meant to be heard. | Deliberation comes off the main chain entirely — a separate call, or the async channel. The voice path stays on the fast tier. |
| Longer answers take longer to start | Not streaming. The whole response is generated, then synthesized, then played. | Stream tokens into TTS. Time-to-first-audio must be independent of answer length. |
| The pause before every reply is identical | A hardcoded VAD silence timeout doing the job of endpointing. | Adaptive endpointing. A fixed 800ms wall is 800ms of dead air on every single turn. |
| Fine alone, sluggish with users on it | Batch-1 serving with a queue behind it. | Continuous batching. Concurrency should cost throughput, not first-audio. |
| Turn one is snappy, turn nine is not | Re-prefilling the whole conversation every turn. | KV reuse and prefix caching. Turn latency should be flat across a session. |
| You interrupt and it keeps talking for a beat | Server-side cancel with no client-side buffer flush. | Drop queued audio locally the instant speech is detected. Cancelling is the easy half. |
| Tool calls create dead air | The conversation halts while a function runs. | Speak through it, or speculate through it. Silence is the one thing a voice product cannot afford. |
| Everything routes to the biggest model | No router. You are paying frontier latency to say "mhm." | Route per utterance. Most turns do not need the expensive rung. |
| The vendor quotes total turn time | They have not optimised first-audio, so they are benchmarking the number that flatters them. | Ask for TTFA p50 and p95 under load. That is the number the user feels. |
You finish rep eight and ask "was that too fast?". At the floor you get an answer as you are racking the bar — useless, the moment passed. Coaching has a deadline the way a conversation does not. The latency budget is not about feeling snappy; it is about whether the answer arrives while it can still change what you do.
| Target | What it takes | |
|---|---|---|
| The floor | 1.5–3s felt reply | What most shipped products do. Cascaded, unoptimised, thinking on-chain. |
| Silver — local | ~40ms TTFT, sub-200ms felt reply | OuroLive on-device. Model resident on the NPU, no network in the loop, no prefill to pay. This is the bar, and it is measured, not projected. |
| Gold — cloud that feels local | ~50ms to first audio | Pre-cached openings, speculative drafting before end-of-turn, and regrounding at commit. See below — this is the interesting one. |
Silver is the honest bar because it removes every excuse. A local model on an NPU answers in 40ms because nothing is in the way — no network, no queue, no cold prefill. Once you have felt that, a 1.2-second cloud reply stops being "the cost of doing business" and starts being what it is: unoptimised.
Three mechanisms, and the third is the one that keeps it honest.
Pre-cached beginnings. Openings are predictable and prosodically neutral — "Right—", "So on that,", "Let me check." Render a library of them ahead of time and start playing one immediately at end-of-turn. Audio begins in ~50ms because nothing is being generated; you are buying 300–600ms of cover for the real answer forming behind it. Keep enough variety that it never sounds canned, and keep them neutral enough to lead into any continuation.
Speculative drafting before the reply is called for. Do not wait for the endpoint to start working. Draft the likely reply continuously against the partial transcript while the user is still talking, and warm retrieval and tools alongside it. Most turns are predictable enough that by the time they stop, the answer is largely formed. You are spending idle compute during a window that was previously dead.
Regrounding at commit — the part that stops this being a liar. Drafting early risks answering the question you predicted instead of the one that was asked. So the draft is never authoritative. At end-of-turn you verify it against the completed transcript and against retrieved truth, then accept, patch, or discard it. Speculation buys latency; regrounding buys correctness. Ship one without the other and you have built something fast and confidently wrong, which is worse than slow.
Where the milliseconds actually go, and which ones you can get back. All figures measured on one serving configuration; treat the shape as transferable and the absolute numbers as ours, not yours.
| Stage | Measured | Notes |
|---|---|---|
| STT | 436ms warm TTFT · 375ms p95 | audio tower folded into the model, resident on the accelerator. Flat from 1 to 16 concurrent seats — that flatness matters more than the number. |
| LLM | 536ms TTFT · 29–70 tok/s | fast enough that it is not the bottleneck, which surprises people. |
| TTS — fast tier | RTF 0.175 | Matcha on CPU. Comfortably realtime with headroom. |
| TTS — balanced | RTF 1.31 | Kokoro. Real-time-ish, better voices, visemes for lip-sync. |
| TTS — expressive | RTF 0.52× realtime | Orpheus-class. Not shippable at this speed; see the graveyard. |
| Felt first audio | ~590ms p50 | at a 12-seat load, inside the 450–600ms cascade floor the good products sit in. |
| On-device, whole turn | ~40ms TTFT · sub-200ms felt | no network, no queue, no cold prefill. The bar that removes every excuse. |
An earlier configuration measured 669.9ms STT + 656.2ms chat + 1449.5ms TTS = 2775.6ms per turn, with TTS at 52% of the budget. Two changes moved it: folding STT into the model on the accelerator removed the separate transcription hop entirely (42.6× faster than host-CPU decode), and picking the right TTS tier rather than the best-sounding one took synthesis off the critical path.
| Lever | Typical gain | Cost to you |
|---|---|---|
| Fold STT into the model | removes a whole hop | needs a model with an audio tower |
| Pick the right TTS tier | can be most of the budget | the expressive voice is usually not worth the seconds |
| Stream tokens into TTS | makes TTFA independent of length | a day of plumbing |
| Pre-cached opening | audio starts at ~50ms | needs regrounding or it lies — see section on standards |
| Route small turns to a small model | TTFT scales with model size | a router |
| A bigger, better LLM | usually nothing | it was not the bottleneck |
The same diagnostic as section 02, on the other axis. These are the tells that a voice product is paying several times what it needs to — and unlike latency, nobody notices until the invoice arrives.
| The smell | What it actually is | The fix |
|---|---|---|
| One model id hardcoded everywhere | You are paying frontier rates to say "mhm." Most turns in a real conversation are acknowledgements and short answers. | Route per utterance. Let a small model take the turns that do not need a big one. |
| Tool schemas in every request | A realistic tool set is 700–1600 tokens. At forty turns a session, re-sent every turn, that is most of your token spend buying nothing. | Declare tools once per session. This is usually the single biggest line item. |
| A realtime socket for a walkie-talkie | A session bills for its duration; a request bills for the turn. Dictation and ordering do not need interruption. | Use the request API. This one is structural — you cannot optimise the wrong rail. |
| The system prompt has the user's name in it | A per-user prefix cannot be cache-shared, so every request pays full prefill. | Static prefix, user specifics retrieved into the turn. |
| Overnight jobs on the interactive tier | You are paying for latency nobody is waiting on. | Mark it batch. Interruptible work belongs on interruptible capacity. |
| The whole document in every prompt | Context is billed per turn, forever, whether or not it was relevant. | Embed once, retrieve the relevant few hundred tokens per turn. |
| Sessions provisioned at the highest tier they might need | Paying peak for the mean. One hard question in a ten-minute call should not price the whole call. | Start cheap, escalate for the turns that need it, drop back down. |
| A reasoning model answering the user directly | Both smells at once — the user waits, and you pay for tokens they never hear. | Fast tier answers; deliberation runs on another clock and is folded in. |
| Nobody can say which tier answered | You have no per-turn cost attribution, so every optimisation is a guess. | Log the trace id on every request before changing anything else. |
A session is forty turns and most of them are "good", "next", "one more", "how many left". Pin a large model and you pay frontier rates for every one of those. Re-send an eight-tool schema each turn and you have billed 56,000 tokens of definitions the model already had. Neither changes a single word SpotBot says.
There is no standard for this and everyone is guessing. Word error rate is necessary and nowhere near sufficient — it measures one stage of a pipeline whose failures mostly happen between stages. Here is a metric set worth arguing with.
| Metric | What it catches | How to measure |
|---|---|---|
| Cut-off rate | endpointing firing while the user is still talking — the complaint people phrase as "it doesn't listen" | % of turns where the transcript ends mid-clause or the user immediately repeats themselves |
| Dead-air | endpointing firing too late; the silence before every reply | median ms from true end-of-speech to first audio out |
| TTFA p50 / p95 under load | the number the user feels, and whether it survives concurrency | at your target seat count, never on an idle box. p95 is the honest one. |
| False barge-in rate | stopping for the TV, a cough, or its own echo | interruptions triggered per minute in a room with background audio and nobody addressing the agent |
| Missed barge-in rate | ignoring a genuine interruption — worse than a false one, because the user knows | % of scripted interruptions where the agent kept speaking past 300ms |
| Tool: valid format | malformed calls | must be 100%. Anything less means no contract exists. |
| Tool: right tool | choosing correctly among candidates | % correct on an ambiguous-by-design set, not on easy cases |
| Tool: exact args | the values that carry the consequences | tool and arguments both correct |
| Negative rate | inventing a call when no tool applies | The one that costs real money in production. Feed it turns where the answer is "just talk" and count the calls it makes anyway. |
| Task completion | whether the user got what they came for | end-to-end, judged, on scripted scenarios — the only metric that is actually the product |
| Coherence over N turns | drift, repetition, echo loops | long scripted sessions, judged. Ours ran 32/32 coherent on one tier and 26/32 on another — the difference was invisible in single-turn tests. |
| WER | transcription only | report it, do not lead with it. A perfect transcript with a wrong tool call is a failed turn. |
Things that sounded right, got built, got measured, and did not work. Every one of these cost real time. Published because a catalogue of only wins is a brochure, and because negatives are the part nobody else will tell you.
| The idea | What happened | What to do instead |
|---|---|---|
| Per-token retrieval blending mix retrieved neighbours
into the logits at each step |
Fired 40 times, corrected zero. Inert exactly when the model was confidently wrong, which is the case you needed it for. | Retrieved-prefix grounding. Same information, measured +40pp, and it works because it changes what the model sees rather than nudging what it says. |
| Speculative drafters on a static accelerator | Two parallel KV caches on a chip with fixed compiled graphs. A memory problem for a gain that was not the bottleneck anyway. | Nothing. The LLM was not the constraint — the voice stages were. |
| Self-speculative early exit | Exit-layer token agreement 0.168 against a draft cost fraction of 0.534. It costs more than it saves. | Drop it. Published because a measured negative is worth as much as a win. |
| Layer-pruned "turbo" models | Faster, and instruction-following destroyed. Fluent, obedient to nothing. | Depth cuts as latency probes, not as shipping models. |
| Quantising a 270M model to 4-bit | Tool matching fell from 25/25 to 4/25. A 270M has no weight redundancy left to give. | fp16 and accept the size. Below it, a deterministic rule parser — never a smaller model. |
| fp8 / int8 to make TTS realtime | fp8 would not compile on the stack at all; int8 gave zero decode speedup — which disproved the bandwidth theory it was based on. | The bind was all-reduce latency across two devices. Fewer devices, not smaller numbers. |
| Emotion tags into an expressive TTS | The "prosody responds" result had been measured on a mock. The real compiled graph ignores prosody fields and breaks on tags. | Ship emotion detection, which works. Emotion voice needs trained latent tokens, not markup. |
| Vocab-trimming the router | Smaller and rejected — trimming remaps token ids and breaks token-space alignment with the tiers it routes to, which is what made handoff free. | Trim the whole family to one shared vocabulary, or none of it. |
| Distilling a mandate router | 83.5% in-eval, 47.3% held out. It memorised the templates. | Disjoint entities are not enough — held-out rows sharing templates with training rows will flatter you. |
| Z-normalising audio for emotion parity | Falsified. Normalising biased every input toward one class. | Raw amplitude with tile-fill. And re-check the "obvious" preprocessing step. |
| The smallest speech-to-text model | Measurably mishears domain-specific phrases — exactly the words your product exists to hear. | Pay the extra memory for the next size up. Fallback only, never default. |
| Level 1 | Level 2 | Level 3 | |
|---|---|---|---|
| Transport | REST | WebSocket | WebSocket, multiplexed |
| Endpointing | client, accumulative VAD | server | server + affect |
| Echo | duck, or mute if prototyping | duck | duck + speaker handling |
| Interruption | none | barge-in | barge-in + interjection |
| Concurrency | one turn | one lane | five lanes |
| Build time | a day | a week | a project |
| Right for | dictation, ordering, logging, forms | tutoring, support, companionship | coaching under load, ops copilots, anything supervising a live process |
POST /v1/generate and
wss /v1/realtime. The level 3 thinking and async
channels are published API design; server support is in progress. We would rather tell you
that than have you find out in production.Everything above describes one session. The reason HawkTalkLive is a platform and not an endpoint is what happens when you have ten thousand of them.
EC2 was never "a server you rent." It was an API for managing compute as a fleet — launch, resize, monitor, place, reclaim. The machine was the commodity; the control plane was the product. Voice AI is at exactly the stage compute was before that idea existed: everyone is selling you one session at a time, and nobody is selling you the thing that runs ten thousand of them.
HawkTalkLive manages both directions. It carries the user-to-model conversation, and because it lives on the cloud side it also manages the fleet of those conversations — placement, tiering, capacity, spend and lifecycle — through the same API.
| EC2 concept | HawkTalkLive equivalent | What it does |
|---|---|---|
| instance | voice session | the unit you provision, meter and reclaim |
| instance type | tier — quick · dank · think | capability class, priced accordingly |
| launch / terminate | session.start / ephemeral teardown | sessions are cattle, not pets |
| resize a running instance | session.update | re-point a live call at a different model without dropping it |
| placement / scheduler | the router, per utterance | picks the cheapest tier that clears the bar |
| autoscaling group | seats per box, replicated data-parallel | 12 seats measured per node, ~144 dense |
| CloudWatch | trace_id on every turn | full audit of tokens, tool calls, channel activity |
| spot vs on-demand | priority: batch | interruptible work at interruptible prices |
| security groups | plan entitlement clamp | a key can never be routed above what it bought |
| AMI | the model registry | same id resolves to whatever silicon the node has |
session.update
re-points a live socket at a different brain mid-conversation without dropping the call. That
is the elastic in elastic compute, and applied to voice it means you can start every
call on the cheap tier and only pay for the expensive one during the ninety seconds that
actually needed it.| You want to | The platform does |
|---|---|
| cap what a customer can spend | entitlement clamp applied after routing, never a suggestion to the model |
| survive a traffic spike | bias the router down a rung under load — "good enough now" beats "better in four seconds" |
| know why a call cost what it cost | per-turn trace: which tier answered, which tool ran, which channel was open |
| run 10k sessions on fixed silicon | continuous batching plus KV paging, seats as the unit of capacity planning |
| degrade instead of failing | the ladder has rungs; a saturated top tier steps down rather than erroring |
| reclaim a wedged session | ephemeral by construction — sessions have a lifecycle, not just a socket |
None of that is available to you if your voice AI is an endpoint you post to. It is only available if the thing running your sessions is designed to run a fleet of them, and exposes that fleet to you. That is the difference between renting a machine and having a control plane.
trace_id on every response, entitlement clamping, and measured capacity at 12
seats per node. The fleet-level API surface — enumerate sessions, set
autoscaling policy, query capacity programmatically — is in build. The architecture assumes
it; the endpoints are not published yet.