The workhorse. One synchronous request in, one complete response out — text generation and function-calling over the HTTP your stack already speaks. Stateless, drop-in, and everywhere a server can make a POST.
Correction, before anything else. If you are working from an older page that
documents POST /v1/generate with an input / output /
usage body, that endpoint does not exist. It never shipped under that name.
The request tier is POST /v1/chat/completions and it takes the OpenAI
chat-completions shape — messages in, choices out. Anything you have
written against an OpenAI-compatible client already speaks it.
Two things to set: the base URL and the key.
| what | value | notes |
|---|---|---|
| base URL | https://api.hawktalk.ai | A local node is http://HOST:8890 — same routes, same shapes, no TLS. Put it in an env var; you will point at both. |
| auth | Authorization: Bearer sk-... | Every route except GET /health, which takes no auth and is your boot-time probe. |
| model | "auto" | Runs the router per utterance. Always start here. Pins exist — ouro, quick, dank, think, cloud (aliases self, route, ouromega, live, specialist, t0–t3) — but a pin is a decision to pay one tier's price for every turn, including "mhm". |
Do not hardcode a registry id from this page or any other. Nodes serve different
models. Call GET /v1/models and read what your node actually has, or send
"auto" and let the router pick. Ids printed below came from one node's registry and
are illustrative only.
# The whole thing. Nothing else is required. export HAWKTALK_API_KEY="sk-..." export HAWKTALK_BASE="https://api.hawktalk.ai" # local node: http://HOST:8890 curl -s "$HAWKTALK_BASE/v1/chat/completions" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [ {"role": "system", "content": "You are terse."}, {"role": "user", "content": "Name the highest tide on earth."} ] }'
The response, unedited. Every field OpenAI defines is where OpenAI puts it; HawkTalk adds two top-level extras.
{
"id": "chatcmpl-8f2b1c0a",
"object": "chat.completion",
"created": 1786060800,
"model": "hawkalphaquick",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The Bay of Fundy, about 16 m."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 24,
"completion_tokens": 11,
"total_tokens": 35
},
"x_timing": {
"ttft_ms": 536.1,
"decode_tps": 41.7,
"total_ms": 1180.4,
"backend": "llama-server/ggml-hexagon"
},
"x_compute": { "mode": "casual", "engaged": false }
}
# NOTE: "model" is what the router CHOSE, which is not what you SENT. You sent
# "auto"; the node answered with the id that served the turn. Log both.
# "hawkalphaquick" is one node's id. Read yours from GET /v1/models.
The two extras are additive — ignoring them is safe and nothing breaks. Do not ignore them. They are the only honest answer to "which silicon served that turn, and what did it cost me". Parse permissively while you are at it: a node may add top-level fields this page does not document, and an unknown field must never crash your turn.
| field | shape | notes |
|---|---|---|
| x_timing.ttft_ms | number | null | Time to first token, measured on the node. |
| x_timing.decode_tps | number | null | Decode throughput, tokens/second. |
| x_timing.total_ms | number | null | Wall-clock for the turn on the server side. Your client-observed latency will be larger; the gap is the network. |
| x_timing.backend | string | null | Which engine served it, e.g. "llama-server/ggml-hexagon". Node-specific string — log it, do not switch on it. |
| x_compute.mode | string | null — today "casual" or "pro" | Whether the heavy compute path ran for this turn. Treat it as an open string: unmeasured comes back null or "unknown", and a node may add modes. Log it; do not branch on an exhaustive set. |
| x_compute.engaged | bool | null | false is a fact, not a fallback or a failure. |
The one rule that will bite you. An unmeasured value comes back null, or the
string "unknown" — never a fabricated 0. That invariant holds
across the whole API. If your metrics layer coerces missing numbers to zero you will graph a
0 ms time-to-first-token that never happened, and then optimise against it. Keep the nulls null
and drop the sample. Every example below is written to preserve that distinction; that is why
they all reach for Option, *float64, num? and
?? null rather than a default.
Errors arrive with a non-2xx status and one consistent body shape. The two you will
meet first are 401 invalid_api_key (the key) and 429
rate_limit_exceeded, which carries a Retry-After header — honour it rather
than inventing a backoff.
{"error": {
"message": "Invalid API key.",
"type": "invalid_request_error",
"code": "invalid_api_key",
"param": null
}}
# pip install openai (the OpenAI SDK works unmodified — just repoint base_url) import os from openai import OpenAI client = OpenAI( api_key=os.environ["HAWKTALK_API_KEY"], # The /v1 IS part of base_url for this SDK. Local node: "http://HOST:8890/v1". base_url=os.environ.get("HAWKTALK_BASE", "https://api.hawktalk.ai") + "/v1", ) r = client.chat.completions.create( model="auto", # router picks per utterance — the default advice messages=[ {"role": "system", "content": "You are terse."}, {"role": "user", "content": "Name the highest tide on earth."}, ], ) print(r.choices[0].message.content) print("served by:", r.model) # the id the ROUTER chose, not "auto" # The SDK parks fields it does not know about in model_extra. That is where # x_timing / x_compute land — they are real, just not in its schema. extra = r.model_extra or {} timing = extra.get("x_timing") or {} print("backend", timing.get("backend")) ttft = timing.get("ttft_ms") # NOT .get("ttft_ms", 0) if ttft is not None: # null means unmeasured; skip the sample print(f"ttft {ttft:.0f} ms")
// No dependency needed — fetch is built in from Node 18. The `openai` npm // package also works: new OpenAI({ baseURL: base + "/v1", apiKey }). const key = process.env.HAWKTALK_API_KEY!; const base = process.env.HAWKTALK_BASE ?? "https://api.hawktalk.ai"; // The SDK's types do not know about these. Declare them yourself. type HawkCompletion = { model: string; choices: { message: { role: string; content: string }; finish_reason: string }[]; usage?: { prompt_tokens: number; completion_tokens: number; total_tokens: number }; x_timing?: { ttft_ms: number | null; decode_tps: number | null; total_ms: number | null; backend: string | null }; x_compute?: { mode: string | null; engaged: boolean | null }; }; const res = await fetch(`${base}/v1/chat/completions`, { method: "POST", headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json", }, body: JSON.stringify({ model: "auto", messages: [ { role: "system", content: "You are terse." }, { role: "user", content: "Name the highest tide on earth." }, ], }), }); if (!res.ok) { // The error body is JSON on every documented failure. Read it before // throwing — {error:{message,type,code,param}} says exactly what went wrong. const body = await res.text(); throw new Error(`hawktalk ${res.status}: ${body}`); } const r = (await res.json()) as HawkCompletion; console.log(r.choices[0].message.content); // ?? null, never ?? 0 — a coerced zero is a measurement you did not take. console.log({ served_by: r.model, backend: r.x_timing?.backend ?? null, ttft_ms: r.x_timing?.ttft_ms ?? null, compute: r.x_compute?.mode ?? null, });
// pubspec: http: ^1.2.0 import 'dart:convert'; import 'dart:io' show Platform; import 'package:http/http.dart' as http; final base = Platform.environment['HAWKTALK_BASE'] ?? 'https://api.hawktalk.ai'; final key = Platform.environment['HAWKTALK_API_KEY']!; Future<void> main() async { final res = await http.post( Uri.parse('$base/v1/chat/completions'), headers: { 'Authorization': 'Bearer $key', 'Content-Type': 'application/json', }, body: jsonEncode({ 'model': 'auto', 'messages': [ {'role': 'system', 'content': 'You are terse.'}, {'role': 'user', 'content': 'Name the highest tide on earth.'}, ], }), ); // bodyBytes + utf8.decode, not res.body: http's default fallback charset // mangles non-ASCII replies. final body = jsonDecode(utf8.decode(res.bodyBytes)) as Map<String, dynamic>; if (res.statusCode != 200) { final err = body['error'] as Map<String, dynamic>?; throw Exception('hawktalk ${res.statusCode} ' '${err?["code"] ?? "unknown"}: ${err?["message"] ?? body}'); } final choices = body['choices'] as List<dynamic>; final text = (choices.first as Map)['message']['content'] as String; print(text); // Nullable on purpose. A null ttft is "not measured" and the UI must render // it as an em dash, never as 0 ms. final timing = body['x_timing'] as Map<String, dynamic>?; final num? ttftMs = timing?['ttft_ms'] as num?; print('served by ${body["model"]} ' 'backend=${timing?["backend"] ?? "unknown"} ' 'ttft=${ttftMs?.toStringAsFixed(0) ?? "—"}ms'); }
// Cargo.toml: // tokio = { version = "1", features = ["macros", "rt-multi-thread"] } // reqwest = { version = "0.12", features = ["json"] } // serde = { version = "1", features = ["derive"] } // serde_json = "1" use serde::Deserialize; use serde_json::json; // Option<T> everywhere: serde maps JSON null -> None, which is exactly the // distinction the API is drawing. Never #[serde(default)] these to 0.0. #[derive(Deserialize, Debug)] struct XTiming { ttft_ms: Option<f64>, decode_tps: Option<f64>, total_ms: Option<f64>, backend: Option<String>, } #[derive(Deserialize, Debug)] struct XCompute { mode: Option<String>, engaged: Option<bool> } #[derive(Deserialize, Debug)] struct Message { content: String } #[derive(Deserialize, Debug)] struct Choice { message: Message, finish_reason: Option<String> } #[derive(Deserialize, Debug)] struct Completion { model: String, choices: Vec<Choice>, x_timing: Option<XTiming>, x_compute: Option<XCompute>, } #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let key = std::env::var("HAWKTALK_API_KEY")?; let base = std::env::var("HAWKTALK_BASE") .unwrap_or_else(|_| "https://api.hawktalk.ai".to_string()); let res = reqwest::Client::new() .post(format!("{base}/v1/chat/completions")) .bearer_auth(&key) .json(&json!({ "model": "auto", "messages": [ {"role": "system", "content": "You are terse."}, {"role": "user", "content": "Name the highest tide on earth."} ] })) .send() .await?; // Read the body before erroring: the JSON says which code it was. let status = res.status(); let raw = res.text().await?; if !status.is_success() { return Err(format!("hawktalk {status}: {raw}").into()); } let c: Completion = serde_json::from_str(&raw)?; println!("{}", c.choices[0].message.content); println!( "served_by={} backend={:?} ttft_ms={:?} compute={:?}", c.model, c.x_timing.as_ref().and_then(|t| t.backend.as_deref()), c.x_timing.as_ref().and_then(|t| t.ttft_ms), // stays None if unmeasured c.x_compute.as_ref().and_then(|x| x.mode.as_deref()), ); Ok(()) }
// Standard library only. package main import ( "bytes" "encoding/json" "fmt" "io" "log" "net/http" "os" "time" ) // Pointers, not values: a nil *float64 is "not measured". A plain float64 // would decode a missing field to 0 and invent a measurement. type xTiming struct { TTFTms *float64 `json:"ttft_ms"` DecodeTPS *float64 `json:"decode_tps"` TotalMs *float64 `json:"total_ms"` Backend *string `json:"backend"` } type completion struct { Model string `json:"model"` Choices []struct { Message struct { Content string `json:"content"` } `json:"message"` FinishReason string `json:"finish_reason"` } `json:"choices"` Timing *xTiming `json:"x_timing"` Compute *struct { Mode *string `json:"mode"` Engaged *bool `json:"engaged"` } `json:"x_compute"` } func main() { base := os.Getenv("HAWKTALK_BASE") if base == "" { base = "https://api.hawktalk.ai" } payload, err := json.Marshal(map[string]any{ "model": "auto", "messages": []map[string]string{ {"role": "system", "content": "You are terse."}, {"role": "user", "content": "Name the highest tide on earth."}, }, }) if err != nil { log.Fatalf("marshal: %v", err) } req, err := http.NewRequest("POST", base+"/v1/chat/completions", bytes.NewReader(payload)) if err != nil { log.Fatalf("request: %v", err) } req.Header.Set("Authorization", "Bearer "+os.Getenv("HAWKTALK_API_KEY")) req.Header.Set("Content-Type", "application/json") // A routed turn can take seconds on a cold tier. Do not use a 2s timeout. client := &http.Client{Timeout: 120 * time.Second} res, err := client.Do(req) if err != nil { log.Fatalf("post: %v", err) } defer res.Body.Close() raw, err := io.ReadAll(res.Body) if err != nil { log.Fatalf("read: %v", err) } if res.StatusCode != http.StatusOK { log.Fatalf("hawktalk %d: %s", res.StatusCode, raw) } var c completion if err := json.Unmarshal(raw, &c); err != nil { log.Fatalf("decode: %v", err) } if len(c.Choices) == 0 { log.Fatalf("no choices in response: %s", raw) } fmt.Println(c.Choices[0].Message.Content) // Only record what was actually measured. if c.Timing != nil && c.Timing.TTFTms != nil { fmt.Printf("ttft %.0f ms\n", *c.Timing.TTFTms) } if c.Compute != nil && c.Compute.Mode != nil { fmt.Printf("compute %s\n", *c.Compute.Mode) } fmt.Printf("served_by=%s\n", c.Model) }
Add "stream": true to the same request. The response becomes
text/event-stream: a sequence of data: lines, each carrying one
chat.completion.chunk object, terminated by the literal line
data: [DONE]. This is byte-compatible with OpenAI's stream — if you already have an
SSE parser, it works untouched.
Four facts decide whether your parser is correct:
| fact | consequence |
|---|---|
Events are separated by a blank line; the payload is everything after data: . | Split on newlines and skip empty lines. Do not assume one chunk per TCP read — a read can split a JSON object in half, so buffer. |
data: [DONE] is not JSON. | Check for it before you call your JSON parser, or the last event of every stream throws. |
The first chunk usually carries delta.role and no content. Later chunks carry delta.content and no role. Some chunks carry an empty delta. | Treat delta.content as optional on every chunk. Append when present; never index into it blind. |
x_timing, x_compute and usage ride the FINAL chunk — the one with finish_reason: "stop" — not every chunk. | Read the telemetry off the chunk you are already holding, and keep reading until [DONE]. A parser that breaks out of the loop the instant it sees finish_reason, before handling that same chunk's body, throws the telemetry away. |
curl -N -s "$HAWKTALK_BASE/v1/chat/completions" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ -H "Accept: text/event-stream" \ -d '{ "model": "auto", "stream": true, "messages": [{"role": "user", "content": "Name the highest tide on earth."}] }' # -N (--no-buffer) is not optional. Without it curl buffers the whole stream # and you see nothing until the turn ends, which looks exactly like a hang.
The bytes on the wire, complete, from first chunk to terminator:
data: {"id":"chatcmpl-8f2b1c0a","object":"chat.completion.chunk","created":1786060800, "model":"hawkalphaquick", "choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]} data: {"id":"chatcmpl-8f2b1c0a","object":"chat.completion.chunk","created":1786060800, "model":"hawkalphaquick", "choices":[{"index":0,"delta":{"content":"The Bay"},"finish_reason":null}]} data: {"id":"chatcmpl-8f2b1c0a","object":"chat.completion.chunk","created":1786060800, "model":"hawkalphaquick", "choices":[{"index":0,"delta":{"content":" of Fundy"},"finish_reason":null}]} data: {"id":"chatcmpl-8f2b1c0a","object":"chat.completion.chunk","created":1786060800, "model":"hawkalphaquick", "choices":[{"index":0,"delta":{"content":", about 16 m."},"finish_reason":null}]} data: {"id":"chatcmpl-8f2b1c0a","object":"chat.completion.chunk","created":1786060800, "model":"hawkalphaquick", "choices":[{"index":0,"delta":{},"finish_reason":"stop"}], "usage":{"prompt_tokens":18,"completion_tokens":11,"total_tokens":29}, "x_timing":{"ttft_ms":214.6,"decode_tps":43.2,"total_ms":512.9, "backend":"llama-server/ggml-hexagon"}, "x_compute":{"mode":"casual","engaged":false}} data: [DONE] # Wrapped here for the page. On the wire each `data:` event is ONE line. # Note ttft_ms 214 streamed vs 536 unstreamed on the same question: streaming # does not make the turn faster, it makes the wait visible. Use it for anything # a human is watching; skip it for machine-to-machine calls, where the extra # parsing buys nothing.
# pip install httpx — raw SSE, so you can see exactly what arrives. # (The openai SDK also streams: `for chunk in client.chat.completions.create( # ..., stream=True)`. It swallows [DONE] and parks x_timing in # chunk.model_extra on the final chunk.) import json, os import httpx BASE = os.environ.get("HAWKTALK_BASE", "https://api.hawktalk.ai") KEY = os.environ["HAWKTALK_API_KEY"] body = { "model": "auto", "stream": True, "messages": [{"role": "user", "content": "Name the highest tide on earth."}], } text, timing, compute = [], None, None # timeout=None on read, or a long turn dies mid-stream at the default 5s. with httpx.stream( "POST", f"{BASE}/v1/chat/completions", headers={"Authorization": f"Bearer {KEY}"}, json=body, timeout=httpx.Timeout(10.0, read=None), ) as r: if r.status_code != 200: r.read() # body is lazy on a stream raise RuntimeError(f"hawktalk {r.status_code}: {r.text}") # iter_lines() already handles the buffering and the newline split. for line in r.iter_lines(): if not line or not line.startswith("data:"): continue # blank separators, comments, keepalives payload = line[5:].strip() if payload == "[DONE]": break # NOT json — check before parsing chunk = json.loads(payload) choice = (chunk.get("choices") or [{}])[0] piece = (choice.get("delta") or {}).get("content") if piece: # role-only and empty deltas are normal text.append(piece) print(piece, end="", flush=True) # Telemetry rides the FINAL chunk. Read it on every chunk and keep the # last non-null — cheaper than special-casing finish_reason. timing = chunk.get("x_timing") or timing compute = chunk.get("x_compute") or compute print() print("backend", (timing or {}).get("backend"), "ttft_ms", (timing or {}).get("ttft_ms"), # None stays None "mode", (compute or {}).get("mode"))
// fetch + ReadableStream. The parsing works in Node 18+ and in the browser // unchanged — but never put a real sk- key in browser code; proxy the call. // (process.stdout below is Node-only; in a browser append to the DOM.) const res = await fetch(`${base}/v1/chat/completions`, { method: "POST", headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" }, body: JSON.stringify({ model: "auto", stream: true, messages: [{ role: "user", content: "Name the highest tide on earth." }], }), }); if (!res.ok || !res.body) { const body = await res.text(); throw new Error(`hawktalk ${res.status}: ${body}`); } const reader = res.body.getReader(); const decoder = new TextDecoder(); let buffer = "", text = ""; let timing: unknown = null, compute: unknown = null; outer: while (true) { const { done, value } = await reader.read(); if (done) break; // stream:true on the decoder keeps multi-byte characters intact when a // chunk boundary lands mid-codepoint. Without it you get mojibake. buffer += decoder.decode(value, { stream: true }); // A network read can end mid-JSON. Only consume COMPLETE lines; whatever // is left after the last newline stays in the buffer for the next read. const lines = buffer.split("\n"); buffer = lines.pop() ?? ""; for (const raw of lines) { const line = raw.trim(); if (!line.startsWith("data:")) continue; const payload = line.slice(5).trim(); if (payload === "[DONE]") break outer; // not JSON const chunk = JSON.parse(payload); const piece = chunk.choices?.[0]?.delta?.content; if (piece) { text += piece; process.stdout.write(piece); } timing = chunk.x_timing ?? timing; // final chunk only compute = chunk.x_compute ?? compute; } } console.log("\n", { chars: text.length, timing, compute });
// pubspec: http: ^1.2.0 import 'dart:convert'; import 'dart:io' show stdout; import 'package:http/http.dart' as http; Future<void> streamTurn(String base, String key, String prompt) async { final client = http.Client(); try { final req = http.Request('POST', Uri.parse('$base/v1/chat/completions')) ..headers['Authorization'] = 'Bearer $key' ..headers['Content-Type'] = 'application/json' ..body = jsonEncode({ 'model': 'auto', 'stream': true, 'messages': [{'role': 'user', 'content': prompt}], }); // send(), not post() — post() waits for the whole body and you lose the // only thing streaming buys you. final res = await client.send(req); if (res.statusCode != 200) { final err = await res.stream.bytesToString(); throw Exception('hawktalk ${res.statusCode}: $err'); } final buf = StringBuffer(); Map<String, dynamic>? timing, compute; // LineSplitter does the buffering across network chunk boundaries. await for (final line in res.stream .transform(utf8.decoder) .transform(const LineSplitter())) { if (!line.startsWith('data:')) continue; final payload = line.substring(5).trim(); if (payload == '[DONE]') break; // not JSON final chunk = jsonDecode(payload) as Map<String, dynamic>; final choices = chunk['choices'] as List<dynamic>?; final delta = choices?.isNotEmpty == true ? (choices!.first as Map)['delta'] as Map<String, dynamic>? : null; final piece = delta?['content'] as String?; if (piece != null && piece.isNotEmpty) { buf.write(piece); stdout.write(piece); } timing = chunk['x_timing'] as Map<String, dynamic>? ?? timing; compute = chunk['x_compute'] as Map<String, dynamic>? ?? compute; } print('\nbackend=${timing?["backend"] ?? "unknown"} ' 'mode=${compute?["mode"] ?? "unknown"}'); } finally { client.close(); // or the socket leaks } }
// Cargo.toml adds: futures-util = "0.3" (reqwest "stream" feature) // reqwest = { version = "0.12", features = ["json", "stream"] } use futures_util::StreamExt; use serde_json::{json, Value}; #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let key = std::env::var("HAWKTALK_API_KEY")?; let base = std::env::var("HAWKTALK_BASE") .unwrap_or_else(|_| "https://api.hawktalk.ai".to_string()); let res = reqwest::Client::new() .post(format!("{base}/v1/chat/completions")) .bearer_auth(&key) .json(&json!({ "model": "auto", "stream": true, "messages": [{"role": "user", "content": "Name the highest tide on earth."}] })) .send() .await?; let status = res.status(); if !status.is_success() { return Err(format!("hawktalk {status}: {}", res.text().await?).into()); } let mut stream = res.bytes_stream(); let mut buf: Vec<u8> = Vec::new(); let mut text = String::new(); let mut timing: Option<Value> = None; 'outer: while let Some(bytes) = stream.next().await { // Buffer BYTES, not str. A network chunk boundary can land in the // middle of a multi-byte character, and from_utf8 on the raw chunk // would fail on a perfectly valid stream. buf.extend_from_slice(&bytes?); // Consume complete lines only; keep the tail for the next read. while let Some(nl) = buf.iter().position(|&b| b == b'\n') { let raw: Vec<u8> = buf.drain(..=nl).collect(); let line = String::from_utf8(raw)?; let line = line.trim(); let Some(payload) = line.strip_prefix("data:") else { continue }; let payload = payload.trim(); if payload == "[DONE]" { break 'outer; } // not JSON let chunk: Value = serde_json::from_str(payload)?; if let Some(piece) = chunk["choices"][0]["delta"]["content"].as_str() { text.push_str(piece); print!("{piece}"); } if !chunk["x_timing"].is_null() { // final chunk only timing = Some(chunk["x_timing"].clone()); } } } println!("\n{} chars, x_timing={:?}", text.len(), timing); Ok(()) }
// Standard library. bufio.Scanner does the line buffering for you. package main import ( "bufio" "bytes" "encoding/json" "fmt" "io" "log" "net/http" "os" "strings" ) func main() { base := os.Getenv("HAWKTALK_BASE") if base == "" { base = "https://api.hawktalk.ai" } payload, err := json.Marshal(map[string]any{ "model": "auto", "stream": true, "messages": []map[string]string{ {"role": "user", "content": "Name the highest tide on earth."}, }, }) if err != nil { log.Fatalf("marshal: %v", err) } req, err := http.NewRequest("POST", base+"/v1/chat/completions", bytes.NewReader(payload)) if err != nil { log.Fatalf("request: %v", err) } req.Header.Set("Authorization", "Bearer "+os.Getenv("HAWKTALK_API_KEY")) req.Header.Set("Content-Type", "application/json") req.Header.Set("Accept", "text/event-stream") // No client Timeout here: it caps the WHOLE stream, not the dial, and // would kill a long turn mid-sentence. Use a context deadline if you // need a ceiling. res, err := http.DefaultClient.Do(req) if err != nil { log.Fatalf("post: %v", err) } defer res.Body.Close() if res.StatusCode != http.StatusOK { raw, _ := io.ReadAll(res.Body) log.Fatalf("hawktalk %d: %s", res.StatusCode, raw) } var text strings.Builder var timing json.RawMessage sc := bufio.NewScanner(res.Body) // Default token limit is 64 KB. A long delta plus the final chunk's // telemetry can exceed it; raise it or the scan stops with an error. sc.Buffer(make([]byte, 0, 64*1024), 1024*1024) for sc.Scan() { line := strings.TrimSpace(sc.Text()) if !strings.HasPrefix(line, "data:") { continue } data := strings.TrimSpace(strings.TrimPrefix(line, "data:")) if data == "[DONE]" { break // not JSON — check before Unmarshal } var chunk struct { Choices []struct { Delta struct { Content string `json:"content"` } `json:"delta"` FinishReason *string `json:"finish_reason"` } `json:"choices"` Timing json.RawMessage `json:"x_timing"` } if err := json.Unmarshal([]byte(data), &chunk); err != nil { log.Printf("skipping unparseable chunk: %v", err) continue } if len(chunk.Choices) > 0 && chunk.Choices[0].Delta.Content != "" { text.WriteString(chunk.Choices[0].Delta.Content) fmt.Print(chunk.Choices[0].Delta.Content) } if chunk.Timing != nil { timing = chunk.Timing // arrives on the final chunk } } if err := sc.Err(); err != nil { log.Fatalf("stream: %v", err) } fmt.Printf("\nx_timing=%s\n", timing) }
The request tier is stateless. There is no conversation id, no thread handle, nothing on the server that remembers your last turn. Continuity is a client responsibility: you keep the list, and you send the whole list every time. That is the entire mechanism.
| role | who writes it | notes |
|---|---|---|
| system | you | Zero or one, first. Instructions and persona. See the cost note below — this one has a price attached. |
| user | you | What the human said. |
| assistant | the model, echoed back by you | Append choices[0].message verbatim. Do not paraphrase it, do not strip it, do not re-wrap it — the model's own prior words are the context. |
The four-message request below is what turn two looks like on the wire. Note that
"model": "auto" is sent again: the router re-decides per utterance, so a
conversation can be served by a small tier for "thanks" and a larger one for the question after
it. That is the point — and it is why model in the response can differ from
turn to turn within one conversation. Log it per turn.
curl -s "$HAWKTALK_BASE/v1/chat/completions" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [ {"role": "system", "content": "You are terse."}, {"role": "user", "content": "Name the highest tide on earth."}, {"role": "assistant", "content": "The Bay of Fundy, about 16 m."}, {"role": "user", "content": "Which country is that in?"} ] }' | jq -r '.choices[0].message.content, .model, .usage.prompt_tokens' # Canada. # hawkalphaquick <- the router's pick for THIS turn; it can change # 47 <- prompt_tokens grew: you re-sent the whole history
The cost consequence, and it is the big one. Every turn re-sends every prior message,
so prompt_tokens climbs with the conversation and you pay prefill on all of it,
every turn. The fix is not to send less history — it is to make the front of the prompt
identical across users so the serving fleet can reuse a shared prefix instead of
prefilling yours from scratch.
Never interpolate per-user context into the system prompt. A system prefix that
contains a user's name, plan, locale or history is unique to that user, so nothing in front of
the first differing token can be shared — that request prefills from the very first token, on
every turn, forever. The same words placed a few messages later cost a fraction of that. The
response carries no per-request cache-hit field, so you will see this in your
prompt_tokens curve and your latency, not in a flag. So:
| do this | not this |
|---|---|
| One static system prompt, byte-identical for every user of the app. | f"You are helping {user.name}, a {user.plan} customer in {user.city}." |
| User specifics retrieved into the turn — a user message just before the question, carrying only the facts this question needs. | The user's whole profile pasted into the prefix on turn one and re-sent on turn forty. |
The whole document embedded once (see POST /v1/embeddings) and the relevant few hundred tokens retrieved per turn. | The whole document in the prompt, billed per turn, whether or not it was relevant. |
It reads like a style preference. It is a cost decision: the shared prefix is the part you stop paying to recompute.
# KEY and BASE as in 01.1. from openai import OpenAI client = OpenAI(api_key=KEY, base_url=BASE + "/v1") # STATIC. No f-string, no .format(), no user data. This exact byte sequence is # the front of every request from every user, which is what makes it cheap. SYSTEM = "You are terse. Answer in one sentence unless asked for more." def facts_for(user_id: str, question: str) -> str | None: """Only the profile fields THIS question needs. Usually a line or two.""" return retrieve(user_id, question) # your store; may return None class Conversation: def __init__(self, user_id: str): self.user_id = user_id self.messages = [{"role": "system", "content": SYSTEM}] def ask(self, question: str) -> str: # Per-user context goes HERE, next to the question that needs it — # never merged into SYSTEM, which would make the prefix unshareable. facts = facts_for(self.user_id, question) added = 0 if facts: self.messages.append({"role": "user", "content": f"Context: {facts}"}) added += 1 self.messages.append({"role": "user", "content": question}) added += 1 try: r = client.chat.completions.create(model="auto", messages=self.messages) except Exception: # Drop what we appended, or a retry doubles the history. del self.messages[-added:] raise msg = r.choices[0].message # Echo the assistant turn back verbatim — this IS the memory. self.messages.append({"role": "assistant", "content": msg.content}) # prompt_tokens grows every turn. Watch it; it is your bill. Guard the # usage block rather than assuming it: missing is not zero. if r.usage is not None: print(f"[{r.model}] prompt={r.usage.prompt_tokens} " f"completion={r.usage.completion_tokens}") return msg.content c = Conversation("u_318") c.ask("Name the highest tide on earth.") c.ask("Which country is that in?")
type Msg = { role: "system" | "user" | "assistant"; content: string }; // A module-level const, not a template literal built per request. If this // string ever contains `${user...}` the prefix stops being shareable and // every turn prefills from the first token. const SYSTEM = "You are terse. Answer in one sentence unless asked for more."; class Conversation { private messages: Msg[] = [{ role: "system", content: SYSTEM }]; constructor(private userId: string) {} async ask(question: string): Promise<string> { const facts = await retrieve(this.userId, question); // your store if (facts) this.messages.push({ role: "user", content: `Context: ${facts}` }); this.messages.push({ role: "user", content: question }); const res = await fetch(`${base}/v1/chat/completions`, { method: "POST", headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" }, body: JSON.stringify({ model: "auto", messages: this.messages }), }); if (!res.ok) { // Drop what we optimistically appended, or a retry re-sends a // question that was never answered and doubles the history. this.messages.length -= facts ? 2 : 1; const body = await res.text(); throw new Error(`hawktalk ${res.status}: ${body}`); } const r = await res.json(); const content: string = r.choices[0].message.content; // Verbatim, including whitespace. Paraphrasing it corrupts the context. this.messages.push({ role: "assistant", content }); console.log(r.model, "prompt_tokens", r.usage?.prompt_tokens ?? null); return content; } } const c = new Conversation("u_318"); await c.ask("Name the highest tide on earth."); await c.ask("Which country is that in?");
import 'dart:convert'; import 'package:http/http.dart' as http; // const, so the analyser stops you the moment someone tries to interpolate // a user into it. That compile error is the guardrail. const systemPrompt = 'You are terse. Answer in one sentence unless asked for more.'; class Conversation { Conversation(this.base, this.key, this.userId); final String base, key, userId; final List<Map<String, String>> _messages = [ {'role': 'system', 'content': systemPrompt}, ]; Future<String> ask(String question) async { final facts = await retrieve(userId, question); // your store, may be null if (facts != null) { _messages.add({'role': 'user', 'content': 'Context: $facts'}); } _messages.add({'role': 'user', 'content': question}); final res = await http.post( Uri.parse('$base/v1/chat/completions'), headers: { 'Authorization': 'Bearer $key', 'Content-Type': 'application/json', }, body: jsonEncode({'model': 'auto', 'messages': _messages}), ); final body = jsonDecode(utf8.decode(res.bodyBytes)) as Map<String, dynamic>; if (res.statusCode != 200) { // Roll back the messages we optimistically appended, or the next // attempt re-sends a question that was never answered. _messages.removeLast(); if (facts != null) _messages.removeLast(); throw Exception('hawktalk ${res.statusCode}: ${body["error"]}'); } final content = ((body['choices'] as List).first as Map)['message']['content'] as String; _messages.add({'role': 'assistant', 'content': content}); final usage = body['usage'] as Map<String, dynamic>?; print('[${body["model"]}] prompt=${usage?["prompt_tokens"] ?? "unknown"}'); return content; } }
use serde::{Deserialize, Serialize}; use serde_json::json; // A const &str, not a String built per request. The type system now makes it // awkward to format!() a user into the prefix — which is the point. const SYSTEM: &str = "You are terse. Answer in one sentence unless asked for more."; #[derive(Serialize, Deserialize, Clone, Debug)] struct Msg { role: String, content: String } #[derive(Deserialize)] struct Usage { prompt_tokens: Option<u32> } #[derive(Deserialize)] struct Choice { message: Msg } #[derive(Deserialize)] struct Reply { model: String, choices: Vec<Choice>, usage: Option<Usage>, } struct Conversation { client: reqwest::Client, base: String, key: String, messages: Vec<Msg>, } impl Conversation { fn new(base: String, key: String) -> Self { Self { client: reqwest::Client::new(), base, key, messages: vec![Msg { role: "system".into(), content: SYSTEM.into() }], } } async fn ask(&mut self, question: &str, facts: Option<&str>) -> Result<String, Box<dyn std::error::Error>> { // Retrieved context rides in the turn, not in SYSTEM. if let Some(f) = facts { self.messages.push(Msg { role: "user".into(), content: format!("Context: {f}"), }); } self.messages.push(Msg { role: "user".into(), content: question.into() }); let res = self.client .post(format!("{}/v1/chat/completions", self.base)) .bearer_auth(&self.key) .json(&json!({ "model": "auto", "messages": &self.messages })) .send() .await?; let status = res.status(); let raw = res.text().await?; if !status.is_success() { // Unwind the pushes so a retry does not double the history. self.messages.pop(); if facts.is_some() { self.messages.pop(); } return Err(format!("hawktalk {status}: {raw}").into()); } let r: Reply = serde_json::from_str(&raw)?; let content = r.choices[0].message.content.clone(); self.messages.push(Msg { role: "assistant".into(), content: content.clone() }); println!("[{}] prompt={:?}", r.model, r.usage.and_then(|u| u.prompt_tokens)); Ok(content) } } #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let mut c = Conversation::new( std::env::var("HAWKTALK_BASE") .unwrap_or_else(|_| "https://api.hawktalk.ai".into()), std::env::var("HAWKTALK_API_KEY")?, ); println!("{}", c.ask("Name the highest tide on earth.", None).await?); println!("{}", c.ask("Which country is that in?", None).await?); Ok(()) }
// Drop this in its own file alongside a main(); it is a library type, not a // program. package main import ( "bytes" "encoding/json" "fmt" "io" "net/http" "time" ) // A package-level const. Never fmt.Sprintf a user into this — a per-user // prefix cannot be shared and every turn then prefills from the first token. const systemPrompt = "You are terse. Answer in one sentence unless asked for more." type Msg struct { Role string `json:"role"` Content string `json:"content"` } type Conversation struct { Base, Key string Client *http.Client messages []Msg } func NewConversation(base, key string) *Conversation { return &Conversation{ Base: base, Key: key, Client: &http.Client{Timeout: 120 * time.Second}, messages: []Msg{{Role: "system", Content: systemPrompt}}, } } // facts is the retrieved per-user context for THIS question, or "". func (c *Conversation) Ask(question, facts string) (string, error) { added := 1 if facts != "" { c.messages = append(c.messages, Msg{Role: "user", Content: "Context: " + facts}) added++ } c.messages = append(c.messages, Msg{Role: "user", Content: question}) // On any failure, drop what we appended so a retry is not a duplicate. rollback := func() { c.messages = c.messages[:len(c.messages)-added] } payload, err := json.Marshal(map[string]any{ "model": "auto", "messages": c.messages, }) if err != nil { rollback() return "", fmt.Errorf("marshal: %w", err) } req, err := http.NewRequest("POST", c.Base+"/v1/chat/completions", bytes.NewReader(payload)) if err != nil { rollback() return "", fmt.Errorf("request: %w", err) } req.Header.Set("Authorization", "Bearer "+c.Key) req.Header.Set("Content-Type", "application/json") res, err := c.Client.Do(req) if err != nil { rollback() return "", fmt.Errorf("post: %w", err) } defer res.Body.Close() raw, err := io.ReadAll(res.Body) if err != nil { rollback() return "", fmt.Errorf("read: %w", err) } if res.StatusCode != http.StatusOK { rollback() return "", fmt.Errorf("hawktalk %d: %s", res.StatusCode, raw) } var r struct { Model string `json:"model"` Choices []struct { Message Msg `json:"message"` } `json:"choices"` Usage *struct { PromptTokens *int `json:"prompt_tokens"` CompletionTokens *int `json:"completion_tokens"` } `json:"usage"` } if err := json.Unmarshal(raw, &r); err != nil { rollback() return "", fmt.Errorf("decode: %w", err) } if len(r.Choices) == 0 { rollback() return "", fmt.Errorf("no choices in response") } // The assistant turn, echoed back verbatim, IS the conversation memory. c.messages = append(c.messages, r.Choices[0].Message) if r.Usage != nil && r.Usage.PromptTokens != nil { fmt.Printf("[%s] prompt=%d\n", r.Model, *r.Usage.PromptTokens) } return r.Choices[0].Message.Content, nil }
One last consequence to plan for. The history you keep is unbounded and the context
window is not. When prompt_tokens approaches your model's limit the turn fails with
a 400 invalid_request rather than silently truncating. Decide your policy before you
ship: drop the oldest turns, or summarise them into a single assistant message. Whatever you
choose, never drop the system message — it is the one part of the prompt you want re-sent
unchanged every time.
A tool round trip is two POSTs to the same endpoint,
POST /v1/chat/completions. There is no separate tools route. You send the
schemas on the first call; the model answers with a structured
tool_calls array instead of prose; you execute the call yourself; you send the
whole conversation back — including the assistant's own message, verbatim — with the result
attached; the second response is the sentence your user reads.
Read the structured field. Never regex the prose. This is the single instruction on this page that will save you a production incident, and there are three independent reasons for it:
| because | what happens if you don't |
|---|---|
On a tool turn there is no prose. message.content comes back null — a real null, not "", per the API's no-fabrication invariant. | Your regex runs over nothing, matches nothing, and your app reports "the model gave an empty answer" on the turn where it worked perfectly. |
auto re-routes per utterance. The tier that answers "tide at Saint John" is not necessarily the tier that answers the next question. | A regex fitted to one tier's phrasing is one routing decision away from silently failing. The tool_calls array is the contract; phrasing never was. |
arguments is a JSON string, and its contents can hold braces, quotes and commas of their own. | Any delimiter your pattern picks appears inside real user data eventually. Parse it with a JSON parser, always. |
Branch on the presence of a non-empty choices[0].message.tool_calls
array. Treat absent and null as identical — nodes differ on which they
emit for a plain answer, and both mean "no call". finish_reason is a useful log
field ("tool_calls" when calls were emitted, "stop" for a finished
sentence, "length" when the token budget cut it off) but branch on the array, not
on the string.
The tool schema is nested here, and flat on the socket. REST follows the OpenAI
chat shape — {"type":"function","function":{"name",...}}. The realtime
socket follows the OpenAI realtime shape, where name sits at the top
level of the tool object. Porting a working schema between the two tiers without re-nesting it
is the most common reason a node appears to "ignore tools": an unrecognised schema is not an
error, you simply never get a call back.
| field | type | notes |
|---|---|---|
| id | string | Opaque. Echo it back verbatim as tool_call_id on the tool message. Do not parse it, do not shorten it, do not generate your own. |
| type | string | "function" today. Check it before reading function rather than assuming — an unknown type should be answered with an error result, not crashed on and not silently dropped. |
| function.name | string | One of the names you declared. Dispatch through a map you own. Never eval, never build a call path out of it. |
| function.arguments | string | JSON encoded as a string, not an object. May legitimately be "{}" for a no-argument tool. JSON-parse it, then validate — a returned call is a request, not an authorisation. Parse defensively: a turn cut off by finish_reason:"length" can leave the string truncated. |
The whole round trip, in real payloads. One tool, one call, one answer:
// 1. -> POST /v1/chat/completions {"model":"auto", "messages":[{"role":"user","content":"How high is the tide at Saint John right now?"}], "tools":[{"type":"function","function":{ "name":"tide_height", "description":"Current tide height in metres for a named port.", "parameters":{"type":"object", "properties":{"port":{"type":"string"}}, "required":["port"]}}}]} // 2. <- 200. content is null. THIS is what you read. {"id":"chatcmpl-7f3a91","object":"chat.completion","created":1771027200, "model":"hawkalphaquick", "choices":[{"index":0, "message":{"role":"assistant","content":null, "tool_calls":[{"id":"call_a19","type":"function", "function":{"name":"tide_height", "arguments":"{\"port\":\"Saint John\"}"}}]}, "finish_reason":"tool_calls"}], "usage":{"prompt_tokens":118,"completion_tokens":24,"total_tokens":142}, "x_timing":{"ttft_ms":412.8,"decode_tps":44.1,"total_ms":655.2, "backend":"llama-server/ggml-hexagon"}, "x_compute":{"mode":"casual","engaged":false}} // "hawkalphaquick" is what ONE node's registry returned for model:"auto" on // this turn. Do not hardcode it. Call GET /v1/models and read what yours serves. // x_timing / x_compute are the only additions to the OpenAI shape. Any other // side-band key is node-specific: read it if it is there, never require it. // 3. -> POST /v1/chat/completions again: same user turn, the assistant message // copied back VERBATIM, then one tool message per call_id. {"model":"auto", "messages":[ {"role":"user","content":"How high is the tide at Saint John right now?"}, {"role":"assistant","content":null, "tool_calls":[{"id":"call_a19","type":"function", "function":{"name":"tide_height","arguments":"{\"port\":\"Saint John\"}"}}]}, {"role":"tool","tool_call_id":"call_a19", "content":"{\"port\":\"Saint John\",\"metres\":11.7,\"rising\":true}"}], "tools":[/* the same array again — REST is stateless */]} // 4. <- 200. Now there is prose, and tool_calls is gone. {"id":"chatcmpl-7f3a92","object":"chat.completion","created":1771027202, "model":"hawkalphaquick", "choices":[{"index":0, "message":{"role":"assistant", "content":"The tide at Saint John is 11.7 m and still rising."}, "finish_reason":"stop"}], "usage":{"prompt_tokens":171,"completion_tokens":14,"total_tokens":185}, "x_timing":{"ttft_ms":298.4,"decode_tps":46.0,"total_ms":511.7, "backend":"llama-server/ggml-hexagon"}, "x_compute":{"mode":"casual","engaged":false}}
Four rules that turn into 400s if you break them. Every id in the
assistant's tool_calls needs exactly one matching tool message before
the follow-up — answer even the ones that failed, and even the ones you refused to run, with
{"error":"..."}, rather than dropping them. The tool message's
content is a string; stringify your result object rather than nesting it.
The assistant message goes back as received, content:null and all —
reconstructing it from parts is how call_ids stop matching. And keep the
messages in order: user, assistant, tool, tool, …
These examples are non-streaming. With stream:true the same two-POST
round trip holds, but the first response arrives as chat.completion.chunk SSE
events terminated by data: [DONE], and the calls come fragmented: each chunk
carries choices[0].delta.tool_calls, an array of partial objects keyed by
an index field, whose function.arguments fragments you concatenate
per index until finish_reason is "tool_calls". A chunk has no
message.tool_calls — looking for one, finding nothing, and falling back to
pattern-matching the deltas is exactly the failure this section exists to prevent. Assemble
the array first, then run the branch below on it unchanged.
What this costs. A tool turn is two billed requests, and because REST is stateless
the tool schema is re-sent and re-billed as input tokens on every request in the
conversation — twice per tool round trip, and again on every later turn that still declares
it. Long descriptions and deep parameters objects are a per-turn tax on every
user; write them tight. Latency adds the same way: two ttft_ms, plus however long
your own function takes, before the user hears anything. If you are voicing the reply, that
whole sum lands before first audio — which is exactly why the socket tier declares tools once
per session instead.
# The shell version of the full round trip. jq does the JSON assembly, because # hand-escaping an assistant message back into a request is how call_ids break. # Local node: http://HOST:8890/v1/chat/completions API="https://api.hawktalk.ai/v1/chat/completions" Q="How high is the tide at Saint John right now?" cat > tools.json <<'EOF' [{"type":"function","function":{ "name":"tide_height", "description":"Current tide height in metres for a named port.", "parameters":{"type":"object", "properties":{"port":{"type":"string"}}, "required":["port"]}}}] EOF # --- call 1 ------------------------------------------------------------- jq -n --arg q "$Q" --slurpfile tools tools.json \ '{model:"auto", messages:[{role:"user",content:$q}], tools:$tools[0]}' > req1.json # --fail-with-body: still writes the {"error":{…}} body, but exits non-zero, so # a 401/429/503 stops the script instead of feeding jq a null downstream. curl -sS --fail-with-body "$API" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ --data-binary @req1.json > r1.json \ || { jq . r1.json; exit 1; } # READ THE STRUCTURED FIELD. There is no prose on this turn to grep. jq '.choices[0] | {finish_reason, content: .message.content, calls: .message.tool_calls}' r1.json # { # "finish_reason": "tool_calls", # "content": null, # "calls": [ { "id": "call_a19", "type": "function", # "function": { "name": "tide_height", # "arguments": "{\"port\":\"Saint John\"}" } } ] # } # --- run the tool ------------------------------------------------------- # fromjson, not sed: arguments is JSON *inside* a string. CALL_ID=$(jq -r '.choices[0].message.tool_calls[0].id' r1.json) PORT=$(jq -r '.choices[0].message.tool_calls[0].function.arguments | fromjson | .port' r1.json) RESULT=$(jq -cn --arg port "$PORT" \ '{port:$port, metres:11.7, rising:true}') # your real lookup goes here # --- call 2: user turn, assistant message VERBATIM, then the tool result -- jq -n --arg q "$Q" --arg id "$CALL_ID" --arg result "$RESULT" \ --slurpfile tools tools.json --slurpfile r1 r1.json \ '{model:"auto", messages:[{role:"user",content:$q}, $r1[0].choices[0].message, {role:"tool", tool_call_id:$id, content:$result}], tools:$tools[0]}' > req2.json curl -sS --fail-with-body "$API" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ --data-binary @req2.json \ | jq -r '.choices[0].message.content' # The tide at Saint John is 11.7 m and still rising.
# pip install "openai>=1.40" # The OpenAI SDK is a client for the *shape*, not the company — point base_url # at HawkTalk and everything below is stock. Local node: http://HOST:8890/v1 import json, logging, os from openai import OpenAI log = logging.getLogger("hawktalk") client = OpenAI( api_key=os.environ["HAWKTALK_API_KEY"], base_url="https://api.hawktalk.ai/v1", ) TOOLS = [{"type": "function", "function": { "name": "tide_height", "description": "Current tide height in metres for a named port.", "parameters": {"type": "object", "properties": {"port": {"type": "string"}}, "required": ["port"]}}}] def tide_height(port: str) -> dict: """Your real lookup. Keep it total: return an error dict, never raise past the dispatcher, because every call_id still needs an answer.""" return {"port": port, "metres": 11.7, "rising": True} IMPLS = {"tide_height": tide_height} messages = [{"role": "user", "content": "How high is the tide at Saint John right now?"}] # model="auto" runs the router for THIS utterance. Leave it on auto: a pin # costs you the router's cheap-tier hits on every turn that never needed one. first = client.chat.completions.create(model="auto", messages=messages, tools=TOOLS) msg = first.choices[0].message # THE STRUCTURED FIELD. msg.content is None on a tool turn — there is nothing # to regex, and a regex over a routed model's prose is not a contract anyway. calls = msg.tool_calls or [] if not calls: # Not an error. See 01.5 — this is the common, correct case. print(msg.content) else: # The assistant message goes back verbatim. exclude_unset keeps exactly the # keys the node sent — content:None among them — and invents none. Do NOT # use exclude_none: it deletes the null content this API guarantees. messages.append(msg.model_dump(exclude_unset=True)) for c in calls: if c.type != "function": # Answer it anyway: a call_id with no tool message 400s the next call. messages.append({"role": "tool", "tool_call_id": c.id, "content": json.dumps( {"error": f"unsupported tool_call type: {c.type}"})}) continue fn = IMPLS.get(c.function.name) # dispatch map you own; no eval try: # Inside the try: arguments is a STRING of JSON and a length-truncated # turn can leave it unparseable. That is a tool result, not a crash. args = json.loads(c.function.arguments or "{}") out = fn(**args) if fn else {"error": f"no such tool: {c.function.name}"} except Exception as e: # answer anyway — a missing out = {"error": str(e)} # tool_call_id is a 400 next call messages.append({"role": "tool", "tool_call_id": c.id, "content": json.dumps(out)}) # content is a STRING # Second billed request. TOOLS is re-sent because REST is stateless — the # schema is charged as input tokens again, so keep descriptions short. second = client.chat.completions.create(model="auto", messages=messages, tools=TOOLS) print(second.choices[0].message.content) # Side-band telemetry rides both responses. Nulls stay null — never `or 0`. x = second.model_extra or {} t = x.get("x_timing") or {} log.info("model=%s ttft_ms=%s decode_tps=%s compute=%s", second.model, t.get("ttft_ms"), t.get("decode_tps"), (x.get("x_compute") or {}).get("mode"))
// npm i openai (ESM, Node 18+, top-level await) import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.HAWKTALK_API_KEY!, baseURL: "https://api.hawktalk.ai/v1", // local node: http://HOST:8890/v1 }); const tools: OpenAI.Chat.Completions.ChatCompletionTool[] = [{ type: "function", // Nested under `function` — this is the chat shape, not the realtime shape. function: { name: "tide_height", description: "Current tide height in metres for a named port.", parameters: { type: "object", properties: { port: { type: "string" } }, required: ["port"], }, }, }]; const impls: Record<string, (a: any) => Promise<unknown>> = { tide_height: async ({ port }: { port: string }) => ({ port, metres: 11.7, rising: true }), }; const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [ { role: "user", content: "How high is the tide at Saint John right now?" }, ]; const first = await client.chat.completions.create({ model: "auto", messages, tools }); const msg = first.choices[0].message; // The structured field. `?? []` covers both spellings of "no call": the key // absent, and the key present as null. Both mean the same thing. const calls = msg.tool_calls ?? []; if (calls.length === 0) { console.log(msg.content); // 01.5: correct, not a failure } else { messages.push(msg); // verbatim — do not rebuild it for (const c of calls) { let out: unknown; if (c.type !== "function") { // Never `continue` past a call: every id owes a tool message. out = { error: `unsupported tool_call type: ${c.type}` }; } else { const fn = impls[c.function.name]; try { // JSON.parse, never a regex: arguments is JSON inside a string and // real user data contains braces and quotes. out = fn ? await fn(JSON.parse(c.function.arguments || "{}")) : { error: `no such tool: ${c.function.name}` }; } catch (e) { out = { error: String(e) }; // still answer this call_id } } messages.push({ role: "tool", tool_call_id: c.id, content: JSON.stringify(out) }); } const second = await client.chat.completions.create({ model: "auto", messages, tools }); console.log(second.choices[0].message.content); }
// pubspec: http: ^1.2.0 — no SDK needed, the shape is just JSON. import 'dart:convert'; import 'dart:io'; import 'package:http/http.dart' as http; final _api = Uri.parse('https://api.hawktalk.ai/v1/chat/completions'); final _key = Platform.environment['HAWKTALK_API_KEY']!; const tools = [ { 'type': 'function', 'function': { 'name': 'tide_height', 'description': 'Current tide height in metres for a named port.', 'parameters': { 'type': 'object', 'properties': {'port': {'type': 'string'}}, 'required': ['port'], }, }, }, ]; Future<Map<String, dynamic>> complete(List<Map<String, dynamic>> messages) async { final res = await http.post( _api, headers: { 'Authorization': 'Bearer $_key', 'Content-Type': 'application/json', }, body: jsonEncode({'model': 'auto', 'messages': messages, 'tools': tools}), ); final body = jsonDecode(utf8.decode(res.bodyBytes)) as Map<String, dynamic>; if (res.statusCode != 200) { // {"error":{"message","type","code","param"}} — code is the machine-readable // half. 429 also carries Retry-After; honour it instead of tight-looping. final err = body['error'] as Map<String, dynamic>?; throw HttpException( '${res.statusCode} ${err?['code'] ?? 'unknown'}: ${err?['message'] ?? res.body}'); } return body; } Map<String, dynamic> runTool(String name, Map<String, dynamic> args) { switch (name) { case 'tide_height': return {'port': args['port'], 'metres': 11.7, 'rising': true}; default: return {'error': 'no such tool: $name'}; } } Future<void> main() async { final messages = <Map<String, dynamic>>[ {'role': 'user', 'content': 'How high is the tide at Saint John right now?'}, ]; final first = await complete(messages); final msg = ((first['choices'] as List).first as Map<String, dynamic>)['message'] as Map<String, dynamic>; // Structured, and nullable on purpose: absent and null both mean "no call". final calls = (msg['tool_calls'] as List?) ?? const []; if (calls.isEmpty) { // content may itself be null — render '' or a placeholder, never 'error'. print(msg['content'] ?? ''); return; } messages.add(msg); // verbatim, content:null and all for (final c in calls.cast<Map<String, dynamic>>()) { Map<String, dynamic> out; if (c['type'] != 'function') { final kind = c['type']; out = {'error': 'unsupported tool_call type: $kind'}; // never dropped } else { final f = c['function'] as Map<String, dynamic>; try { final args = jsonDecode((f['arguments'] as String?) ?? '{}') as Map<String, dynamic>; out = runTool(f['name'] as String, args); // consent-gate real actuators } catch (e) { // Truncated or non-object arguments still owe this call_id an answer. out = {'error': 'bad arguments: $e'}; } } messages.add({ 'role': 'tool', 'tool_call_id': c['id'], 'content': jsonEncode(out), // a String, not a Map }); } final second = await complete(messages); print(((second['choices'] as List).first as Map<String, dynamic>)['message']['content']); }
// Cargo.toml // tokio = { version = "1", features = ["macros", "rt-multi-thread"] } // reqwest = { version = "0.12", features = ["json"] } // serde_json = "1" // anyhow = "1" use anyhow::{anyhow, Result}; use serde_json::{json, Value}; const API: &str = "https://api.hawktalk.ai/v1/chat/completions"; fn tools() -> Value { json!([{ "type": "function", "function": { "name": "tide_height", "description": "Current tide height in metres for a named port.", "parameters": { "type": "object", "properties": { "port": { "type": "string" } }, "required": ["port"] } } }]) } async fn complete(http: &reqwest::Client, key: &str, messages: &Value) -> Result<Value> { let res = http .post(API) .bearer_auth(key) .json(&json!({ "model": "auto", "messages": messages, "tools": tools() })) .send() .await?; let status = res.status(); let body: Value = res.json().await?; if !status.is_success() { // {"error":{"message","type","code","param"}}. Match on `code`: on this // endpoint 503 model_unavailable / backend_unavailable / compute_not_wired // and 502 backend_failed are the retryable ones; 400/401/404 are not. return Err(anyhow!( "{} {}: {}", status.as_u16(), body["error"]["code"].as_str().unwrap_or("unknown"), body["error"]["message"].as_str().unwrap_or("") )); } Ok(body) } fn run_tool(name: &str, args: &Value) -> Value { match name { "tide_height" => json!({ "port": args["port"], "metres": 11.7, "rising": true }), other => json!({ "error": format!("no such tool: {other}") }), } } #[tokio::main] async fn main() -> Result<()> { let key = std::env::var("HAWKTALK_API_KEY")?; let http = reqwest::Client::new(); let mut messages = json!([ { "role": "user", "content": "How high is the tide at Saint John right now?" } ]); let first = complete(&http, &key, &messages).await?; let msg = first["choices"][0]["message"].clone(); // The structured field. Missing key and null both yield None here, which // is the whole point: one code path for "no call". let calls = msg["tool_calls"].as_array().cloned().unwrap_or_default(); if calls.is_empty() { // 01.5. `content` is a real string here; on a tool turn it is null. println!("{}", msg["content"].as_str().unwrap_or("")); return Ok(()); } let msgs = messages .as_array_mut() .ok_or_else(|| anyhow!("messages must be an array"))?; msgs.push(msg.clone()); // the assistant message, verbatim for c in &calls { // No `continue` and no `?` inside this loop: every call_id owes exactly // one tool message, so a bad call becomes an error result, not an exit. let out = if c["type"] != "function" { json!({ "error": format!("unsupported tool_call type: {}", c["type"]) }) } else if let Some(name) = c["function"]["name"].as_str() { // arguments is a JSON *string*: as_str() first, then parse it. A // length-truncated one is an error result, not a dead turn. let raw = c["function"]["arguments"].as_str().unwrap_or("{}"); match serde_json::from_str::<Value>(raw) { Ok(args) => run_tool(name, &args), Err(e) => json!({ "error": format!("bad arguments: {e}") }), } } else { json!({ "error": "tool_call with no function.name" }) }; msgs.push(json!({ "role": "tool", "tool_call_id": c["id"], // echoed exactly, never regenerated "content": out.to_string() // serialised: content is a string })); } let second = complete(&http, &key, &messages).await?; println!( "{}", second["choices"][0]["message"]["content"] .as_str() .unwrap_or("") ); Ok(()) }
package main import ( "bytes" "encoding/json" "fmt" "io" "net/http" "os" "time" ) const api = "https://api.hawktalk.ai/v1/chat/completions" type toolCall struct { ID string `json:"id"` Type string `json:"type"` Function struct { Name string `json:"name"` Arguments string `json:"arguments"` // JSON encoded AS A STRING } `json:"function"` } // Content is *string, not string: null is a real value on a tool turn and must // survive the round trip back to the server unchanged. type message struct { Role string `json:"role"` Content *string `json:"content"` ToolCalls []toolCall `json:"tool_calls,omitempty"` ToolCallID string `json:"tool_call_id,omitempty"` } type completion struct { Model string `json:"model"` Choices []struct { Message message `json:"message"` FinishReason string `json:"finish_reason"` } `json:"choices"` Error *struct { Message string `json:"message"` Type string `json:"type"` Code string `json:"code"` } `json:"error"` } var toolsJSON = json.RawMessage(`[{"type":"function","function":{ "name":"tide_height", "description":"Current tide height in metres for a named port.", "parameters":{"type":"object", "properties":{"port":{"type":"string"}}, "required":["port"]}}}]`) func complete(c *http.Client, key string, msgs []message) (*completion, error) { body, err := json.Marshal(map[string]any{ "model": "auto", "messages": msgs, "tools": toolsJSON, }) if err != nil { return nil, fmt.Errorf("encode request: %w", err) } req, err := http.NewRequest(http.MethodPost, api, bytes.NewReader(body)) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+key) req.Header.Set("Content-Type", "application/json") res, err := c.Do(req) if err != nil { return nil, err } defer res.Body.Close() raw, err := io.ReadAll(res.Body) if err != nil { return nil, err } var out completion if err := json.Unmarshal(raw, &out); err != nil { return nil, fmt.Errorf("http %d, unparseable body: %s", res.StatusCode, raw) } if res.StatusCode != http.StatusOK { if out.Error != nil { // 429 carries Retry-After; 502 backend_failed and 503 model_unavailable / // backend_unavailable / compute_not_wired are the other retryables. return nil, fmt.Errorf("http %d %s: %s (retry-after %q)", res.StatusCode, out.Error.Code, out.Error.Message, res.Header.Get("Retry-After")) } return nil, fmt.Errorf("http %d: %s", res.StatusCode, raw) } if len(out.Choices) == 0 { return nil, fmt.Errorf("no choices in response") } return &out, nil } func runTool(name string, args []byte) ([]byte, error) { switch name { case "tide_height": var a struct { Port string `json:"port"` } if err := json.Unmarshal(args, &a); err != nil { // Malformed arguments still deserve a tool message — otherwise the // follow-up call 400s on the unanswered tool_call_id. return json.Marshal(map[string]any{"error": "bad arguments: " + err.Error()}) } return json.Marshal(map[string]any{ "port": a.Port, "metres": 11.7, "rising": true, }) default: return json.Marshal(map[string]any{"error": "no such tool: " + name}) } } func main() { key := os.Getenv("HAWKTALK_API_KEY") if key == "" { fmt.Fprintln(os.Stderr, "HAWKTALK_API_KEY is unset") os.Exit(1) } c := &http.Client{Timeout: 60 * time.Second} q := "How high is the tide at Saint John right now?" msgs := []message{{Role: "user", Content: &q}} first, err := complete(c, key, msgs) if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } m := first.Choices[0].Message // The structured field. An empty slice covers both "key absent" and "null". if len(m.ToolCalls) == 0 { if m.Content != nil { // 01.5: a correct answer, not a failure fmt.Println(*m.Content) } return } msgs = append(msgs, m) // verbatim for _, tc := range m.ToolCalls { var out []byte var err error if tc.Type != "function" { // Answered, not skipped: an unanswered tool_call_id 400s the next call. out, err = json.Marshal(map[string]any{ "error": "unsupported tool_call type: " + tc.Type, }) } else { args := tc.Function.Arguments if args == "" { args = "{}" } out, err = runTool(tc.Function.Name, []byte(args)) } if err != nil { fmt.Fprintln(os.Stderr, "encode tool result:", err) os.Exit(1) } s := string(out) // content is a string, not an object msgs = append(msgs, message{Role: "tool", ToolCallID: tc.ID, Content: &s}) } second, err := complete(c, key, msgs) if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } if content := second.Choices[0].Message.Content; content != nil { fmt.Println(*content) } fmt.Fprintf(os.Stderr, "model=%s finish=%s\n", second.Model, second.Choices[0].FinishReason) } // One caveat on modelling messages as structs: encoding/json drops any field // you did not declare, so the assistant message you echo back is only as // faithful as this struct. If your node starts carrying a field that matters // to the follow-up turn, keep the choice's message as json.RawMessage and // splice the raw bytes back in instead of re-encoding it.
Declare a tool and ask a question it cannot answer, and a working model returns
no tool_calls at all — just a sentence. That is not a degraded response,
a routing miss, or a small-model failure. It is the model being right. Yet it is the branch
most clients get wrong, and the one that shows up on the invoice.
Here is exactly what comes back. Same request as 01.4 — the tide_height tool
is still declared — but the user asked something else:
// -> same tools array, different question {"model":"auto", "messages":[{"role":"user","content":"Thanks — what's the capital of Nova Scotia?"}], "tools":[{"type":"function","function":{"name":"tide_height","description":"...", "parameters":{"type":"object","properties":{"port":{"type":"string"}}, "required":["port"]}}}]} // <- 200. No tool_calls key at all. content is prose. finish_reason "stop". {"id":"chatcmpl-7f3a95","object":"chat.completion","created":1771027260, "model":"hawkalphaquick", "choices":[{"index":0, "message":{"role":"assistant","content":"Halifax."}, "finish_reason":"stop"}], "usage":{"prompt_tokens":112,"completion_tokens":3,"total_tokens":115}, "x_timing":{"ttft_ms":184.2,"decode_tps":47.3,"total_ms":241.9, "backend":"llama-server/ggml-hexagon"}, "x_compute":{"mode":"casual","engaged":false}} // Some nodes emit the key as an explicit null instead of omitting it: // "message":{"role":"assistant","content":"Halifax.","tool_calls":null} // Both mean NO CALL. Never branch differently on absent vs null. // And the declining-with-an-explanation variant — still no tool_calls: // "content": "I can look up tide height for a named port, but not water // temperature. Which port did you mean?"
| the mistake | what it actually does |
|---|---|
| Treating no-call as an error and retrying. The client expects a tool call, finds none, throws, and the retry wrapper fires. | You pay for the identical turn two or three times and the user waits for all of it. Worse, retries are indistinguishable from load at the node, so a chatty hour turns into 429 rate_limit_exceeded for everyone. Retry 429, 502 and 503. Never retry a 200. |
Escalating the tier. "The small model didn't call the tool, so re-ask on think or cloud." | This is the expensive one. Most turns in a real conversation need no tool, so this rule promotes most of your traffic to your most expensive tier to reproduce an answer the cheap tier already got right. It also inverts what auto is for: the router already chose the tier for this utterance. |
Regexing the prose for a call that isn't there. Pattern-matching content for something that looks like a function name. | A sentence that merely mentions the tool ("I could check the tide for you") matches, and you fire a real call with arguments you invented. If the tool spends money, moves something, or sends a message, you have just done it on a hallucination. This is the failure that is not merely expensive. |
Forcing a call to make the branch go away. Pinning tool_choice to a specific function so a call always comes back. | Now "what's the capital of Nova Scotia?" produces tide_height({"port":"Nova Scotia"}). You removed the branch and kept the bug, with fabricated arguments. tool_choice:"auto" is the default and is almost always what you want; verify any other value against your own node before relying on it. |
The correct handling is boring. No calls means you already have the final answer:
render content and stop. There is no second request, no retry, and no escalation
— the cheap path is also the correct one. Log the outcome so you can see the ratio; if a tool
you expect is never being called, the fix is the tool's description and its
parameter names, not a bigger model.
One null to respect: on a no-call turn content is normally a string, but a node
that produced nothing measurable will send null rather than "".
Render that as empty or as a placeholder. Do not coerce it, and do not report it as an
error — the invariant across this whole API is that unknown is null or
"unknown", never a fabricated value.
# Same client, TOOLS and `log` as 01.4. This is the whole no-call path. messages = [{"role": "user", "content": "Thanks — what's the capital of Nova Scotia?"}] r = client.chat.completions.create(model="auto", messages=messages, tools=TOOLS) msg = r.choices[0].message calls = msg.tool_calls or [] # None and [] collapse to the same branch if not calls: # DONE. One request, one answer. No retry, no second call, no escalation. # content may be None -> render "", never the string "None". print(msg.content or "") log.info("tool_used=none finish=%s model=%s", r.choices[0].finish_reason, r.model) else: handle_calls(calls) # the 01.4 path # WRONG — every one of these is a real bug someone has shipped: # # if not calls: raise RuntimeError("model failed to call the tool") # -> turns a correct answer into a retry, and you pay for it twice. # # if not calls: # r = client.chat.completions.create(model="think", ...) # or "cloud" # -> promotes every chit-chat turn to your dearest tier. This is the # line that shows up on the bill, because most turns need no tool. # # m = re.search(r"tide_height\((.*)\)", msg.content or "") # -> fires a real actuator on a sentence that merely mentioned the tool.
// Same client and tools as 01.4. `render` and `logger` are yours. const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [ { role: "user", content: "Thanks — what's the capital of Nova Scotia?" }, ]; const r = await client.chat.completions.create({ model: "auto", messages, tools }); const choice = r.choices[0]; const calls = choice.message.tool_calls ?? []; // absent === null === no call if (calls.length === 0) { // Terminal. Render and stop — this turn is finished and already paid for. render(choice.message.content ?? ""); logger.info({ tool_used: "none", finish_reason: choice.finish_reason, model: r.model }); } else { await handleCalls(calls); // the 01.4 path } // Two guards worth writing down, because both look reasonable in review: // // if (!calls.length) return retryWithBiggerTier(messages); // Your retry budget now fires on the happy path. Under model:"auto" the // router already picked the tier for this utterance; second-guessing it // per-turn is how a cheap conversation becomes an expensive one. // // if (choice.message.content?.includes("tide_height")) callTool(...); // The model talking ABOUT a tool is not the model calling it. Only the // tool_calls array is an instruction; content is never one.
// Same tools array and complete() from 01.4. This is the whole no-call path. import 'dart:convert'; import 'dart:io'; import 'package:http/http.dart' as http; final _api = Uri.parse('https://api.hawktalk.ai/v1/chat/completions'); final _key = Platform.environment['HAWKTALK_API_KEY']!; const tools = [ { 'type': 'function', 'function': { 'name': 'tide_height', 'description': 'Current tide height in metres for a named port.', 'parameters': { 'type': 'object', 'properties': {'port': {'type': 'string'}}, 'required': ['port'], }, }, }, ]; Future<Map<String, dynamic>> complete(List<Map<String, dynamic>> messages) async { final res = await http.post( _api, headers: { 'Authorization': 'Bearer $_key', 'Content-Type': 'application/json', }, body: jsonEncode({'model': 'auto', 'messages': messages, 'tools': tools}), ); final body = jsonDecode(utf8.decode(res.bodyBytes)) as Map<String, dynamic>; if (res.statusCode != 200) { final err = body['error'] as Map<String, dynamic>?; throw HttpException( '${res.statusCode} ${err?['code'] ?? 'unknown'}: ${err?['message'] ?? res.body}'); } return body; } Future<void> handleCalls(List<Map<String, dynamic>> calls) async { // the 01.4 path: run actuators and submit role: "tool" responses } Future<void> main() async { final messages = <Map<String, dynamic>>[ {'role': 'user', 'content': "Thanks — what's the capital of Nova Scotia?"}, ]; final r = await complete(messages); final choice = (r['choices'] as List).first as Map<String, dynamic>; final msg = choice['message'] as Map<String, dynamic>; final calls = (msg['tool_calls'] as List?)?.cast<Map<String, dynamic>>() ?? const []; if (calls.isEmpty) { // Terminal. Render and stop — this turn is finished and already paid for. // content may be null -> render '', never 'null'. print(msg['content'] ?? ''); print('tool_used=none finish=${choice["finish_reason"]} model=${r["model"]}'); } else { await handleCalls(calls); // the 01.4 path } // Two guards worth writing down, because both look reasonable in review: // // if (calls.isEmpty) return complete(messages); // Your retry budget now fires on the happy path. Under model:"auto" the // router already picked the tier for this utterance; second-guessing it // per-turn is how a cheap conversation becomes an expensive one. // // if ((msg['content'] as String?)?.contains('tide_height') ?? false) callTool(); // The model talking ABOUT a tool is not the model calling it. Only the // tool_calls array is an instruction; content is never one. }
POST /v1/audio/transcriptionsOne endpoint, three accepted bodies, one response shape. Send audio, get
{"text", "x_stt"} back. This is the batch committer: it answers once, when
the whole clip is decoded. There are no partial results here — live partials come from the
socket tier, not from REST.
There is no model routing on this endpoint. A node has one STT engine, wired or not.
The OpenAI SDKs insist on sending a model field; the gateway accepts it and
ignores it. Sending model=auto here is harmless and selects nothing — do not build
a tier ladder on top of it.
| form | Content-Type | body |
|---|---|---|
| multipart | multipart/form-data |
a part named file, with a filename. The default — use this unless
something stops you. |
| JSON base64 | application/json |
{"audio":"<base64>","filename":"clip.wav"}. The key may be
audio, file, audio_b64 or data; a
data: URI prefix is stripped for you. |
| raw | audio/wav, audio/webm, … or
application/octet-stream |
the bytes themselves, nothing wrapped around them. |
| anything else | — | 415. There is no sniffing: the Content-Type decides which parser runs. |
Two limits that bite before the model does. The body cap is a per-node setting — over
it you get 413, and the fix is to split the clip, not to retry. Read the cap off
the node you are actually calling rather than off this page. And the request must carry a
Content-Length: a chunked upload reads as a zero-byte body and comes back 400 "empty
body", which looks nothing like the streaming bug it actually is. Every client library
below sets it for you; a hand-rolled streaming io.Reader does not.
Format matters more than it should. A node with ffmpeg decodes whatever you send
down to 16 kHz mono; a node without it accepts wav only — ideally already 16 kHz mono —
and answers 502 stt_failed for anything else. So there are two different "best"
answers and they pull in opposite directions: 16 kHz mono wav is the format that works
everywhere, and it is what to send when you do not know the node. But it is raw PCM at
~32 kB per second, so on a metered or slow phone link it is the largest thing you can
send — an opus/webm capture of the same utterance is roughly an order of magnitude smaller.
Compress for the link only once /health has told you the node has ffmpeg.
# BASE is https://api.hawktalk.ai, or http://HOST:8890 for a local node. curl -sS "$BASE/v1/audio/transcriptions" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -F "file=@clip.wav;type=audio/wav" # 200 — a real local transcription. Never a placeholder, never an empty string # standing in for "it didn't work". { "text": "launch at first light, wind is twelve knots", "x_stt": { "engine": "whisper-server", "decode": "ffmpeg-16k-mono", "whisper_ms": 412.7, "bytes_in": 221484, "total_ms": 470.3 } }
| field | type | notes |
|---|---|---|
| text | string | the transcript. Always present on a 200. |
| x_stt.engine | string | which committer ran — e.g.
whisper-server (persistent, warm) or whisper.cpp (per-call
process, cold-loads the model). |
| x_stt.decode | string | ffmpeg-16k-mono or
wav-passthrough. Passthrough means the node has no ffmpeg — send wav. |
| x_stt.whisper_ms | number | time the recogniser itself spent on the
clip. Every number here is measured inside the gateway, so
total_ms − whisper_ms is resample plus gateway overhead — it is not your
upload time. The gateway's clock starts when the body has already arrived; only your own
client can time the wire. |
| x_stt.total_ms | number | gateway wall-clock for the call. |
| x_stt.bytes_in | number | bytes the gateway actually received — the cheapest way to catch a truncated upload. |
The key set differs per node. A CLI-backed node adds bin and
model; a long-form node adds chunks, audio_sec and
segments. Read the keys you need and keep the rest as an opaque map. A timing the
node did not measure is absent or null — never 0, and your
client must not turn it into one either.
If the node cannot wire an STT engine, the endpoint says so, loudly, before it does anything else:
# HTTP/1.1 501 Not Implemented {"error": { "message": "STT seam not wired on this node: whisper binary not found — the client must fall back to the browser Web Speech API (SpeechRecognition)", "type": "not_implemented", "code": "stt_not_wired", "param": null }}
Never synthesise a transcript. Not an empty string, not a "[inaudible]", not the last thing the user said. A fabricated transcript propagates into the chat turn, the tool call and the trace, and nothing downstream can tell it from a real one.
The honest fallback is device speech recognition — and note what that actually means:
the device recognizer listens to a microphone, it does not transcribe the file you are
holding. So the fallback is not "decode this clip another way", it is "re-route capture for
the next utterance": browser SpeechRecognition, Flutter
speech_to_text, Android SpeechRecognizer. On a server there is no
microphone and therefore no fallback at all — surface the seam to your caller and let it decide.
Returning null is a correct answer; inventing text is not.
501 is not the only failure, and the other one is easier to hit.
502 stt_failed means the engine is wired and choked on these particular
bytes — nine times out of ten a webm/opus capture sent to a node with no ffmpeg. It is worth one
retry after re-encoding to 16 kHz mono wav, and after that it is worth exactly the same
treatment as a 501: surface it, or re-route capture. It is never worth a placeholder. Handle
both codes or your "it works on my node" client will return silence on someone else's.
Probe the seam once at startup instead of discovering it 30 MB into an upload.
GET /health needs no auth:
# is this node's ear wired at all? curl -sS "$BASE/health" | jq '.voice.stt | {wired, engine, via, error}' { "wired": true, "engine": "whisper-server", "via": "whisper-server @ http://127.0.0.1:8910 (persistent)", "error": null } # unwired looks like this — and `error` tells you why, which is the one thing # a support ticket always leaves out: {"wired": false, "engine": "whisper.cpp", "via": null, "error": "no whisper binary on PATH; no model file"}
# pip install httpx (Python 3.9+ for the builtin generics below) import os import httpx BASE = os.environ.get("HAWKTALK_BASE", "https://api.hawktalk.ai") KEY = os.environ["HAWKTALK_API_KEY"] class SttNotWired(Exception): """501 — this node has no ear. Re-route capture to the device recognizer. Deliberately its own exception type: it is NOT a failure to retry, and it is NOT a reason to invent a transcript. It is a capability answer. """ class SttFailed(Exception): """502 — the engine is wired and failed on THESE bytes. Different from 501: the node can hear, it just couldn't decode what you sent (usually webm on a node with no ffmpeg). Worth one retry as 16 kHz mono wav. Still not a reason to invent a transcript. """ def transcribe(path: str, client: httpx.Client) -> tuple[str, dict]: with open(path, "rb") as f: audio = f.read() # httpx sets the multipart boundary itself — never set Content-Type here. # It also sets a real Content-Length, which this endpoint requires. r = client.post( f"{BASE}/v1/audio/transcriptions", headers={"Authorization": f"Bearer {KEY}"}, files={"file": (os.path.basename(path), audio, "audio/wav")}, ) if r.status_code == 501: raise SttNotWired(r.json()["error"]["message"]) if r.status_code == 502: raise SttFailed(r.json()["error"]["message"]) if r.status_code == 413: raise ValueError(f"clip too large for this node: {r.json()['error']['message']}") r.raise_for_status() body = r.json() # x_stt is telemetry, not contract: default it, never require it. return body["text"], body.get("x_stt") or {} if __name__ == "__main__": # A cold whisper.cpp node loads the model on the first call. 30s is too # tight and you will blame the network for it. with httpx.Client(timeout=120.0) as client: try: text, x_stt = transcribe("clip.wav", client) except (SttNotWired, SttFailed) as e: # Server-side: there is no microphone here, so there is no fallback. # Say so and stop. Do not write a placeholder into your pipeline. print(f"stt unavailable: {e}") raise SystemExit(0) print(text) # .get() with no default: an unmeasured timing prints as None, which is # the truth. `x_stt.get("whisper_ms", 0)` would be a lie in a dashboard. print("engine", x_stt.get("engine", "unknown"), "whisper_ms", x_stt.get("whisper_ms"), "bytes_in", x_stt.get("bytes_in"))
// Node 18+, ESM (.mjs or "type": "module" — the top-level await needs it). // Global fetch, FormData and Blob — no dependencies. import { readFile } from "node:fs/promises"; import { basename } from "node:path"; const BASE = process.env.HAWKTALK_BASE ?? "https://api.hawktalk.ai"; const KEY = process.env.HAWKTALK_API_KEY!; export class SttNotWired extends Error {} // 501: no engine at all export class SttFailed extends Error {} // 502: engine choked on these bytes export type Transcription = { text: string; xStt: Record<string, unknown> }; export async function transcribe(path: string): Promise<Transcription> { const bytes = await readFile(path); const form = new FormData(); // The third argument is the filename. It is not decoration: the node keys // its decode off the extension, and a missing one degrades to "raw.bin". form.append("file", new Blob([new Uint8Array(bytes)], { type: "audio/wav" }), basename(path)); const r = await fetch(`${BASE}/v1/audio/transcriptions`, { method: "POST", // Authorization ONLY. Setting Content-Type yourself overwrites the // boundary fetch generated and the node sees a malformed multipart body. headers: { Authorization: `Bearer ${KEY}` }, body: form, signal: AbortSignal.timeout(120_000), }); if (r.status === 501 || r.status === 502) { const e = await r.json() as { error?: { message?: string } }; const msg = e.error?.message ?? `stt ${r.status}`; throw r.status === 501 ? new SttNotWired(msg) : new SttFailed(msg); } if (!r.ok) throw new Error(`transcriptions ${r.status}: ${await r.text()}`); const body = await r.json() as { text: string; x_stt?: Record<string, unknown> }; return { text: body.text, xStt: body.x_stt ?? {} }; } const out = await transcribe("clip.wav"); console.log(out.text); // ?? "unknown" on a string, nothing on a number: an absent whisper_ms logs as // undefined, which is what happened. `?? 0` would invent a measurement. console.log(out.xStt.engine ?? "unknown", out.xStt.whisper_ms);
// In a browser you DO have a device recognizer, so the seam has a real // answer: stop uploading, start listening. Nothing here ever produces text // the machine did not hear. const BASE = "https://api.hawktalk.ai"; // null means "this node cannot transcribe this for you" — the caller must // switch to the mic path. It never means "the clip was silent". async function transcribeBlob(blob: Blob, key: string): Promise<string | null> { const form = new FormData(); // MediaRecorder gives you webm/opus. A node with ffmpeg takes it; a node // without answers 502 stt_failed — check /health once, at startup. form.append("file", blob, "capture.webm"); const r = await fetch(`${BASE}/v1/audio/transcriptions`, { method: "POST", headers: { Authorization: `Bearer ${key}` }, body: form, }); if (r.status === 501) return null; // no engine: never retry if (r.status === 502) { // engine choked — usually no ffmpeg console.warn("stt_failed:", await r.text()); return null; // re-encode to 16k wav, or use the mic } if (!r.ok) throw new Error(`transcriptions ${r.status}`); return ((await r.json()) as { text: string }).text; } function deviceListen(onFinal: (text: string) => void): boolean { const Rec = (window as any).SpeechRecognition ?? (window as any).webkitSpeechRecognition; if (!Rec) return false; // no ear anywhere: tell the user, type instead const rec = new Rec(); rec.lang = navigator.language; rec.interimResults = false; rec.onresult = (ev: any) => onFinal(ev.results[0][0].transcript); rec.start(); return true; }
// pubspec: http: ^1.2.0 (Dart 3 — the record return type below needs it) import 'dart:convert'; import 'dart:io'; import 'package:http/http.dart' as http; class SttNotWired implements Exception { final String message; SttNotWired(this.message); @override String toString() => 'SttNotWired: $message'; } class SttFailed implements Exception { final String message; SttFailed(this.message); @override String toString() => 'SttFailed: $message'; } /// Uploads [clip] and returns the transcript plus the raw x_stt map. /// Throws [SttNotWired] on 501 and [SttFailed] on 502 — in both cases the /// caller re-routes to the device recognizer (package:speech_to_text). /// It never returns invented text. Future<({String text, Map<String, dynamic> xStt})> transcribe( File clip, { required String base, required String apiKey, http.Client? client, }) async { final owned = client == null; final c = client ?? http.Client(); try { final req = http.MultipartRequest( 'POST', Uri.parse('$base/v1/audio/transcriptions')) ..headers['authorization'] = 'Bearer $apiKey' ..files.add(http.MultipartFile.fromBytes( 'file', await clip.readAsBytes(), filename: clip.uri.pathSegments.last)); final res = await http.Response.fromStream( await c.send(req).timeout(const Duration(seconds: 120))); if (res.statusCode == 501 || res.statusCode == 502) { final err = jsonDecode(res.body) as Map<String, dynamic>; final msg = err['error']?['message'] as String? ?? 'stt unavailable'; throw res.statusCode == 501 ? SttNotWired(msg) : SttFailed(msg); } if (res.statusCode != 200) { throw HttpException('transcriptions ${res.statusCode}: ${res.body}'); } final body = jsonDecode(res.body) as Map<String, dynamic>; return ( text: body['text'] as String, xStt: (body['x_stt'] as Map?)?.cast<String, dynamic>() ?? <String, dynamic>{}, ); } finally { if (owned) c.close(); } } Future<void> main() async { final base = Platform.environment['HAWKTALK_BASE'] ?? 'https://api.hawktalk.ai'; final key = Platform.environment['HAWKTALK_API_KEY']!; try { final r = await transcribe(File('clip.wav'), base: base, apiKey: key); print(r.text); // whisper_ms is num? — print it as it came. Do not `?? 0`. print('engine=${r.xStt['engine'] ?? 'unknown'} ms=${r.xStt['whisper_ms']}'); } on SttNotWired catch (e) { // In a Flutter app this is where you flip to speech_to_text and listen. // In a CLI there is nothing to flip to — report and exit non-zero. stderr.writeln(e); exit(1); } on SttFailed catch (e) { // Wired but unhappy: re-encode to 16 kHz mono wav and try once more. stderr.writeln(e); exit(1); } }
// Cargo.toml // reqwest = { version = "0.12", features = ["json", "multipart"] } // tokio = { version = "1", features = ["full"] } // serde_json = "1" use std::{env, error::Error, fmt, path::Path, time::Duration}; use reqwest::{multipart, Client, StatusCode}; use serde_json::Value; /// 501 — the node has no STT engine. Its own type so `?` propagates it and a /// caller can `downcast_ref` to decide whether a device recognizer exists. #[derive(Debug)] pub struct SttNotWired(pub String); impl fmt::Display for SttNotWired { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "stt_not_wired: {}", self.0) } } impl Error for SttNotWired {} /// 502 — a wired engine failed on these bytes. Separate from SttNotWired /// because it is worth exactly one retry as 16 kHz mono wav. #[derive(Debug)] pub struct SttFailed(pub String); impl fmt::Display for SttFailed { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "stt_failed: {}", self.0) } } impl Error for SttFailed {} pub async fn transcribe( client: &Client, base: &str, key: &str, path: &Path, ) -> Result<(String, Value), Box<dyn Error>> { let bytes = tokio::fs::read(path).await?; let name = path .file_name() .and_then(|s| s.to_str()) .unwrap_or("clip.wav") .to_string(); let part = multipart::Part::bytes(bytes) .file_name(name) .mime_str("audio/wav")?; let form = multipart::Form::new().part("file", part); let res = client .post(format!("{base}/v1/audio/transcriptions")) .bearer_auth(key) .multipart(form) // sets Content-Type + boundary + a real Content-Length .send() .await?; let status = res.status(); if status == StatusCode::NOT_IMPLEMENTED || status == StatusCode::BAD_GATEWAY { let v: Value = res.json().await?; let msg = v["error"]["message"] .as_str() .unwrap_or("stt unavailable") .to_string(); return if status == StatusCode::NOT_IMPLEMENTED { Err(Box::new(SttNotWired(msg))) } else { Err(Box::new(SttFailed(msg))) }; } let res = res.error_for_status()?; // 400/401/413/415 all land here let v: Value = res.json().await?; let text = v["text"] .as_str() .ok_or("200 response had no `text` field")? .to_string(); // x_stt stays a Value. Missing keys read as Value::Null and print as // `null` — the honest answer for a timing nobody measured. Ok((text, v["x_stt"].clone())) } #[tokio::main] async fn main() -> Result<(), Box<dyn Error>> { let base = env::var("HAWKTALK_BASE") .unwrap_or_else(|_| "https://api.hawktalk.ai".to_string()); let key = env::var("HAWKTALK_API_KEY")?; let client = Client::builder() .timeout(Duration::from_secs(120)) .build()?; match transcribe(&client, &base, &key, Path::new("clip.wav")).await { Ok((text, x_stt)) => { println!("{text}"); println!( "engine={} whisper_ms={}", x_stt["engine"].as_str().unwrap_or("unknown"), x_stt["whisper_ms"] ); } Err(e) if e.downcast_ref::<SttNotWired>().is_some() || e.downcast_ref::<SttFailed>().is_some() => { eprintln!("{e} — no server-side fallback exists; not inventing one"); } Err(e) => return Err(e), } Ok(()) }
// Go 1.21+, stdlib only. package main import ( "bytes" "encoding/json" "errors" "fmt" "io" "mime/multipart" "net/http" "os" "path/filepath" "time" ) // The two seams, wrapped so callers use errors.Is instead of string-matching a // message that differs per node. ErrSttNotWired is "no engine, ever"; // ErrSttFailed is "engine present, these bytes lost". var ( ErrSttNotWired = errors.New("stt_not_wired") ErrSttFailed = errors.New("stt_failed") ) type Transcription struct { Text string `json:"text"` XStt map[string]any `json:"x_stt"` // any: values may be null. Keep them null. } // apiMessage pulls {"error":{"message":...}} out of a body, falling back to the // raw bytes. Some errors carry a null "code", so never key off code alone. func apiMessage(raw []byte) string { var e struct { Error struct { Message string `json:"message"` Code string `json:"code"` } `json:"error"` } if err := json.Unmarshal(raw, &e); err == nil && e.Error.Message != "" { return e.Error.Message } return string(raw) } func transcribe(hc *http.Client, base, key, path string) (Transcription, error) { var out Transcription audio, err := os.ReadFile(path) if err != nil { return out, fmt.Errorf("read %s: %w", path, err) } var body bytes.Buffer mw := multipart.NewWriter(&body) part, err := mw.CreateFormFile("file", filepath.Base(path)) if err != nil { return out, err } if _, err := part.Write(audio); err != nil { return out, err } // Close writes the terminating boundary. Forget it and the node sees a // truncated multipart body and answers 400, not 500. if err := mw.Close(); err != nil { return out, err } // A *bytes.Buffer body gives the request a known ContentLength. A streaming // io.Reader would be sent chunked, which this endpoint reads as an empty // body and rejects with 400 "empty body". req, err := http.NewRequest(http.MethodPost, base+"/v1/audio/transcriptions", &body) if err != nil { return out, err } req.Header.Set("Authorization", "Bearer "+key) req.Header.Set("Content-Type", mw.FormDataContentType()) res, err := hc.Do(req) if err != nil { return out, fmt.Errorf("post transcriptions: %w", err) } defer res.Body.Close() raw, err := io.ReadAll(res.Body) if err != nil { return out, fmt.Errorf("read body: %w", err) } switch res.StatusCode { case http.StatusOK: case http.StatusNotImplemented: return out, fmt.Errorf("%w: %s", ErrSttNotWired, apiMessage(raw)) case http.StatusBadGateway: return out, fmt.Errorf("%w: %s", ErrSttFailed, apiMessage(raw)) default: return out, fmt.Errorf("transcriptions %d: %s", res.StatusCode, apiMessage(raw)) } if err := json.Unmarshal(raw, &out); err != nil { return out, fmt.Errorf("decode transcription: %w", err) } return out, nil } func main() { base := os.Getenv("HAWKTALK_BASE") if base == "" { base = "https://api.hawktalk.ai" } hc := &http.Client{Timeout: 120 * time.Second} out, err := transcribe(hc, base, os.Getenv("HAWKTALK_API_KEY"), "clip.wav") if errors.Is(err, ErrSttNotWired) || errors.Is(err, ErrSttFailed) { // No microphone in a server process, so no device fallback. Report the // seam upward. Writing "" into your pipeline would be a silent lie. fmt.Fprintln(os.Stderr, err) os.Exit(2) } if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } fmt.Println(out.Text) // out.XStt["whisper_ms"] is nil when unmeasured; %v prints "<nil>". // Coercing it to 0 here is how a node with no timings starts looking fast. fmt.Printf("engine=%v whisper_ms=%v bytes_in=%v\n", out.XStt["engine"], out.XStt["whisper_ms"], out.XStt["bytes_in"]) }
# Use this when your transport is JSON-only: a queue message, a webhook body, # a form you cannot make multipart. It costs ~33% more bytes on the wire and # those bytes count against the node's body cap — check the cap before you # pick this form for anything long. # Build the body with a real JSON encoder. Shell-quoting base64 by hand is # how you get a 400 you cannot see. python3 - 'clip.wav' > body.json <<'PY' import base64, json, sys raw = open(sys.argv[1], "rb").read() json.dump({"audio": base64.b64encode(raw).decode(), "filename": "clip.wav"}, sys.stdout) PY curl -sS "$BASE/v1/audio/transcriptions" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ --data-binary @body.json {"text": "launch at first light, wind is twelve knots", "x_stt": {"engine": "whisper-server", "decode": "ffmpeg-16k-mono", "whisper_ms": 409.1, "bytes_in": 221484, "total_ms": 468.8}} # Notes that save an afternoon: # - the key may be "audio", "file", "audio_b64" or "data" — first one wins. # - a browser data: URI works as-is: "data:audio/webm;base64,GkXf..." — the # prefix is stripped server-side. Do not strip it twice. # - "filename" is optional but send it anyway: the extension is what tells # the node how to decode. Without it the node guesses ".bin" and a # no-ffmpeg node fails the clip it could have played. # - bad base64 is 400 with the decoder's own complaint, not a 500.
audio/*# The smallest form: no wrapper, no inflation, one header. Best for a device # pushing captures over a metered link. curl -sS "$BASE/v1/audio/transcriptions" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: audio/wav" \ --data-binary @clip.wav {"text": "launch at first light, wind is twelve knots", "x_stt": {"engine": "whisper-server", "decode": "ffmpeg-16k-mono", "whisper_ms": 405.6, "bytes_in": 221484, "total_ms": 441.2}} # The Content-Type header is NOT optional. curl's default for --data-binary is # application/x-www-form-urlencoded, which this endpoint answers with: {"error": {"message": "unsupported Content-Type 'application/x-www-form-urlencoded' — send multipart/form-data 'file', JSON base64, or raw audio/*", "type": "invalid_request_error", "code": null, "param": null}} # There is no filename in this form, so the node derives the extension from # the Content-Type: audio/webm -> raw.webm, audio/wav -> raw.wav. Send the # type that matches the bytes. application/octet-stream is accepted too, but # it decodes as raw.octet-stream and only works on an ffmpeg node.
POST /v1/audio/speechJSON in, real audio/wav bytes out. Not base64, not a URL to fetch later,
not JSON with audio inside it — the body of a 200 is a RIFF/WAVE file you can write to disk or
hand to a player. That is a contract: if this endpoint cannot produce genuine audio it
returns an error, never silence and never a placeholder tone.
| field | type | notes |
|---|---|---|
| text | string | required, non-empty. input is accepted
as an alias so OpenAI-shaped code ports unchanged, but text is the name the
endpoint's own 400 message teaches — prefer it. |
| voice | string | optional. Voice ids are a property of the node's
engine, not of this page. Read GET /health →
voice.tts.default_voice and use that until you have a reason not to. |
| speed | number | optional, 1.0 is normal. Like
voice, whether it does anything is a property of the node's engine — listen to
the output before you build a UI slider on it. |
| model | string | accepted and ignored, exactly as on
/v1/audio/transcriptions. A node has one voice engine; there is no routing and
no tier ladder here. The OpenAI SDKs require the field, so it is tolerated. |
The whole clip is synthesised before the first byte is sent, so latency scales with the
length of text — a paragraph is not a sentence. REST speech does not stream and
cannot be barged into. If a person needs to interrupt the machine mid-sentence, that is the
socket tier's job, not this endpoint's.
Advisory response headers, when the node sets them — read them if present, never require them:
# HTTP/1.1 200 OK content-type: audio/wav x-engine: kokoro-onnx x-voice: af_heart x-seconds: 2.4 # duration of the returned audio x-synth-ms: 381.6 # wall-clock to make it; compare with x-seconds for headroom
And the seam, identical in shape to the STT one:
# HTTP/1.1 501 Not Implemented {"error": { "message": "TTS seam not wired on this node: kokoro model dir missing — the client must fall back to the browser Web Speech API (speechSynthesis)", "type": "not_implemented", "code": "tts_not_wired", "param": null }}
There are two failure codes here, not one. 501 tts_not_wired is "this node
has no voice, and never will during this process" — never retry it. 502 tts_failed
is "the voice is wired and this synthesis blew up" — a model file half-loaded, a phoneme the
engine hated, an ONNX session that died. Both mean the words did not get spoken by the
server, and both take the same fallback. A client that only branches on 501 will throw a raw
exception the first time it meets a 502 and the user will get silence with no explanation.
The bug this section exists to prevent: a 200 is audio, every error is JSON, and a
client that writes the response body to reply.wav without checking produces a
"corrupt audio file" that is really an error message. Check the status, then check that the
bytes start with RIFF. Both, every time.
On 501/502 the fallback is device synthesis — speechSynthesis in a browser,
flutter_tts or the platform TTS in an app. Unlike the STT seam this fallback is
genuinely equivalent: the words still get spoken, just in the device's voice. On a server with
no speaker, propagate the seam and let the caller decide whether to show text instead.
curl -sS -X POST "$BASE/v1/audio/speech" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ -d '{"text":"Wind is twelve knots, gusting eighteen."}' \ -D headers.txt -o reply.wav # -o writes WHATEVER came back, including a JSON error envelope. Verify: head -c 4 reply.wav; echo # RIFF -> real audio, play it # {"er -> an error envelope with a .wav name; `cat reply.wav` to read it # -E: POSIX ERE, portable. The BRE form '^\(content-type\|x-\)' is a GNU # extension and silently matches nothing on BSD/macOS grep. grep -iE '^(content-type|x-)' headers.txt # content-type: audio/wav # x-engine: kokoro-onnx # x-voice: af_heart # x-seconds: 2.4 # x-synth-ms: 381.6 # Pick a voice the NODE serves — never one copied from a doc page: curl -sS "$BASE/health" | jq '.voice.tts | {wired, engine, default_voice, error}' # {"wired": true, "engine": "kokoro-onnx", "default_voice": "af_heart", "error": null} # An empty text is a 400, not an empty wav: curl -sS -X POST "$BASE/v1/audio/speech" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" -d '{"text":" "}' {"error": {"message": "'text' is required (example: {\"text\": \"hello\", \"voice\": \"af_heart\"})", "type": "invalid_request_error", "code": null, "param": null}}
# pip install httpx (Python 3.10+ for the `str | None` annotation) import os import httpx BASE = os.environ.get("HAWKTALK_BASE", "https://api.hawktalk.ai") KEY = os.environ["HAWKTALK_API_KEY"] class TtsUnavailable(Exception): """501 (no voice wired) or 502 (wired voice failed). One type, because the caller's answer is the same for both: speak with the device, or show the text. `.status` is kept so a retry policy can tell the permanent seam from the transient failure. """ def __init__(self, status: int, message: str): self.status = status super().__init__(f"{status}: {message}") def speak(text: str, path: str = "reply.wav", voice: str | None = None) -> dict: body: dict = {"text": text} if voice: body["voice"] = voice # omit entirely to take the node's default # Synthesis is one blocking round trip for the WHOLE clip. A long # paragraph on a CPU-only node can take tens of seconds — size the # timeout by the text you actually send. r = httpx.post(f"{BASE}/v1/audio/speech", headers={"Authorization": f"Bearer {KEY}"}, json=body, timeout=120.0) if r.status_code in (501, 502): # Errors are JSON even though a 200 is not. Guard the parse anyway: # a proxy can return an HTML 502 that never touched the gateway. try: msg = r.json()["error"]["message"] except Exception: msg = r.text[:200] raise TtsUnavailable(r.status_code, msg) r.raise_for_status() # Belt and braces: the header says what it is, the magic says what it is. ctype = r.headers.get("content-type", "") if not ctype.startswith("audio/") or not r.content.startswith(b"RIFF"): raise RuntimeError(f"expected wav bytes, got {ctype}: {r.content[:120]!r}") with open(path, "wb") as f: f.write(r.content) # Headers are strings or None. Left as-is on purpose: a node that reports # no duration reports None, and None is not 0.0 seconds of audio. return {"path": path, "bytes": len(r.content), "engine": r.headers.get("x-engine", "unknown"), "voice": r.headers.get("x-voice", "unknown"), "seconds": r.headers.get("x-seconds"), "synth_ms": r.headers.get("x-synth-ms")} if __name__ == "__main__": try: print(speak("Wind is twelve knots, gusting eighteen.")) except TtsUnavailable as e: # Nothing here can make audio honestly, so it does not try. print(f"no cloud voice ({e}); print the line instead")
// Plays HawkTalk audio when the node has a voice, falls back to the device // voice when it does not. There is no third path where audio is invented. const BASE = "https://api.hawktalk.ai"; export async function speak(text: string, key: string): Promise<"cloud" | "device" | "silent"> { const r = await fetch(`${BASE}/v1/audio/speech`, { method: "POST", headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" }, body: JSON.stringify({ text }), }); // 501 tts_not_wired (no voice at all) and 502 tts_failed (voice wired, this // synthesis died) take the SAME fallback. Branching on 501 alone is how a // half-broken node turns into unexplained silence. if (r.status === 501 || r.status === 502) { console.warn(`tts ${r.status}:`, await r.text()); return deviceSpeak(text); } if (!r.ok) throw new Error(`speech ${r.status}: ${await r.text()}`); const blob = await r.blob(); // A JSON error with a 200 should be impossible, but a proxy in the middle // can make it possible. One cheap check beats one silent bug. if (!blob.type.startsWith("audio/")) throw new Error(`expected audio, got ${blob.type}`); const url = URL.createObjectURL(blob); const el = new Audio(url); // Revoke on end AND on error, or every reply leaks a blob for the session. el.onended = el.onerror = () => URL.revokeObjectURL(url); try { // Autoplay policy: this must run inside a user gesture the first time, // or play() rejects and you will think the audio was bad. await el.play(); return "cloud"; } catch { URL.revokeObjectURL(url); return deviceSpeak(text); } } function deviceSpeak(text: string): "device" | "silent" { if (!("speechSynthesis" in window)) return "silent"; // show the text; say nothing window.speechSynthesis.cancel(); // don't queue behind a stale line window.speechSynthesis.speak(new SpeechSynthesisUtterance(text)); return "device"; }
// pubspec: http: ^1.2.0 (and flutter_tts for the fallback leg) import 'dart:convert'; import 'dart:typed_data'; import 'package:http/http.dart' as http; /// 501 (no voice wired) or 502 (wired voice failed). One type: the caller's /// answer is the same for both. [status] survives so a retry policy can tell /// the permanent seam from the transient failure. class TtsUnavailable implements Exception { final int status; final String message; TtsUnavailable(this.status, this.message); @override String toString() => 'TtsUnavailable($status): $message'; } /// Returns real WAV bytes, or throws [TtsUnavailable] so the caller can hand /// the line to flutter_tts. It never returns empty bytes to paper over a seam. Future<Uint8List> synthesize( String text, { required String base, required String apiKey, String? voice, http.Client? client, }) async { final owned = client == null; final c = client ?? http.Client(); try { final res = await c .post( Uri.parse('$base/v1/audio/speech'), headers: { 'authorization': 'Bearer $apiKey', 'content-type': 'application/json', }, body: jsonEncode({'text': text, if (voice != null) 'voice': voice}), ) .timeout(const Duration(seconds: 120)); if (res.statusCode == 501 || res.statusCode == 502) { String msg; try { final err = jsonDecode(res.body) as Map<String, dynamic>; msg = err['error']?['message'] as String? ?? 'tts unavailable'; } catch (_) { // A proxy 502 is often HTML, not our envelope. Don't crash on it. msg = res.body.length > 200 ? res.body.substring(0, 200) : res.body; } throw TtsUnavailable(res.statusCode, msg); } if (res.statusCode != 200) { throw Exception('speech ${res.statusCode}: ${res.body}'); } final bytes = res.bodyBytes; // 'RIFF' == 0x52 0x49 0x46 0x46. res.body on binary is mojibake, so check // the BYTES, not the string. final looksWav = bytes.length >= 4 && bytes[0] == 0x52 && bytes[1] == 0x49 && bytes[2] == 0x46 && bytes[3] == 0x46; if (!looksWav) { throw Exception('expected wav bytes, got ${res.headers['content-type']}'); } return bytes; } finally { if (owned) c.close(); } } // Call site: one voice, two possible bodies, no invented audio. // // try { // final wav = await synthesize(line, base: base, apiKey: key); // await player.play(BytesSource(wav)); // package:audioplayers // } on TtsUnavailable { // await FlutterTts().speak(line); // the device says it // }
POST /v1/embeddingsThe OpenAI shape, unchanged: {"model", "input"} in,
{"object":"list","data":[…],"model","usage"} out. input takes a
string or an array of strings — batch it. Sixty-four short strings in one array is one
round trip; sixty-four calls is sixty-four, and the difference is the whole cost of indexing a
corpus.
Embeddings are served by a separate backend from chat. The chat tiers cannot embed, so
the tier vocabulary does not apply here and the model you send is not a router
instruction — the node has one embedding model and it tells you which in the response. Send
"auto", then read the response's top-level model and store it beside every vector
you keep: dimension and geometry belong to that model, and mixing two of them in one index
produces confident nonsense. When the node has no embedding backend at all you get a
503, and GET /health → embeddings.wired tells you before
you try.
| field | type | notes |
|---|---|---|
| input | string | string[] | required. No empty strings — an empty
member is a 400 with param: "input". Token arrays are accepted
only where the backend supports them; GET /health →
embeddings.accepts_tokens is the answer for the node in front of you. |
| model | string | required by the OpenAI contract.
"auto" is fine; the response reports what actually ran. |
| encoding_format | string | "float" is the default and
the only one every node can serve. "base64" — packed little-endian float32 —
is part of the OpenAI contract but is a per-node backend capability, so probe it
before you depend on it. And note what it changes: data[].embedding comes back
as a string, not number[], which will break every cosine routine on this
page unless you unpack it first. |
| dimensions | int | optional truncation, per-node. Not every
embedding backend implements truncate-and-renormalize, and a backend that does not will
simply hand you its native vector. Send it, then verify the length you actually got
(.data[0].embedding | length) before you pin an index schema to it. Never
assume a shorter vector came back, and never expect padding in the other direction. |
| usage.prompt_tokens | int | null | may be null. A backend that reports no token count is reported as null, not as 0. Sum it defensively. |
curl -sS "$BASE/v1/embeddings" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"auto","input":["the tide turns at 06:12","the gate code is 4417"]}' # The "…382 more" below is an ELISION for this page, not a value the API # sends. A real body is 384 floats per row and nothing else. { "object": "list", "data": [ {"object": "embedding", "index": 0, "embedding": [-0.0231, 0.0714, "…382 more"]}, {"object": "embedding", "index": 1, "embedding": [0.0118, -0.0442, "…382 more"]} ], "model": "all-MiniLM-L6-v2", "usage": {"prompt_tokens": 14, "total_tokens": 14} } # That model id came from ONE node's embedding backend. Yours may differ — # read it from the response, never from this page. Check the dimension the # same way, once, and pin your index schema to it: curl -sS "$BASE/v1/embeddings" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"auto","input":"dimension probe"}' \ | jq '{model, dims: (.data[0].embedding | length), usage}' # {"model": "all-MiniLM-L6-v2", "dims": 384, "usage": {"prompt_tokens": 2, "total_tokens": 2}} # Same probe is how you test `dimensions` support instead of assuming it: # if dims comes back 384 when you asked for 256, this node ignored you. curl -sS "$BASE/v1/embeddings" \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"auto","input":"dimension probe","dimensions":256}' \ | jq '.data[0].embedding | length' # Is the seam even wired on this node? (no auth needed) curl -sS "$BASE/health" | jq .embeddings # {"wired": true, "backend": "RetrieveServerClient", "model": "all-MiniLM-L6-v2", "accepts_tokens": false} # unwired: {"wired": false, "reason": "no embedding backend configured (…)"}
Embeddings on their own answer nothing. The pattern that makes them useful is three steps:
embed your corpus once and keep the vectors; embed the question at query time and
take the nearest few chunks; prepend those chunks to the chat call as context the model
must answer from. The model stays on "auto" — grounding is a prompt-construction
job, not a model-selection one.
Two costs to keep honest about. The retrieved prefix is prompt tokens you pay on
every turn, so keep k at 3–5 and truncate each chunk — an unbounded prefix
is the most common reason a cheap turn stops being cheap. And re-embedding your whole corpus on
every process start is a bill you did not need: vectors are stable for a given
model, so persist them with that id and re-embed only what changed.
The two examples below embed the corpus inline so they run as one paste. That is the one thing you must not copy. In a real service the corpus embed happens in an indexing job and the query path only ever embeds the question.
# pip install httpx — no numpy, so this drops into any service. # Python 3.10+ (the `int | None` annotations are evaluated at def time). import math import os import httpx BASE = os.environ.get("HAWKTALK_BASE", "https://api.hawktalk.ai") KEY = os.environ["HAWKTALK_API_KEY"] HEADERS = {"Authorization": f"Bearer {KEY}"} def embed(texts: list[str], client: httpx.Client) -> tuple[list[list[float]], str, int | None]: """One call for the whole batch. Returns (vectors, model_id, prompt_tokens|None).""" # No encoding_format: the default "float" gives number[], which is what # cosine() below expects. "base64" would return a string and break it. r = client.post(f"{BASE}/v1/embeddings", headers=HEADERS, json={"model": "auto", "input": texts}) if r.status_code == 503: raise RuntimeError(f"no embedding backend on this node: {r.json()['error']['message']}") r.raise_for_status() body = r.json() # data comes back with an explicit `index`. Sort by it rather than trusting # array order — the contract promises the index, not the ordering. rows = sorted(body["data"], key=lambda d: d["index"]) vectors = [row["embedding"] for row in rows] # May be None. It stays None — see accumulate() below. tokens = (body.get("usage") or {}).get("prompt_tokens") return vectors, body["model"], tokens def cosine(a: list[float], b: list[float]) -> float: # Vectors from this endpoint are usually L2-normalized already, but "usually" # is not a contract. Dividing by the real norms costs nothing and is right # on every backend. dot = sum(x * y for x, y in zip(a, b)) na = math.sqrt(sum(x * x for x in a)) nb = math.sqrt(sum(y * y for y in b)) return 0.0 if na == 0 or nb == 0 else dot / (na * nb) def ground_and_ask(question: str, chunks: list[str], client: httpx.Client, k: int = 4, chunk_chars: int = 600) -> str: # 1) index — INLINE HERE ONLY so this file runs standalone. In production # this line lives in an indexing job and (vectors, model_id) are read back # from your store; re-embedding the corpus per question is the single most # expensive mistake in this pattern. corpus_vecs, corpus_model, _ = embed(chunks, client) # 2) retrieve — the question must be embedded by the SAME model as the # corpus. If your store's model_id differs from this one, re-index; do not # compare across models. q_vecs, q_model, _ = embed([question], client) if q_model != corpus_model: raise RuntimeError(f"index built with {corpus_model}, node now serves {q_model}") ranked = sorted( ((cosine(q_vecs[0], v), chunks[i]) for i, v in enumerate(corpus_vecs)), reverse=True, )[:k] # 3) ground — a bounded prefix. Truncating each chunk is what keeps the # prompt (and the bill) from growing with your corpus. context = "\n\n".join(f"[{i + 1}] {text[:chunk_chars]}" for i, (_score, text) in enumerate(ranked)) r = client.post(f"{BASE}/v1/chat/completions", headers=HEADERS, json={ "model": "auto", # the router picks the tier; grounding is not a tier "messages": [ {"role": "system", "content": ("Answer only from the numbered context. " "If it is not there, say you do not know.")}, {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}, ], }) r.raise_for_status() return r.json()["choices"][0]["message"]["content"] def accumulate(total: int, unknown: int, tokens: int | None) -> tuple[int, int]: """Add a possibly-null token count without lying about it.""" # `total += tokens or 0` silently turns "we don't know" into "it was free". # Counting the unknowns separately keeps the number you report defensible. return (total, unknown + 1) if tokens is None else (total + tokens, unknown) if __name__ == "__main__": docs = [ "The tide turns at 06:12; launch window closes at 07:40.", "The gate code is 4417. It rotates on the first of the month.", "Fuel is cash only at the north dock.", ] with httpx.Client(timeout=60.0) as client: print(ground_and_ask("When do I have to be off the water?", docs, client))
// Node 18+, no dependencies. const BASE = process.env.HAWKTALK_BASE ?? "https://api.hawktalk.ai"; const KEY = process.env.HAWKTALK_API_KEY!; const HEADERS = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" }; type EmbeddingResponse = { data: { index: number; embedding: number[] }[]; model: string; usage?: { prompt_tokens: number | null; total_tokens: number | null }; }; export type Indexed = { vectors: number[][]; model: string; promptTokens: number | null }; export async function embed(input: string[]): Promise<Indexed> { const r = await fetch(`${BASE}/v1/embeddings`, { method: "POST", headers: HEADERS, // One request for the whole batch — N requests is N round trips. // No encoding_format: the default "float" is what `number[]` above means. body: JSON.stringify({ model: "auto", input }), }); if (!r.ok) throw new Error(`embeddings ${r.status}: ${await r.text()}`); const body = await r.json() as EmbeddingResponse; const vectors = [...body.data] .sort((a, b) => a.index - b.index) .map((d) => d.embedding); // `?? null`, never `?? 0`: an unreported count is unknown, not zero. return { vectors, model: body.model, promptTokens: body.usage?.prompt_tokens ?? null }; } function cosine(a: number[], b: number[]): number { let dot = 0, na = 0, nb = 0; for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] ** 2; nb += b[i] ** 2; } return na === 0 || nb === 0 ? 0 : dot / (Math.sqrt(na) * Math.sqrt(nb)); } export async function groundedAnswer(question: string, chunks: string[], k = 4) { // Inline for the demo only — in a service this comes from your vector store. const corpus = await embed(chunks); const query = await embed([question]); // Same-model check. Vectors from two different models are not comparable, // and nothing downstream will tell you they were mixed. if (corpus.model !== query.model) { throw new Error(`index model ${corpus.model} != query model ${query.model}`); } const context = chunks .map((text, i) => ({ text, score: cosine(query.vectors[0], corpus.vectors[i]) })) .sort((a, b) => b.score - a.score) .slice(0, k) .map((hit, i) => `[${i + 1}] ${hit.text.slice(0, 600)}`) // bounded prefix .join("\n\n"); const r = await fetch(`${BASE}/v1/chat/completions`, { method: "POST", headers: HEADERS, body: JSON.stringify({ model: "auto", messages: [ { role: "system", content: "Answer only from the numbered context. If it is not there, say you do not know." }, { role: "user", content: `Context:\n${context}\n\nQuestion: ${question}` }, ], }), }); if (!r.ok) throw new Error(`chat ${r.status}: ${await r.text()}`); const body = await r.json() as { choices: { message: { content: string } }[]; usage?: { prompt_tokens: number | null }; }; // Report the unknown as unknown. A dashboard that shows 0 tokens for a // grounded turn is worse than one that shows a gap. console.log("prompt_tokens:", body.usage?.prompt_tokens ?? "unknown"); return body.choices[0].message.content; }
// pubspec: http: ^1.2.0 import 'dart:convert'; import 'dart:io'; import 'dart:math' as math; import 'package:http/http.dart' as http; final _base = Platform.environment['HAWKTALK_BASE'] ?? 'https://api.hawktalk.ai'; final _key = Platform.environment['HAWKTALK_API_KEY']!; final class Indexed { final List<List<double>> vectors; final String model; final int? promptTokens; const Indexed({ required this.vectors, required this.model, required this.promptTokens, }); } Future<Indexed> embed(List<String> input, {http.Client? client}) async { final httpClient = client ?? http.Client(); final shouldClose = client == null; try { // One request for the whole batch — N requests is N round trips. // No encoding_format: default "float" is what List<double> expects. final res = await httpClient.post( Uri.parse('$_base/v1/embeddings'), headers: { 'Authorization': 'Bearer $_key', 'Content-Type': 'application/json', }, body: jsonEncode({'model': 'auto', 'input': input}), ); final body = jsonDecode(utf8.decode(res.bodyBytes)) as Map<String, dynamic>; if (res.statusCode == 503) { final err = body['error'] as Map<String, dynamic>?; throw HttpException( 'no embedding backend on this node: ${err?["message"] ?? res.body}'); } if (res.statusCode != 200) { final err = body['error'] as Map<String, dynamic>?; throw HttpException( 'embeddings ${res.statusCode} ${err?["code"] ?? "unknown"}: ${err?["message"] ?? res.body}'); } // data comes back with explicit `index`. Sort by it rather than trusting // array order — the contract promises the index, not the ordering. final rawData = (body['data'] as List<dynamic>).cast<Map<String, dynamic>>(); final sortedRows = [...rawData]..sort( (a, b) => (a['index'] as num).compareTo(b['index'] as num), ); final vectors = sortedRows .map((row) => (row['embedding'] as List<dynamic>) .map((v) => (v as num).toDouble()) .toList()) .toList(); final usage = body['usage'] as Map<String, dynamic>?; // `?? null`, never `?? 0`: an unreported count is unknown, not zero. final int? promptTokens = usage?['prompt_tokens'] as int?; return Indexed( vectors: vectors, model: body['model'] as String, promptTokens: promptTokens, ); } finally { if (shouldClose) httpClient.close(); } } double cosine(List<double> a, List<double> b) { // Vectors from this endpoint are usually L2-normalized already, but "usually" // is not a contract. Dividing by the real norms costs nothing and is right on every backend. var dot = 0.0; var na = 0.0; var nb = 0.0; for (var i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i]; } return (na == 0.0 || nb == 0.0) ? 0.0 : dot / (math.sqrt(na) * math.sqrt(nb)); } Future<String> groundAndAsk( String question, List<String> chunks, { http.Client? client, int k = 4, int chunkChars = 600, }) async { final httpClient = client ?? http.Client(); final shouldClose = client == null; try { // 1) index — INLINE HERE ONLY so this file runs standalone. In production // this line lives in an indexing job and (vectors, model) are read back // from your store; re-embedding the corpus per question is the single most // expensive mistake in this pattern. final corpus = await embed(chunks, client: httpClient); // 2) retrieve — the question must be embedded by the SAME model as the corpus. final query = await embed([question], client: httpClient); if (query.model != corpus.model) { throw StateError( 'index model ${corpus.model} != query model ${query.model}'); } final queryVector = query.vectors.first; final scored = <({double score, String text})>[]; for (var i = 0; i < chunks.length; i++) { scored.add((score: cosine(queryVector, corpus.vectors[i]), text: chunks[i])); } scored.sort((a, b) => b.score.compareTo(a.score)); // 3) ground — a bounded prefix. Truncating each chunk is what keeps the // prompt (and the bill) from growing with your corpus. final topK = scored.take(k); final context = topK .toList() .asMap() .entries .map((entry) { final i = entry.key; final item = entry.value; final truncated = item.text.length > chunkChars ? item.text.substring(0, chunkChars) : item.text; return '[${i + 1}] $truncated'; }) .join('\n\n'); final res = await httpClient.post( Uri.parse('$_base/v1/chat/completions'), headers: { 'Authorization': 'Bearer $_key', 'Content-Type': 'application/json', }, body: jsonEncode({ 'model': 'auto', // the router picks the tier; grounding is not a tier 'messages': [ { 'role': 'system', 'content': 'Answer only from the numbered context. If it is not there, say you do not know.', }, { 'role': 'user', 'content': 'Context:\n$context\n\nQuestion: $question', }, ], }), ); final body = jsonDecode(utf8.decode(res.bodyBytes)) as Map<String, dynamic>; if (res.statusCode != 200) { final err = body['error'] as Map<String, dynamic>?; throw HttpException( 'chat ${res.statusCode} ${err?["code"] ?? "unknown"}: ${err?["message"] ?? res.body}'); } final usage = body['usage'] as Map<String, dynamic>?; // Report the unknown as unknown. A dashboard that shows 0 tokens for a // grounded turn is worse than one that shows a gap. final promptTokens = usage?['prompt_tokens']; print('prompt_tokens: ${promptTokens ?? "unknown"}'); final choices = body['choices'] as List<dynamic>; final first = choices.first as Map<String, dynamic>; final message = first['message'] as Map<String, dynamic>; return message['content'] as String; } finally { if (shouldClose) httpClient.close(); } } Future<void> main() async { final docs = [ 'The tide turns at 06:12; launch window closes at 07:40.', 'The gate code is 4417. It rotates on the first of the month.', 'Fuel is cash only at the north dock.', ]; final client = http.Client(); try { final answer = await groundAndAsk( 'When do I have to be off the water?', docs, client: client, ); print(answer); } finally { client.close(); } }
Every error on this API is the same envelope, on every endpoint:
{"error": {"message": "…", "type": "…", "code": "…", "param": "…"}}
Branch on the HTTP status first and the code second. code is
genuinely null on several errors — a plain 400, a 413, a
415 — so a client that switches on code alone falls through its own
default case on the three errors it is most likely to hit. param is set when the
fault is a specific field ("input", "dimensions", "model")
and null otherwise.
| status | code | type | what to do |
|---|---|---|---|
| 400 | null (usually) | invalid_request_error |
Your request is wrong. Never retry. Read param, fix the field. |
| 401 | invalid_api_key | invalid_request_error |
Bad or missing key. Go dark — never retry. A retry loop on a bad key is how a key gets locked out. |
| 404 | model_not_found | invalid_request_error |
That id or pin is not served here. Fall back to "auto" once, then call
GET /v1/models and read what this node has. |
| 413 | null | invalid_request_error |
Audio over the node's cap. Split the clip; retrying is pointless. |
| 415 | null | invalid_request_error |
Content-Type the audio endpoint does not parse. Fix the header, not the bytes. |
| 429 | rate_limit_exceeded | rate_limit_error |
Carries Retry-After, in seconds. Honour it — do not substitute
your own backoff, and add jitter so a fleet does not resynchronise on it. |
| 501 | stt_not_wired / tts_not_wired | not_implemented |
The seam is absent, not busy. Never retry. Fall back to device speech (§01.6, §01.7) and never fabricate audio or a transcript. |
| 502 | backend_failed / stt_failed / tts_failed | backend_error |
A wired backend answered badly. On chat: step down one tier and retry once — repeating the same request against the same rung usually reproduces it. On audio: there is no ladder, so re-encode once (16 kHz mono wav) and otherwise take the same device fallback as a 501. |
| 503 | model_unavailable / backend_unavailable / compute_not_wired | backend_error |
Capacity or wiring, not your request. Retry on "auto" after a
short backoff — the router will pick something that is actually up. |
The tier ladder, cheap → dear: ouro · quick · dank ·
think · cloud (aliases self, route,
ouromega, live, specialist, t0–t3).
These are the literal values of model — "model": "quick", bare,
with no prefix and no namespace. There is no tier: form; inventing one earns a
404 model_not_found. "Stepping down" means moving toward ouro.
"auto" is the shipping default: it runs the router per request, and it is the
only value every node accepts. Treat a pin as a temporary, deliberate escalation — and if a node
answers 404 model_not_found to a tier pin, that node is not behind the router: stop
pinning and go back to "auto". Never hardcode a concrete registry id off this page;
call GET /v1/models and read what your node serves.
The rule that outranks all of this: never coerce an unknown to a number.
usage.prompt_tokens, x_timing.ttft_ms, x_stt.whisper_ms
and friends are null or "unknown" when the node did not measure them.
tokens or 0, ?? 0, float(x or 0) — each one converts "we
don't know" into "it was free and instant", and it converts silently, in the one direction that
flatters you.
# pip install httpx from __future__ import annotations import os import random import time import httpx # cheap -> dear. These are the literal `model` values: bare names, no prefix. # "Stepping down" walks left. LADDER = ("ouro", "quick", "dank", "think", "cloud") class HawkTalkError(Exception): """One error type carrying the whole envelope, so callers can branch.""" def __init__(self, status: int, body: dict | None, text: str = ""): err = (body or {}).get("error") or {} # code and param are None on plenty of real errors. Keep them None. self.status = status self.code = err.get("code") self.type = err.get("type") self.param = err.get("param") self.message = err.get("message") or text or f"HTTP {status}" super().__init__(f"{status} {self.code or self.type or 'error'}: {self.message}") def _retry_after(res: httpx.Response, attempt: int) -> float: """Seconds to wait. The server's number wins; ours is only the fallback.""" raw = res.headers.get("Retry-After") if raw: try: # This gateway sends integer seconds. (HTTP also permits an # HTTP-date; if you meet one, parse it rather than guessing.) return max(0.0, float(raw.strip())) except ValueError: pass return min(30.0, 2.0 ** attempt) def _step_down(model: str) -> str | None: """One rung cheaper, or None when there is nowhere left to go.""" # A tier pin IS the bare name — "quick", not "tier:quick". Any prefix you # bolt on becomes an unknown model id and comes back 404. if model in LADDER: i = LADDER.index(model) return None if i == 0 else LADDER[i - 1] # "auto" (or a concrete registry id, or an alias) just failed, so the rung # it chose is the one that's broken. Drop to the cheapest rung, which is # the most likely to be alive AND the cheapest place to discover it isn't. return LADDER[0] class HawkTalk: """A chat client that degrades instead of failing the turn. 429 -> wait exactly as long as the server said. 502 -> step down a tier, once per rung. 503 -> hand it back to the router ("auto") and back off. 401/400/404-on-auto -> stop. Retrying those is how you get locked out or spin forever on a request that will never become valid. Non-streaming only: `chat()` parses one JSON body. For stream=True the reply is SSE `chat.completion.chunk` lines terminated by `data: [DONE]`, which needs a line reader, not `.json()` — see the streaming section. """ def __init__(self, base: str | None = None, key: str | None = None, max_attempts: int = 5, timeout: float = 120.0): self.base = (base or os.environ.get("HAWKTALK_BASE") or "https://api.hawktalk.ai").rstrip("/") self.key = key or os.environ["HAWKTALK_API_KEY"] self.max_attempts = max_attempts # One pooled client: TLS handshakes dominate short calls. self.http = httpx.Client(timeout=timeout, headers={"Authorization": f"Bearer {self.key}"}) def close(self) -> None: self.http.close() def chat(self, messages: list[dict], model: str = "auto", **extra) -> dict: if extra.get("stream"): raise ValueError("chat() is the non-streaming path; use the SSE reader") current = model for attempt in range(self.max_attempts): res = self.http.post(f"{self.base}/v1/chat/completions", json={"model": current, "messages": messages, **extra}) if res.status_code == 200: return res.json() try: body = res.json() except ValueError: body = None err = HawkTalkError(res.status_code, body, res.text) # --- fatal: fix the caller, not the call --------------------- if err.status in (400, 401, 413, 415, 501): raise err # --- 404: a pin this node doesn't serve ---------------------- if err.status == 404: if current == "auto": # Even the router said no. Nothing to fall back to. raise err current = "auto" continue # --- 429: the server named the price of waiting -------------- if err.status == 429: delay = _retry_after(res, attempt) # Jitter: without it, every worker that got 429 at the same # second retries at the same second. time.sleep(delay + random.uniform(0, 0.5)) continue # --- 502: that backend is sick; try a cheaper one ------------ if err.status == 502: nxt = _step_down(current) if nxt is None: raise err # bottom rung failed: the node is down current = nxt continue # --- 503: capacity/wiring; let the router choose ------------- if err.status == 503: current = "auto" time.sleep(min(30.0, 2.0 ** attempt) + random.uniform(0, 0.5)) continue raise err # unknown status: fail loud, do not guess a policy raise HawkTalkError(503, {"error": { "message": f"gave up after {self.max_attempts} attempts", "type": "backend_error", "code": "backend_unavailable"}}) def read_usage(body: dict) -> dict: """Pull the numbers out WITHOUT inventing any.""" usage = body.get("usage") or {} timing = body.get("x_timing") or {} # Every one of these may be None or the string "unknown". Both mean the # same thing and neither is 0. Pass them through; let the sink decide how # to render a gap. return { "prompt_tokens": usage.get("prompt_tokens"), "completion_tokens": usage.get("completion_tokens"), "total_tokens": usage.get("total_tokens"), "ttft_ms": timing.get("ttft_ms"), "total_ms": timing.get("total_ms"), "backend": timing.get("backend", "unknown"), } if __name__ == "__main__": hawk = HawkTalk() try: out = hawk.chat([{"role": "user", "content": "One line: why is the sky blue?"}]) print(out["choices"][0]["message"]["content"]) print(read_usage(out)) except HawkTalkError as e: # status/code/param are all on the exception — log the whole envelope. print(f"failed: status={e.status} code={e.code} param={e.param}: {e.message}") finally: hawk.close()
// pubspec: http: ^1.2.0 import 'dart:convert'; import 'dart:io'; import 'dart:math' as math; import 'package:http/http.dart' as http; // cheap -> dear. These are the literal `model` values: bare names, no prefix. // "Stepping down" walks toward index 0. const ladder = ['ouro', 'quick', 'dank', 'think', 'cloud']; /// One error type carrying the whole envelope, so callers can branch. final class HawkTalkException implements Exception { final int status; final String? code; final String? type; final String? param; final String message; const HawkTalkException({ required this.status, required this.message, this.code, this.type, this.param, }); factory HawkTalkException.fromResponse(int status, Map<String, dynamic>? body, String rawText) { final err = body?['error'] as Map<String, dynamic>?; final code = err?['code'] as String?; final type = err?['type'] as String?; final param = err?['param'] as String?; final msg = (err?['message'] as String?) ?? (rawText.isNotEmpty ? rawText : 'HTTP $status'); return HawkTalkException( status: status, code: code, type: type, param: param, message: msg, ); } @override String toString() { final label = code ?? type ?? 'error'; return 'hawktalk $status $label: $message'; } } Duration _retryAfter(http.Response res, int attempt, math.Random rng) { final header = res.headers['retry-after']; if (header != null) { final secs = num.tryParse(header.trim()); if (secs != null && secs >= 0) { return Duration(milliseconds: (secs * 1000).round()); } } return _backoff(attempt, rng); } Duration _backoff(int attempt, math.Random rng) { final secs = math.min(30.0, math.pow(2.0, attempt).toDouble()); final jitterMs = rng.nextInt(500); return Duration(milliseconds: (secs * 1000).round() + jitterMs); } /// Returns the next cheaper rung, or null when there is nowhere left to go. /// A tier pin is the bare name ("quick"), never "tier:quick" — this API has no /// prefixed form, and sending one earns a 404 model_not_found. String? _stepDown(String model) { final idx = ladder.indexOf(model); if (idx > 0) return ladder[idx - 1]; if (idx == 0) return null; // already cheapest: nowhere left to fall // "auto", an alias, or a concrete id failed: restart from the cheapest rung return ladder[0]; } final class HawkTalk { final String base; final String key; final int maxAttempts; final http.Client _client; final math.Random _rng = math.Random(); HawkTalk({ String? base, String? key, this.maxAttempts = 5, http.Client? client, }) : base = (base ?? Platform.environment['HAWKTALK_BASE'] ?? 'https://api.hawktalk.ai') .replaceAll(RegExp(r'/+$'), ''), key = key ?? Platform.environment['HAWKTALK_API_KEY']!, _client = client ?? http.Client(); void close() => _client.close(); Future<Map<String, dynamic>> chat( List<Map<String, dynamic>> messages, { String model = 'auto', Map<String, dynamic>? extra, }) async { var current = model; HawkTalkException? lastErr; for (var attempt = 0; attempt < maxAttempts; attempt++) { final payload = <String, dynamic>{ 'model': current, 'messages': messages, if (extra != null) ...extra, }; http.Response res; try { res = await _client.post( Uri.parse('$base/v1/chat/completions'), headers: { 'Authorization': 'Bearer $key', 'Content-Type': 'application/json', }, body: jsonEncode(payload), ); } catch (e) { // Transport error (DNS, TLS, socket drop). Back off and retry. if (attempt == maxAttempts - 1) rethrow; await Future<void>.delayed(_backoff(attempt, _rng)); continue; } if (res.statusCode == 200) { return jsonDecode(utf8.decode(res.bodyBytes)) as Map<String, dynamic>; } Map<String, dynamic>? body; try { body = jsonDecode(utf8.decode(res.bodyBytes)) as Map<String, dynamic>?; } catch (_) { body = null; } final err = HawkTalkException.fromResponse(res.statusCode, body, res.body); lastErr = err; switch (res.statusCode) { case 400 || 401 || 413 || 415 || 501: // Fatal: bad request, invalid key, clip over cap, bad content-type, unwired seam. throw err; case 404: // model_not_found if (current == 'auto') throw err; // router itself said no current = 'auto'; // node does not serve the pinned tier continue; case 429: // rate_limit_exceeded final delay = _retryAfter(res, attempt, _rng); final jitter = Duration(milliseconds: _rng.nextInt(500)); await Future<void>.delayed(delay + jitter); continue; case 502: // backend_failed final next = _stepDown(current); if (next == null) throw err; // bottom rung failed: node is down current = next; continue; case 503: // model_unavailable current = 'auto'; // hand tier choice back to the router await Future<void>.delayed(_backoff(attempt, _rng)); continue; default: throw err; // unknown status: fail loud } } throw lastErr ?? const HawkTalkException( status: 503, code: 'backend_unavailable', type: 'backend_error', message: 'gave up after max attempts', ); } } String tokensOrUnknown(int? v) => v == null ? 'unknown' : v.toString(); (int total, int unknown) addKnown((int, int) acc, int? v) => v == null ? (acc.$1, acc.$2 + 1) : (acc.$1 + v, acc.$2); Future<void> main() async { final hawk = HawkTalk(); try { final out = await hawk.chat([ {'role': 'user', 'content': 'One line: why is the sky blue?'}, ], model: 'auto'); final choices = out['choices'] as List<dynamic>; final msg = (choices.first as Map<String, dynamic>)['message'] as Map<String, dynamic>; print(msg['content']); final usage = out['usage'] as Map<String, dynamic>?; final timing = out['x_timing'] as Map<String, dynamic>?; print('prompt=${tokensOrUnknown(usage?["prompt_tokens"] as int?)} ' 'completion=${tokensOrUnknown(usage?["completion_tokens"] as int?)} ' 'ttft_ms=${timing?["ttft_ms"] ?? "unknown"} ' 'backend=${timing?["backend"] ?? "unknown"}'); } on HawkTalkException catch (e) { // status/code/param are all on the exception — log the whole envelope. stderr.writeln('failed: status=${e.status} code=${e.code} param=${e.param}: ${e.message}'); } finally { hawk.close(); } }
// Go 1.21+. Stdlib only. package hawktalk import ( "bytes" "context" "encoding/json" "fmt" "io" "math" "math/rand" "net/http" "os" "strconv" "strings" "time" ) // ladder is cheap -> dear; stepping down walks toward index 0. These strings // are the literal `model` values — bare names, no "tier:" prefix. var ladder = []string{"ouro", "quick", "dank", "think", "cloud"} // APIError is the whole envelope. Code and Param are legitimately empty on // 400/413/415, so callers switch on Status first. type APIError struct { Status int `json:"-"` Message string `json:"message"` Type string `json:"type"` Code string `json:"code"` Param string `json:"param"` } func (e *APIError) Error() string { code := e.Code if code == "" { // The API sends a null code on 400/413/415. Print the gap as a gap. code = "(no code)" } return fmt.Sprintf("hawktalk %d %s: %s", e.Status, code, e.Message) } func parseAPIError(status int, raw []byte) *APIError { var env struct { Error APIError `json:"error"` } if err := json.Unmarshal(raw, &env); err != nil || env.Error.Message == "" { return &APIError{Status: status, Message: strings.TrimSpace(string(raw))} } env.Error.Status = status return &env.Error } // Usage mirrors the server's honesty: pointers, so an unmeasured value stays // nil instead of decoding to Go's zero value and reading as a real 0. type Usage struct { PromptTokens *int `json:"prompt_tokens"` CompletionTokens *int `json:"completion_tokens"` TotalTokens *int `json:"total_tokens"` } type ChatResponse struct { Choices []struct { Message struct { Content string `json:"content"` } `json:"message"` } `json:"choices"` Usage Usage `json:"usage"` XTiming map[string]any `json:"x_timing"` // values may be null or "unknown" XCompute map[string]any `json:"x_compute"` } type Client struct { Base string Key string HTTP *http.Client MaxAttempts int } func New() *Client { base := os.Getenv("HAWKTALK_BASE") if base == "" { base = "https://api.hawktalk.ai" } return &Client{ Base: strings.TrimRight(base, "/"), Key: os.Getenv("HAWKTALK_API_KEY"), HTTP: &http.Client{Timeout: 120 * time.Second}, MaxAttempts: 5, } } // retryAfter honours the server's number and only falls back to our own. func retryAfter(res *http.Response, attempt int) time.Duration { if v := res.Header.Get("Retry-After"); v != "" { if secs, err := strconv.Atoi(strings.TrimSpace(v)); err == nil && secs >= 0 { return time.Duration(secs) * time.Second } } return backoff(attempt) } func backoff(attempt int) time.Duration { d := time.Duration(math.Min(30, math.Pow(2, float64(attempt)))) * time.Second // Jitter, so a fleet that got limited together doesn't retry together. return d + time.Duration(rand.Int63n(int64(500*time.Millisecond))) } // stepDown returns the next cheaper rung, or "" when there is none. // A tier pin is the bare name ("quick"), never "tier:quick" — this API has no // prefixed form, and sending one earns a 404 model_not_found. func stepDown(model string) string { for i, rung := range ladder { if rung != model { continue } if i == 0 { return "" // already cheapest: nowhere left to fall } return ladder[i-1] } // "auto", an alias, or a concrete id failed — the rung it picked is the // broken one, so start again from the cheapest. return ladder[0] } func (c *Client) Chat(ctx context.Context, messages []map[string]any, model string) (*ChatResponse, error) { if model == "" { model = "auto" // the default that works on every node } current := model var last error for attempt := 0; attempt < c.MaxAttempts; attempt++ { payload, err := json.Marshal(map[string]any{"model": current, "messages": messages}) if err != nil { return nil, err } req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.Base+"/v1/chat/completions", bytes.NewReader(payload)) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+c.Key) req.Header.Set("Content-Type", "application/json") res, err := c.HTTP.Do(req) if err != nil { // Transport failure (DNS, TLS, timeout). Retriable, but respect ctx. last = err select { case <-ctx.Done(): return nil, ctx.Err() case <-time.After(backoff(attempt)): } continue } raw, readErr := io.ReadAll(res.Body) res.Body.Close() if readErr != nil { return nil, fmt.Errorf("read body: %w", readErr) } if res.StatusCode == http.StatusOK { var out ChatResponse if err := json.Unmarshal(raw, &out); err != nil { return nil, fmt.Errorf("decode chat response: %w", err) } return &out, nil } apiErr := parseAPIError(res.StatusCode, raw) last = apiErr switch res.StatusCode { case 400, 401, 413, 415, 501: // Nothing a retry can fix: a bad body, a bad key, a clip too big, // a wrong content type, an absent seam. return nil, apiErr case http.StatusNotFound: // model_not_found if current == "auto" { return nil, apiErr // the router itself said no } current = "auto" // this node isn't behind the pin you used case http.StatusTooManyRequests: select { case <-ctx.Done(): return nil, ctx.Err() case <-time.After(retryAfter(res, attempt)): } case http.StatusBadGateway: next := stepDown(current) if next == "" { return nil, apiErr // cheapest rung is the one that failed } current = next case http.StatusServiceUnavailable: current = "auto" // hand tier choice back to the router select { case <-ctx.Done(): return nil, ctx.Err() case <-time.After(backoff(attempt)): } default: return nil, apiErr // unknown status: fail loud } } return nil, fmt.Errorf("gave up after %d attempts: %w", c.MaxAttempts, last) } // TokensOrUnknown renders a possibly-nil count for a log line. This is the // whole discipline in four lines: a nil count prints "unknown", and it is // never added into a total as 0. func TokensOrUnknown(v *int) string { if v == nil { return "unknown" } return strconv.Itoa(*v) } // AddKnown accumulates only what was actually measured, and counts the rest. func AddKnown(total, unknown int, v *int) (int, int) { if v == nil { return total, unknown + 1 } return total + *v, unknown }
package main import ( "context" "errors" "fmt" "os" "time" "example.com/hawktalk" ) func main() { ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute) defer cancel() c := hawktalk.New() out, err := c.Chat(ctx, []map[string]any{ {"role": "user", "content": "One line: why is the sky blue?"}, }, "auto") var apiErr *hawktalk.APIError switch { case errors.As(err, &apiErr): // The envelope survived the retry loop intact — log all of it. fmt.Fprintf(os.Stderr, "status=%d code=%q type=%q param=%q msg=%s\n", apiErr.Status, apiErr.Code, apiErr.Type, apiErr.Param, apiErr.Message) os.Exit(1) case err != nil: fmt.Fprintln(os.Stderr, err) os.Exit(1) } fmt.Println(out.Choices[0].Message.Content) // x_timing values are `any`: a number, or null, or the string "unknown". // Print them as they came. Formatting them as %.1f would force a lie. fmt.Printf("prompt=%s completion=%s ttft_ms=%v backend=%v\n", hawktalk.TokensOrUnknown(out.Usage.PromptTokens), hawktalk.TokensOrUnknown(out.Usage.CompletionTokens), out.XTiming["ttft_ms"], out.XTiming["backend"]) }
HawkTalk's REST tier is the OpenAI chat-completions shape at POST /v1/chat/completions. If your code already calls OpenAI through the official SDK, the migration is a base URL and a key — no request rewriting, no response rewriting, no new client library, no vendor SDK to learn. Everything below is the three places where that is not quite the whole story, and the checklist that closes them.
Base URL for an SDK is https://api.hawktalk.ai/v1 — the SDK appends /chat/completions itself. On a local node it is http://HOST:8890/v1. Same key, same header: Authorization: Bearer sk-....
One correction before you start. If you are following an older page documenting POST /v1/generate with an input / output / usage shape, that endpoint does not exist — it is POST /v1/chat/completions with the OpenAI chat shape. There is nothing to port from that older shape except the prompt string itself.
# before curl https://api.openai.com/v1/chat/completions \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Summarize this thread in one line."}] }' # after — two tokens changed: the host, and the key curl https://api.hawktalk.ai/v1/chat/completions \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "Summarize this thread in one line."}], "max_tokens": 320, "temperature": 0.2 }'
# pip install openai — the official SDK, unchanged import os from openai import OpenAI # before: client = OpenAI() # api.openai.com client = OpenAI( base_url="https://api.hawktalk.ai/v1", api_key=os.environ["HAWKTALK_API_KEY"], ) r = client.chat.completions.create( model="auto", # not a hardcoded id — see below messages=[{"role": "user", "content": "Summarize this thread in one line."}], max_tokens=320, temperature=0.2, ) print(r.choices[0].message.content)
// npm i openai — the official SDK, unchanged import OpenAI from "openai"; // before: const client = new OpenAI(); // api.openai.com const client = new OpenAI({ baseURL: "https://api.hawktalk.ai/v1", apiKey: process.env.HAWKTALK_API_KEY, }); const r = await client.chat.completions.create({ model: "auto", messages: [{ role: "user", content: "Summarize this thread in one line." }], max_tokens: 320, temperature: 0.2, }); console.log(r.choices[0].message.content);
// pubspec: http: ^1.2.0 — no HawkTalk SDK required, and none exists import 'dart:convert'; import 'package:http/http.dart' as http; final _client = http.Client(); // reuse it — one socket, many turns Future<String> ask(String prompt, String key) async { final res = await _client.post( Uri.parse('https://api.hawktalk.ai/v1/chat/completions'), headers: { 'Authorization': 'Bearer $key', 'Content-Type': 'application/json', }, body: jsonEncode({ 'model': 'auto', 'messages': [{'role': 'user', 'content': prompt}], 'max_tokens': 320, }), ); final body = jsonDecode(utf8.decode(res.bodyBytes)) as Map<String, dynamic>; if (res.statusCode != 200) { final e = (body['error'] as Map?) ?? const {}; throw Exception('hawktalk ${res.statusCode} ${e["code"]}: ${e["message"]}'); } return body['choices'][0]['message']['content'] as String; }
// Cargo.toml: reqwest = { version = "0.12", features = ["json"] } // tokio = { version = "1", features = ["full"] } use serde_json::{json, Value}; // Read the error body before you throw the response away — it carries // {"error":{"message","type","code","param"}} and the code is what you branch on. async fn ask(client: &reqwest::Client, key: &str, prompt: &str) -> Result<String, Box<dyn std::error::Error>> { let res = client .post("https://api.hawktalk.ai/v1/chat/completions") .bearer_auth(key) .json(&json!({ "model": "auto", "messages": [{ "role": "user", "content": prompt }], "max_tokens": 320 })) .send() .await?; let status = res.status(); let raw = res.text().await?; if !status.is_success() { return Err(format!("hawktalk {status}: {raw}").into()); } let body: Value = serde_json::from_str(&raw)?; Ok(body["choices"][0]["message"]["content"] .as_str().unwrap_or_default().to_string()) }
package hawktalk import ( "bytes" "encoding/json" "fmt" "io" "net/http" ) type chatResp struct { Choices []struct { Message struct { Content string `json:"content"` } `json:"message"` } `json:"choices"` } func Ask(c *http.Client, key, prompt string) (string, error) { body, err := json.Marshal(map[string]any{ "model": "auto", "messages": []map[string]string{{"role": "user", "content": prompt}}, "max_tokens": 320, }) if err != nil { return "", err } req, err := http.NewRequest("POST", "https://api.hawktalk.ai/v1/chat/completions", bytes.NewReader(body)) if err != nil { return "", err } req.Header.Set("Authorization", "Bearer "+key) req.Header.Set("Content-Type", "application/json") res, err := c.Do(req) if err != nil { return "", err } defer res.Body.Close() raw, err := io.ReadAll(res.Body) if err != nil { return "", err } if res.StatusCode != http.StatusOK { return "", fmt.Errorf("hawktalk %d: %s", res.StatusCode, raw) } var out chatResp if err := json.Unmarshal(raw, &out); err != nil { return "", err } if len(out.Choices) == 0 { return "", fmt.Errorf("hawktalk: no choices in response") } return out.Choices[0].Message.Content, nil }
These are the same bytes on the wire as your current OpenAI-shaped integration. If your code touches them, your code does not change.
| what | status | notes |
|---|---|---|
| messages array | identical | system / user / assistant / tool roles, same ordering rules. Multi-turn history is carried by the caller, exactly as now. |
| tools / tool_calls | identical | Same JSON-schema function definitions up, same tool_calls down, same tool-role message to feed the result back. Support is per model — see capabilities.tools in GET /v1/models. |
| stream: true | identical | SSE of chat.completion.chunk objects, terminated by data: [DONE]. Your existing SSE parser works untouched. |
| temperature, max_tokens | identical | Same meaning, same range. |
| choices[0].message.content | identical | Plus finish_reason and usage in the same places. |
| Authorization header | identical | Bearer sk-.... One key for REST, WebSockets and Live — there is never a second secret to manage. |
| error envelope | identical shape | {"error": {"message", "type", "code", "param"}}. Status codes: 400 · 401 invalid_api_key · 404 model_not_found · 429 rate_limit_exceeded (carries Retry-After) · 502 · 503. |
Structured output is capability-gated, not assumed. response_format and JSON-schema output are advertised per model in GET /v1/models under capabilities.response_format and capabilities.json_schema, and a node advertises only what it can honour natively. Read the flag; do not assume JSON mode because your old provider had it.
The swap above is a one-liner only if you are already on the OpenAI shape. From the Anthropic Messages API or Google's generateContent, the conversation model carries over but the envelope does not. This is the whole of the translation:
| your code today | what it becomes | notes |
|---|---|---|
Anthropic system: "..." top-level param | a {"role":"system"} entry at the head of messages | The single most common migration bug: dropping the system prompt entirely because it lived outside the array. |
Anthropic content blocks ([{type:"text",...}]) | a plain content string | Join the text blocks. Non-text blocks have no equivalent on this tier. |
Anthropic tool_use / tool_result blocks | tool_calls on the assistant message, and a {"role":"tool","tool_call_id":...} message back | Same two-step protocol, different container. |
Anthropic max_tokens (required) | max_tokens (optional) | Keep sending it. An unbounded generation is an unbounded bill. |
Google contents[].parts[], roles user/model | messages[].content, roles user/assistant | Rename the role; flatten the parts. |
Google systemInstruction | {"role":"system"} message | Same trap as Anthropic's. |
Anthropic SSE (content_block_delta)Google streamed GenerateContentResponse | chat.completion.chunk then data: [DONE] | Your stream parser is rewritten. Budget for it — it is the only real work in this migration. |
Fastest honest path: install the official openai SDK, point it at https://api.hawktalk.ai/v1, and let the SDK own the envelope. You are not adopting OpenAI — you are adopting the wire format that both ends already speak.
Your current model id will 404. HawkTalk model ids are not OpenAI's, they are not Anthropic's, and they are not fixed across nodes — a phone-NPU node, an inf2 node and a CPU node serve different registries. Call GET /v1/models at startup and read available. Anything else is a 404 model_not_found waiting for a deploy.
Then send "auto" anyway. auto runs the router per utterance: trivial turns are answered by the small tier, hard turns escalate. Pinning the largest model is the single largest source of overspend on this API. Use a pin only when you have a measured reason.
| value for model | meaning | when |
|---|---|---|
| "auto" | router picks per utterance | The default. Start here and stay here. |
"tier:quick" · ouro · dank · think · cloud | pin a rung of the ladder, let the node choose the weights | You need a latency or capability floor but not a specific build. |
a registry id from /v1/models | exactly that model | Reproducibility work, evals, bug reports. Portable only to nodes that serve it. |
The registry entry is richer than OpenAI's — alongside the id it carries availability, the backend serving it and per-model capability flags. The field-by-field shape is documented once, on the REST tier reference; read it there rather than from a copy in a migration guide. The one thing to internalise here: available: false with a non-null error is a model the node knows about and cannot currently serve — that is deliberate honesty, not a bug.
# what does THIS node actually serve? curl -s https://api.hawktalk.ai/v1/models \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ | jq '.data[] | {id, available, default, tools: .capabilities.tools}' # {"id":"hawkalphaquick","available":true,"default":true,"tools":true} # {"id":"qwen3-4b-htp","available":false,"default":false,"tools":false} # this came from one node's registry — read yours. Do not hardcode these ids.
# same SDK call you already have — the extra fields ride along untyped def usable_models(client): out = [] for m in client.models.list().data: extra = m.model_extra or {} if extra.get("available"): out.append(m.id) return out # log it at boot, then send "auto" — this is a guardrail, not a selector print("hawktalk node serves:", usable_models(client))
type HawkModel = { id: string; available?: boolean; capabilities?: { tools?: boolean } }; const page = await client.models.list(); const models = page.data as unknown as HawkModel[]; const usable = models.filter((m) => m.available === true).map((m) => m.id); const canCallTools = models.some((m) => m.available && m.capabilities?.tools); console.log("hawktalk node serves:", usable, "tools:", canCallTools);
Future<List<String>> usableModels(String key) async { final res = await _client.get( Uri.parse('https://api.hawktalk.ai/v1/models'), headers: {'Authorization': 'Bearer $key'}, ); if (res.statusCode != 200) throw Exception('models ${res.statusCode}'); final data = (jsonDecode(res.body)['data'] as List).cast<Map<String, dynamic>>(); return [ for (final m in data) if (m['available'] == true) m['id'] as String, ]; }
use serde::Deserialize; // Only the fields you branch on. serde ignores the rest, so a node that adds // a field tomorrow does not break you today. #[derive(Deserialize)] struct Model { id: String, available: bool } #[derive(Deserialize)] struct ModelList { data: Vec<Model> } async fn usable_models(client: &reqwest::Client, key: &str) -> Result<Vec<String>, reqwest::Error> { let list: ModelList = client .get("https://api.hawktalk.ai/v1/models") .bearer_auth(key) .send().await? .error_for_status()? .json().await?; Ok(list.data.into_iter().filter(|m| m.available).map(|m| m.id).collect()) }
type model struct { ID string `json:"id"` Available bool `json:"available"` Capabilities struct { Tools bool `json:"tools"` } `json:"capabilities"` } func UsableModels(c *http.Client, key string) ([]string, error) { req, err := http.NewRequest("GET", "https://api.hawktalk.ai/v1/models", nil) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+key) res, err := c.Do(req) if err != nil { return nil, err } defer res.Body.Close() if res.StatusCode != http.StatusOK { return nil, fmt.Errorf("models: %d", res.StatusCode) } var out struct { Data []model `json:"data"` } if err := json.NewDecoder(res.Body).Decode(&out); err != nil { return nil, err } ids := []string{} for _, m := range out.Data { if m.Available { ids = append(ids, m.ID) } } return ids, nil }
HawkTalk adds side-band objects to the completion. They are additive: every field OpenAI defines is still where it was, so ignoring them is safe and nothing breaks. Do not ignore them — they are the only honest answer to "which silicon served that turn, and what did it cost me".
There are three, and one line each is enough to migrate against: x_timing is what the node actually measured for the turn (on a stream it rides the final chunk — the one with finish_reason:"stop" — not every chunk), x_compute says whether the heavy compute path was engaged. (A per-turn trace_id is published in the spec but is not returned by the deployed response today — do not build a log pipeline that requires it. Key your logs on the response id.) The field-by-field shapes are documented once, on the REST tier reference — that page owns them; this one only tells you what to do with them.
The one rule that will bite you. An unmeasured value comes back null (or the string "unknown") — never a fabricated 0. That is a deliberate invariant across the whole API. If your metrics layer coerces missing numbers to zero, you will graph a 0ms time-to-first-token that never happened and then optimise against it. Keep the nulls null; drop the sample instead.
curl -s https://api.hawktalk.ai/v1/chat/completions \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"auto","messages":[{"role":"user","content":"hi"}]}' \ # { # "model": "hawkalphaquick", # "x_timing": { "ttft_ms": 536.1, "decode_tps": 41.7, "total_ms": 1180.4, # "backend": "llama-server/ggml-hexagon" }, # "x_compute": { "mode": "casual", "engaged": false } # } # this came from one node's registry — read yours. That model id is not a constant. # a null in x_timing means NOT MEASURED. It does not mean zero.
r = client.chat.completions.create(model="auto", messages=msgs) extra = r.model_extra or {} # the SDK parks unknown fields here timing = extra.get("x_timing") or {} log.info( "hawktalk model=%s backend=%s ttft_ms=%s tps=%s trace=%s", r.model, timing.get("backend"), timing.get("ttft_ms"), # may be None — log it as None timing.get("decode_tps"), ) # WRONG: timing.get("ttft_ms", 0) — invents a measurement that never happened ttft = timing.get("ttft_ms") if ttft is not None: metrics.observe("hawktalk.ttft_ms", ttft)
// The extra fields exist at runtime but are not in the SDK's types. type HawkExtras = { x_timing?: { ttft_ms: number | null; decode_tps: number | null; total_ms: number | null; backend: string | null; }; x_compute?: { mode: string | null; engaged: boolean | null }; }; const r = await client.chat.completions.create({ model: "auto", messages }); const x = r as unknown as HawkExtras; // ?? null, never ?? 0 logger.info({ model: r.model, backend: x.x_timing?.backend ?? null, ttft_ms: x.x_timing?.ttft_ms ?? null, compute: x.x_compute?.mode ?? null, });
final body = jsonDecode(utf8.decode(res.bodyBytes)) as Map<String, dynamic>; // nullable on purpose — a null ttft is "not measured", and the UI must // render it as "—", never as 0 ms. final timing = body['x_timing'] as Map<String, dynamic>?; final num? ttftMs = timing?['ttft_ms'] as num?; final String? backend = timing?['backend'] as String?; debugPrint('hawktalk backend=${backend ?? "unknown"} ' 'ttft=${ttftMs?.toStringAsFixed(0) ?? "—"}ms trace=$traceId');
// Option<T> everywhere. serde maps JSON null -> None, which is exactly the // distinction the API is making. Never #[serde(default)] these to 0.0. #[derive(Deserialize, Debug)] struct XTiming { ttft_ms: Option<f64>, decode_tps: Option<f64>, total_ms: Option<f64>, backend: Option<String>, } #[derive(Deserialize, Debug)] struct Completion { model: String, choices: Vec<Choice>, x_timing: Option<XTiming>, } let c: Completion = serde_json::from_str(&raw)?; if let Some(Some(ttft)) = c.x_timing.as_ref().map(|t| t.ttft_ms) { metrics::histogram!("hawktalk.ttft_ms").record(ttft); }
// Pointers, not values: a *float64 nil is "not measured"; a float64 zero // value would silently become a fabricated 0ms. type xTiming struct { TTFTms *float64 `json:"ttft_ms"` DecodeTPS *float64 `json:"decode_tps"` TotalMs *float64 `json:"total_ms"` Backend *string `json:"backend"` } type completion struct { Model string `json:"model"` Timing *xTiming `json:"x_timing"` } var c completion if err := json.Unmarshal(raw, &c); err != nil { return err } if c.Timing != nil && c.Timing.TTFTms != nil { ttftHist.Observe(*c.Timing.TTFTms) // only record what was measured }
If you are migrating speech-to-text or text-to-speech, read this before you ship. POST /v1/audio/transcriptions and POST /v1/audio/speech exist on every node, but on nodes where the seam is not wired they return 501 with stt_not_wired or tts_not_wired.
That is deliberate. The alternative — synthesising a beep, or returning an empty transcript — would let a broken deployment look healthy all the way to production. HawkTalk would rather your client fall back to browser Web Speech than hand you an invented transcript. Your migration must implement the fallback branch. Treat 501 as a permanent condition for the life of the process: do not retry it, do not back off, just route around it.
Check before you commit: GET /health needs no auth and reports voice.stt.wired and voice.tts.wired. Probe it at boot and pick your path once.
# pre-flight — no auth needed curl -s https://api.hawktalk.ai/health | jq '{stt: .voice.stt.wired, tts: .voice.tts.wired}' # {"stt": true, "tts": false} # and the seam itself curl -s -o speech.wav -w '%{http_code}\n' https://api.hawktalk.ai/v1/audio/speech \ -H "Authorization: Bearer $HAWKTALK_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "auto", "input": "the hawk is on the wire"}' # 200 -> speech.wav holds real audio/wav bytes # 501 -> body is {"error":{"type":"not_implemented","code":"tts_not_wired",...}} # use the client's own speech synthesis. Do not retry.
import httpx def speak(text: str) -> bytes | None: """WAV bytes, or None meaning 'this node cannot speak — use local TTS'.""" r = httpx.post( "https://api.hawktalk.ai/v1/audio/speech", headers={"Authorization": f"Bearer {KEY}"}, json={"model": "auto", "input": text}, timeout=30.0, ) if r.status_code == 501: code = r.json().get("error", {}).get("code") log.info("tts seam unwired (%s) — falling back to local", code) return None # permanent for this node. Do not retry. r.raise_for_status() # 502 tts_failed IS retriable — a real fault return r.content
async function speak(text: string): Promise<ArrayBuffer | null> { const res = await fetch("https://api.hawktalk.ai/v1/audio/speech", { method: "POST", headers: { Authorization: `Bearer ${process.env.HAWKTALK_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ model: "auto", input: text }), }); if (res.status === 501) { // not an outage — this node has no TTS engine. Speak locally instead: // speechSynthesis.speak(new SpeechSynthesisUtterance(text)) return null; } if (!res.ok) { const raw = await res.text(); throw new Error(`hawktalk tts ${res.status}: ${raw}`); } return res.arrayBuffer(); }
// Flutter: null here means "use flutter_tts / the platform voice". Future<Uint8List?> speak(String text, String key) async { final res = await _client.post( Uri.parse('https://api.hawktalk.ai/v1/audio/speech'), headers: { 'Authorization': 'Bearer $key', 'Content-Type': 'application/json', }, body: jsonEncode({'model': 'auto', 'input': text}), ); if (res.statusCode == 501) return null; // tts_not_wired — permanent if (res.statusCode != 200) { throw Exception('hawktalk tts ${res.statusCode}: ${res.body}'); } return res.bodyBytes; // real audio/wav }
// Model the seam in the type system so a caller cannot forget the branch. enum Speech { Wav(Vec<u8>), SeamNotWired } async fn speak(client: &reqwest::Client, key: &str, text: &str) -> Result<Speech, Box<dyn std::error::Error>> { let res = client .post("https://api.hawktalk.ai/v1/audio/speech") .bearer_auth(key) .json(&serde_json::json!({ "model": "auto", "input": text })) .send() .await?; match res.status().as_u16() { 200 => Ok(Speech::Wav(res.bytes().await?.to_vec())), 501 => Ok(Speech::SeamNotWired), // route around it, forever s => Err(format!("hawktalk tts {s}: {}", res.text().await?).into()), } }
// ErrSeamNotWired is a capability answer, not a failure. Never retry it. var ErrSeamNotWired = errors.New("hawktalk: tts seam not wired on this node") func Speak(c *http.Client, key, text string) ([]byte, error) { body, _ := json.Marshal(map[string]string{"model": "auto", "input": text}) req, err := http.NewRequest("POST", "https://api.hawktalk.ai/v1/audio/speech", bytes.NewReader(body)) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+key) req.Header.Set("Content-Type", "application/json") res, err := c.Do(req) if err != nil { return nil, err } defer res.Body.Close() switch res.StatusCode { case http.StatusOK: return io.ReadAll(res.Body) case http.StatusNotImplemented: // 501 return nil, ErrSeamNotWired default: raw, _ := io.ReadAll(res.Body) return nil, fmt.Errorf("hawktalk tts %d: %s", res.StatusCode, raw) } }
Work it top to bottom. If you are on the OpenAI SDK, steps 1–4 are an afternoon and the rest is observation.
| # | change | how you know it worked |
|---|---|---|
| 1 | Point the client at https://api.hawktalk.ai/v1 and swap the key into HAWKTALK_API_KEY. Keep the old key in place — you will want the A/B. | A 200 from POST /v1/chat/completions with a non-empty choices[0].message.content. |
| 2 | Replace every hardcoded model id with "auto". Grep for the old provider's ids; there are always more than you remember. | Zero 404 model_not_found in the logs, and GET /v1/models logged once at boot. |
| 4 | Map the error codes: 401 invalid_api_key goes dark and never retries · 429 rate_limit_exceeded honours Retry-After · 503 degrades to your fallback · 502 retries with backoff. | Force each one in staging. A 401 that retries into a lockout is the classic migration outage. |
| 5 | If you use audio: probe GET /health for voice.stt.wired / voice.tts.wired and implement the 501 fallback branch. | With the seam forced off, the product still talks and still transcribes — via the browser or platform voice. |
| 6 | Re-run your evals against "auto", not against a pin. You are testing the router, because the router is what production will run. | Quality within your tolerance band, at a measured cost you can compare to the old bill. |
| 7 | Confirm your SSE parser is untouched, then leave it alone. chat.completion.chunk → data: [DONE] is byte-compatible. | Streaming output renders identically to the old provider's. |
| watch | why | what good looks like |
|---|---|---|
| x_timing.ttft_ms distribution | The router sends different turns to different rungs, so latency is multi-modal by design — a single p50 hides it. | A stable set of modes. A new slow mode appearing means the router is escalating more than it did; check your prompts grew. |
| x_timing.backend spread | Tells you which silicon actually answered. This is the only ground truth for "did auto do the right thing". | Most turns on the fast tier, escalations where you would have escalated by hand. |
429 rate and Retry-After | Per-key RPM limiting is shipped and enforced now; monthly token quota enforcement in the request path is roadmap, not live. | Near-zero 429s at steady state. A cluster of them means your concurrency, not your volume, is the problem. |
503 model_unavailable / backend_unavailable | A node lost a backend. Your fallback path is now load-bearing. | Rare, and invisible to your users because step 4 was done properly. |
| null rate in telemetry | A rising null rate is a node reporting less than it used to — worth an email, not an outage. | Low and steady. Never zero — nulls are honest, not broken. |
| cost per thousand turns | The reason to have migrated. Compare against the same week on the old provider, same traffic. | Lower, with the same eval scores. If it is not, you pinned a model somewhere — go back to step 2. |
Everything above is shipped: REST chat, models, health, per-key RPM limiting and the usage ledger. These are not, and moving them now will cost you a rollback.
| surface | status | what to do instead |
|---|---|---|
| WS /live/brain — the HawkTalkLive lane mux | preview: loopback only, no auth wired, not served on api.hawktalk.ai | Do not put a product on it. If you need concurrent cognition lanes, build them client-side and keep the seam behind a flag. The shipped socket is WS /v1/realtime, which is a real OpenAI-Realtime session with one cognition lane. Lane-by-lane status is the table at the top of the HawkTalkLive page. |
| REST STT / TTS on an unwired node | seam: 501 by design | Migrate the code path, ship the fallback, and switch the default on only after GET /health reports wired: true on the node you actually deploy against. Nodes differ. |
| Monthly token quota enforcement | roadmap: not in the request path | Per-key RPM limiting is live and the usage ledger records spend, but nothing stops a runaway loop at a monthly ceiling today. Keep your own budget guard until it lands. |
| Binary WebSocket frames | rejected by design | If your current realtime client sends raw PCM as binary frames, that code does not port. Audio is base64 pcm16 inside JSON text frames — input defaults to 24000 Hz, and output frames carry x_sample_rate_hz. |
| Server-side voice activity detection | not offered | End-of-speech is your VAD, committed with input_audio_buffer.commit. The server does semantic endpointing only and never emits speech_started / speech_stopped. If your current provider ends turns for you, that logic moves into your client. |
The published tier order is REST → WebSockets → Live, and moving up is additive: the same key, the same model registry and the same tool definitions carry over. Migrate REST first, run it for a week against the checklist above, and only then open a socket.