Reduce TTS API Latency: Streaming, Caching, and Pre-Generation Strategies

Time to first audio (TTFA) is the only latency metric that matters to users. Everything before the first byte of sound plays is dead silence — and in voice applications, silence feels like failure.

Here's how to minimize it.

The Latency Stack

TTS latency has three components:

1. Network round-trip: your server → TTS API → your server. Typically 50–200ms depending on region.
2. Model inference: the TTS model converts text to audio. For neural models like Kokoro-82M, 200–500ms for a short phrase.
3. Audio transfer: how long it takes to receive enough audio to start playback.

Total for a typical short TTS request (under 100 words): 400–900ms before the user hears anything. That's acceptable for async use cases. It's too slow for real-time voice agents or conversational interfaces.

Strategy 1: Streaming (Lowest TTFA for Dynamic Content)

Streaming returns audio chunks as they're generated rather than waiting for the complete file. The user starts hearing the first syllable while the API is still generating the end of the sentence.

With Speeko's streaming endpoint:

import requests

def stream_tts(text: str, api_key: str):
    response = requests.post(
        "https://api.speekoapp.com/v1/tts/stream",
        headers={"X-API-Key": api_key, "Content-Type": "application/json"},
        json={"text": text, "voice": "en-US-neural-1", "format": "mp3"},
        stream=True
    )
    
    audio_buffer = b""
    for chunk in response.iter_content(chunk_size=4096):
        audio_buffer += chunk
        # pipe to audio player as buffer fills
        yield chunk

TTFA with streaming: typically 100–300ms — the time for the first chunk to arrive, not the full audio file. For a 10-second audio clip, that's a 3–7x improvement in perceived responsiveness.

When to use streaming:

• Voice agents and chatbots
• Real-time narration
• Any interface where the user triggered speech generation and is waiting

When not to use streaming:

• Pre-generating content for storage
• Batch jobs where you need the complete file before processing

Strategy 2: Pre-Generation (Zero TTFA for Static Content)

If the text is known ahead of time — article narration, product descriptions, fixed IVR prompts — generate the audio when the content is published, not when the user requests it.

import requests
from pathlib import Path

def pre_generate_audio(text: str, output_path: str, api_key: str):
    response = requests.post(
        "https://api.speekoapp.com/v1/tts",
        headers={"X-API-Key": api_key},
        json={"text": text, "voice": "en-US-neural-1", "format": "mp3"}
    )
    Path(output_path).write_bytes(response.content)

# Generate at publish time, serve from CDN
pre_generate_audio(article_body, f"cdn/audio/{article_id}.mp3", API_KEY)

Serve from a CDN. TTFA becomes CDN edge latency — typically 10–50ms globally, depending on edge proximity.

For a 500-article knowledge base at ~9,000 chars each: one-time generation cost of $135 at $0.03/1K chars. After that, every listen is free and near-instant.

Strategy 3: Pre-Generate Common Phrases

IVR systems, chatbots, and customer service bots repeat the same phrases constantly. "Your account balance is..." — the variable part is the number, not the preamble.

Split generation:

# Pre-generate at startup
COMMON_PHRASES = {
    "greeting": "Thank you for calling. How can I help you today?",
    "hold": "Please hold while I look that up.",
    "goodbye": "Thank you for calling. Have a great day.",
}

def init_phrase_cache(api_key: str) -> dict:
    cache = {}
    for key, text in COMMON_PHRASES.items():
        response = requests.post(
            "https://api.speekoapp.com/v1/tts",
            headers={"X-API-Key": api_key},
            json={"text": text, "voice": "en-US-neural-1", "format": "mp3"}
        )
        cache[key] = response.content
    return cache

PHRASE_CACHE = init_phrase_cache(API_KEY)

Serve PHRASE_CACHE["greeting"] from memory. Zero API calls, zero latency for those phrases.

Strategy 4: Connection Pooling

If you're making multiple TTS requests, HTTP connection overhead adds up. Use a session object (Python requests) or persistent connections (Node.js http.Agent) to reuse TCP connections.

import requests

session = requests.Session()
session.headers.update({
    "X-API-Key": API_KEY,
    "Content-Type": "application/json"
})

# Reuse session for all requests — avoids TCP handshake overhead
def tts_with_session(text: str) -> bytes:
    response = session.post(
        "https://api.speekoapp.com/v1/tts",
        json={"text": text, "voice": "en-US-neural-1", "format": "mp3"}
    )
    return response.content

Savings: 50–150ms per request by skipping TCP and TLS handshake. Adds up fast in high-frequency applications.

Strategy 5: Regional Proximity

Place your backend in the same region as the TTS API endpoint. A 200ms round-trip from US-East to EU adds latency you can't optimize away.

Check the API's available regions and deploy accordingly. For Speeko, the API endpoint is in US-East — co-locate your backend there if latency is critical.

Measuring TTFA

Don't guess — measure. In Python:

import time
import requests

start = time.perf_counter()
response = requests.post(
    "https://api.speekoapp.com/v1/tts/stream",
    headers={"X-API-Key": API_KEY},
    json={"text": "Hello world.", "voice": "en-US-neural-1", "format": "mp3"},
    stream=True
)

first_chunk = True
for chunk in response.iter_content(chunk_size=1024):
    if first_chunk:
        ttfa = time.perf_counter() - start
        print(f"TTFA: {ttfa*1000:.0f}ms")
        first_chunk = False
    # process chunk

Log TTFA percentiles (p50, p95, p99) — not just averages. A p95 of 800ms means 5% of users wait nearly a second. That's the number to optimize.

Choosing the Right Strategy

Start with pre-generation for anything static. Add streaming for anything dynamic. Connection pooling is free — always enable it.

See also: async TTS job queue guide for handling high-volume batch generation without blocking your API.