Implement real streaming TTS for ElevenLabs to cut voice-assistant reply latency

The AudioSocket/media_runtime playback pipeline already supports chunked
TTS streaming (voice_v2_streaming_tts), but every provider inherited the
base TTSProvider.synthesize_chunks(), which just called the blocking
synthesize() and yielded the entire finished audio as a single "chunk" -
so the caller waited for full-utterance synthesis before any playback
could start regardless of the flag.

ElevenLabs is the production default (AI_VOICE_TTS_PROVIDER=elevenlabs in
deployment/docker-compose.server.yml), so give it a real implementation
that POSTs to the /stream endpoint and yields audio as network chunks
arrive, instead of waiting for the whole response body. Chunk boundaries
are re-aligned to whole 16-bit PCM samples so a split sample at a network
read boundary can't corrupt playback. The full synthesized audio is still
written to the on-disk cache afterwards so repeat phrases stay fast and
skip the vendor call entirely, matching the existing synthesize() cache
behavior.

Added test_elevenlabs_tts_provider_streams_chunks_and_caches_full_audio to
cover: chunk splitting mid-sample gets re-aligned, all yielded chunks are
sample-aligned, the full audio round-trips through the cache, and a
cached synthesis is replayed without invoking the streaming endpoint
again.

Verified via tests/test_ai_voice_tts_provider.py (9/9 pass) and a wider
voice/tts-filtered run across the suite: the only failures present are
the same pre-existing, already-documented ones (sales_service test
cross-file isolation ordering, one known persona-prompt assertion) -
identical set to before this change, no new failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Didar Kozhikov
2026-08-19 21:40:35 +05:00
co-authored by Claude Sonnet 5
parent cdabe61bc2
commit a9be976845
2 changed files with 134 additions and 0 deletions
@@ -777,6 +777,67 @@ class ElevenLabsTTSProvider(TTSProvider):
self._write_cached_synthesis(synthesis, language=language)
return synthesis
def synthesize_chunks(
self,
text: str,
*,
language: str | None = None,
style_hints: dict[str, object] | None = None,
):
del style_hints
if not text:
return
cached = self._load_cached_synthesis(text, language=language)
if cached is not None:
if cached.audio_bytes:
yield cached
return
if not self._api_key:
raise RuntimeError("AI_VOICE_TTS_ELEVENLABS_API_KEY is required for ElevenLabs TTS")
# Raw PCM16 samples must stay 2-byte aligned across network chunk
# boundaries, or a split sample corrupts playback at that boundary.
collected = bytearray()
leftover = b""
with httpx.Client(timeout=self._timeout_seconds) as client:
with client.stream(
"POST",
f"{self._api_base}/v1/text-to-speech/{self._voice(language)}/stream",
headers={
"xi-api-key": self._api_key,
"Accept": "application/octet-stream",
"Content-Type": "application/json",
},
params={"output_format": self._output_format},
json={
"text": text,
"model_id": self._model(language),
"language_code": self._language_code(language),
},
) as response:
response.raise_for_status()
for raw_chunk in response.iter_bytes():
if not raw_chunk:
continue
data = leftover + raw_chunk
if len(data) % 2:
leftover = data[-1:]
data = data[:-1]
else:
leftover = b""
if not data:
continue
collected.extend(data)
yield TTSSynthesis(text=text, audio_bytes=data, sample_rate_hz=self._sample_rate_hz)
if leftover:
collected.extend(leftover)
if collected:
full_synthesis = TTSSynthesis(text=text, audio_bytes=bytes(collected), sample_rate_hz=self._sample_rate_hz)
with self._cache_lock:
if self._load_cached_synthesis(text, language=language) is None:
self._write_cached_synthesis(full_synthesis, language=language)
def build_tts_provider(name: str) -> TTSProvider:
normalized = str(name or "stub").strip().lower()