Implement real streaming TTS for ElevenLabs to cut voice-assistant reply latency
The AudioSocket/media_runtime playback pipeline already supports chunked TTS streaming (voice_v2_streaming_tts), but every provider inherited the base TTSProvider.synthesize_chunks(), which just called the blocking synthesize() and yielded the entire finished audio as a single "chunk" - so the caller waited for full-utterance synthesis before any playback could start regardless of the flag. ElevenLabs is the production default (AI_VOICE_TTS_PROVIDER=elevenlabs in deployment/docker-compose.server.yml), so give it a real implementation that POSTs to the /stream endpoint and yields audio as network chunks arrive, instead of waiting for the whole response body. Chunk boundaries are re-aligned to whole 16-bit PCM samples so a split sample at a network read boundary can't corrupt playback. The full synthesized audio is still written to the on-disk cache afterwards so repeat phrases stay fast and skip the vendor call entirely, matching the existing synthesize() cache behavior. Added test_elevenlabs_tts_provider_streams_chunks_and_caches_full_audio to cover: chunk splitting mid-sample gets re-aligned, all yielded chunks are sample-aligned, the full audio round-trips through the cache, and a cached synthesis is replayed without invoking the streaming endpoint again. Verified via tests/test_ai_voice_tts_provider.py (9/9 pass) and a wider voice/tts-filtered run across the suite: the only failures present are the same pre-existing, already-documented ones (sales_service test cross-file isolation ordering, one known persona-prompt assertion) - identical set to before this change, no new failures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
cdabe61bc2
commit
a9be976845
@@ -777,6 +777,67 @@ class ElevenLabsTTSProvider(TTSProvider):
|
||||
self._write_cached_synthesis(synthesis, language=language)
|
||||
return synthesis
|
||||
|
||||
def synthesize_chunks(
|
||||
self,
|
||||
text: str,
|
||||
*,
|
||||
language: str | None = None,
|
||||
style_hints: dict[str, object] | None = None,
|
||||
):
|
||||
del style_hints
|
||||
if not text:
|
||||
return
|
||||
cached = self._load_cached_synthesis(text, language=language)
|
||||
if cached is not None:
|
||||
if cached.audio_bytes:
|
||||
yield cached
|
||||
return
|
||||
if not self._api_key:
|
||||
raise RuntimeError("AI_VOICE_TTS_ELEVENLABS_API_KEY is required for ElevenLabs TTS")
|
||||
|
||||
# Raw PCM16 samples must stay 2-byte aligned across network chunk
|
||||
# boundaries, or a split sample corrupts playback at that boundary.
|
||||
collected = bytearray()
|
||||
leftover = b""
|
||||
with httpx.Client(timeout=self._timeout_seconds) as client:
|
||||
with client.stream(
|
||||
"POST",
|
||||
f"{self._api_base}/v1/text-to-speech/{self._voice(language)}/stream",
|
||||
headers={
|
||||
"xi-api-key": self._api_key,
|
||||
"Accept": "application/octet-stream",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
params={"output_format": self._output_format},
|
||||
json={
|
||||
"text": text,
|
||||
"model_id": self._model(language),
|
||||
"language_code": self._language_code(language),
|
||||
},
|
||||
) as response:
|
||||
response.raise_for_status()
|
||||
for raw_chunk in response.iter_bytes():
|
||||
if not raw_chunk:
|
||||
continue
|
||||
data = leftover + raw_chunk
|
||||
if len(data) % 2:
|
||||
leftover = data[-1:]
|
||||
data = data[:-1]
|
||||
else:
|
||||
leftover = b""
|
||||
if not data:
|
||||
continue
|
||||
collected.extend(data)
|
||||
yield TTSSynthesis(text=text, audio_bytes=data, sample_rate_hz=self._sample_rate_hz)
|
||||
if leftover:
|
||||
collected.extend(leftover)
|
||||
|
||||
if collected:
|
||||
full_synthesis = TTSSynthesis(text=text, audio_bytes=bytes(collected), sample_rate_hz=self._sample_rate_hz)
|
||||
with self._cache_lock:
|
||||
if self._load_cached_synthesis(text, language=language) is None:
|
||||
self._write_cached_synthesis(full_synthesis, language=language)
|
||||
|
||||
|
||||
def build_tts_provider(name: str) -> TTSProvider:
|
||||
normalized = str(name or "stub").strip().lower()
|
||||
|
||||
Reference in New Issue
Block a user