Two bugs found via a live test with two real registered browser softphones:
1. retry_escalation_no_answer() re-resolved the client channel via
_resolve_handoff_channel() -> a live AMI CoreShowChannels round-trip that
can take ~10s. The new mvpcc-transfer dialplan wait window (MusicOnHold,
also ~10s, added to give the backend time to redirect before the final
Hangup) was consistently LOST to this exact same duration: the backend's
AMI Redirect fired against a channel the dialplan had already hung up
('Channel does not exist: PJSIP/...', confirmed in escalation timeline).
Fixed by reusing the actively-maintained AsteriskCallLinkRow.channel_name
directly (unchanged for a PJSIP channel across Redirect between contexts
of the same call) instead of re-discovering it, falling back to the slow
path only if that field is empty.
2. VoiceEventIn.event_type is a pydantic Literal restricted to 6 legacy
values (call.started/ivr.completed/...). None of the Phase 2 event
catalog names (AgentReserved/AgentRinging/AgentNoAnswer/AgentConnected/
TransferCompleted/TransferFailed) were ever in it, so every single
_emit_escalation_event() call has been failing with 422 since Phase 2
shipped (swallowed silently by the broad except there) - confirmed by
calling app._emit_voice_event() directly against the running service.
Extended the Literal to include all six.
process_agent_dial_outcome only recognized hangup causes 17/18/19/21/34/38.
When the reserved agent's AOR has zero registered contacts (e.g. the
softphone dropped, or nobody ever registered), Asterisk immediately
hangs up with cause 3 (no route to destination) instead of running a
Dial() long enough to produce a DialEnd/NOANSWER at all - so the retry
listener silently ignored it and the escalation was left dangling in
'ringing' status (the agent itself still got released via the
call-ended fallback path, but no retry to the next agent was ever
attempted and the escalation record never reflects the failure).
Added causes 1 (unallocated number), 3 (no route), 20 (subscriber
absent), 22 (number changed) alongside the existing set.
The speculative "early plan" turn (computed on partial ASR, before the
caller finishes talking) can win the race and get spoken as the actual
reply, but it unconditionally skipped KB search and answered common
questions (schedule/address/price/status/problem) with a hardcoded
clarifying question even when the FAQ already had the answer.
KB search is a cheap in-memory lexical scan over a DB-cached row set,
so it fits the early-plan latency budget unlike a real LLM call. Now
early-plan runs it and, on a match, answers from the KB snippet
(intent resolved via normalize_intent) instead of guessing a generic
clarifying question; with no match it falls back to the prior
behavior unchanged. operator_request is unaffected.
Centralizes fixed control intents and adds a data-driven intent_code
field on kb_articles so many phrasings of the same FAQ question
resolve to one stable code (e.g. VOUCHER_ACTIVATION) instead of a
free-form, unvalidated string the LLM invented on the fly.
- services/shared/intents.py: CONTROL_INTENTS + normalize_intent()
- kb_articles.intent_code column (ORM + dev/sqlite runtime compat +
migrations/sql/0034_* for postgres/sqlite)
- kb_service CRUD exposes intent_code
- orchestrator surfaces intent_code to the LLM and validates its
intent output against control intents + the KB codes shown that turn
- voice.py: _voice_early_intent_bucket renamed to _voice_ack_topic_bucket
to stop it being conflated with the canonical FAQ intent
request_handoff() (called by ai_voice_runtime_service for every real
call handoff, the only handoff path production calls actually use)
reserved an agent from the same routing pool as create_escalation()
but never created an EscalationRow, so the Phase 2 no-answer-retry
listener (DialEnd/Hangup) could never find it. A failed transfer
(no SIP registration, no answer, redirect error) left the agent
stuck in RESERVED forever with no retry to the next agent.
Now creates an EscalationRow (status=ringing) alongside the agent
reservation, releases the agent + marks the escalation failed if the
AMI Redirect itself errors immediately, and lets the existing
DialEnd/Hangup handler drive no-answer retry / release exactly like
the /escalations endpoint already does.
Reproduced live: call handed off to extension 2002 with no SIP
contact registered -> immediate hangup, cause=3, both pool agents
stuck in RESERVED indefinitely (had to release manually via psql).
Phase 2 of the L1->L2 routing engine (Phase 1: MR!4).
- ami_loop() now also captures native AMI DialEnd/Hangup frames (not
only UserEvent), needed to detect that an escalated agent did not
answer. No dialplan change required - Redirect already routes the
client channel into an existing Dial()-based transfer context, so
Asterisk emits these events on its own; the listener just wasn't
reading them before.
- retry_escalation_no_answer(): on NOANSWER/BUSY/CANCEL/CHANUNAVAIL/
CONGESTION, releases the non-answering agent, excludes it, and
reserves+redirects to the next available agent via the routing
engine's existing exclude_agent_ids support. Exhausted pool marks
the escalation failed and leaves the call with the AI instead of
dropping the client (ТЗ §32).
- Agent status now actually moves through
RESERVED -> RINGING -> TALKING -> AFTER_CALL_WORK -> AVAILABLE
instead of staying stuck on RESERVED for the whole call; a new
acw_sweep_loop background thread (same pattern as the existing
failed_retry_loop) times out AFTER_CALL_WORK back to AVAILABLE.
- escalations gains attempt_count/real_agent_id/attempted_agent_ids_json
(migration 0033); fixes a latent bug where assigned_agent_id stored
the SIP extension instead of the real agent_id despite routing-service
already returning it in RoutingAgentReserveOut.
- Every transition now records an interaction timeline entry and
publishes the ТЗ §25 event catalog (AgentReserved/AgentRinging/
AgentNoAnswer/AgentConnected/TransferCompleted/TransferFailed)
through the existing emit_voice_event/EventOutboxRow idempotent path.
Not in this MR (see plan): SLA config, Callback, L3 (needs real
technical agents from the business), metrics.
Both services/ai_orchestrator_service/voice.py and the aimaq persona/DOMAIN
SCOPE prompt were bind-mounted straight from the host on the aimaq stack,
bypassing git and CI entirely since they were first hand-edited in prod.
voice.py: merged the host's live business logic (gas/aimaq domain keyword
list, off-domain Kazakh/Russian replies, disabled re-correction of an already
obtained name) with the timeout_seconds fix from 09bcf74 that never reached
aimaq because the bind mount blocked it.
aimaq.env.production: replaced the AI_OPERATOR_* env values (which were never
interpolated -- {agent_name}/{company_name} would have been read literally)
with the final resolved Zhanna/Kazakgaz Aimaq text including the DOMAIN SCOPE
clause, matching what was actually live on the host.
Parametrize ai_operator_default_config() via AI_OPERATOR_* env vars,
falling back to the existing hardcoded defaults so the other two stacks
(call-center, sales-call-center) that share this code are unaffected.
Set aimaq-only overrides matching the client-provided script (Скрипт
ии-оператора КГА.docx): agent renamed to Жанна, company to Казакгаз
Аймак, the voice greeting now asks the caller whether Russian or
Kazakh is more convenient before anything else, and the base system
prompt instructs the model to commit to whichever language the caller
picks for the rest of the call, ask how to address them, and confirm
there's nothing else before saying goodbye.
RuntimeConfiguredTTSProvider (the live 'dynamic' TTS provider, resolves
voice from DB-backed config) never overrode cache_fingerprint(), so it
fell back to the base class's empty string. PrebakedAckBank keys its
on-disk cache on that fingerprint, so short filler phrases like
'Секунду' kept serving audio baked with the previous ElevenLabs voice
even after a voice change, while full LLM replies (cached inside the
resolved provider itself, keyed on its own voice id) already used the
new voice — explaining why callers heard two different voices in the
same call. Fix: delegate cache_fingerprint() to the resolved provider.
Also extend the voice delivery_hint so the model transliterates website
addresses and English words/abbreviations into spoken Cyrillic instead
of leaving raw Latin text for the TTS engine to mangle (egov.kz was
coming out as 'эговкз').
TTS reads raw text digit-by-digit with no number/date normalization layer.
Short numbers like 1414 were read as a single 4-digit number (tysyacha
chetyresta chetyrnadtsat) instead of a spoken code, and dates like '25
chisla' were read in the wrong grammatical case (dvadtsat pyat chislo
instead of dvadtsat pyatogo chisla). Extend the voice delivery_hint with
explicit spell-out rules so the model itself produces already-correct
spoken-form text.
Found during production smoke test: if the AMI Redirect call in
create_escalation() raises (channel gone, AMI hiccup), the agent stays
RESERVED forever with no owning call — orphaned out of the pool until
someone fixes it by hand. Now releases the agent and marks the
escalation failed before re-raising as a 502.
- routing-service: GET /escalations (list recent escalation attempts)
- supervisor UI: new panel showing the real agent pool (status, level,
tenant, skills, calls handled) fed by GET /agents, with a form to add
operators to the pool
- supervisor UI: new live escalation feed (AI->L2 handoffs, status,
assigned operator), auto-refreshed every 5s alongside live calls
Replaces the hardcoded single-extension redirect for AI->human call
escalation with a real Agent Pool + Routing Engine:
- agents/escalations/routing_rules tables (migration 0031), asterisk_call_links
gains tenant_id/current_level/required_skills_json/priority.
- services/routing_service/engine.py: level/tenant/skill filtered agent
selection with atomic (CAS) reservation, no double-booking.
- routing-service: /agents CRUD + /internal/routing/reserve-agent and
/internal/routing/release-agent.
- asterisk-bridge-service: voice_ai.request_handoff now uses the Routing
Engine automatically for any queue_code configured in
ASTERISK_QUEUE_LEVEL_MAP_JSON (all other queue_codes keep the existing
static ASTERISK_TRANSFER_TARGET_MAP_JSON behavior unchanged); new
POST /asterisk/live-calls/{call_id}/escalations entrypoint; agent is
released back to AVAILABLE and the escalation closed when the call ends.
Targets the Tele2 Kazgaz DID +77476456048 (from-tele2-kazgaz context) as the
first queue wired to real L2 routing instead of AI-only.
Known gap (documented in docs/architecture/l1-l2-routing-engine.md):
automatic no-answer retry-to-next-agent needs a small, separately reviewed
dialplan change and is left for a follow-up MR rather than guessed at blind.
Tests: services/routing_service/engine.py covered by
tests/test_routing_engine.py (selection filtering, atomic reservation,
release); existing test_asterisk_bridge_service.py and
test_routing_service_pg_counter.py suites still pass unmodified.
Add sync_ai_operator_config_from_code() which overwrites the DB-cached
ai_operator_settings row from ai_operator_default_config() on startup
of ai_orchestrator_service and ai_voice_runtime_service. Greeting and
system prompt changes now go through git + deploy instead of manual
psql/API edits to prod. Also adds a root README pointing to existing docs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous marker was added to operator_persona.py's inline fallback,
which only fires when config is None. Every real call path loads a
populated AIOperatorConfig via load_effective_ai_operator_config(),
so ai_operator_default_config() in services/shared/ai_operator_config.py
is the default that's actually served.
Two independent latency fixes for the voice-assistant reply pipeline,
both scoped to the parts of the flow that run regardless of whether
voice_v2 is enabled for a queue:
1. media_runtime._process_utterance: the v1/fallback turn path (used by
any queue not covered by AI_VOICE_V2_QUEUE_CODES) silently awaited the
full LLM decision with no audio playing at all, unlike the v2 path
which already has a decision-timeout ack. Give v1 the same behavior:
wait up to 600ms (_v1_ack_wait_seconds) for the decision, and if it's
still not ready, play a short "Секунду." filler via the existing
_emit_early_ack before the real reply, instead of leaving the caller
in silence for the full LLM+TTS round trip. Reuses the same ack
selection/playback code path v2 already exercises, so no new failure
modes - just an added timeout branch mirroring the existing v2 one.
2. ai_orchestrator_service._kb_search: every voice/chat turn re-ran a
full-table scan of kb_articles (all columns, including body text) and
rescored every row in Python, even though the KB rarely changes
mid-conversation. Added an in-process cache keyed by language, gated
on a cheap content fingerprint (row count + max id + max updated_at +
summed title/body/tags length, all computed server-side without
transferring the text columns). A fingerprint mismatch always
triggers a fresh fetch, so this can never serve stale results after
an insert/update/delete - unlike a naive TTL cache, which would have
been be wrong the moment a test (or a real KB edit) changed the table
within the cache window.
Note the first fingerprint design (count + max id + max updated_at
only) was insufficient: utc_now_iso() truncates to whole seconds and
SQLite reuses primary keys after a full-table delete, so two
different row sets written in the same wall-clock second could share
a fingerprint. Caught this via a real test failure
(test_ai_whatsapp_relaxed_kb_search_answers_phrase_query breaking
only when run after test_ai_orchestrator_service.py in the same
process) before it could reach production; the summed content-length
term closes the gap.
Added test_media_runtime_plays_filler_ack_when_v1_decision_is_slow
(asserts greeting -> ack -> reply delivery order when process_turn is
slow) and verified the KB cache against the full
test_ai_orchestrator_service.py + test_ai_whatsapp_orchestrator_service.py
suite plus a wider kb/orchestrator/whatsapp/telegram/voice-filtered run:
only the same pre-existing, already-documented failures remain (unrelated
sales_service test-isolation ordering, one known persona-prompt
assertion) - no new failures from either change.
Streaming the LLM decision itself (start speaking reply_text before the
full structured JSON response finishes generating) was scoped but
deliberately deferred: it needs incremental JSON parsing on top of SSE
streaming to detect when just the reply_text field is complete, shared
across both voice and text-channel decision paths - a separate,
higher-risk change that deserves its own PR and testing pass rather than
being bundled here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The AudioSocket/media_runtime playback pipeline already supports chunked
TTS streaming (voice_v2_streaming_tts), but every provider inherited the
base TTSProvider.synthesize_chunks(), which just called the blocking
synthesize() and yielded the entire finished audio as a single "chunk" -
so the caller waited for full-utterance synthesis before any playback
could start regardless of the flag.
ElevenLabs is the production default (AI_VOICE_TTS_PROVIDER=elevenlabs in
deployment/docker-compose.server.yml), so give it a real implementation
that POSTs to the /stream endpoint and yields audio as network chunks
arrive, instead of waiting for the whole response body. Chunk boundaries
are re-aligned to whole 16-bit PCM samples so a split sample at a network
read boundary can't corrupt playback. The full synthesized audio is still
written to the on-disk cache afterwards so repeat phrases stay fast and
skip the vendor call entirely, matching the existing synthesize() cache
behavior.
Added test_elevenlabs_tts_provider_streams_chunks_and_caches_full_audio to
cover: chunk splitting mid-sample gets re-aligned, all yielded chunks are
sample-aligned, the full audio round-trips through the cache, and a
cached synthesis is replayed without invoking the streaming endpoint
again.
Verified via tests/test_ai_voice_tts_provider.py (9/9 pass) and a wider
voice/tts-filtered run across the suite: the only failures present are
the same pre-existing, already-documented ones (sales_service test
cross-file isolation ordering, one known persona-prompt assertion) -
identical set to before this change, no new failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- docs: longread.md — deep architecture/flow review of the whole
platform (services, event bus reality vs docs, AI/ML stack honesty
check, tech debt inventory)
- ai_orchestrator_service/voice.py: drop _voice_decision_legacy
(unreferenced) and the shadowed first _voice_decision definition
(silently overwritten by the real one, dead code)
- ui/analyst/app.js: drop duplicate dead definitions of
loadSavedAnalyticsViews/saveAnalyticsView/deleteAnalyticsView and
the first loadAnalyticsTrend implementation, all shadowed by later
declarations in the same file; kept the intentional AI-mode
drilldown wrapper layer (openAnalyticsDrilldown/exportAnalyticsDrilldownCsv/etc.)
since that duplication is deliberate delegation, not dead code
- ui/operator/vendor/sip-0.21.2.min.js: remove byte-identical orphaned
duplicate of ui/operator/sip-0.21.2.min.js (unreferenced anywhere)
Verified via full pytest run: identical set of 97 pre-existing
failures before and after (sales_* test-isolation ordering issue and
one known persona-prompt test), no new regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
arys
didar
arystanbek