ai-operator/docs/language-selection.md

Language Selection

Purpose

Language Selection recognizes an explicit customer choice between Russian (ru) and Kazakh (kk) before business help is allowed. It is implemented as internal/dialogue/language and is independent from Asterisk, OpenAI, and the media pipeline.

Supported Languages

• ru: Russian
• kk: Kazakh

The stored language is always one of these strict internal codes.

Detection Pipeline

1. Normalize text: trim, lowercase, normalize ё to е, collapse spaces, remove harmless punctuation, preserve Kazakh Cyrillic letters.
2. Match exact codes and phrase tables.
3. Detect mixed RU/KK and ethnicity ambiguity.
4. Apply confidence and reason codes.
5. Run selection policy based on conversation state.

Russian Phrases

Examples include ru, rus, русский, русский язык, на русском, по-русски, хочу на русском, говорите на русском, russian, speak russian, russki, and ASR-like variants such as руский, русски, порусски.

Kazakh Phrases

Examples include kk, kz as alias to kk, қазақша, қазақ тілі, қазақ тілінде, қазақша сөйлейік, казахский, на казахском, qazaqsha, kazakh, kazaksha, and ASR-like variants such as қазакша, казакша, показахски.

Ambiguous Cases

The detector asks for clarification for cases such as:

• я русский
• я казах
• русский или қазақша?
• сначала русский потом казахский

These are not applied as automatic language selection.

False-Positive Protection

The detector avoids language selection for terms such as Казахтелеком, русский клиент спрашивает, У меня вопрос на русском сайте, embedded ru/kk substrings, and ordinary business questions.

Language Change Rules

• In LANGUAGE_SELECTION, clear RU/KK choices are applied.
• In REGION_SELECTION, language changes require a clear phrase.
• In READY_TO_HELP and QUESTION_ANSWERING, language changes require explicit switch intent.
• In HANDOFF, CLOSING, and ENDED, language changes are denied.

Integration

The Dialogue Orchestrator exposes HandleUserText. It passes transcripts or CLI input through the detector, applies selection policy, then calls the same set_language transition path. VoiceEventUserTranscriptDone is wired to this path. Assistant transcript/audio events do not trigger language detection.

CLI

./bin/ai-operator doctor --env /etc/ai-operator/ai-operator.env --check-language
./bin/ai-operator language-self-test --env /etc/ai-operator/ai-operator.env
./bin/ai-operator language-self-test --env /etc/ai-operator/ai-operator.env --json

Not Implemented Yet

• Region Resolver
• Knowledge Base/RAG
• final business prompt
• production routing
• real ASR accuracy validation

TZ-08 Region Resolver Update

TZ-08 adds an offline Kazakhstan Region Resolver with 20 enabled regions, stable region_code values, Almaty city/region clarification, false-positive protection, and explicit region-change policy. Production routing remains disabled; OpenAI, PostgreSQL, and Knowledge Base/RAG are still not used at this stage.

TZ-09 Knowledge Base Update

TZ-09 adds PostgreSQL + pgvector, KGA JSONL ingestion, fake embeddings, hybrid KB search, cross-language fallback for KK queries, citations, and search_knowledge_base integration behind the existing language/region guardrail. OpenAI is not called automatically and production routing remains disabled.

TZ-10 Agent Integration

The agent prompt keeps language selection as the first required action. Business questions in LANGUAGE_SELECTION must repeat the language request and must not trigger KB search.