82 lines
3.8 KiB
Markdown
82 lines
3.8 KiB
Markdown
# Language Selection
|
||
|
||
## Purpose
|
||
|
||
Language Selection recognizes an explicit customer choice between Russian (`ru`) and Kazakh (`kk`) before business help is allowed. It is implemented as `internal/dialogue/language` and is independent from Asterisk, OpenAI, and the media pipeline.
|
||
|
||
## Supported Languages
|
||
|
||
- `ru`: Russian
|
||
- `kk`: Kazakh
|
||
|
||
The stored language is always one of these strict internal codes.
|
||
|
||
## Detection Pipeline
|
||
|
||
1. Normalize text: trim, lowercase, normalize `ё` to `е`, collapse spaces, remove harmless punctuation, preserve Kazakh Cyrillic letters.
|
||
2. Match exact codes and phrase tables.
|
||
3. Detect mixed RU/KK and ethnicity ambiguity.
|
||
4. Apply confidence and reason codes.
|
||
5. Run selection policy based on conversation state.
|
||
|
||
## Russian Phrases
|
||
|
||
Examples include `ru`, `rus`, `русский`, `русский язык`, `на русском`, `по-русски`, `хочу на русском`, `говорите на русском`, `russian`, `speak russian`, `russki`, and ASR-like variants such as `руский`, `русски`, `порусски`.
|
||
|
||
## Kazakh Phrases
|
||
|
||
Examples include `kk`, `kz` as alias to `kk`, `қазақша`, `қазақ тілі`, `қазақ тілінде`, `қазақша сөйлейік`, `казахский`, `на казахском`, `qazaqsha`, `kazakh`, `kazaksha`, and ASR-like variants such as `қазакша`, `казакша`, `показахски`.
|
||
|
||
## Ambiguous Cases
|
||
|
||
The detector asks for clarification for cases such as:
|
||
|
||
- `я русский`
|
||
- `я казах`
|
||
- `русский или қазақша?`
|
||
- `сначала русский потом казахский`
|
||
|
||
These are not applied as automatic language selection.
|
||
|
||
## False-Positive Protection
|
||
|
||
The detector avoids language selection for terms such as `Казахтелеком`, `русский клиент спрашивает`, `У меня вопрос на русском сайте`, embedded `ru`/`kk` substrings, and ordinary business questions.
|
||
|
||
## Language Change Rules
|
||
|
||
- In `LANGUAGE_SELECTION`, clear RU/KK choices are applied.
|
||
- In `REGION_SELECTION`, language changes require a clear phrase.
|
||
- In `READY_TO_HELP` and `QUESTION_ANSWERING`, language changes require explicit switch intent.
|
||
- In `HANDOFF`, `CLOSING`, and `ENDED`, language changes are denied.
|
||
|
||
## Integration
|
||
|
||
The Dialogue Orchestrator exposes `HandleUserText`. It passes transcripts or CLI input through the detector, applies selection policy, then calls the same `set_language` transition path. `VoiceEventUserTranscriptDone` is wired to this path. Assistant transcript/audio events do not trigger language detection.
|
||
|
||
## CLI
|
||
|
||
```bash
|
||
./bin/ai-operator doctor --env /etc/ai-operator/ai-operator.env --check-language
|
||
./bin/ai-operator language-self-test --env /etc/ai-operator/ai-operator.env
|
||
./bin/ai-operator language-self-test --env /etc/ai-operator/ai-operator.env --json
|
||
```
|
||
|
||
## Not Implemented Yet
|
||
|
||
- Region Resolver
|
||
- Knowledge Base/RAG
|
||
- final business prompt
|
||
- production routing
|
||
- real ASR accuracy validation
|
||
|
||
## TZ-08 Region Resolver Update
|
||
|
||
TZ-08 adds an offline Kazakhstan Region Resolver with 20 enabled regions, stable `region_code` values, Almaty city/region clarification, false-positive protection, and explicit region-change policy. Production routing remains disabled; OpenAI, PostgreSQL, and Knowledge Base/RAG are still not used at this stage.
|
||
|
||
## TZ-09 Knowledge Base Update
|
||
|
||
TZ-09 adds PostgreSQL + pgvector, KGA JSONL ingestion, fake embeddings, hybrid KB search, cross-language fallback for KK queries, citations, and `search_knowledge_base` integration behind the existing language/region guardrail. OpenAI is not called automatically and production routing remains disabled.
|
||
## TZ-10 Agent Integration
|
||
|
||
The agent prompt keeps language selection as the first required action. Business questions in `LANGUAGE_SELECTION` must repeat the language request and must not trigger KB search.
|