sync: migrate ai-operator to Gitea (2026-08-10)
This commit is contained in:
konturai-ops
@@ -0,0 +1,134 @@
|
||||
# Knowledge Base / RAG
|
||||
|
||||
## Purpose
|
||||
|
||||
TZ-09 adds a PostgreSQL + pgvector backed Knowledge Base / RAG layer for the AI Operator. The initial production-like data source is the KGA JSONL import under `/opt/ai-operator/knowledge/import/jsonl`.
|
||||
|
||||
## Architecture
|
||||
|
||||
The KB layer contains:
|
||||
|
||||
- PostgreSQL with pgvector, bound to `127.0.0.1` only.
|
||||
- `knowledge_documents` for source JSONL records.
|
||||
- `knowledge_chunks` for searchable text chunks and embeddings.
|
||||
- `knowledge_ingest_runs` for import accounting.
|
||||
- `knowledge_search_logs` for sanitized search metadata.
|
||||
- EmbeddingProvider abstraction with `fake` default and OpenAI skeleton.
|
||||
- Hybrid search: vector score, keyword/full-text score, and region boost.
|
||||
|
||||
## Deployment
|
||||
|
||||
Docker Compose file:
|
||||
|
||||
```bash
|
||||
/opt/ai-operator/deploy/docker-compose.postgres.yml
|
||||
```
|
||||
|
||||
Postgres env file:
|
||||
|
||||
```bash
|
||||
/etc/ai-operator/postgres.env
|
||||
```
|
||||
|
||||
The env file is root-owned and mode `600`. The database port is bound as:
|
||||
|
||||
```text
|
||||
127.0.0.1:${AI_OPERATOR_POSTGRES_PORT:-5432}:5432
|
||||
```
|
||||
|
||||
## Source Files
|
||||
|
||||
Required KGA JSONL files:
|
||||
|
||||
```text
|
||||
/opt/ai-operator/knowledge/import/jsonl/kga_kb_rag.jsonl
|
||||
/opt/ai-operator/knowledge/import/jsonl/kga_branches_by_region.jsonl
|
||||
```
|
||||
|
||||
`kga_kb_rag.jsonl` contains the main FAQ/RAG records. `kga_branches_by_region.jsonl` contains regional branch contacts.
|
||||
|
||||
Required fields are `external_id`, `title`, `language`, `region_code`, `status`, `content`, and `source_hash`. Unknown fields are preserved in document metadata.
|
||||
|
||||
## Chunking
|
||||
|
||||
The chunker is UTF-8 safe and uses deterministic character windows:
|
||||
|
||||
- target: 1200 chars
|
||||
- max: 2000 chars
|
||||
- overlap: 150 chars
|
||||
|
||||
Chunk text keeps title, question, short answer, full answer, keywords, source, and content context.
|
||||
|
||||
## Embeddings
|
||||
|
||||
Default provider: `fake`.
|
||||
|
||||
The fake provider is deterministic, local-only, 1536-dimensional, and suitable for tests/MVP plumbing. The OpenAI provider is implemented as a low-level HTTP skeleton but is not called automatically.
|
||||
|
||||
## Search
|
||||
|
||||
Search filters:
|
||||
|
||||
- `status='published'`
|
||||
- `valid_from` / `valid_to`
|
||||
- selected language first
|
||||
- selected region plus `global`
|
||||
|
||||
For Kazakh (`kk`), if no KK result exists, cross-language fallback searches Russian (`ru`) and marks returned results with `cross_language_fallback=true`.
|
||||
|
||||
Ranking:
|
||||
|
||||
```text
|
||||
score = 0.65 * vector_score + 0.25 * keyword_score + 0.10 * region_boost
|
||||
```
|
||||
|
||||
Region boost is `1.0` for exact region and `0.6` for global fallback.
|
||||
|
||||
## Tool Integration
|
||||
|
||||
`search_knowledge_base` is only allowed after language and region are selected and the conversation state is `READY_TO_HELP` or `QUESTION_ANSWERING`.
|
||||
|
||||
If no result exists, the tool returns:
|
||||
|
||||
```json
|
||||
{"ok": false, "reason_code": "no_relevant_knowledge", "message_key": "knowledge.no_answer"}
|
||||
```
|
||||
|
||||
The dialogue layer does not generate a final natural-language answer in TZ-09. TZ-10 will use the returned chunks and citations to produce user-facing answers.
|
||||
|
||||
## CLI
|
||||
|
||||
```bash
|
||||
./bin/ai-operator kb-migrate --env /etc/ai-operator/ai-operator.env
|
||||
./bin/ai-operator kb-health --env /etc/ai-operator/ai-operator.env
|
||||
./bin/ai-operator kb-ingest --env /etc/ai-operator/ai-operator.env --path /opt/ai-operator/knowledge/import/jsonl --format jsonl --embedding-provider fake
|
||||
./bin/ai-operator kb-search --env /etc/ai-operator/ai-operator.env --query "Сколько стоит первичное подключение газа?" --language ru --region-code almaty_city --limit 5
|
||||
./bin/ai-operator kb-self-test --env /etc/ai-operator/ai-operator.env
|
||||
```
|
||||
|
||||
## Safety
|
||||
|
||||
- No OpenAI call is made unless an explicit live embedding command is run with an API key.
|
||||
- DB credentials are masked in logs and reports.
|
||||
- No raw audio or base64 audio is stored.
|
||||
- No Asterisk configs, dialplan, SIP, ARI, AMI, or RTP configs are changed.
|
||||
- Production routing remains disabled.
|
||||
|
||||
## Not Implemented Yet
|
||||
|
||||
- final agent prompt
|
||||
- real business tools
|
||||
- human handoff transfer
|
||||
- web/admin UI
|
||||
- document approval workflow
|
||||
## TZ-10 Tool Answer Payload
|
||||
|
||||
`search_knowledge_base` now returns caller-facing `answer_text` in addition to KB results and citations. The answer text is generated only from returned KB content. Cross-language fallback can return Russian source content for a Kazakh caller, with Kazakh framing and no unsupported facts.
|
||||
## TZ-11 Fallback
|
||||
|
||||
If KB is unavailable or repeated searches return no relevant answer, fallback can offer human handoff. The AI must still not invent an answer.
|
||||
|
||||
|
||||
## Audit Integration
|
||||
|
||||
KB searches write `ai_kb_audit` rows containing redacted query text, language, region, result count, top score, citation count and cross-language fallback status.
|
||||
Reference in New Issue
Block a user