# Knowledge Base / RAG

## Purpose

TZ-09 adds a PostgreSQL + pgvector backed Knowledge Base / RAG layer for the AI Operator. The initial production-like data source is the KGA JSONL import under `/opt/ai-operator/knowledge/import/jsonl`.

## Architecture

The KB layer contains:

- PostgreSQL with pgvector, bound to `127.0.0.1` only.
- `knowledge_documents` for source JSONL records.
- `knowledge_chunks` for searchable text chunks and embeddings.
- `knowledge_ingest_runs` for import accounting.
- `knowledge_search_logs` for sanitized search metadata.
- EmbeddingProvider abstraction with `fake` default and OpenAI skeleton.
- Hybrid search: vector score, keyword/full-text score, and region boost.

## Deployment

Docker Compose file:

```bash
/opt/ai-operator/deploy/docker-compose.postgres.yml
```

Postgres env file:

```bash
/etc/ai-operator/postgres.env
```

The env file is root-owned and mode `600`. The database port is bound as:

```text
127.0.0.1:${AI_OPERATOR_POSTGRES_PORT:-5432}:5432
```

## Source Files

Required KGA JSONL files:

```text
/opt/ai-operator/knowledge/import/jsonl/kga_kb_rag.jsonl
/opt/ai-operator/knowledge/import/jsonl/kga_branches_by_region.jsonl
```

`kga_kb_rag.jsonl` contains the main FAQ/RAG records. `kga_branches_by_region.jsonl` contains regional branch contacts.

Required fields are `external_id`, `title`, `language`, `region_code`, `status`, `content`, and `source_hash`. Unknown fields are preserved in document metadata.

## Chunking

The chunker is UTF-8 safe and uses deterministic character windows:

- target: 1200 chars
- max: 2000 chars
- overlap: 150 chars

Chunk text keeps title, question, short answer, full answer, keywords, source, and content context.

## Embeddings

Default provider: `fake`.

The fake provider is deterministic, local-only, 1536-dimensional, and suitable for tests/MVP plumbing. The OpenAI provider is implemented as a low-level HTTP skeleton but is not called automatically.

## Search

Search filters:

- `status='published'`
- `valid_from` / `valid_to`
- selected language first
- selected region plus `global`

For Kazakh (`kk`), if no KK result exists, cross-language fallback searches Russian (`ru`) and marks returned results with `cross_language_fallback=true`.

Ranking:

```text
score = 0.65 * vector_score + 0.25 * keyword_score + 0.10 * region_boost
```

Region boost is `1.0` for exact region and `0.6` for global fallback.

## Tool Integration

`search_knowledge_base` is only allowed after language and region are selected and the conversation state is `READY_TO_HELP` or `QUESTION_ANSWERING`.

If no result exists, the tool returns:

```json
{"ok": false, "reason_code": "no_relevant_knowledge", "message_key": "knowledge.no_answer"}
```

The dialogue layer does not generate a final natural-language answer in TZ-09. TZ-10 will use the returned chunks and citations to produce user-facing answers.

## CLI

```bash
./bin/ai-operator kb-migrate --env /etc/ai-operator/ai-operator.env
./bin/ai-operator kb-health --env /etc/ai-operator/ai-operator.env
./bin/ai-operator kb-ingest --env /etc/ai-operator/ai-operator.env --path /opt/ai-operator/knowledge/import/jsonl --format jsonl --embedding-provider fake
./bin/ai-operator kb-search --env /etc/ai-operator/ai-operator.env --query "Сколько стоит первичное подключение газа?" --language ru --region-code almaty_city --limit 5
./bin/ai-operator kb-self-test --env /etc/ai-operator/ai-operator.env
```

## Safety

- No OpenAI call is made unless an explicit live embedding command is run with an API key.
- DB credentials are masked in logs and reports.
- No raw audio or base64 audio is stored.
- No Asterisk configs, dialplan, SIP, ARI, AMI, or RTP configs are changed.
- Production routing remains disabled.

## Not Implemented Yet

- final agent prompt
- real business tools
- human handoff transfer
- web/admin UI
- document approval workflow
## TZ-10 Tool Answer Payload

`search_knowledge_base` now returns caller-facing `answer_text` in addition to KB results and citations. The answer text is generated only from returned KB content. Cross-language fallback can return Russian source content for a Kazakh caller, with Kazakh framing and no unsupported facts.
## TZ-11 Fallback

If KB is unavailable or repeated searches return no relevant answer, fallback can offer human handoff. The AI must still not invent an answer.


## Audit Integration

KB searches write `ai_kb_audit` rows containing redacted query text, language, region, result count, top score, citation count and cross-language fallback status.