call-center/docs/runbooks/asterisk-lab-linux-vm.md

# Asterisk Lab on Linux VM

Use this runbook to stand up the Track 9 human baseline, the IVR staging entrypoint, and the Voice AI V1 staging entrypoint on the same Asterisk VM.

## Target

- Host: dedicated Linux VM
- OS: Ubuntu 22.04 LTS or Ubuntu 24.04 LTS
- Asterisk: 20 LTS
- Integration mode: AMI + custom `UserEvent`
- Call source: softphone lab only
- Human baseline entrypoint: `7000`
- IVR staging entrypoint: `7200`
- Voice AI staging entrypoint: `7100`

## Prepare the VM

1. Install Asterisk 20 LTS.
2. Ensure these ports are reachable from the platform host:
   - `5060/udp` for SIP
   - `5038/tcp` for AMI
   - `22/tcp` for SFTP
3. Create the recording directory:
   - `/var/spool/asterisk/monitor/mvpcc`

## Apply repo configs

Copy these files from the repository:

- `deployment/asterisk/pjsip.conf`
- `deployment/asterisk/extensions.conf`
- `deployment/asterisk/manager.conf`

Before reload:

1. Replace default passwords.
2. Confirm extension `1001`.
3. Keep extension `7000` as the human-only inbound test entrypoint.
4. Keep extension `7100` reserved for Voice AI staging.
5. Keep extension `7200` reserved for IVR staging.

Voice AI dialplan notes:

- `deployment/asterisk/extensions.conf` now defines two queue codes:
  - `voice_lab` for `7000`
  - `voice_lab_ai` for `7100`
- `7100` falls back to the human dial targets unless `MVPCC_AI_AUDIOSOCKET_SERVICE` is populated.
- When you are ready to test the media bridge, load `app_audiosocket.so` and set `MVPCC_AI_AUDIOSOCKET_SERVICE=<media-host>:<port>`.
- Asterisk `AudioSocket()` requires a raw TCP media listener and a UUID per call. The template generates the UUID with `uuidgen`.
- Official app reference:
  - [Asterisk AudioSocket application](https://docs.asterisk.org/Latest_API/API_Documentation/Dialplan_Applications/AudioSocket/)

IVR dialplan notes:

- `deployment/asterisk/extensions.conf` defines `voice_lab_ivr` for `7200`.
- `7200` calls FastAGI at `MVPCC_IVR_FASTAGI_HOSTPORT` and expects `prompt_audio_key` values to match pre-provisioned Asterisk sound files.
- If FastAGI does not return `MVPCC_IVR_TARGET_EXTENSION`, the call falls back to the human dial targets.

## Configure the platform bridge

Set these environment variables for `asterisk-bridge-service`:

- `ASTERISK_BRIDGE_ENABLED=1`
- `ASTERISK_AMI_HOST=<linux-vm-host>`
- `ASTERISK_AMI_PORT=5038`
- `ASTERISK_AMI_USERNAME=mvpcc`
- `ASTERISK_AMI_SECRET=<ami-secret>`
- `ASTERISK_AMI_EVENT_PREFIX=MVPCC`
- `ASTERISK_QUEUE_MAP_JSON={"voice_lab":"<platform_queue_id>","voice_lab_ai":"<platform_ai_queue_id_or_same_queue_id>","voice_lab_ivr":"<platform_ivr_queue_id>"}`
- `ASTERISK_BRIDGE_AUTH_MODE=legacy_headers` (or `bearer_first` / `bearer`)
- `ASTERISK_BRIDGE_AUTH_FALLBACK_LEGACY=1` (used only with `bearer_first`)
- `ASTERISK_BRIDGE_AUTH_SUBJECT=svc:asterisk-bridge`
- `ASTERISK_BRIDGE_AUTH_USER=asterisk-bridge`
- `ASTERISK_BRIDGE_AUTH_ROLE=admin`
- `ASTERISK_BRIDGE_AUTH_TOKEN_TTL_SECONDS=300`
- `ASTERISK_IVR_FASTAGI_ENABLED=1`
- `ASTERISK_IVR_FASTAGI_HOST=0.0.0.0` (or the explicit bridge bind IP)
- `ASTERISK_IVR_FASTAGI_PORT=4573`
- `ASTERISK_IVR_DTMF_TIMEOUT_SECONDS=5`
- `ASTERISK_IVR_MAX_NO_INPUT_RETRIES=2`
- `ASTERISK_IVR_MAX_INVALID_RETRIES=2`
- `IVR_RUNTIME_TRUSTED_SERVICE_SUBJECTS=svc:asterisk-bridge`
- `VOICE_ADAPTER_TRUSTED_SERVICE_SUBJECTS=svc:asterisk-bridge,svc:ivr-service`
- `RECORDING_IMPORT_TRUSTED_SERVICE_SUBJECTS=svc:asterisk-bridge`
- `RECORDING_IMPORT_ALLOW_ADMIN=1` (set `0` in strict mode)
- `ASTERISK_SFTP_HOST=<linux-vm-host>`
- `ASTERISK_SFTP_PORT=22`
- `ASTERISK_SFTP_USERNAME=<linux-user>`
- `ASTERISK_SFTP_PASSWORD=<linux-password>`
- `ASTERISK_SFTP_BASE_PATH=/var/spool/asterisk/monitor`
- `ASTERISK_TRANSFER_TARGET_MAP_JSON={"voice_lab":"2001","voice_lab_ai":"2001","voice_lab_ivr":"7200"}`
- `AI_VOICE_ENABLED=1`
- `AI_VOICE_QUEUE_CONFIG_JSON={"voice_lab_ai":{"mode":"ai_first","agent_profile":"voice_support","handoff_queue_code":"voice_lab","language":"ru"}}`
- `AI_VOICE_RUNTIME_SERVICE_URL=http://ai-voice-runtime-service:8000`
- `AI_VOICE_RUNTIME_TRUSTED_SERVICE_SUBJECTS=svc:ai-voice-runtime`
- `AI_VOICE_ASR_PROVIDER=openai`
- `AI_VOICE_TTS_PROVIDER=openai`
- `AI_VOICE_MAX_CONTEXT_SEGMENTS=8`
- `AI_VOICE_HANDOFF_TIMEOUT_SECONDS=8`

For Track 9 QA acceptance baseline, keep:

- bridge auth mode in compatibility headers (`X-User` / `X-Role`)
- this is expected for Track 9 baseline and moved to hardening in Track 9.1

For Track 9.1 hardening, switch to:

- `ASTERISK_BRIDGE_AUTH_MODE=bearer_first` first
- then `ASTERISK_BRIDGE_AUTH_MODE=bearer` after successful smoke and no auth regressions
- set `RECORDING_IMPORT_ALLOW_ADMIN=0`
- in Helm-based environments, use `deployment/helm/values.track9-strict.yaml` as an overlay

Voice AI V1 staging guidance:

- Start with `MVPCC_AI_AUDIOSOCKET_SERVICE=` blank in `extensions.conf`.
- This lets `7100` validate the AI control-plane wiring while still falling back to the existing human voice path.
- Only point `MVPCC_AI_AUDIOSOCKET_SERVICE` to a live AudioSocket listener after the media listener is reachable from the VM.
- `AI_VOICE_RUNTIME_SERVICE_URL` is not the AudioSocket address; it is only the HTTP control-plane URL.

## Register the softphone

In your softphone:

- username: `1001`
- password: the value set in `pjsip.conf`
- server: the Linux VM IP/hostname

## QA preflight (required)

1. Copy [`.env.production.template`](/e:/Zhan/.env.production.template) to local `.env.production`.
2. Complete [`.env.production.checklist.md`](/e:/Zhan/docs/runbooks/.env.production.checklist.md).
3. Start the stack with production env loaded:

```powershell
powershell -ExecutionPolicy Bypass -File scripts\start_track9_qa.ps1
```

4. Run:

```powershell
python scripts\migrate_core_db.py
pytest -q
python scripts\track9_preflight.py --base-url http://127.0.0.1:8080 --check-sftp
python scripts\track9_preflight.py --base-url http://127.0.0.1:8080 --check-sftp --require-strict-service-auth
```

## Run the first test call

1. Start the platform.
2. Start `asterisk-bridge-service`.
3. Call `7000` from extension `1001`.
4. Hang up after the playback finishes.

Expected result:

- `asterisk-bridge-service` receives:
  - `MVPCCCallStarted`
  - `MVPCCCallEnded`
  - `MVPCCRecordingReady`
- platform creates a new `voice` interaction
- `voice_events` contain:
  - `call.started`
  - `call.ended`
- one recording is uploaded into `recording-service`

## Run the IVR-over-Asterisk smoke

1. Create or activate an IVR flow for the queue mapped from `voice_lab_ivr`.
2. Make sure every live node uses a valid `prompt_audio_key`, and the audio files are present in Asterisk sounds.
3. Call `7200` from extension `1001`.
4. Test both branches:
   - enter `1` or `2` to complete IVR and transfer into the mapped queue target
   - stay silent until retry exhaustion to verify fallback into the human baseline

Expected result:

- `asterisk-bridge-service` receives `MVPCCCallStarted` with `QueueCode=voice_lab_ivr`.
- `ivr_sessions` contains a live session for the `call_id`.
- interaction timeline contains `ivr.session.started` and `ivr.step.no_input` or `ivr.step.completed`.
- terminal completion writes `voice_events.event_type="ivr.completed"`.
- telephony returns to `mvpcc-transfer` for the resolved queue target, or to the human baseline on retry exhaustion.

## Run the Voice AI control-plane smoke

Use this smoke before turning on the real AudioSocket media path.

1. Keep `MVPCC_AI_AUDIOSOCKET_SERVICE=` blank in `extensions.conf`.
2. Start the platform and `ai-voice-runtime-service`.
3. Call `7100` from extension `1001`.
4. Let the call ring through to the fallback human dial targets.

Expected result:

- `asterisk-bridge-service` receives `MVPCCCallStarted` with `QueueCode=voice_lab_ai`.
- platform creates:
  - a `voice` interaction
  - a `voice_ai_sessions` row
  - an `ai_sessions` row with `channel=voice`
- operator UI shows the AI badge on the live/recent call card
- normal call recording flow stays green

This smoke proves queue selection, bridge-to-runtime session creation, timeline events, and safe human fallback. It does not prove realtime ASR/TTS yet.

## Verify from the platform

1. Check:
   - `GET /proxy/asterisk-bridge/asterisk/status`
2. Then run:

```powershell
python scripts\track9_preflight.py --base-url http://127.0.0.1:8080 --check-sftp
```

3. For smoke validation:

```powershell
python scripts\asterisk_lab_smoke.py --base-url http://127.0.0.1:8080 --database-url <database-url> --require-recording
```

4. For formal validation:

```powershell
python scripts\track9_check.py --base-url http://127.0.0.1:8080 --database-url <database-url> --require-recording
```

5. For Voice AI control-plane validation, also verify:
   - `GET /proxy/asterisk-bridge/asterisk/live-calls`
   - `GET /proxy/asterisk-bridge/asterisk/live-calls/{call_id}/ai-summary`
   - `ai.session_started` exists in the interaction timeline for the `7100` call

## Build formal acceptance evidence pack

After smoke/check pass:

```powershell
python scripts\track9_collect_evidence.py --base-url http://127.0.0.1:8080 --database-url <database-url> --run-checks --require-recording
```

Then fill:

- `docs/acceptance/track9/<timestamp>/track9-acceptance.md`

At minimum include:

- bridge status snapshot
- smoke output
- track9_check output
- sample rows from:
  - `asterisk_event_log`
  - `voice_events`
  - recording linkage

## Retry semantics and duplicate safety

Bridge event statuses:

- `received`: event accepted by bridge, no forward result yet.
- `forwarded`: event processed successfully and forwarded to platform services.
- `failed`: forwarding failed; inspect `last_error`, fix root cause, then retry.

Retry endpoint:

- `POST /proxy/asterisk-bridge/asterisk/events/{bridge_event_id}/retry`

Expected behavior:

1. Retry is safe for the same `bridge_event_id`.
2. Existing `call_id -> interaction_id` links prevent duplicate interaction creation.
3. Recording retry will update missing upload state if SFTP/upload issue is fixed.
4. Keep retries manual in QA until root cause is verified.

## Timeout / retry policy baseline (QA)

| Area | Current baseline | Operator action |
|---|---|---|
| AMI reconnect | `ASTERISK_BRIDGE_POLL_INTERVAL_SECONDS` (default `1s`) reconnect loop | Verify `ami_connected=true` in `/asterisk/status` |
| SFTP fetch | Single attempt per event processing | Fix credentials/path/network, then call retry endpoint |
| Recording upload | Single upload attempt per event processing | Fix `recording-service` reachability/auth, then retry |
| Failed event aging | No auto-waiver | Track 9 is not accepted with unresolved failed events |

## Troubleshooting

- `AMI is not connected`
  - verify `manager.conf`, firewall, and credentials
- `Unknown QueueCode`
  - ensure dialplan `QueueCode` matches `ASTERISK_QUEUE_MAP_JSON`
  - for Voice AI staging, verify both `voice_lab` and `voice_lab_ai`
  - verify mapped `queue_id` exists in platform:
    - `GET /proxy/routing/queues`
  - re-run failed event with retry endpoint
- `7100` immediately falls back to a human and no AI badge appears
  - verify `AI_VOICE_ENABLED=1`
  - verify `AI_VOICE_QUEUE_CONFIG_JSON` contains `voice_lab_ai`
  - verify the `MVPCCCallStarted` event shows `QueueCode=voice_lab_ai`
- `7100` errors as soon as AudioSocket is enabled
  - verify `app_audiosocket.so` is loaded on Asterisk
  - verify `MVPCC_AI_AUDIOSOCKET_SERVICE` points to a raw TCP media listener, not `http://...:8000`
  - verify the media host and port are reachable from the Asterisk VM
- `No uploaded recordings linked to Asterisk events`
  - verify SFTP credentials and `/var/spool/asterisk/monitor/mvpcc`
  - verify `ASTERISK_SFTP_BASE_PATH`
  - verify `POST /proxy/recording/recordings/import-upload` path is reachable from bridge host
  - retry failed `MVPCCRecordingReady` events after fix
- `Failed bridge events present`
  - inspect:
    - `GET /proxy/asterisk-bridge/asterisk/events?status=failed`
  - then retry with:
    - `POST /proxy/asterisk-bridge/asterisk/events/{bridge_event_id}/retry`

## Track 9.2 production cutover

After QA acceptance is signed, switch to controlled production rollout:

- [track9-2-production-cutover.md](/e:/Zhan/docs/runbooks/track9-2-production-cutover.md)
- `scripts/track9_2_cutover.ps1`