call-center/docs/runbooks/asterisk-lab-linux-vm.md

Asterisk Lab on Linux VM

Use this runbook to stand up the Track 9 human baseline, the IVR staging entrypoint, and the Voice AI V1 staging entrypoint on the same Asterisk VM.

Target

• Host: dedicated Linux VM
• OS: Ubuntu 22.04 LTS or Ubuntu 24.04 LTS
• Asterisk: 20 LTS
• Integration mode: AMI + custom UserEvent
• Call source: softphone lab only
• Human baseline entrypoint: 7000
• IVR staging entrypoint: 7200
• Voice AI staging entrypoint: 7100

Prepare the VM

1. Install Asterisk 20 LTS.
2. Ensure these ports are reachable from the platform host:
   • 5060/udp for SIP
   • 5038/tcp for AMI
   • 22/tcp for SFTP
3. Create the recording directory:
   • /var/spool/asterisk/monitor/mvpcc

Apply repo configs

Copy these files from the repository:

• deployment/asterisk/pjsip.conf
• deployment/asterisk/extensions.conf
• deployment/asterisk/manager.conf

Before reload:

1. Replace default passwords.
2. Confirm extension 1001.
3. Keep extension 7000 as the human-only inbound test entrypoint.
4. Keep extension 7100 reserved for Voice AI staging.
5. Keep extension 7200 reserved for IVR staging.

Voice AI dialplan notes:

• deployment/asterisk/extensions.conf now defines two queue codes:
  • voice_lab for 7000
  • voice_lab_ai for 7100
• 7100 falls back to the human dial targets unless MVPCC_AI_AUDIOSOCKET_SERVICE is populated.
• When you are ready to test the media bridge, load app_audiosocket.so and set MVPCC_AI_AUDIOSOCKET_SERVICE=<media-host>:<port>.
• Asterisk AudioSocket() requires a raw TCP media listener and a UUID per call. The template generates the UUID with uuidgen.
• Official app reference:
  • Asterisk AudioSocket application

IVR dialplan notes:

• deployment/asterisk/extensions.conf defines voice_lab_ivr for 7200.
• 7200 calls FastAGI at MVPCC_IVR_FASTAGI_HOSTPORT and expects prompt_audio_key values to match pre-provisioned Asterisk sound files.
• If FastAGI does not return MVPCC_IVR_TARGET_EXTENSION, the call falls back to the human dial targets.

Configure the platform bridge

Set these environment variables for asterisk-bridge-service:

• ASTERISK_BRIDGE_ENABLED=1
• ASTERISK_AMI_HOST=<linux-vm-host>
• ASTERISK_AMI_PORT=5038
• ASTERISK_AMI_USERNAME=mvpcc
• ASTERISK_AMI_SECRET=<ami-secret>
• ASTERISK_AMI_EVENT_PREFIX=MVPCC
• ASTERISK_QUEUE_MAP_JSON={"voice_lab":"<platform_queue_id>","voice_lab_ai":"<platform_ai_queue_id_or_same_queue_id>","voice_lab_ivr":"<platform_ivr_queue_id>"}
• ASTERISK_BRIDGE_AUTH_MODE=legacy_headers (or bearer_first / bearer)
• ASTERISK_BRIDGE_AUTH_FALLBACK_LEGACY=1 (used only with bearer_first)
• ASTERISK_BRIDGE_AUTH_SUBJECT=svc:asterisk-bridge
• ASTERISK_BRIDGE_AUTH_USER=asterisk-bridge
• ASTERISK_BRIDGE_AUTH_ROLE=admin
• ASTERISK_BRIDGE_AUTH_TOKEN_TTL_SECONDS=300
• ASTERISK_IVR_FASTAGI_ENABLED=1
• ASTERISK_IVR_FASTAGI_HOST=0.0.0.0 (or the explicit bridge bind IP)
• ASTERISK_IVR_FASTAGI_PORT=4573
• ASTERISK_IVR_DTMF_TIMEOUT_SECONDS=5
• ASTERISK_IVR_MAX_NO_INPUT_RETRIES=2
• ASTERISK_IVR_MAX_INVALID_RETRIES=2
• IVR_RUNTIME_TRUSTED_SERVICE_SUBJECTS=svc:asterisk-bridge
• VOICE_ADAPTER_TRUSTED_SERVICE_SUBJECTS=svc:asterisk-bridge,svc:ivr-service
• RECORDING_IMPORT_TRUSTED_SERVICE_SUBJECTS=svc:asterisk-bridge
• RECORDING_IMPORT_ALLOW_ADMIN=1 (set 0 in strict mode)
• ASTERISK_SFTP_HOST=<linux-vm-host>
• ASTERISK_SFTP_PORT=22
• ASTERISK_SFTP_USERNAME=<linux-user>
• ASTERISK_SFTP_PASSWORD=<linux-password>
• ASTERISK_SFTP_BASE_PATH=/var/spool/asterisk/monitor
• ASTERISK_TRANSFER_TARGET_MAP_JSON={"voice_lab":"2001","voice_lab_ai":"2001","voice_lab_ivr":"7200"}
• AI_VOICE_ENABLED=1
• AI_VOICE_QUEUE_CONFIG_JSON={"voice_lab_ai":{"mode":"ai_first","agent_profile":"voice_support","handoff_queue_code":"voice_lab","language":"ru"}}
• AI_VOICE_RUNTIME_SERVICE_URL=http://ai-voice-runtime-service:8000
• AI_VOICE_RUNTIME_TRUSTED_SERVICE_SUBJECTS=svc:ai-voice-runtime
• AI_VOICE_ASR_PROVIDER=openai
• AI_VOICE_TTS_PROVIDER=openai
• AI_VOICE_MAX_CONTEXT_SEGMENTS=8
• AI_VOICE_HANDOFF_TIMEOUT_SECONDS=8

For Track 9 QA acceptance baseline, keep:

• bridge auth mode in compatibility headers (X-User / X-Role)
• this is expected for Track 9 baseline and moved to hardening in Track 9.1

For Track 9.1 hardening, switch to:

• ASTERISK_BRIDGE_AUTH_MODE=bearer_first first
• then ASTERISK_BRIDGE_AUTH_MODE=bearer after successful smoke and no auth regressions
• set RECORDING_IMPORT_ALLOW_ADMIN=0
• in Helm-based environments, use deployment/helm/values.track9-strict.yaml as an overlay

Voice AI V1 staging guidance:

• Start with MVPCC_AI_AUDIOSOCKET_SERVICE= blank in extensions.conf.
• This lets 7100 validate the AI control-plane wiring while still falling back to the existing human voice path.
• Only point MVPCC_AI_AUDIOSOCKET_SERVICE to a live AudioSocket listener after the media listener is reachable from the VM.
• AI_VOICE_RUNTIME_SERVICE_URL is not the AudioSocket address; it is only the HTTP control-plane URL.

Register the softphone

In your softphone:

• username: 1001
• password: the value set in pjsip.conf
• server: the Linux VM IP/hostname

QA preflight (required)

1. Copy .env.production.template to local .env.production.
2. Complete .env.production.checklist.md.
3. Start the stack with production env loaded:

powershell -ExecutionPolicy Bypass -File scripts\start_track9_qa.ps1

4. Run:

python scripts\migrate_core_db.py
pytest -q
python scripts\track9_preflight.py --base-url http://127.0.0.1:8080 --check-sftp
python scripts\track9_preflight.py --base-url http://127.0.0.1:8080 --check-sftp --require-strict-service-auth

Run the first test call

1. Start the platform.
2. Start asterisk-bridge-service.
3. Call 7000 from extension 1001.
4. Hang up after the playback finishes.

Expected result:

• asterisk-bridge-service receives:
  • MVPCCCallStarted
  • MVPCCCallEnded
  • MVPCCRecordingReady
• platform creates a new voice interaction
• voice_events contain:
  • call.started
  • call.ended
• one recording is uploaded into recording-service

Run the IVR-over-Asterisk smoke

1. Create or activate an IVR flow for the queue mapped from voice_lab_ivr.
2. Make sure every live node uses a valid prompt_audio_key, and the audio files are present in Asterisk sounds.
3. Call 7200 from extension 1001.
4. Test both branches:
   • enter 1 or 2 to complete IVR and transfer into the mapped queue target
   • stay silent until retry exhaustion to verify fallback into the human baseline

Expected result:

• asterisk-bridge-service receives MVPCCCallStarted with QueueCode=voice_lab_ivr.
• ivr_sessions contains a live session for the call_id.
• interaction timeline contains ivr.session.started and ivr.step.no_input or ivr.step.completed.
• terminal completion writes voice_events.event_type="ivr.completed".
• telephony returns to mvpcc-transfer for the resolved queue target, or to the human baseline on retry exhaustion.

Run the Voice AI control-plane smoke

Use this smoke before turning on the real AudioSocket media path.

1. Keep MVPCC_AI_AUDIOSOCKET_SERVICE= blank in extensions.conf.
2. Start the platform and ai-voice-runtime-service.
3. Call 7100 from extension 1001.
4. Let the call ring through to the fallback human dial targets.

Expected result:

• asterisk-bridge-service receives MVPCCCallStarted with QueueCode=voice_lab_ai.
• platform creates:
  • a voice interaction
  • a voice_ai_sessions row
  • an ai_sessions row with channel=voice
• operator UI shows the AI badge on the live/recent call card
• normal call recording flow stays green

This smoke proves queue selection, bridge-to-runtime session creation, timeline events, and safe human fallback. It does not prove realtime ASR/TTS yet.

Verify from the platform

1. Check:
   • GET /proxy/asterisk-bridge/asterisk/status
2. Then run:

python scripts\track9_preflight.py --base-url http://127.0.0.1:8080 --check-sftp

3. For smoke validation:

python scripts\asterisk_lab_smoke.py --base-url http://127.0.0.1:8080 --database-url <database-url> --require-recording

4. For formal validation:

python scripts\track9_check.py --base-url http://127.0.0.1:8080 --database-url <database-url> --require-recording

5. For Voice AI control-plane validation, also verify:
   • GET /proxy/asterisk-bridge/asterisk/live-calls
   • GET /proxy/asterisk-bridge/asterisk/live-calls/{call_id}/ai-summary
   • ai.session_started exists in the interaction timeline for the 7100 call

Build formal acceptance evidence pack

After smoke/check pass:

python scripts\track9_collect_evidence.py --base-url http://127.0.0.1:8080 --database-url <database-url> --run-checks --require-recording

Then fill:

• docs/acceptance/track9/<timestamp>/track9-acceptance.md

At minimum include:

• bridge status snapshot
• smoke output
• track9_check output
• sample rows from:
  • asterisk_event_log
  • voice_events
  • recording linkage

Retry semantics and duplicate safety

Bridge event statuses:

• received: event accepted by bridge, no forward result yet.
• forwarded: event processed successfully and forwarded to platform services.
• failed: forwarding failed; inspect last_error, fix root cause, then retry.

Retry endpoint:

• POST /proxy/asterisk-bridge/asterisk/events/{bridge_event_id}/retry

Expected behavior:

1. Retry is safe for the same bridge_event_id.
2. Existing call_id -> interaction_id links prevent duplicate interaction creation.
3. Recording retry will update missing upload state if SFTP/upload issue is fixed.
4. Keep retries manual in QA until root cause is verified.

Timeout / retry policy baseline (QA)

Area	Current baseline	Operator action
AMI reconnect	ASTERISK_BRIDGE_POLL_INTERVAL_SECONDS (default 1s) reconnect loop	Verify ami_connected=true in /asterisk/status
SFTP fetch	Single attempt per event processing	Fix credentials/path/network, then call retry endpoint
Recording upload	Single upload attempt per event processing	Fix recording-service reachability/auth, then retry
Failed event aging	No auto-waiver	Track 9 is not accepted with unresolved failed events

Troubleshooting

• AMI is not connected
  • verify manager.conf, firewall, and credentials
• Unknown QueueCode
  • ensure dialplan QueueCode matches ASTERISK_QUEUE_MAP_JSON
  • for Voice AI staging, verify both voice_lab and voice_lab_ai
  • verify mapped queue_id exists in platform:
    • GET /proxy/routing/queues
  • re-run failed event with retry endpoint
• 7100 immediately falls back to a human and no AI badge appears
  • verify AI_VOICE_ENABLED=1
  • verify AI_VOICE_QUEUE_CONFIG_JSON contains voice_lab_ai
  • verify the MVPCCCallStarted event shows QueueCode=voice_lab_ai
• 7100 errors as soon as AudioSocket is enabled
  • verify app_audiosocket.so is loaded on Asterisk
  • verify MVPCC_AI_AUDIOSOCKET_SERVICE points to a raw TCP media listener, not http://...:8000
  • verify the media host and port are reachable from the Asterisk VM
• No uploaded recordings linked to Asterisk events
  • verify SFTP credentials and /var/spool/asterisk/monitor/mvpcc
  • verify ASTERISK_SFTP_BASE_PATH
  • verify POST /proxy/recording/recordings/import-upload path is reachable from bridge host
  • retry failed MVPCCRecordingReady events after fix
• Failed bridge events present
  • inspect:
    • GET /proxy/asterisk-bridge/asterisk/events?status=failed
  • then retry with:
    • POST /proxy/asterisk-bridge/asterisk/events/{bridge_event_id}/retry

Track 9.2 production cutover

After QA acceptance is signed, switch to controlled production rollout:

• track9-2-production-cutover.md
• scripts/track9_2_cutover.ps1