call-center/docs/runbooks/track9-2-production-cutover.md

Wave 2 / Track 9.2 - Production Cutover (K8s Helm, Controlled Window)

This runbook executes Track 9.2 as an operations cutover, not a feature track.

Goal

Move the accepted Asterisk bridge flow from QA to production Helm deployment with:

• strict bridge auth (bearer only),
• one real softphone call validation,
• recording import + supervisor playback proof,
• formal GO/NO-GO evidence and rollback path.

Scope

Includes:

• production Helm upgrade in strict mode,
• post-cutover checks:
  • track9_preflight --check-sftp --require-strict-service-auth
  • asterisk_lab_smoke --require-recording
  • track9_check --require-recording
• evidence package collection.

Excludes:

• ARI/call-control,
• new voice features,
• business API contract changes.

Required inputs

Fill track9-cutover-sheet.template.md first.

Mandatory values:

• KUBE_CONTEXT
• NAMESPACE
• RELEASE
• GATEWAY_BASE_URL
• DATABASE_URL (PostgreSQL in production)
• IMAGE_TAG
• ASTERISK_AMI_HOST, ASTERISK_AMI_USERNAME, ASTERISK_AMI_SECRET
• ASTERISK_SFTP_HOST, ASTERISK_SFTP_USERNAME, ASTERISK_SFTP_PASSWORD
• QUEUE_ID for voice_lab
• APP_TOKEN_SECRET

Phase 0 - Entry checks (T-1 day)

1. Baseline reference is fixed:
   • docs/acceptance/track9/track9-acceptance.md
2. Confirm voice_lab -> queue_id exists in target routing DB.
3. Confirm cluster can reach Asterisk VM:
   • AMI 5038/tcp
   • SFTP 22/tcp
4. Confirm maintenance window and owners.

Phase 1 - Pre-cutover dry-run (T-4h to T-1h)

Run dry-run command (no deploy):

powershell -ExecutionPolicy Bypass -File scripts\track9_2_cutover.ps1 `
  -KubeContext <KUBE_CONTEXT> `
  -Namespace <NAMESPACE> `
  -Release <RELEASE> `
  -GatewayBaseUrl <GATEWAY_BASE_URL> `
  -DatabaseUrl <DATABASE_URL> `
  -ImageTag <IMAGE_TAG> `
  -AmiHost <ASTERISK_AMI_HOST> `
  -AmiUser <ASTERISK_AMI_USERNAME> `
  -AmiSecret <ASTERISK_AMI_SECRET> `
  -SftpHost <ASTERISK_SFTP_HOST> `
  -SftpUser <ASTERISK_SFTP_USERNAME> `
  -SftpPassword <ASTERISK_SFTP_PASSWORD> `
  -QueueId <QUEUE_ID> `
  -AppTokenSecret <APP_TOKEN_SECRET>

Expected:

• helm lint PASS
• helm template PASS
• rendered manifest contains:
  • ALLOW_LEGACY_HEADER_AUTH=0
  • ASTERISK_BRIDGE_AUTH_MODE=bearer
  • ASTERISK_BRIDGE_AUTH_FALLBACK_LEGACY=0
  • VOICE_ADAPTER_TRUSTED_SERVICE_SUBJECTS with svc:asterisk-bridge
  • RECORDING_IMPORT_TRUSTED_SERVICE_SUBJECTS with svc:asterisk-bridge
  • RECORDING_IMPORT_ALLOW_ADMIN=0

Artifacts are saved under:

• .artifacts/track9_2/<timestamp>/

Phase 2 - Controlled cutover execute (T0)

Run same command with -Execute:

powershell -ExecutionPolicy Bypass -File scripts\track9_2_cutover.ps1 `
  -KubeContext <KUBE_CONTEXT> `
  -Namespace <NAMESPACE> `
  -Release <RELEASE> `
  -GatewayBaseUrl <GATEWAY_BASE_URL> `
  -DatabaseUrl <DATABASE_URL> `
  -ImageTag <IMAGE_TAG> `
  -AmiHost <ASTERISK_AMI_HOST> `
  -AmiUser <ASTERISK_AMI_USERNAME> `
  -AmiSecret <ASTERISK_AMI_SECRET> `
  -SftpHost <ASTERISK_SFTP_HOST> `
  -SftpUser <ASTERISK_SFTP_USERNAME> `
  -SftpPassword <ASTERISK_SFTP_PASSWORD> `
  -QueueId <QUEUE_ID> `
  -AppTokenSecret <APP_TOKEN_SECRET> `
  -Execute

If Helm reports conflicts on HPA-managed Deployment.spec.replicas, rerun with:

... -Execute -ForceUpgrade

Script actions:

1. snapshot:
   • kubectl get pods -o wide
   • helm history (before)
2. helm upgrade --install with strict overlay
3. wait for the migration Job to complete
4. rollout wait:
   • api-gateway
   • asterisk-bridge-service
   • voice-adapter-service
   • recording-service
5. strict preflight + smoke + track9_check
6. evidence package via track9_collect_evidence
7. writes rollback hint from Helm history

Phase 3 - Live acceptance (mandatory)

During window, place one real call:

• softphone 1001 -> 7000

Then confirm:

• call.started from Asterisk source exists
• call.ended for same call exists
• recording.ready uploaded to recording-service
• supervisor playback is HTTP 200 and audio/*

Phase 4 - Evidence and GO/NO-GO

Use generated evidence folder from script and finalize:

• track9-acceptance.md
• playback-proof.md
• optional playback-proof.json

GO only if all true:

1. preflight PASS
2. smoke PASS
3. track9_check PASS
4. failed bridge events = 0
5. supervisor playback confirmed

Rollback policy

Rollback triggers:

• preflight FAIL
• track9_check FAIL
• no recording import after live call
• failed bridge events keep growing

Rollback command pattern:

helm -n <NAMESPACE> rollback <RELEASE> <PREVIOUS_REVISION>
kubectl -n <NAMESPACE> rollout status deploy/api-gateway
kubectl -n <NAMESPACE> rollout status deploy/asterisk-bridge-service

After rollback:

• rerun track9_preflight in previous baseline,
• log incident note + root cause item in cutover sheet.