call-center/docs/runbooks/deployment-onprem.md

Runbook - On-Prem Deployment

Helm

• helm upgrade --install mvp-cc ./deployment/helm -n mvp-cc --create-namespace
• For PostgreSQL-backed Helm releases, the chart now runs scripts/migrate_core_db.py automatically via a pre-install / pre-upgrade migration Job.

Raw manifests

• kubectl apply -f deployment/kubernetes/mvp-cc-platform.yaml
• Raw manifests remain a minimal reference/bootstrap path.
• Helm is the canonical production-scale deployment path for Wave 2 / Track 7.

Core DB migration

• Helm path:
  • keep migrations.enabled=true
  • keep SCHEMA_MANAGEMENT_MODE=migrations
  • the Helm hook Job applies migrations before rollout
• Raw manifests / manual path:
  • run python scripts/migrate_core_db.py
  • then start services with SCHEMA_MANAGEMENT_MODE=migrations

Wave 2 / Track 7 scale baseline

Production-like K8s scale validation requires:

• DATABASE_URL set to a shared PostgreSQL endpoint
• DB_POOL_SIZE
• DB_MAX_OVERFLOW
• DB_POOL_TIMEOUT_SECONDS
• DB_POOL_RECYCLE_SECONDS
• a working metrics-server in the target cluster

Recommended Wave 2 / Track 7 rollout:

1. Render Helm config for the scale profile.
2. Confirm DATABASE_URL points to PostgreSQL (not SQLite).
3. Confirm SCHEMA_MANAGEMENT_MODE=migrations.
4. Confirm migrations.enabled=true.
5. Deploy via Helm.
6. Verify the migration Job completed successfully.
7. Run python scripts/load_test.py --base-url http://<gateway-host> --profile step_250_250 --auth-mode bearer.
8. Run python scripts/track7_check.py --namespace mvp-cc --report-dir <report_dir>.
9. Repeat with --profile target_500_500.

For the complete procedure, use:

• track7-scale-validation.md

Wave 2 / Track 1 auth configuration

Minimum enterprise auth variables:

• APP_TOKEN_SECRET
• APP_TOKEN_TTL_SECONDS (default 3600)
• ALLOW_LEGACY_HEADER_AUTH=0
• OIDC_ENABLED=1
• OIDC_PROVIDER=keycloak
• OIDC_ISSUER_URL
• OIDC_CLIENT_ID
• OIDC_CLIENT_SECRET
• OIDC_REDIRECT_URI
• OIDC_SCOPES (default openid profile email)
• OIDC_ROLE_CLAIM (default groups)
• OIDC_ROLE_MAP_JSON

Helm and raw Kubernetes manifests now include placeholder values for these settings. Replace all change-me-* values before deploying outside local/demo.

Recommended rollout sequence:

1. For Helm, keep migration hook enabled so schema upgrades run before deploy.
2. Deploy with OIDC_ENABLED=1 and ALLOW_LEGACY_HEADER_AUTH=1 in QA.
3. Run python scripts/oidc_smoke.py --base-url http://<gateway-host> --require-enabled.
4. Validate a real Keycloak login round trip.
5. Switch enterprise pilot to ALLOW_LEGACY_HEADER_AUTH=0.

Post-deploy checks

• Pods ready state
• /health probes green
• Gateway registry endpoint available
• GET /proxy/auth/auth/oidc/config returns enabled=true
• GET /proxy/auth/auth/oidc/health returns provider metadata status

See also:

• keycloak-oidc.md
• postgres-dev-cutover.md
• postgres-server-docker.md
• postgres-server-parallel-stack.md

Wave 2 / Track 9.1 strict bridge auth cutover

Use Helm overlays to force strict bridge auth mode in QA/enterprise environments:

helm upgrade --install mvp-cc deployment\helm `
  -f deployment\helm\values.scale500.yaml `
  -f deployment\helm\values.track9-strict.yaml `
  --set-string auth.appTokenSecret=<shared-app-token-secret> `
  --set-string asteriskBridge.enabled=1 `
  --set-string asteriskBridge.amiHost=<asterisk-vm-host> `
  --set-string asteriskBridge.amiUsername=<ami-user> `
  --set-string asteriskBridge.amiSecret=<ami-secret> `
  --set-string asteriskBridge.queueMapJson='{\"voice_lab\":\"<queue_id>\"}' `
  --set-string asteriskBridge.sftpHost=<asterisk-vm-host> `
  --set-string asteriskBridge.sftpUsername=<sftp-user> `
  --set-string asteriskBridge.sftpPassword=<sftp-password> `
  -n mvp-cc --create-namespace

Expected strict baseline after deploy:

• ALLOW_LEGACY_HEADER_AUTH=0
• ASTERISK_BRIDGE_AUTH_MODE=bearer
• ASTERISK_BRIDGE_AUTH_FALLBACK_LEGACY=0
• VOICE_ADAPTER_TRUSTED_SERVICE_SUBJECTS contains svc:asterisk-bridge
• RECORDING_IMPORT_TRUSTED_SERVICE_SUBJECTS contains svc:asterisk-bridge
• RECORDING_IMPORT_ALLOW_ADMIN=0

Post-cutover validation:

python scripts\track9_preflight.py --base-url http://<gateway-host> --check-sftp
python scripts\asterisk_lab_smoke.py --base-url http://<gateway-host> --database-url <database-url> --require-recording
python scripts\track9_check.py --base-url http://<gateway-host> --database-url <database-url> --require-recording

Wave 2 / Track 9.2 controlled production cutover

Use the dedicated cutover runbook and script:

• track9-2-production-cutover.md
• scripts/track9_2_cutover.ps1

Dry-run (render/lint/snapshot only):

powershell -ExecutionPolicy Bypass -File scripts\track9_2_cutover.ps1 `
  -KubeContext <context> `
  -Namespace <namespace> `
  -Release <release> `
  -GatewayBaseUrl http://<gateway-host> `
  -DatabaseUrl postgresql://<...> `
  -ImageTag <image-tag> `
  -AmiHost <asterisk-vm-host> `
  -AmiUser <ami-user> `
  -AmiSecret <ami-secret> `
  -SftpHost <asterisk-vm-host> `
  -SftpUser <sftp-user> `
  -SftpPassword <sftp-password> `
  -QueueId <voice-lab-queue-id> `
  -AppTokenSecret <shared-app-token-secret>

Execute cutover:

powershell -ExecutionPolicy Bypass -File scripts\track9_2_cutover.ps1 `
  -KubeContext <context> `
  -Namespace <namespace> `
  -Release <release> `
  -GatewayBaseUrl http://<gateway-host> `
  -DatabaseUrl postgresql://<...> `
  -ImageTag <image-tag> `
  -AmiHost <asterisk-vm-host> `
  -AmiUser <ami-user> `
  -AmiSecret <ami-secret> `
  -SftpHost <asterisk-vm-host> `
  -SftpUser <sftp-user> `
  -SftpPassword <sftp-password> `
  -QueueId <voice-lab-queue-id> `
  -AppTokenSecret <shared-app-token-secret> `
  -Execute

If Helm reports HPA scale field conflicts, rerun the same command with -ForceUpgrade.