call-center/docs/runbooks/deployment-onprem.md

# Runbook - On-Prem Deployment

## Helm
- `helm upgrade --install mvp-cc ./deployment/helm -n mvp-cc --create-namespace`
- For PostgreSQL-backed Helm releases, the chart now runs `scripts/migrate_core_db.py` automatically via a pre-install / pre-upgrade migration Job.

## Raw manifests
- `kubectl apply -f deployment/kubernetes/mvp-cc-platform.yaml`
- Raw manifests remain a minimal reference/bootstrap path.
- Helm is the canonical production-scale deployment path for Wave 2 / Track 7.

## Core DB migration
- Helm path:
  - keep `migrations.enabled=true`
  - keep `SCHEMA_MANAGEMENT_MODE=migrations`
  - the Helm hook Job applies migrations before rollout
- Raw manifests / manual path:
  - run `python scripts/migrate_core_db.py`
  - then start services with `SCHEMA_MANAGEMENT_MODE=migrations`

## Wave 2 / Track 7 scale baseline

Production-like K8s scale validation requires:

- `DATABASE_URL` set to a shared PostgreSQL endpoint
- `DB_POOL_SIZE`
- `DB_MAX_OVERFLOW`
- `DB_POOL_TIMEOUT_SECONDS`
- `DB_POOL_RECYCLE_SECONDS`
- a working `metrics-server` in the target cluster

Recommended Wave 2 / Track 7 rollout:

1. Render Helm config for the scale profile.
2. Confirm `DATABASE_URL` points to PostgreSQL (not SQLite).
3. Confirm `SCHEMA_MANAGEMENT_MODE=migrations`.
4. Confirm `migrations.enabled=true`.
5. Deploy via Helm.
6. Verify the migration Job completed successfully.
7. Run `python scripts/load_test.py --base-url http://<gateway-host> --profile step_250_250 --auth-mode bearer`.
8. Run `python scripts/track7_check.py --namespace mvp-cc --report-dir <report_dir>`.
9. Repeat with `--profile target_500_500`.

For the complete procedure, use:

- [track7-scale-validation.md](/e:/Zhan/docs/runbooks/track7-scale-validation.md)

## Wave 2 / Track 1 auth configuration

Minimum enterprise auth variables:

- `APP_TOKEN_SECRET`
- `APP_TOKEN_TTL_SECONDS` (default `3600`)
- `ALLOW_LEGACY_HEADER_AUTH=0`
- `OIDC_ENABLED=1`
- `OIDC_PROVIDER=keycloak`
- `OIDC_ISSUER_URL`
- `OIDC_CLIENT_ID`
- `OIDC_CLIENT_SECRET`
- `OIDC_REDIRECT_URI`
- `OIDC_SCOPES` (default `openid profile email`)
- `OIDC_ROLE_CLAIM` (default `groups`)
- `OIDC_ROLE_MAP_JSON`

Helm and raw Kubernetes manifests now include placeholder values for these settings.
Replace all `change-me-*` values before deploying outside local/demo.

Recommended rollout sequence:

1. For Helm, keep migration hook enabled so schema upgrades run before deploy.
2. Deploy with `OIDC_ENABLED=1` and `ALLOW_LEGACY_HEADER_AUTH=1` in QA.
3. Run `python scripts/oidc_smoke.py --base-url http://<gateway-host> --require-enabled`.
4. Validate a real Keycloak login round trip.
5. Switch enterprise pilot to `ALLOW_LEGACY_HEADER_AUTH=0`.

## Post-deploy checks
- Pods ready state
- `/health` probes green
- Gateway registry endpoint available
- `GET /proxy/auth/auth/oidc/config` returns `enabled=true`
- `GET /proxy/auth/auth/oidc/health` returns provider metadata status

See also:

- [keycloak-oidc.md](/e:/Zhan/docs/runbooks/keycloak-oidc.md)
- [postgres-dev-cutover.md](/e:/Zhan/docs/runbooks/postgres-dev-cutover.md)
- [postgres-server-docker.md](/e:/Zhan/docs/runbooks/postgres-server-docker.md)
- [postgres-server-parallel-stack.md](/e:/Zhan/docs/runbooks/postgres-server-parallel-stack.md)

## Wave 2 / Track 9.1 strict bridge auth cutover

Use Helm overlays to force strict bridge auth mode in QA/enterprise environments:

```powershell
helm upgrade --install mvp-cc deployment\helm `
  -f deployment\helm\values.scale500.yaml `
  -f deployment\helm\values.track9-strict.yaml `
  --set-string auth.appTokenSecret=<shared-app-token-secret> `
  --set-string asteriskBridge.enabled=1 `
  --set-string asteriskBridge.amiHost=<asterisk-vm-host> `
  --set-string asteriskBridge.amiUsername=<ami-user> `
  --set-string asteriskBridge.amiSecret=<ami-secret> `
  --set-string asteriskBridge.queueMapJson='{\"voice_lab\":\"<queue_id>\"}' `
  --set-string asteriskBridge.sftpHost=<asterisk-vm-host> `
  --set-string asteriskBridge.sftpUsername=<sftp-user> `
  --set-string asteriskBridge.sftpPassword=<sftp-password> `
  -n mvp-cc --create-namespace
```

Expected strict baseline after deploy:

- `ALLOW_LEGACY_HEADER_AUTH=0`
- `ASTERISK_BRIDGE_AUTH_MODE=bearer`
- `ASTERISK_BRIDGE_AUTH_FALLBACK_LEGACY=0`
- `VOICE_ADAPTER_TRUSTED_SERVICE_SUBJECTS` contains `svc:asterisk-bridge`
- `RECORDING_IMPORT_TRUSTED_SERVICE_SUBJECTS` contains `svc:asterisk-bridge`
- `RECORDING_IMPORT_ALLOW_ADMIN=0`

Post-cutover validation:

```powershell
python scripts\track9_preflight.py --base-url http://<gateway-host> --check-sftp
python scripts\asterisk_lab_smoke.py --base-url http://<gateway-host> --database-url <database-url> --require-recording
python scripts\track9_check.py --base-url http://<gateway-host> --database-url <database-url> --require-recording
```

## Wave 2 / Track 9.2 controlled production cutover

Use the dedicated cutover runbook and script:

- [track9-2-production-cutover.md](/e:/Zhan/docs/runbooks/track9-2-production-cutover.md)
- `scripts/track9_2_cutover.ps1`

Dry-run (render/lint/snapshot only):

```powershell
powershell -ExecutionPolicy Bypass -File scripts\track9_2_cutover.ps1 `
  -KubeContext <context> `
  -Namespace <namespace> `
  -Release <release> `
  -GatewayBaseUrl http://<gateway-host> `
  -DatabaseUrl postgresql://<...> `
  -ImageTag <image-tag> `
  -AmiHost <asterisk-vm-host> `
  -AmiUser <ami-user> `
  -AmiSecret <ami-secret> `
  -SftpHost <asterisk-vm-host> `
  -SftpUser <sftp-user> `
  -SftpPassword <sftp-password> `
  -QueueId <voice-lab-queue-id> `
  -AppTokenSecret <shared-app-token-secret>
```

Execute cutover:

```powershell
powershell -ExecutionPolicy Bypass -File scripts\track9_2_cutover.ps1 `
  -KubeContext <context> `
  -Namespace <namespace> `
  -Release <release> `
  -GatewayBaseUrl http://<gateway-host> `
  -DatabaseUrl postgresql://<...> `
  -ImageTag <image-tag> `
  -AmiHost <asterisk-vm-host> `
  -AmiUser <ami-user> `
  -AmiSecret <ami-secret> `
  -SftpHost <asterisk-vm-host> `
  -SftpUser <sftp-user> `
  -SftpPassword <sftp-password> `
  -QueueId <voice-lab-queue-id> `
  -AppTokenSecret <shared-app-token-secret> `
  -Execute
```

If Helm reports HPA scale field conflicts, rerun the same command with `-ForceUpgrade`.