# Runbook - Track 7 Scale Validation

## Goal

Validate the production-like `scale500` profile for Wave 2 / Track 7 on `K8s on-prem`.

This runbook assumes:

- Helm is the canonical deployment path
- `DATABASE_URL` points to a shared PostgreSQL endpoint
- `metrics-server` is installed and healthy
- Helm migration hook is enabled

## Prerequisites

- `kubectl` access to the target cluster
- `helm` installed locally
- namespace chosen (examples below use `mvp-cc`)
- Helm chart configured with:
  - `global.schemaManagementMode=migrations`
  - `migrations.enabled=true`

## Configure Helm values

Before deployment, confirm:

- `profiles.scale500.enabled=true`
- `global.databaseUrl` is set to PostgreSQL
- `global.schemaManagementMode=migrations`
- `migrations.enabled=true`
- `db.poolSize`, `db.maxOverflow`, `db.poolTimeoutSeconds`, `db.poolRecycleSeconds` match the target baseline

Render and inspect:

```powershell
helm lint deployment\helm
helm template mvp-cc deployment\helm -n mvp-cc
```

## Deploy

```powershell
helm upgrade --install mvp-cc deployment\helm -n mvp-cc --create-namespace
```

Wait until:

- migration Job `mvp-cc-db-migrate` completed successfully
- all expected pods are `Ready`
- HPA objects are created for the hot-path services
- `recording-service` stays singleton

## Step validation (`250 + 250`)

Run the staged profile first:

```powershell
python scripts\load_test.py --base-url http://<gateway-host> --profile step_250_250 --auth-mode bearer
python scripts\track7_check.py --namespace mvp-cc --report-dir .artifacts\track7\<timestamp> --require-success-rate 99 --require-p95-seconds 1.5 --require-p99-seconds 3.0
```

Expected:

- success rate `>= 99%`
- no crash loops or OOM kills
- no unexpected pod restarts

You can also run the staged helper, which writes a formal acceptance pack automatically:

```powershell
python scripts\track7_stage.py --base-url http://<gateway-host> --namespace mvp-cc --auth-mode bearer
```

This creates:

- `.artifacts/track7/staged_<timestamp>/step_250_250`
- `.artifacts/track7/staged_<timestamp>/target_500_500`
- `.artifacts/track7/staged_<timestamp>/acceptance_summary.json`
- `.artifacts/track7/staged_<timestamp>/acceptance_summary.md`

## Target validation (`500 + 500`)

Only after the step profile passes:

```powershell
python scripts\load_test.py --base-url http://<gateway-host> --profile target_500_500 --auth-mode bearer
python scripts\track7_check.py --namespace mvp-cc --report-dir .artifacts\track7\<timestamp> --require-success-rate 99 --require-p95-seconds 2.0 --require-p99-seconds 3.5
```

Expected:

- `500 voice + 500 digital` mixed profile
- success rate `>= 99%`
- p95 `<= 2.0s`
- p99 `<= 3.5s`
- `5xx` / transport failure rate `<= 0.5%`

## Evidence pack

Keep the following together:

- rendered Helm values/profile
- `summary.json`
- `latency_samples.csv`
- `error_samples.json`
- `mix_profile.json`
- output from `track7_check.py`
- rollout notes (cluster, namespace, date, profile, result)

## Rollback

If the target profile fails:

1. Roll back the Helm release:

```powershell
helm rollback mvp-cc
```

2. Keep the failed report directory for analysis.
3. Reduce to the last passing profile and re-run validation only after tuning.

## Canonical scale500 values

Use the built-in preset as the base overlay for the production-like scale profile:

```powershell
helm upgrade --install mvp-cc deployment\helm -f deployment\helm\values.scale500.yaml -n mvp-cc --create-namespace
```

Then override at least:

- `global.databaseUrl`
- image tag / repository as needed for the target cluster