135 lines
3.5 KiB
Markdown
135 lines
3.5 KiB
Markdown
# Runbook - Track 7 Scale Validation
|
|
|
|
## Goal
|
|
|
|
Validate the production-like `scale500` profile for Wave 2 / Track 7 on `K8s on-prem`.
|
|
|
|
This runbook assumes:
|
|
|
|
- Helm is the canonical deployment path
|
|
- `DATABASE_URL` points to a shared PostgreSQL endpoint
|
|
- `metrics-server` is installed and healthy
|
|
- Helm migration hook is enabled
|
|
|
|
## Prerequisites
|
|
|
|
- `kubectl` access to the target cluster
|
|
- `helm` installed locally
|
|
- namespace chosen (examples below use `mvp-cc`)
|
|
- Helm chart configured with:
|
|
- `global.schemaManagementMode=migrations`
|
|
- `migrations.enabled=true`
|
|
|
|
## Configure Helm values
|
|
|
|
Before deployment, confirm:
|
|
|
|
- `profiles.scale500.enabled=true`
|
|
- `global.databaseUrl` is set to PostgreSQL
|
|
- `global.schemaManagementMode=migrations`
|
|
- `migrations.enabled=true`
|
|
- `db.poolSize`, `db.maxOverflow`, `db.poolTimeoutSeconds`, `db.poolRecycleSeconds` match the target baseline
|
|
|
|
Render and inspect:
|
|
|
|
```powershell
|
|
helm lint deployment\helm
|
|
helm template mvp-cc deployment\helm -n mvp-cc
|
|
```
|
|
|
|
## Deploy
|
|
|
|
```powershell
|
|
helm upgrade --install mvp-cc deployment\helm -n mvp-cc --create-namespace
|
|
```
|
|
|
|
Wait until:
|
|
|
|
- migration Job `mvp-cc-db-migrate` completed successfully
|
|
- all expected pods are `Ready`
|
|
- HPA objects are created for the hot-path services
|
|
- `recording-service` stays singleton
|
|
|
|
## Step validation (`250 + 250`)
|
|
|
|
Run the staged profile first:
|
|
|
|
```powershell
|
|
python scripts\load_test.py --base-url http://<gateway-host> --profile step_250_250 --auth-mode bearer
|
|
python scripts\track7_check.py --namespace mvp-cc --report-dir .artifacts\track7\<timestamp> --require-success-rate 99 --require-p95-seconds 1.5 --require-p99-seconds 3.0
|
|
```
|
|
|
|
Expected:
|
|
|
|
- success rate `>= 99%`
|
|
- no crash loops or OOM kills
|
|
- no unexpected pod restarts
|
|
|
|
You can also run the staged helper, which writes a formal acceptance pack automatically:
|
|
|
|
```powershell
|
|
python scripts\track7_stage.py --base-url http://<gateway-host> --namespace mvp-cc --auth-mode bearer
|
|
```
|
|
|
|
This creates:
|
|
|
|
- `.artifacts/track7/staged_<timestamp>/step_250_250`
|
|
- `.artifacts/track7/staged_<timestamp>/target_500_500`
|
|
- `.artifacts/track7/staged_<timestamp>/acceptance_summary.json`
|
|
- `.artifacts/track7/staged_<timestamp>/acceptance_summary.md`
|
|
|
|
## Target validation (`500 + 500`)
|
|
|
|
Only after the step profile passes:
|
|
|
|
```powershell
|
|
python scripts\load_test.py --base-url http://<gateway-host> --profile target_500_500 --auth-mode bearer
|
|
python scripts\track7_check.py --namespace mvp-cc --report-dir .artifacts\track7\<timestamp> --require-success-rate 99 --require-p95-seconds 2.0 --require-p99-seconds 3.5
|
|
```
|
|
|
|
Expected:
|
|
|
|
- `500 voice + 500 digital` mixed profile
|
|
- success rate `>= 99%`
|
|
- p95 `<= 2.0s`
|
|
- p99 `<= 3.5s`
|
|
- `5xx` / transport failure rate `<= 0.5%`
|
|
|
|
## Evidence pack
|
|
|
|
Keep the following together:
|
|
|
|
- rendered Helm values/profile
|
|
- `summary.json`
|
|
- `latency_samples.csv`
|
|
- `error_samples.json`
|
|
- `mix_profile.json`
|
|
- output from `track7_check.py`
|
|
- rollout notes (cluster, namespace, date, profile, result)
|
|
|
|
## Rollback
|
|
|
|
If the target profile fails:
|
|
|
|
1. Roll back the Helm release:
|
|
|
|
```powershell
|
|
helm rollback mvp-cc
|
|
```
|
|
|
|
2. Keep the failed report directory for analysis.
|
|
3. Reduce to the last passing profile and re-run validation only after tuning.
|
|
|
|
## Canonical scale500 values
|
|
|
|
Use the built-in preset as the base overlay for the production-like scale profile:
|
|
|
|
```powershell
|
|
helm upgrade --install mvp-cc deployment\helm -f deployment\helm\values.scale500.yaml -n mvp-cc --create-namespace
|
|
```
|
|
|
|
Then override at least:
|
|
|
|
- `global.databaseUrl`
|
|
- image tag / repository as needed for the target cluster
|