call-center/docs/runbooks/track7-scale-validation.md

Runbook - Track 7 Scale Validation

Goal

Validate the production-like scale500 profile for Wave 2 / Track 7 on K8s on-prem.

This runbook assumes:

• Helm is the canonical deployment path
• DATABASE_URL points to a shared PostgreSQL endpoint
• metrics-server is installed and healthy
• Helm migration hook is enabled

Prerequisites

• kubectl access to the target cluster
• helm installed locally
• namespace chosen (examples below use mvp-cc)
• Helm chart configured with:
  • global.schemaManagementMode=migrations
  • migrations.enabled=true

Configure Helm values

Before deployment, confirm:

• profiles.scale500.enabled=true
• global.databaseUrl is set to PostgreSQL
• global.schemaManagementMode=migrations
• migrations.enabled=true
• db.poolSize, db.maxOverflow, db.poolTimeoutSeconds, db.poolRecycleSeconds match the target baseline

Render and inspect:

helm lint deployment\helm
helm template mvp-cc deployment\helm -n mvp-cc

Deploy

helm upgrade --install mvp-cc deployment\helm -n mvp-cc --create-namespace

Wait until:

• migration Job mvp-cc-db-migrate completed successfully
• all expected pods are Ready
• HPA objects are created for the hot-path services
• recording-service stays singleton

Step validation (250 + 250)

Run the staged profile first:

python scripts\load_test.py --base-url http://<gateway-host> --profile step_250_250 --auth-mode bearer
python scripts\track7_check.py --namespace mvp-cc --report-dir .artifacts\track7\<timestamp> --require-success-rate 99 --require-p95-seconds 1.5 --require-p99-seconds 3.0

Expected:

• success rate >= 99%
• no crash loops or OOM kills
• no unexpected pod restarts

You can also run the staged helper, which writes a formal acceptance pack automatically:

python scripts\track7_stage.py --base-url http://<gateway-host> --namespace mvp-cc --auth-mode bearer

This creates:

• .artifacts/track7/staged_<timestamp>/step_250_250
• .artifacts/track7/staged_<timestamp>/target_500_500
• .artifacts/track7/staged_<timestamp>/acceptance_summary.json
• .artifacts/track7/staged_<timestamp>/acceptance_summary.md

Target validation (500 + 500)

Only after the step profile passes:

python scripts\load_test.py --base-url http://<gateway-host> --profile target_500_500 --auth-mode bearer
python scripts\track7_check.py --namespace mvp-cc --report-dir .artifacts\track7\<timestamp> --require-success-rate 99 --require-p95-seconds 2.0 --require-p99-seconds 3.5

Expected:

• 500 voice + 500 digital mixed profile
• success rate >= 99%
• p95 <= 2.0s
• p99 <= 3.5s
• 5xx / transport failure rate <= 0.5%

Evidence pack

Keep the following together:

• rendered Helm values/profile
• summary.json
• latency_samples.csv
• error_samples.json
• mix_profile.json
• output from track7_check.py
• rollout notes (cluster, namespace, date, profile, result)

Rollback

If the target profile fails:

1. Roll back the Helm release:

helm rollback mvp-cc

2. Keep the failed report directory for analysis.
3. Reduce to the last passing profile and re-run validation only after tuning.

Canonical scale500 values

Use the built-in preset as the base overlay for the production-like scale profile:

helm upgrade --install mvp-cc deployment\helm -f deployment\helm\values.scale500.yaml -n mvp-cc --create-namespace

Then override at least:

• global.databaseUrl
• image tag / repository as needed for the target cluster