Upgrading

Forginate upgrades are plain helm upgrade operations. Database migrations run automatically and idempotently on api-gateway boot, so a normal upgrade is: bump image tags, apply, watch pods.

Image tagging

CI builds and pushes images to GHCR tagged sha-<full-git-sha>:

• ghcr.io/the-pizza/forginate-api-gateway
• ghcr.io/the-pizza/forginate-orchestrator
• ghcr.io/the-pizza/forginate-factory-runner
• ghcr.io/the-pizza/forginate-qa-runner
• ghcr.io/the-pizza/forginate-chat-editor

Pin SHA tags, not latest

The chart defaults to latest, which is fine for a first look but wrong for operations: with imagePullPolicy: IfNotPresent a node that already has latest cached will never pull the new build. Pin sha-… tags in your values file so every rollout is explicit and reproducible.

Keep all five tags on the same SHA. The runner images matter even though nothing is running them at upgrade time — the orchestrator reads RUNNER_IMAGE / QA_RUNNER_IMAGE from its env and pulls them at Job-spawn time, so bumping orchestrator.runnerImage.tag / qaRunnerImage.tag takes effect on the next run, not on any pod restart.

Standard upgrade procedure

# values-prod.yaml — bump all five tags together
apiGateway:
  image: { repository: ghcr.io/the-pizza/forginate-api-gateway, tag: "sha-<sha>" }
orchestrator:
  image:         { repository: ghcr.io/the-pizza/forginate-orchestrator,   tag: "sha-<sha>" }
  runnerImage:   { repository: ghcr.io/the-pizza/forginate-factory-runner, tag: "sha-<sha>" }
  qaRunnerImage: { repository: ghcr.io/the-pizza/forginate-qa-runner,      tag: "sha-<sha>" }
chatEditor:
  image: { repository: ghcr.io/the-pizza/forginate-chat-editor, tag: "sha-<sha>" }

helm upgrade --install forginate deploy/helm/forginate \
  -n forginate -f values-prod.yaml --wait --timeout 5m

# Verify every pod landed on the new SHA
kubectl -n forginate get pods \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

# Health
curl -s https://forginate.example.com/healthz

What happens during the rollout

• api-gateway — brief connection churn as pods roll; the SPA reconnects WebSockets automatically. Migrations run on the first new pod's boot.
• chat-editor — RollingUpdate with maxSurge: 1, maxUnavailable: 0 and a 90 s terminationGracePeriodSeconds: draining pods keep serving in-flight SSE streams while new pods come up.
• orchestrator — safe to restart at any time. Run state is in Postgres; a restarted orchestrator re-adopts running Jobs, and the stuck-run reaper only fails runs whose heartbeat has been silent past the timeout.
• In-flight factory runs are untouched. Runner pods are one-shot Jobs already running the image they started with; they finish on the old code. Only runs started after the upgrade use the new runner image. Version-skew on the runner ↔ gateway contract is handled defensively (e.g. legacy usage payloads are accepted), but don't let runner and gateway tags drift for weeks.

Migrations and rollback

Migrations are forward-only SQL applied idempotently at gateway boot (saas/shared/db/src/migrations/). Consequences:

• Upgrades: no manual migration step, ever.
• Rollbacks: helm rollback forginate <rev> restores old images, but the schema stays at the newer version. Rolling back across a release that added a migration is usually safe (old code ignores new columns) but is not a tested path — prefer roll-forward. For anything risky, take a manual backup first:

kubectl create job --from=cronjob/forginate-pg-backup "pre-upgrade-$(date +%s)" -n forginate

Upgrading the dependencies

• CNPG operator: upgrade per the CNPG docs; the forginate-db Cluster CR is untouched by chart upgrades. Postgres major version bumps (the postgres.image value) are a CNPG-managed operation — read the CNPG major-upgrade guide before touching it; don't just bump the tag.
• Redis / MinIO images: chart values; both are single-replica StatefulSets, so a bump means a short outage for sessions/rate-limits (Redis) and object storage (MinIO) while the pod restarts. Data persists on the PVCs.
• cert-manager / ingress controller / prometheus-operator: normal cluster operations, independent of the release.

Post-upgrade checklist

1. All pods on the expected SHA (jsonpath command above).
2. /healthz green through the ingress.
3. Sign in and start a small test run — verifies gateway ↔ orchestrator ↔ runner on the new build end to end.
4. Check gateway logs for migration output/errors: kubectl -n forginate logs deploy/api-gateway | head -50.
5. If the release touched the backup CronJob or CNPG values, trigger a manual backup and confirm [backup] done.

Watch a canary run

The single most informative post-upgrade check is a real factory run from a test org: it exercises auth, manifest fetch, the internal listener, Job scheduling, the runner image, event streaming, and usage reporting in one shot.