Backups

The chart ships a daily logical Postgres backup out of the box. Object storage and Kubernetes secrets need operator-side rituals. This page covers all three.

What's backed up automatically

Layer	Mechanism	Destination	Retention
Postgres (forginate DB)	pg_dump | gzip via CronJob	minio/forginate-backups/<env>/daily/forginate-<ts>.sql.gz	backup.retentionDays (14)
MinIO objects (artifacts, docs, attachments)	not automated — mirror it yourself	off-cluster S3/rsync	your policy
Kubernetes secrets	not automated — export ritual below	GPG file off-cluster	indefinite

The backup CronJob

Values under backup.*: enabled (default true), image (postgres:17-bookworm), schedule ("17 3 * * *" — daily 03:17 UTC, offset from cron herds), bucket (forginate-backups), retentionDays (14).

Mechanics (see deploy/helm/forginate/templates/backup-cronjob.yaml):

• Connects to PGHOST=<postgres.name>-rw with creds from forginate-db-app-user.
• Installs curl and fetches the MinIO mc client into /tmp on first run (~10 s prep; avoids maintaining a custom backup image). This needs outbound internet from the backup pod.
• Sanity check: aborts if the gzipped dump is under 1 KB (catches "auth worked but DB is empty").
• Creates the bucket idempotently (mc mb --ignore-existing), uploads, then prunes objects older than RETENTION_DAYS with mc rm --older-than.
• Finished pods are kept 24 h (ttlSecondsAfterFinished) so logs stay inspectable.

CNPG WAL archiving is deliberately not enabled by the chart. Logical dumps restore cleanly into any compatible Postgres — even outside the cluster. For production, consider enabling .spec.backup.barmanObjectStore on the CNPG Cluster for point-in-time recovery and keep this CronJob as belt-and-suspenders.

Manual operations

Substitute your namespace for forginate throughout.

Trigger a backup now

kubectl create job --from=cronjob/forginate-pg-backup "manual-$(date +%s)" -n forginate
kubectl logs -n forginate -l app.kubernetes.io/component=pg-backup --tail=200 -f

List backups

USER=$(kubectl get secret -n forginate forginate-minio -o jsonpath='{.data.root-user}' | base64 -d)
PASS=$(kubectl get secret -n forginate forginate-minio -o jsonpath='{.data.root-password}' | base64 -d)
kubectl run mc-ls --rm -it --restart=Never -n forginate --image=minio/mc:latest \
  --env="MC_HOST_minio=http://${USER}:${PASS}@minio:9000" \
  -- ls minio/forginate-backups/prod/daily/

Pull a backup to your workstation

kubectl port-forward -n forginate svc/minio 9000:9000 &
mc alias set local http://localhost:9000 "$USER" "$PASS"
mc cp local/forginate-backups/prod/daily/forginate-<TS>.sql.gz ./

Restore drill (run monthly)

Restore into a throwaway database in the same CNPG cluster, sanity-check, drop:

DUMP=forginate-<TS>.sql.gz
kubectl exec -n forginate forginate-db-1 -c postgres -- \
  psql -U postgres -c 'CREATE DATABASE forginate_restore_drill;'
gunzip -c "$DUMP" | kubectl exec -i -n forginate forginate-db-1 -c postgres -- \
  psql -U postgres -d forginate_restore_drill
kubectl exec -n forginate forginate-db-1 -c postgres -- \
  psql -U postgres -d forginate_restore_drill -c "
  SELECT 'orgs',  count(*) FROM organizations
  UNION ALL SELECT 'users', count(*) FROM users
  UNION ALL SELECT 'runs',  count(*) FROM runs
  UNION ALL SELECT 'audit', count(*) FROM audit_events;"
kubectl exec -n forginate forginate-db-1 -c postgres -- \
  psql -U postgres -c 'DROP DATABASE forginate_restore_drill;'

A backup is verified when the CronJob log ends [backup] done, the object exists with size > 1 KB, and the drill returns non-zero counts on organizations, users, and audit_events. An unverified backup is no backup.

Real disaster restore

1. Stop writers: kubectl scale deploy/api-gateway deploy/orchestrator -n forginate --replicas=0
2. Pull the most recent good dump.
3. Cluster alive but data corrupted: DROP DATABASE forginate; CREATE DATABASE forginate OWNER forginate; as postgres, then stream the dump in. Cluster gone: helm upgrade recreates a fresh CNPG cluster; wait for READY, then stream the dump in.
4. Scale writers back up. Boot-time migrations are idempotent against the restored schema.

MinIO mirror (operator-side)

Run artifacts, docs (forginate-docs), and chat attachments (forginate-chat-attachments) are not inside the pg dumps. Mirror them off-cluster, e.g. nightly:

mc alias set off https://s3.example.com <key> <secret>
mc mirror --remove --quiet --overwrite minio/forginate-docs off/forginate-docs-mirror

Secret export ritual (quarterly)

Secrets live only in etcd. Some are unrecoverable if lost:

Secret	Why it matters
forginate-credential-key	Encrypts all tenant BYO provider keys — loss = every tenant re-enters credentials
forginate-minio	MinIO data unreadable without it
forginate-db-app-user	Database access
forginate-github-app	Rotate-able but disruptive
forginate-stripe	Account state not re-mintable

mkdir -p /tmp/forginate-export
for s in forginate-stripe forginate-github-app forginate-credential-key \
         forginate-minio forginate-db-app-user; do
  kubectl get secret -n forginate "$s" -o yaml > /tmp/forginate-export/${s}.yaml
done
tar czf - -C /tmp forginate-export \
  | gpg --symmetric --cipher-algo AES256 -o ~/forginate-secrets-$(date +%Y%m%d).tar.gz.gpg
rm -rf /tmp/forginate-export
# Move the .gpg file OFF the cluster and off the admin workstation.

WARNING

Store the GPG passphrase separately from the archive. An encrypted backup you can't decrypt — or one sitting next to its key — is theatre.