Observability

Forginate is instrumented end to end: Prometheus metrics, OTLP traces with cross-service continuity into runner pods, structured JSON logs, and an optional Grafana dashboard. All of it degrades gracefully — no observability stack is required to run the platform.

What the chart assumes (all optional)

Piece	Assumption	Values key
Metrics	prometheus-operator (ServiceMonitor CRD) watching your namespace	apiGateway.serviceMonitor.enabled, chatEditor.serviceMonitor.enabled
Dashboard	kube-prometheus-stack Grafana sidecar picking up ConfigMaps labelled grafana_dashboard: "1" in any namespace	rendered with the gateway ServiceMonitor
Traces	an OTLP gRPC collector (e.g. Grafana Alloy → Tempo)	observability.otlpEndpoint
Logs UI	Loki reachable in-cluster	apiGateway.lokiUrl

ServiceMonitor default is on

apiGateway.serviceMonitor.enabled defaults to true. Without the monitoring.coreos.com/v1 CRD installed, the Helm release fails to apply. Set it to false if you don't run prometheus-operator. If your Prometheus is installed via kube-prometheus-stack with Helm-values-scoped selectors, you also need serviceMonitorSelectorNilUsesHelmValues=false (or a matching label selector) for it to pick up ServiceMonitors in the Forginate namespace.

Metrics

• api-gateway exposes /metrics on its HTTP port (8080). The ServiceMonitor scrapes it every 30 s. Key series: forginate_http_requests_total (by route), forginate_http_request_duration_seconds_bucket (latency histograms), plus run counters used by the shipped dashboard.
• chat-editor exposes /metrics on 8090, including the chat_editor_active_streams gauge — this drives its HPA, so if you enable chatEditor.autoscaling with the custom metric you need a Prometheus adapter exposing pod metrics; otherwise only the CPU target fires.
• The Grafana dashboard ("Forginate · Overview", uid forginate-overview) is shipped as a ConfigMap with grafana_dashboard: "1"; the kube-prometheus-stack sidecar auto-provisions it. Panels: HTTP req/s by route, p50/p95/p99 latency, runs enqueued, and more.
• The ops screens in the admin UI can query Prometheus directly if you set a PROMETHEUS_URL env var on the gateway/orchestrator (not a chart value; empty disables those queries).

Traces (OTLP)

Every service boots an OTel SDK pointed at observability.otlpEndpoint (OTEL_EXPORTER_OTLP_ENDPOINT, default http://alloy.monitoring.svc.cluster.local:4317 — override to wherever your collector lives, or leave it dangling and spans are silently dropped).

Service names are fixed per component: forginate-api-gateway, forginate-orchestrator, forginate-chat-editor. Resource attributes include k8s.namespace.name, k8s.pod.name, and deployment.environment.

Trace continuity into runs: the orchestrator stamps TRACEPARENT and RUNNER_OTLP_ENDPOINT onto every factory-runner Job it creates, so runner spans join the request → orchestrator → runner trace. Runner spans carry forginate.run.id, forginate.org.id, and forginate.project.id attributes — filter by run ID in Tempo to see one run end to end.

The runs-namespace NetworkPolicy opens egress from runner pods to port 4317 on pods labelled app.kubernetes.io/name: alloy in a namespace named monitoring. If your collector lives elsewhere, adjust the policy in deploy/helm/forginate/templates/orchestrator.yaml or runner spans won't escape.

Logs

All services log structured JSON (pino) to stdout — collect with whatever you already run (Alloy/Promtail → Loki, Fluent Bit, etc.). Levels are controlled with LOG_LEVEL (gateway/orchestrator/chat-editor; the templates set info) and FORGINATE_LOG_LEVEL on the factory-runner.

If Loki is present, the admin UI's System Logs tab queries it via apiGateway.lokiUrl. Note the default targets the Loki gateway service on port 80, not the Loki API port 3100 — set it to whatever answers LogQL queries in your stack.

Health endpoints

Service	Endpoint	Used by
api-gateway	:8080/healthz	readiness + liveness probes, your external monitoring
chat-editor	:8090/healthz	probes (long timeouts — pod may be mid-stream)
minio	:9000/minio/health/ready	readiness probe
redis	redis-cli ping exec probe	readiness
deploy-agent (customer side)	:8087/healthz	returns 503 while disconnected; only a hung process restarts it

Minimal external monitoring

If you only wire up three alerts, make them:

1. https://forginate.example.com/healthz returns non-200 (edge + gateway).
2. The forginate-pg-backup CronJob's last Job failed or hasn't succeeded in > 26 h.
3. CNPG cluster forginate-db not Cluster in healthy state (kubectl get cluster -n forginate).

Start with metrics only

Traces and Loki are the most valuable during incident debugging but the least required day one. A fine minimal stack: kube-prometheus-stack + the shipped ServiceMonitor and dashboard, everything else off (observability.otlpEndpoint pointing at nothing costs you nothing but dropped spans).