Erich Blume
8e6a803076
Extend Alloy blackbox probes:
- Add prometheus, loki, grafana, teslamate, immich, navidrome
- Now probing 11 services (was 5), covering most HTTP checks from
services-check
Add alert rules:
- PostgresClusterUnhealthy: cnpg_collector_up < 1 for 3m (critical)
- PodNotReady: kube_pod_status_ready{condition="true"} == 0 for 5m
Add runbooks:
- runbook-postgres-unhealthy.md
- runbook-pod-not-ready.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
677 B
677 B
| title | modified | tags | |
|---|---|---|---|
| Observability | 2026-03-22 |
|
Observability
Metrics, logs, traces, and dashboards for BlumeOps infrastructure.
Components
- prometheus - Metrics storage and querying
- loki - Log aggregation
- tempo - Distributed tracing
- alloy - Metrics, log, and trace collection
- grafana - Dashboards and visualization
Alerting
- deploy-infra-alerting - Alerting pipeline (Grafana Unified Alerting → ntfy)
- runbook-service-probe-failure - Service health check failure runbook
- runbook-postgres-unhealthy - PostgreSQL cluster health runbook
- runbook-pod-not-ready - Pod not ready runbook