Erich Blume
8e6a803076
Extend Alloy blackbox probes:
- Add prometheus, loki, grafana, teslamate, immich, navidrome
- Now probing 11 services (was 5), covering most HTTP checks from
services-check
Add alert rules:
- PostgresClusterUnhealthy: cnpg_collector_up < 1 for 3m (critical)
- PodNotReady: kube_pod_status_ready{condition="true"} == 0 for 5m
Add runbooks:
- runbook-postgres-unhealthy.md
- runbook-pod-not-ready.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
25 lines
677 B
Markdown
25 lines
677 B
Markdown
---
|
|
title: Observability
|
|
modified: 2026-03-22
|
|
tags:
|
|
- operations
|
|
---
|
|
|
|
# Observability
|
|
|
|
Metrics, logs, traces, and dashboards for BlumeOps infrastructure.
|
|
|
|
## Components
|
|
|
|
- [[prometheus]] - Metrics storage and querying
|
|
- [[loki]] - Log aggregation
|
|
- [[tempo]] - Distributed tracing
|
|
- [[alloy|Alloy]] - Metrics, log, and trace collection
|
|
- [[grafana]] - Dashboards and visualization
|
|
|
|
## Alerting
|
|
|
|
- [[deploy-infra-alerting]] - Alerting pipeline (Grafana Unified Alerting → ntfy)
|
|
- [[runbook-service-probe-failure]] - Service health check failure runbook
|
|
- [[runbook-postgres-unhealthy]] - PostgreSQL cluster health runbook
|
|
- [[runbook-pod-not-ready]] - Pod not ready runbook
|