Erich Blume
2fa536e547
- TextfileStale: fires when a .prom textfile on indri hasn't been updated in 1 hour (node_textfile_mtime_seconds). Covers borgmatic, zot, minikube, jellyfin exporters. - FrigateCameraDown: fires when frigate_camera_fps drops to 0 for 5m. - Add runbooks for both alerts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
27 lines
809 B
Markdown
27 lines
809 B
Markdown
---
|
|
title: Observability
|
|
modified: 2026-03-22
|
|
tags:
|
|
- operations
|
|
---
|
|
|
|
# Observability
|
|
|
|
Metrics, logs, traces, and dashboards for BlumeOps infrastructure.
|
|
|
|
## Components
|
|
|
|
- [[prometheus]] - Metrics storage and querying
|
|
- [[loki]] - Log aggregation
|
|
- [[tempo]] - Distributed tracing
|
|
- [[alloy|Alloy]] - Metrics, log, and trace collection
|
|
- [[grafana]] - Dashboards and visualization
|
|
|
|
## Alerting
|
|
|
|
- [[deploy-infra-alerting]] - Alerting pipeline (Grafana Unified Alerting → ntfy)
|
|
- [[runbook-service-probe-failure]] - Service health check failure runbook
|
|
- [[runbook-postgres-unhealthy]] - PostgreSQL cluster health runbook
|
|
- [[runbook-pod-not-ready]] - Pod not ready runbook
|
|
- [[runbook-textfile-stale]] - Metrics textfile freshness runbook
|
|
- [[runbook-frigate-camera-down]] - Frigate camera health runbook
|