eblume/blumeops

Erich Blume 957ee90fa2 C2(deploy-infra-alerting): impl add ArgoCD scrape and sync alert

- Add ArgoCD metrics scrape target to Prometheus (argocd-metrics:8082)
- Add ArgoCDAppOutOfSync alert: fires when argocd_app_info has
  sync_status != Synced for 30 minutes
- Add runbook with diagnostic steps and common fixes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-22 13:45:34 -07:00

871 B

Raw Blame History

title

modified

tags

Observability

2026-03-22

operations

Observability

Metrics, logs, traces, and dashboards for BlumeOps infrastructure.

Components

prometheus - Metrics storage and querying
loki - Log aggregation
tempo - Distributed tracing
alloy - Metrics, log, and trace collection
grafana - Dashboards and visualization

Alerting

deploy-infra-alerting - Alerting pipeline (Grafana Unified Alerting → ntfy)
runbook-service-probe-failure - Service health check failure runbook
runbook-postgres-unhealthy - PostgreSQL cluster health runbook
runbook-pod-not-ready - Pod not ready runbook
runbook-textfile-stale - Metrics textfile freshness runbook
runbook-frigate-camera-down - Frigate camera health runbook
runbook-argocd-out-of-sync - ArgoCD sync status runbook