eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/docs/how-to/runbooks/runbook-frigate-camera-down.md
Erich Blume 6d65e6928c C2: Deploy infrastructure alerting pipeline (#303) 
## Summary

Mikado chain to replace `mise run services-check` with Grafana Unified Alerting backed by ntfy push notifications.

**Design:**
- Grafana Unified Alerting evaluates rules against Prometheus/Loki
- ntfy webhook contact point delivers iOS notifications
- Anti-noise policy: page once per 24h per alert group
- Every alert links to a runbook in `docs/how-to/alerts/`
- services-check eventually queries the alerting API instead of doing its own probes

**Chain (bottom-up):**
1. `configure-grafana-alerting-pipeline` — enable alerting, ntfy contact point, notification policy
2. `first-alert-and-runbook` — end-to-end proof of concept with blackbox probe failure
3. `port-services-check-alerts` — migrate all services-check probes to alert rules + runbooks
4. `refactor-services-check-to-query-alerts` — rewrite services-check to query Grafana API
5. `deploy-infra-alerting` — goal card

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #303
2026-03-22 14:52:56 -07:00

1.2 KiB
Raw Blame History

title modified tags
Runbook: Frigate Camera Down 2026-03-22
how-to
alerting
runbook

Runbook: Frigate Camera Down

Alert name: FrigateCameraDown

A Frigate camera has reported 0 FPS for 5+ minutes, meaning the camera feed is not being received.

Diagnostic Steps

  1. Check Frigate UI — https://nvr.ops.eblu.me — look at the camera thumbnail and status
  2. Check Frigate API stats: 
    curl -s https://nvr.ops.eblu.me/api/stats | python3 -m json.tool
  3. Check Frigate pod logs on ringtail: 
    kubectl logs -n frigate -l app=frigate --context=k3s-ringtail --tail=30
  4. Check the camera itself — verify it's powered on and network-connected. Try accessing the RTSP stream directly.

Common Causes

  • Camera offline — power outage, network issue, or camera crash
  • NFS mount lost — Frigate storage on sifaka; if the NFS mount drops, recording stops and FPS may drop
  • Frigate pod restart — during restart, camera FPS briefly drops to 0
  • RTSP stream timeout — camera firmware issue; power cycle the camera

Related