Erich Blume
6d65e6928c
## Summary Mikado chain to replace `mise run services-check` with Grafana Unified Alerting backed by ntfy push notifications. **Design:** - Grafana Unified Alerting evaluates rules against Prometheus/Loki - ntfy webhook contact point delivers iOS notifications - Anti-noise policy: page once per 24h per alert group - Every alert links to a runbook in `docs/how-to/alerts/` - services-check eventually queries the alerting API instead of doing its own probes **Chain (bottom-up):** 1. `configure-grafana-alerting-pipeline` — enable alerting, ntfy contact point, notification policy 2. `first-alert-and-runbook` — end-to-end proof of concept with blackbox probe failure 3. `port-services-check-alerts` — migrate all services-check probes to alert rules + runbooks 4. `refactor-services-check-to-query-alerts` — rewrite services-check to query Grafana API 5. `deploy-infra-alerting` — goal card 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #303
39 lines
1.2 KiB
Markdown
39 lines
1.2 KiB
Markdown
---
|
|
title: "Runbook: Frigate Camera Down"
|
|
modified: 2026-03-22
|
|
tags:
|
|
- how-to
|
|
- alerting
|
|
- runbook
|
|
---
|
|
|
|
# Runbook: Frigate Camera Down
|
|
|
|
**Alert name:** `FrigateCameraDown`
|
|
|
|
A Frigate camera has reported 0 FPS for 5+ minutes, meaning the camera feed is not being received.
|
|
|
|
## Diagnostic Steps
|
|
|
|
1. **Check Frigate UI** — https://nvr.ops.eblu.me — look at the camera thumbnail and status
|
|
2. **Check Frigate API stats**:
|
|
```fish
|
|
curl -s https://nvr.ops.eblu.me/api/stats | python3 -m json.tool
|
|
```
|
|
3. **Check Frigate pod logs** on ringtail:
|
|
```fish
|
|
kubectl logs -n frigate -l app=frigate --context=k3s-ringtail --tail=30
|
|
```
|
|
4. **Check the camera itself** — verify it's powered on and network-connected. Try accessing the RTSP stream directly.
|
|
|
|
## Common Causes
|
|
|
|
- **Camera offline** — power outage, network issue, or camera crash
|
|
- **NFS mount lost** — Frigate storage on sifaka; if the NFS mount drops, recording stops and FPS may drop
|
|
- **Frigate pod restart** — during restart, camera FPS briefly drops to 0
|
|
- **RTSP stream timeout** — camera firmware issue; power cycle the camera
|
|
|
|
## Related
|
|
|
|
- [[frigate]] — Frigate NVR reference
|
|
- [[deploy-infra-alerting]] — Alerting pipeline overview
|