Erich Blume
e7d3871144
Add Pyroscope server (StatefulSet on ringtail k3s) and Alloy profiling DaemonSet with pyroscope.ebpf collection. Grafana datasource with traces-to-profiles cross-linking. Docs updated across observability reference card, Alloy, Grafana, apps registry, and README. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
37 lines
1.7 KiB
Markdown
37 lines
1.7 KiB
Markdown
---
|
|
title: Observability
|
|
modified: 2026-03-26
|
|
tags:
|
|
- operations
|
|
---
|
|
|
|
# Observability
|
|
|
|
The four(+) pillars of observability — metrics, logs, traces, and profiles — collected and visualized via the Grafana ecosystem.
|
|
|
|
## Components
|
|
|
|
| Pillar | Backend | Collector | Cluster |
|
|
|--------|---------|-----------|---------|
|
|
| **Metrics** | [[prometheus]] | [[alloy]] | indri |
|
|
| **Logs** | [[loki]] | [[alloy]] | indri |
|
|
| **Traces** | [[tempo]] | [[alloy]] (Beyla eBPF) | indri (backend), ringtail (collection) |
|
|
| **Profiles** | [[pyroscope]] | [[alloy]] (pyroscope.ebpf) | ringtail |
|
|
|
|
All four are visualized in [[grafana]] with cross-signal linking (traces → logs, traces → profiles, traces → metrics).
|
|
|
|
## Future: Frontend Monitoring (RUM)
|
|
|
|
Grafana Faro is a Real User Monitoring SDK that captures page loads, web vitals, errors, and network timings from the browser, feeding into Loki (logs) and Tempo (traces) via Alloy's `faro.receiver` component. This would add an "outside-in" view of service health from the user's perspective.
|
|
|
|
**Not currently deployed.** RUM captures browsing behavior from visitors to public services, creating a data retention liability. Would require careful sanitization before deploying.
|
|
|
|
## Alerting
|
|
|
|
- [[deploy-infra-alerting]] - Alerting pipeline (Grafana Unified Alerting → ntfy)
|
|
- [[runbook-service-probe-failure]] - Service health check failure runbook
|
|
- [[runbook-postgres-unhealthy]] - PostgreSQL cluster health runbook
|
|
- [[runbook-pod-not-ready]] - Pod not ready runbook
|
|
- [[runbook-textfile-stale]] - Metrics textfile freshness runbook
|
|
- [[runbook-frigate-camera-down]] - Frigate camera health runbook
|
|
- [[runbook-argocd-out-of-sync]] - ArgoCD sync status runbook
|