eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Add preserve/* branch protection and document Pyroscope blocker

Browse source 
branch-cleanup: Add PROTECTED_PREFIXES with preserve/* exclusion so
preserved work-in-progress branches are never deleted.

observability.md: Document Pyroscope profiling work on branch
preserve/pyroscope-profiling/pr-313, blocked on ringtail kernel
sysctl settings (kptr_restrict=0, perf_event_paranoid≤1). Also
document Faro/RUM as future potential with privacy considerations.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Erich Blume 2026-03-26 15:32:25 -07:00
commit fc8d2cdb12
3 changed files with 27 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

18
docs/reference/operations/observability.md
View file

@ -1,6 +1,6 @@
---
title: Observability
modified: 2026-03-22
modified: 2026-03-26
tags:
- operations
---
@ -17,6 +17,22 @@ Metrics, logs, traces, and dashboards for BlumeOps infrastructure.
- [[alloy|Alloy]] - Metrics, log, and trace collection
- [[grafana]] - Dashboards and visualization
## Future: Continuous Profiling (Pyroscope)
Full implementation on branch `preserve/pyroscope-profiling/pr-313` (PR #313, closed). Includes Pyroscope server (StatefulSet on ringtail), Alloy profiling DaemonSet (`pyroscope.ebpf`), Grafana datasource with traces-to-profiles linking, Nix container build with embedded frontend, and documentation.
**Blocked on ringtail kernel sysctl settings.** The `pyroscope.ebpf` Alloy component requires:
- `kernel.kptr_restrict = 0` (currently `1` — kallsyms addresses are zeroed)
- `kernel.perf_event_paranoid ≤ 1` (currently `2` — eBPF perf events restricted)
These must be set in ringtail's NixOS configuration (`boot.kernel.sysctl`). Once applied, the branch can be rebased onto main and deployed.
## Future: Frontend Monitoring (RUM)
Grafana Faro is a Real User Monitoring SDK that captures page loads, web vitals, errors, and network timings from the browser, feeding into Loki (logs) and Tempo (traces) via Alloy's `faro.receiver` component. This would add an "outside-in" view of service health from the user's perspective.
**Not currently deployed.** RUM captures browsing behavior from visitors to public services, creating a data retention liability. Would require careful sanitization before deploying.
## Alerting
- [[deploy-infra-alerting]] - Alerting pipeline (Grafana Unified Alerting → ntfy)