Add preserve/* branch protection and document Pyroscope blocker

branch-cleanup: Add PROTECTED_PREFIXES with preserve/* exclusion so
preserved work-in-progress branches are never deleted.

observability.md: Document Pyroscope profiling work on branch
preserve/pyroscope-profiling/pr-313, blocked on ringtail kernel
sysctl settings (kptr_restrict=0, perf_event_paranoid≤1). Also
document Faro/RUM as future potential with privacy considerations.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Erich Blume 2026-03-26 15:32:25 -07:00
commit fc8d2cdb12
3 changed files with 27 additions and 4 deletions

View file

@ -0,0 +1 @@
Add `preserve/*` branch prefix exclusion to `branch-cleanup` task; document Pyroscope profiling work and blockers in observability reference.

View file

@ -1,6 +1,6 @@
---
title: Observability
modified: 2026-03-22
modified: 2026-03-26
tags:
- operations
---
@ -17,6 +17,22 @@ Metrics, logs, traces, and dashboards for BlumeOps infrastructure.
- [[alloy|Alloy]] - Metrics, log, and trace collection
- [[grafana]] - Dashboards and visualization
## Future: Continuous Profiling (Pyroscope)
Full implementation on branch `preserve/pyroscope-profiling/pr-313` (PR #313, closed). Includes Pyroscope server (StatefulSet on ringtail), Alloy profiling DaemonSet (`pyroscope.ebpf`), Grafana datasource with traces-to-profiles linking, Nix container build with embedded frontend, and documentation.
**Blocked on ringtail kernel sysctl settings.** The `pyroscope.ebpf` Alloy component requires:
- `kernel.kptr_restrict = 0` (currently `1` — kallsyms addresses are zeroed)
- `kernel.perf_event_paranoid ≤ 1` (currently `2` — eBPF perf events restricted)
These must be set in ringtail's NixOS configuration (`boot.kernel.sysctl`). Once applied, the branch can be rebased onto main and deployed.
## Future: Frontend Monitoring (RUM)
Grafana Faro is a Real User Monitoring SDK that captures page loads, web vitals, errors, and network timings from the browser, feeding into Loki (logs) and Tempo (traces) via Alloy's `faro.receiver` component. This would add an "outside-in" view of service health from the user's perspective.
**Not currently deployed.** RUM captures browsing behavior from visitors to public services, creating a data retention liability. Would require careful sanitization before deploying.
## Alerting
- [[deploy-infra-alerting]] - Alerting pipeline (Grafana Unified Alerting → ntfy)

View file

@ -44,12 +44,18 @@ from rich.console import Console
from rich.table import Table
PROTECTED_BRANCHES = {"main", "master"}
PROTECTED_PREFIXES = ("preserve/",)
FORGE_API = "https://forge.eblu.me/api/v1"
REPO_OWNER = "eblume"
REPO_NAME = "blumeops"
OP_TOKEN_REF = "op://vg6xf6vvfmoh5hqjjhlhbeoaie/w3663ffnvkewbftncqxtcpeavy/api-token"
def is_protected(name: str) -> bool:
"""Check if a branch is protected by name or prefix."""
return name in PROTECTED_BRANCHES or name.startswith(PROTECTED_PREFIXES)
def run_git(*args: str) -> str:
"""Run a git command and return stdout."""
result = subprocess.run(
@ -113,7 +119,7 @@ def get_git_merged_local_branches() -> set[str]:
branches = set()
for line in output.splitlines():
name = line.strip().lstrip("* ")
if name and name not in PROTECTED_BRANCHES:
if name and not is_protected(name):
branches.add(name)
return branches
@ -143,7 +149,7 @@ def get_all_local_branches() -> set[str]:
branches = set()
for line in output.splitlines():
name = line.strip().lstrip("* ")
if name and name not in PROTECTED_BRANCHES:
if name and not is_protected(name):
branches.add(name)
return branches
@ -164,7 +170,7 @@ def get_api_branches(client: httpx.Client) -> dict[str, str]:
break
for branch in data:
name = branch["name"]
if name not in PROTECTED_BRANCHES:
if not is_protected(name):
date = branch.get("commit", {}).get("timestamp", "")
branches[name] = date
page += 1