eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/docs/reference/services/pyroscope.md
Erich Blume e7d3871144 Deploy Grafana Pyroscope for continuous eBPF profiling on ringtail 
Add Pyroscope server (StatefulSet on ringtail k3s) and Alloy profiling
DaemonSet with pyroscope.ebpf collection. Grafana datasource with
traces-to-profiles cross-linking. Docs updated across observability
reference card, Alloy, Grafana, apps registry, and README.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 14:05:57 -07:00

1.8 KiB
Raw Blame History

title modified tags
Pyroscope 2026-03-26
service
observability

Grafana Pyroscope

Continuous profiling backend for BlumeOps. Stores CPU profiles collected by Alloy's eBPF profiler on ringtail, providing function-level visibility into where compute time is spent.

Quick Reference

Property Value
URL https://pyroscope.tail8d86e.ts.net
Namespace pyroscope
Cluster ringtail (k3s)
Deployment StatefulSet (argocd/manifests/pyroscope/)
Image grafana/pyroscope
Port 4040
Storage 10Gi PVC at /data
Retention 7 days (max_query_lookback: 168h)

Architecture

Pyroscope runs on ringtail because eBPF profiling requires Linux. Grafana on indri queries it via Tailscale Ingress.

Alloy (pyroscope.ebpf on ringtail) → Pyroscope (ringtail) → Grafana (indri, via Tailscale)

Collection

Profiles are collected by the alloy-profiling-ringtail DaemonSet, which runs the pyroscope.ebpf component in privileged mode with hostPID: true. It discovers Kubernetes pods automatically and excludes infrastructure namespaces (kube-system, tailscale) and Alloy pods.

The eBPF profiler works without application instrumentation — it samples CPU stack traces from the kernel, covering native code (Go, C/C++), interpreted languages (Python, Ruby, Node.js), and JIT-compiled runtimes (.NET).

Limitations:

  • GPU workloads (e.g., Frigate inference via CUDA) are invisible to CPU profiling
  • Stripped binaries (no debug symbols) produce opaque stack frames
  • Python frame quality varies depending on runtime version

Related

  • alloy - Collection agent
  • observability - Full observability stack overview
  • grafana - Visualization
  • tempo - Distributed tracing (cross-linked via traces-to-profiles)