Erich Blume
e7d3871144
Add Pyroscope server (StatefulSet on ringtail k3s) and Alloy profiling DaemonSet with pyroscope.ebpf collection. Grafana datasource with traces-to-profiles cross-linking. Docs updated across observability reference card, Alloy, Grafana, apps registry, and README. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
50 lines
1.8 KiB
Markdown
50 lines
1.8 KiB
Markdown
---
|
|
title: Pyroscope
|
|
modified: 2026-03-26
|
|
tags:
|
|
- service
|
|
- observability
|
|
---
|
|
|
|
# Grafana Pyroscope
|
|
|
|
Continuous profiling backend for BlumeOps. Stores CPU profiles collected by Alloy's eBPF profiler on ringtail, providing function-level visibility into where compute time is spent.
|
|
|
|
## Quick Reference
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| **URL** | https://pyroscope.tail8d86e.ts.net |
|
|
| **Namespace** | `pyroscope` |
|
|
| **Cluster** | ringtail (k3s) |
|
|
| **Deployment** | StatefulSet (`argocd/manifests/pyroscope/`) |
|
|
| **Image** | `grafana/pyroscope` |
|
|
| **Port** | 4040 |
|
|
| **Storage** | 10Gi PVC at `/data` |
|
|
| **Retention** | 7 days (`max_query_lookback: 168h`) |
|
|
|
|
## Architecture
|
|
|
|
Pyroscope runs on ringtail because eBPF profiling requires Linux. Grafana on indri queries it via Tailscale Ingress.
|
|
|
|
```
|
|
Alloy (pyroscope.ebpf on ringtail) → Pyroscope (ringtail) → Grafana (indri, via Tailscale)
|
|
```
|
|
|
|
## Collection
|
|
|
|
Profiles are collected by the `alloy-profiling-ringtail` DaemonSet, which runs the `pyroscope.ebpf` component in privileged mode with `hostPID: true`. It discovers Kubernetes pods automatically and excludes infrastructure namespaces (`kube-system`, `tailscale`) and Alloy pods.
|
|
|
|
The eBPF profiler works without application instrumentation — it samples CPU stack traces from the kernel, covering native code (Go, C/C++), interpreted languages (Python, Ruby, Node.js), and JIT-compiled runtimes (.NET).
|
|
|
|
**Limitations:**
|
|
- GPU workloads (e.g., Frigate inference via CUDA) are invisible to CPU profiling
|
|
- Stripped binaries (no debug symbols) produce opaque stack frames
|
|
- Python frame quality varies depending on runtime version
|
|
|
|
## Related
|
|
|
|
- [[alloy]] - Collection agent
|
|
- [[observability]] - Full observability stack overview
|
|
- [[grafana]] - Visualization
|
|
- [[tempo]] - Distributed tracing (cross-linked via traces-to-profiles)
|