blumeops/argocd/manifests/alloy-tracing-ringtail/config.alloy
Erich Blume c281fb5403 Add OpenTelemetry distributed tracing (Tempo + Beyla eBPF) (#286)
## Summary

Adds the third observability pillar — **distributed tracing** — alongside existing metrics (Prometheus) and logs (Loki).

- **Grafana Tempo 2.10.1** on minikube-indri for trace storage with 7d retention, OTLP receivers, and `metrics_generator` that remote-writes span-metrics (RED) to Prometheus
- **Beyla eBPF auto-instrumentation** via a privileged Alloy DaemonSet on ringtail — instruments HTTP services (Frigate, ntfy, Ollama, Immich) without code changes
- **Grafana integration** — Tempo datasource with trace↔log and trace↔metrics correlation, plus Loki derivedFields for trace ID linking
- **Prometheus** scrapes Tempo operational metrics

### Architecture

```
ringtail (k3s)                                indri (minikube)
┌──────────────────────┐                      ┌─────────────────────┐
│ Alloy+Beyla (eBPF)   │──OTLP HTTP────────→ │ Tempo               │
│  ↳ Frigate, ntfy,    │  via tailnet         │  ↳ trace storage    │
│    Ollama, Immich     │                      │  ↳ RED → Prometheus │
└──────────────────────┘                      │                     │
                                              │ Grafana             │
                                              │  ↳ Tempo datasource │
                                              └─────────────────────┘
```

### New files (12)
- `docs/reference/services/tempo.md` — reference doc
- `docs/changelog.d/feature-otel-tracing.feature.md`
- `argocd/apps/tempo.yaml` + `argocd/manifests/tempo/` (6 files)
- `argocd/apps/alloy-tracing-ringtail.yaml` + `argocd/manifests/alloy-tracing-ringtail/` (4 files)

### Modified files (6)
- `argocd/manifests/grafana/datasources.yaml` — Tempo datasource + Loki derivedFields
- `argocd/manifests/prometheus/prometheus.yml` — Tempo scrape target
- `service-versions.yaml` — tempo + alloy-tracing-ringtail entries
- `docs/reference/services/grafana.md` — Tempo in datasources table
- `docs/reference/reference.md` — Tempo in services index
- `docs/reference/operations/observability.md` — Tempo in components list

## Deployment and Testing

- [ ] Sync `apps` app to pick up new Application definitions
- [ ] `argocd app set tempo --revision feature/otel-tracing && argocd app sync tempo`
- [ ] Verify Tempo pod: `kubectl --context=minikube-indri get pods -n monitoring -l app=tempo`
- [ ] Verify Tempo ready: port-forward 3200 and `curl localhost:3200/ready`
- [ ] Verify Tailscale ingresses: `kubectl --context=minikube-indri get ingress -n monitoring`
- [ ] `argocd app set alloy-tracing-ringtail --revision feature/otel-tracing && argocd app sync alloy-tracing-ringtail`
- [ ] Check Beyla discovery in alloy-tracing logs on ringtail
- [ ] Sync grafana-config for updated datasources
- [ ] Sync prometheus for updated scrape config
- [ ] Test Grafana Tempo datasource connection
- [ ] Generate test traffic and search traces in Grafana Explore → Tempo
- [ ] After merge: reset all ArgoCD app revisions back to main

Reviewed-on: #286
2026-03-05 10:51:07 -08:00

93 lines
1.8 KiB
Text

// Alloy tracing configuration for ringtail
// Uses Beyla eBPF to auto-instrument HTTP services and export traces to Tempo on indri
// ============== BEYLA eBPF AUTO-INSTRUMENTATION ==============
beyla.ebpf "http_services" {
discovery {
// Instrument HTTP services on common ports
instrument {
open_ports = "80-9999"
}
// Exclude infrastructure pods
exclude_instrument {
kubernetes {
namespace = "kube-system"
}
}
exclude_instrument {
kubernetes {
namespace = "tailscale"
}
}
exclude_instrument {
kubernetes {
pod_labels = { app = "alloy" }
}
}
exclude_instrument {
kubernetes {
pod_labels = { app = "alloy-tracing" }
}
}
exclude_instrument {
kubernetes {
pod_labels = { app = "kube-state-metrics" }
}
}
exclude_instrument {
kubernetes {
pod_labels = { "app.kubernetes.io/name" = "nvidia-device-plugin" }
}
}
}
attributes {
kubernetes {
enable = "true"
cluster_name = "ringtail"
}
}
traces {
instrumentations = ["http"]
}
output {
traces = [otelcol.processor.batch.default.input]
}
}
// ============== OTEL TRACE PIPELINE ==============
// Batch traces before export
otelcol.processor.batch "default" {
output {
traces = [otelcol.processor.attributes.add_cluster.input]
}
}
// Add cluster label to all spans
otelcol.processor.attributes "add_cluster" {
action {
key = "cluster"
value = "ringtail"
action = "upsert"
}
output {
traces = [otelcol.exporter.otlphttp.tempo.input]
}
}
// Export traces to Tempo on indri via Tailscale
otelcol.exporter.otlphttp "tempo" {
client {
endpoint = "https://tempo-otlp.tail8d86e.ts.net"
tls {
insecure_skip_verify = true
}
}
}