eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/argocd/apps
History
Erich Blume c281fb5403 Add OpenTelemetry distributed tracing (Tempo + Beyla eBPF) (#286) 
## Summary

Adds the third observability pillar — **distributed tracing** — alongside existing metrics (Prometheus) and logs (Loki).

- **Grafana Tempo 2.10.1** on minikube-indri for trace storage with 7d retention, OTLP receivers, and `metrics_generator` that remote-writes span-metrics (RED) to Prometheus
- **Beyla eBPF auto-instrumentation** via a privileged Alloy DaemonSet on ringtail — instruments HTTP services (Frigate, ntfy, Ollama, Immich) without code changes
- **Grafana integration** — Tempo datasource with trace↔log and trace↔metrics correlation, plus Loki derivedFields for trace ID linking
- **Prometheus** scrapes Tempo operational metrics

### Architecture

```
ringtail (k3s)                                indri (minikube)
┌──────────────────────┐                      ┌─────────────────────┐
│ Alloy+Beyla (eBPF)   │──OTLP HTTP────────→ │ Tempo               │
│  ↳ Frigate, ntfy,    │  via tailnet         │  ↳ trace storage    │
│    Ollama, Immich     │                      │  ↳ RED → Prometheus │
└──────────────────────┘                      │                     │
                                              │ Grafana             │
                                              │  ↳ Tempo datasource │
                                              └─────────────────────┘
```

### New files (12)
- `docs/reference/services/tempo.md` — reference doc
- `docs/changelog.d/feature-otel-tracing.feature.md`
- `argocd/apps/tempo.yaml` + `argocd/manifests/tempo/` (6 files)
- `argocd/apps/alloy-tracing-ringtail.yaml` + `argocd/manifests/alloy-tracing-ringtail/` (4 files)

### Modified files (6)
- `argocd/manifests/grafana/datasources.yaml` — Tempo datasource + Loki derivedFields
- `argocd/manifests/prometheus/prometheus.yml` — Tempo scrape target
- `service-versions.yaml` — tempo + alloy-tracing-ringtail entries
- `docs/reference/services/grafana.md` — Tempo in datasources table
- `docs/reference/reference.md` — Tempo in services index
- `docs/reference/operations/observability.md` — Tempo in components list

## Deployment and Testing

- [ ] Sync `apps` app to pick up new Application definitions
- [ ] `argocd app set tempo --revision feature/otel-tracing && argocd app sync tempo`
- [ ] Verify Tempo pod: `kubectl --context=minikube-indri get pods -n monitoring -l app=tempo`
- [ ] Verify Tempo ready: port-forward 3200 and `curl localhost:3200/ready`
- [ ] Verify Tailscale ingresses: `kubectl --context=minikube-indri get ingress -n monitoring`
- [ ] `argocd app set alloy-tracing-ringtail --revision feature/otel-tracing && argocd app sync alloy-tracing-ringtail`
- [ ] Check Beyla discovery in alloy-tracing logs on ringtail
- [ ] Sync grafana-config for updated datasources
- [ ] Sync prometheus for updated scrape config
- [ ] Test Grafana Tempo datasource connection
- [ ] Generate test traffic and search traces in Grafana Explore → Tempo
- [ ] After merge: reset all ArgoCD app revisions back to main

Reviewed-on: #286
2026-03-05 10:51:07 -08:00
..
1password-connect-ringtail.yaml Fix mirror org refs in ArgoCD apps and widen credential template (#266) 2026-02-25 06:55:53 -08:00
1password-connect.yaml Fix mirror org refs in ArgoCD apps and widen credential template (#266) 2026-02-25 06:55:53 -08:00
alloy-k8s.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
alloy-ringtail.yaml Add multi-cluster observability with ringtail metrics and dashboards (#270) 2026-02-25 22:01:00 -08:00
alloy-tracing-ringtail.yaml Add OpenTelemetry distributed tracing (Tempo + Beyla eBPF) (#286) 2026-03-05 10:51:07 -08:00
apps.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
argocd.yaml Upgrade ArgoCD v3.2.6 → v3.3.2 with Server-Side Apply (#272) 2026-02-26 06:51:50 -08:00
authentik.yaml Deploy Authentik identity provider (C2 Mikado) (#227) 2026-02-20 12:55:59 -08:00
blumeops-pg.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
cloudnative-pg.yaml Port CloudNative-PG off Helm to direct release manifest (#268) 2026-02-25 17:37:53 -08:00
cv.yaml Add CV/resume web app at cv.ops.eblu.me (#169) 2026-02-12 11:09:41 -08:00
devpi.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
docs.yaml Phase 1b: Deploy docs hosting with Quartz (#85) 2026-02-03 10:52:20 -08:00
external-secrets-config-ringtail.yaml Add k3s, 1Password Connect, and systemd nix-container-builder to ringtail (#209) 2026-02-18 21:15:30 -08:00
external-secrets-config.yaml Add External Secrets Operator with 1Password Connect (#66) (#66) 2026-01-28 19:30:10 -08:00
external-secrets-crds-ringtail.yaml Fix mirror org refs in ArgoCD apps and widen credential template (#266) 2026-02-25 06:55:53 -08:00
external-secrets-crds.yaml Fix mirror org refs in ArgoCD apps and widen credential template (#266) 2026-02-25 06:55:53 -08:00
external-secrets-ringtail.yaml Fix mirror org refs in ArgoCD apps and widen credential template (#266) 2026-02-25 06:55:53 -08:00
external-secrets.yaml Fix mirror org refs in ArgoCD apps and widen credential template (#266) 2026-02-25 06:55:53 -08:00
forgejo-runner.yaml Switch forgejo-runner ArgoCD app to internal SSH repo URL 2026-03-03 10:43:01 -08:00
frigate.yaml Port Frigate NVR to ringtail k3s with GPU acceleration (#217) 2026-02-19 14:27:04 -08:00
grafana-config.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
grafana.yaml C2: Upgrade Grafana to 12.x with Nix container and Kustomize (#260) 2026-02-23 18:07:18 -08:00
homepage.yaml Replace Homepage Helm chart with kustomize manifests and custom Dockerfile (#221) 2026-02-19 18:29:19 -08:00
immich-storage.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
immich.yaml Fix mirror org refs in ArgoCD apps and widen credential template (#266) 2026-02-25 06:55:53 -08:00
kiwix.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
kube-state-metrics-ringtail.yaml Add multi-cluster observability with ringtail metrics and dashboards (#270) 2026-02-25 22:01:00 -08:00
kube-state-metrics.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
loki.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
miniflux.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
mqtt.yaml Port Mosquitto and ntfy to ringtail k3s, retire Apple Silicon Detector (#216) 2026-02-19 11:22:44 -08:00
navidrome.yaml Add Navidrome music streaming server (#79) 2026-01-31 20:19:31 -08:00
ntfy.yaml Port Mosquitto and ntfy to ringtail k3s, retire Apple Silicon Detector (#216) 2026-02-19 11:22:44 -08:00
nvidia-device-plugin.yaml Port Frigate NVR to ringtail k3s with GPU acceleration (#217) 2026-02-19 14:27:04 -08:00
ollama.yaml Deploy Ollama LLM server on ringtail (#277) 2026-03-02 20:39:51 -08:00
prometheus.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
tailscale-operator-ringtail.yaml Deploy Tailscale operator on ringtail k3s cluster (#215) 2026-02-19 09:33:05 -08:00
tailscale-operator.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00
tempo.yaml Add OpenTelemetry distributed tracing (Tempo + Beyla eBPF) (#286) 2026-03-05 10:51:07 -08:00
teslamate.yaml Doc review: connect-to-postgres, create-release-artifact-workflow, deploy-k8s-service (#191) 2026-02-15 07:42:01 -08:00
torrent.yaml Add Immich photo management + migrate forge URLs (#62) 2026-01-26 11:20:11 -08:00