Erich Blume
c281fb5403
## Summary
Adds the third observability pillar — **distributed tracing** — alongside existing metrics (Prometheus) and logs (Loki).
- **Grafana Tempo 2.10.1** on minikube-indri for trace storage with 7d retention, OTLP receivers, and `metrics_generator` that remote-writes span-metrics (RED) to Prometheus
- **Beyla eBPF auto-instrumentation** via a privileged Alloy DaemonSet on ringtail — instruments HTTP services (Frigate, ntfy, Ollama, Immich) without code changes
- **Grafana integration** — Tempo datasource with trace↔log and trace↔metrics correlation, plus Loki derivedFields for trace ID linking
- **Prometheus** scrapes Tempo operational metrics
### Architecture
```
ringtail (k3s) indri (minikube)
┌──────────────────────┐ ┌─────────────────────┐
│ Alloy+Beyla (eBPF) │──OTLP HTTP────────→ │ Tempo │
│ ↳ Frigate, ntfy, │ via tailnet │ ↳ trace storage │
│ Ollama, Immich │ │ ↳ RED → Prometheus │
└──────────────────────┘ │ │
│ Grafana │
│ ↳ Tempo datasource │
└─────────────────────┘
```
### New files (12)
- `docs/reference/services/tempo.md` — reference doc
- `docs/changelog.d/feature-otel-tracing.feature.md`
- `argocd/apps/tempo.yaml` + `argocd/manifests/tempo/` (6 files)
- `argocd/apps/alloy-tracing-ringtail.yaml` + `argocd/manifests/alloy-tracing-ringtail/` (4 files)
### Modified files (6)
- `argocd/manifests/grafana/datasources.yaml` — Tempo datasource + Loki derivedFields
- `argocd/manifests/prometheus/prometheus.yml` — Tempo scrape target
- `service-versions.yaml` — tempo + alloy-tracing-ringtail entries
- `docs/reference/services/grafana.md` — Tempo in datasources table
- `docs/reference/reference.md` — Tempo in services index
- `docs/reference/operations/observability.md` — Tempo in components list
## Deployment and Testing
- [ ] Sync `apps` app to pick up new Application definitions
- [ ] `argocd app set tempo --revision feature/otel-tracing && argocd app sync tempo`
- [ ] Verify Tempo pod: `kubectl --context=minikube-indri get pods -n monitoring -l app=tempo`
- [ ] Verify Tempo ready: port-forward 3200 and `curl localhost:3200/ready`
- [ ] Verify Tailscale ingresses: `kubectl --context=minikube-indri get ingress -n monitoring`
- [ ] `argocd app set alloy-tracing-ringtail --revision feature/otel-tracing && argocd app sync alloy-tracing-ringtail`
- [ ] Check Beyla discovery in alloy-tracing logs on ringtail
- [ ] Sync grafana-config for updated datasources
- [ ] Sync prometheus for updated scrape config
- [ ] Test Grafana Tempo datasource connection
- [ ] Generate test traffic and search traces in Grafana Explore → Tempo
- [ ] After merge: reset all ArgoCD app revisions back to main
Reviewed-on: #286
47 lines
1.9 KiB
YAML
47 lines
1.9 KiB
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
|
|
namespace: monitoring
|
|
|
|
resources:
|
|
- ingress-tailscale.yaml
|
|
- external-secret-admin.yaml
|
|
- external-secret-teslamate-datasource.yaml
|
|
- external-secret-authentik-oauth.yaml
|
|
# Dashboard ConfigMaps - discovered by Grafana sidecar via label grafana_dashboard=1
|
|
- dashboards/configmap-borgmatic.yaml
|
|
- dashboards/configmap-devpi.yaml
|
|
- dashboards/configmap-loki.yaml
|
|
- dashboards/configmap-macos.yaml
|
|
- dashboards/configmap-kubernetes.yaml
|
|
- dashboards/configmap-jellyfin.yaml
|
|
- dashboards/configmap-postgresql.yaml
|
|
- dashboards/configmap-ringtail.yaml
|
|
- dashboards/configmap-zot.yaml
|
|
- dashboards/configmap-frigate.yaml
|
|
- dashboards/configmap-transmission.yaml
|
|
- dashboards/configmap-cv-apm.yaml
|
|
- dashboards/configmap-docs-apm.yaml
|
|
- dashboards/configmap-flyio.yaml
|
|
- dashboards/configmap-sifaka-disks.yaml
|
|
- dashboards/configmap-forgejo.yaml
|
|
- dashboards/configmap-tempo.yaml
|
|
# TeslaMate dashboards
|
|
- dashboards/configmap-teslamate-overview.yaml
|
|
- dashboards/configmap-teslamate-charges.yaml
|
|
- dashboards/configmap-teslamate-drives.yaml
|
|
- dashboards/configmap-teslamate-efficiency.yaml
|
|
- dashboards/configmap-teslamate-states.yaml
|
|
- dashboards/configmap-teslamate-vampire-drain.yaml
|
|
- dashboards/configmap-teslamate-battery-health.yaml
|
|
- dashboards/configmap-teslamate-statistics.yaml
|
|
- dashboards/configmap-teslamate-charge-level.yaml
|
|
- dashboards/configmap-teslamate-updates.yaml
|
|
- dashboards/configmap-teslamate-trip.yaml
|
|
- dashboards/configmap-teslamate-locations.yaml
|
|
- dashboards/configmap-teslamate-mileage.yaml
|
|
- dashboards/configmap-teslamate-drive-stats.yaml
|
|
- dashboards/configmap-teslamate-charging-stats.yaml
|
|
- dashboards/configmap-teslamate-projected-range.yaml
|
|
- dashboards/configmap-teslamate-timeline.yaml
|
|
- dashboards/configmap-teslamate-visited.yaml
|