Erich Blume
c281fb5403
## Summary
Adds the third observability pillar — **distributed tracing** — alongside existing metrics (Prometheus) and logs (Loki).
- **Grafana Tempo 2.10.1** on minikube-indri for trace storage with 7d retention, OTLP receivers, and `metrics_generator` that remote-writes span-metrics (RED) to Prometheus
- **Beyla eBPF auto-instrumentation** via a privileged Alloy DaemonSet on ringtail — instruments HTTP services (Frigate, ntfy, Ollama, Immich) without code changes
- **Grafana integration** — Tempo datasource with trace↔log and trace↔metrics correlation, plus Loki derivedFields for trace ID linking
- **Prometheus** scrapes Tempo operational metrics
### Architecture
```
ringtail (k3s) indri (minikube)
┌──────────────────────┐ ┌─────────────────────┐
│ Alloy+Beyla (eBPF) │──OTLP HTTP────────→ │ Tempo │
│ ↳ Frigate, ntfy, │ via tailnet │ ↳ trace storage │
│ Ollama, Immich │ │ ↳ RED → Prometheus │
└──────────────────────┘ │ │
│ Grafana │
│ ↳ Tempo datasource │
└─────────────────────┘
```
### New files (12)
- `docs/reference/services/tempo.md` — reference doc
- `docs/changelog.d/feature-otel-tracing.feature.md`
- `argocd/apps/tempo.yaml` + `argocd/manifests/tempo/` (6 files)
- `argocd/apps/alloy-tracing-ringtail.yaml` + `argocd/manifests/alloy-tracing-ringtail/` (4 files)
### Modified files (6)
- `argocd/manifests/grafana/datasources.yaml` — Tempo datasource + Loki derivedFields
- `argocd/manifests/prometheus/prometheus.yml` — Tempo scrape target
- `service-versions.yaml` — tempo + alloy-tracing-ringtail entries
- `docs/reference/services/grafana.md` — Tempo in datasources table
- `docs/reference/reference.md` — Tempo in services index
- `docs/reference/operations/observability.md` — Tempo in components list
## Deployment and Testing
- [ ] Sync `apps` app to pick up new Application definitions
- [ ] `argocd app set tempo --revision feature/otel-tracing && argocd app sync tempo`
- [ ] Verify Tempo pod: `kubectl --context=minikube-indri get pods -n monitoring -l app=tempo`
- [ ] Verify Tempo ready: port-forward 3200 and `curl localhost:3200/ready`
- [ ] Verify Tailscale ingresses: `kubectl --context=minikube-indri get ingress -n monitoring`
- [ ] `argocd app set alloy-tracing-ringtail --revision feature/otel-tracing && argocd app sync alloy-tracing-ringtail`
- [ ] Check Beyla discovery in alloy-tracing logs on ringtail
- [ ] Sync grafana-config for updated datasources
- [ ] Sync prometheus for updated scrape config
- [ ] Test Grafana Tempo datasource connection
- [ ] Generate test traffic and search traces in Grafana Explore → Tempo
- [ ] After merge: reset all ArgoCD app revisions back to main
Reviewed-on: #286
67 lines
1.7 KiB
YAML
67 lines
1.7 KiB
YAML
apiVersion: 1
|
|
datasources:
|
|
- access: proxy
|
|
editable: false
|
|
isDefault: true
|
|
name: Prometheus
|
|
orgId: 1
|
|
type: prometheus
|
|
uid: prometheus
|
|
url: http://prometheus.monitoring.svc.cluster.local:9090
|
|
- access: proxy
|
|
editable: false
|
|
name: Loki
|
|
orgId: 1
|
|
type: loki
|
|
uid: loki
|
|
url: http://loki.monitoring.svc.cluster.local:3100
|
|
jsonData:
|
|
derivedFields:
|
|
- datasourceUid: tempo
|
|
matcherRegex: '"traceID":"(\w+)"'
|
|
name: TraceID
|
|
url: "$${__value.raw}"
|
|
- access: proxy
|
|
editable: false
|
|
name: Tempo
|
|
orgId: 1
|
|
type: tempo
|
|
uid: tempo
|
|
url: http://tempo.monitoring.svc.cluster.local:3200
|
|
jsonData:
|
|
tracesToLogsV2:
|
|
datasourceUid: loki
|
|
filterByTraceID: true
|
|
filterBySpanID: false
|
|
tracesToMetrics:
|
|
datasourceUid: prometheus
|
|
spanStartTimeShift: "-1h"
|
|
spanEndTimeShift: "1h"
|
|
queries:
|
|
- name: Request rate
|
|
query: "sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))"
|
|
- name: Error rate
|
|
query: "sum(rate(traces_spanmetrics_calls_total{$$__tags, status_code=\"STATUS_CODE_ERROR\"}[5m]))"
|
|
- name: Duration (p95)
|
|
query: "histogram_quantile(0.95, sum(rate(traces_spanmetrics_duration_seconds_bucket{$$__tags}[5m])) by (le))"
|
|
serviceMap:
|
|
datasourceUid: prometheus
|
|
nodeGraph:
|
|
enabled: true
|
|
- access: proxy
|
|
database: teslamate
|
|
editable: false
|
|
jsonData:
|
|
database: teslamate
|
|
connMaxLifetime: 14400
|
|
maxIdleConns: 2
|
|
maxOpenConns: 5
|
|
sslmode: disable
|
|
name: TeslaMate
|
|
orgId: 1
|
|
secureJsonData:
|
|
password: $TESLAMATE_DB_PASSWORD
|
|
type: postgres
|
|
uid: TeslaMate
|
|
url: blumeops-pg-rw.databases.svc.cluster.local:5432
|
|
user: teslamate
|