The local-blocks processor requires its own dedicated traces WAL
(traces_storage.path), separate from the ingester WAL and the
metrics generator WAL. Without it, the processor fails with
"local blocks processor requires traces wal".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Required by Grafana's TraceQL metrics queries. Keeps recent
traces in memory for query-time aggregation without
duplicating data to storage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Panels: Storage Used, PVC Utilization (% of 10Gi), Total
Blocks, Heap Usage, Storage Over Time, Span Ingestion Rate,
Ingestion Throughput, and Query Latency (p50/p95).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add PromQL query for checking Tempo storage utilization
against PVC capacity using tempodb_backend_bytes_total.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updates service-versions.yaml, Grafana datasources table,
ArgoCD apps registry, and Tempo image version to 2.10.1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploys a privileged Alloy DaemonSet on ringtail's k3s that
uses Beyla eBPF to auto-instrument HTTP services (Frigate,
ntfy, Ollama, Immich) without code changes. Traces are
exported via OTLP HTTP to Tempo on indri.
Separate from the existing unprivileged alloy-ringtail to
preserve least-privilege for metrics/logs collection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana: Tempo datasource with trace-to-log (Loki) and
trace-to-metrics (Prometheus) correlation. Loki gets
derivedFields to link trace IDs back to Tempo.
Prometheus: scrape Tempo operational metrics on port 3200.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploys Grafana Tempo 2.10.1 on minikube-indri for distributed
trace storage. Includes OTLP receivers (gRPC + HTTP), local
filesystem storage with 7d retention, and metrics_generator
that remote-writes span-metrics to Prometheus.
Two Tailscale Ingresses: tempo (query API) and tempo-otlp
(OTLP HTTP receiver for cross-cluster trace ingestion).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo is the new distributed tracing backend for BlumeOps,
completing the third observability pillar alongside Prometheus
(metrics) and Loki (logs).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
