Erich Blume 40f1568088 Remove unused Mosquitto MQTT broker from ringtail

Mosquitto has been dormant since frigate-notify switched from MQTT to
webapi polling (529ba10). Tear down live infra (ArgoCD app, namespace)
and remove all manifests, service-versions entry, services-check, and
doc references.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-11 18:37:31 -07:00

2.3 KiB

Raw Blame History

title

modified

Grafana Tempo

Distributed tracing backend for BlumeOps infrastructure. Receives traces via OTLP, stores them locally, and generates RED metrics (rate, error, duration) for prometheus.

Quick Reference

Property	Value
URL	https://tempo.ops.eblu.me (when Caddy route added)
Tailscale URL	https://tempo.tail8d86e.ts.net
OTLP Endpoint	https://tempo-otlp.tail8d86e.ts.net
Namespace	`monitoring`
Image	`grafana/tempo:2.10.1`
Storage	10Gi PVC (local filesystem)
Retention	7 days

Architecture

Single-node deployment with local filesystem storage
OTLP receivers: gRPC (4317) and HTTP (4318)
metrics_generator produces span-metrics and service-graphs, remote-written to prometheus
Queried via grafana Tempo datasource
Two Tailscale Ingresses: one for query API (3200), one for OTLP HTTP receiver (4318)

Trace Sources

From ringtail (via Beyla eBPF in Alloy):

Service	Protocol	Coverage
frigate	HTTP REST	Request rate, error rate, latency, trace spans
ntfy	HTTP	Same
ollama	HTTP REST	Same (model inference latency)
immich	HTTP REST	Same

Beyla auto-instruments HTTP services via eBPF kernel hooks — no code changes needed.

Future: SDK instrumentation Services with OTel SDK support (e.g., Hermes) can send traces directly to the OTLP endpoint for deeper internal spans (DB queries, business logic) alongside eBPF envelope traces.

Storage Monitoring

Tempo exposes tempodb_backend_bytes_total via its /metrics endpoint (scraped by prometheus). To check storage utilization against the 10Gi PVC:

tempodb_backend_bytes_total / 10737418240 * 100

Full PVC-level monitoring (via kubelet volume stats) is not yet available — see backlog.

Grafana Integration

Tempo datasource with trace-to-log and trace-to-metrics correlation
Service map and node graph visualization
Loki derived fields link trace IDs in logs back to Tempo

alloy - Trace collector (Beyla eBPF on ringtail)
prometheus - Receives span-metrics from Tempo
loki - Log correlation via trace IDs
grafana - Trace visualization

2.3 KiB Raw Blame History