Erich Blume ab8ea6f301 Bump Grafana Alloy to v1.14.0 (#292 )

## Summary
- Bump alloy-k8s, alloy-ringtail, and alloy-tracing-ringtail image tags from v1.13.1 to v1.14.0
- Mark indri alloy (ansible) as reviewed at v1.14.0 — source rebuild from forge mirror needed
- Add missing alloy-ringtail entry to service-versions.yaml
- Update alloy reference doc

## Breaking changes reviewed
- `loki.secretfilter` options removed — not used in our configs
- OTel Collector upgraded to v0.142.0 — Kafka receiver changes don't affect us
- Exporter queue default changes — our tracing pipeline (Beyla → batch → otlphttp) uses simple config, low risk

## Deployment and Testing
- [ ] Sync alloy-k8s: `argocd app set alloy-k8s --revision bump/alloy-v1.14.0 && argocd app sync alloy-k8s`
- [ ] Sync alloy-ringtail: `argocd app set alloy-ringtail --revision bump/alloy-v1.14.0 --server ringtail-argocd && argocd app sync alloy-ringtail`
- [ ] Sync alloy-tracing-ringtail similarly
- [ ] Verify metrics flowing in Grafana
- [ ] Verify traces flowing to Tempo (ringtail)
- [ ] Rebuild indri alloy from source (`v1.14.0` tag on forge mirror), SCP to indri, restart
- [ ] After merge: reset ArgoCD revisions to main, re-sync

Reviewed-on: #292

2026-03-13 16:25:27 -07:00

2.3 KiB

Raw Blame History

title

modified

Grafana Alloy

Unified observability collector for metrics and logs with three deployments:

Indri (host) - System metrics and service logs from macOS host
Kubernetes (DaemonSet) - Automatic pod log collection and service health probes
Fly.io proxy (embedded) - nginx access log metrics and log forwarding from flyio-proxy

Quick Reference

Property	Value
Indri Binary	`~/.local/bin/alloy`
Indri Config	`~/.config/grafana-alloy/config.alloy`
K8s Namespace	`alloy`
K8s Image	`grafana/alloy:v1.14.0`
ArgoCD App	`alloy-k8s`
Fly.io Config	`fly/alloy.river`
Fly.io Image	`grafana/alloy:v1.5.1` (binary copied into nginx container)

Metrics Collected

From Indri

System metrics via prometheus.exporter.unix
Textfile collector: minikube.prom, borgmatic.prom, zot.prom, jellyfin.prom
Zot registry metrics from http://localhost:5050/metrics
Pushed to prometheus via remote_write

From Kubernetes

All pod logs via loki.source.kubernetes
Service health probes: miniflux, kiwix, transmission, devpi, argocd

From Fly.io Proxy

flyio_nginx_http_requests_total — request rate by status/method/host
flyio_nginx_http_request_duration_seconds — latency histogram
flyio_nginx_http_response_bytes_total — response bandwidth
flyio_nginx_cache_requests_total — cache HIT/MISS/EXPIRED counts
Pushed to prometheus via remote_write through caddy

Logs Collected

Brew services: forgejo, tailscale

mcquack LaunchAgents: alloy, borgmatic, zot, jellyfin

Logs pushed to loki at https://loki.tail8d86e.ts.net/loki/api/v1/push.

Fly.io proxy: nginx JSON access logs pushed to loki at https://loki.ops.eblu.me/loki/api/v1/push (via caddy).

Why Built from Source

The Homebrew bottle uses CGO_ENABLED=0, which breaks Tailscale MagicDNS. Building with CGO_ENABLED=1 uses the macOS native resolver.

Note: This may no longer be needed now that services use *.ops.eblu.me URLs (routed via Caddy) instead of *.tail8d86e.ts.net. Should be tested in the future.

prometheus - Metrics storage
loki - Log storage
grafana - Visualization

2.3 KiB Raw Blame History