Erich Blume bb55fa9566 Recurring review sweep: 4 doc cards + nvidia-device-plugin v0.19.2 (#366 )

Knocks out the two daily recurring review tasks (doc review + service review) in one PR.

## Doc review (4 never-reviewed reference cards, `last-reviewed: 2026-06-04`)
- **cluster.md** — Kubernetes version v1.34.0 → **v1.35.0**; refreshed the stale ringtail workload list and noted the in-progress minikube→k3s migration (points to `[[ringtail]]` as the canonical list).
- **ntfy.md / tempo.md / alloy.md** — corrected image references: these are now **locally-built `registry.ops.eblu.me/blumeops/*` nix containers** (ntfy v2.19.2, tempo v2.10.3, alloy-k8s v1.16.0), not upstream Docker Hub. Fly.io alloy binary bumped to v1.16.1.

## Service review
- **nvidia-device-plugin** (ringtail GPU): v0.19.0 → **v0.19.2**. Upstream patch releases — CDI/Tegra fixes + dependency bumps, no breaking changes for our manifest-based CDI + RuntimeClass setup (the service-account change in the notes is helm-only).

## Not in this PR (need container rebuilds, deferred)
The other stale services are locally-built nix images, so upgrading them is a forge-runner rebuild rather than a clean tag bump — left untouched (not date-bumped, so they resurface): **prometheus** (v3.10.0→v3.12.0), **loki** (3.6.7→3.7.2), **kube-state-metrics**, **homepage**. Happy to do these as a follow-up rebuild PR.

## Deploy / verify
Not yet deployed — `nvidia-device-plugin` still points at `main`. After review:
```
argocd app set nvidia-device-plugin --revision reviews-jun4 && argocd app sync nvidia-device-plugin
# after merge:
argocd app set nvidia-device-plugin --revision main && argocd app sync nvidia-device-plugin
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #366

2026-06-04 13:37:02 -07:00

2.4 KiB

Raw Blame History

title

modified

last-reviewed

Grafana Alloy

Unified observability collector for metrics and logs with three deployments:

Indri (host) - System metrics and service logs from macOS host
Kubernetes (DaemonSet) - Automatic pod log collection and service health probes
Fly.io proxy (embedded) - nginx access log metrics and log forwarding from flyio-proxy

Quick Reference

Property	Value
Indri Binary	`~/.local/bin/alloy`
Indri Config	`~/.config/grafana-alloy/config.alloy`
K8s Namespace	`alloy`
K8s Image	`registry.ops.eblu.me/blumeops/alloy:v1.16.0-9564435` (locally built)
ArgoCD App	`alloy-k8s`
Fly.io Config	`fly/alloy.river`
Fly.io Image	`grafana/alloy:v1.16.1` (binary copied into nginx container, sha-pinned)

Metrics Collected

From Indri

System metrics via prometheus.exporter.unix
Textfile collector: minikube.prom, borgmatic.prom, zot.prom, jellyfin.prom
Zot registry metrics from http://localhost:5050/metrics
Pushed to prometheus via remote_write

From Kubernetes

All pod logs via loki.source.kubernetes
Service health probes: miniflux, kiwix, transmission, devpi, argocd

From Fly.io Proxy

flyio_nginx_http_requests_total — request rate by status/method/host
flyio_nginx_http_request_duration_seconds — latency histogram
flyio_nginx_http_response_bytes_total — response bandwidth
flyio_nginx_cache_requests_total — cache HIT/MISS/EXPIRED counts
Pushed to prometheus via remote_write through caddy

Logs Collected

Brew services: forgejo, tailscale

mcquack LaunchAgents: alloy, borgmatic, zot, jellyfin

Logs pushed to loki at https://loki.tail8d86e.ts.net/loki/api/v1/push.

Fly.io proxy: nginx JSON access logs pushed to loki at https://loki.ops.eblu.me/loki/api/v1/push (via caddy).

Why Built from Source

The Homebrew bottle uses CGO_ENABLED=0, which breaks Tailscale MagicDNS. Building with CGO_ENABLED=1 uses the macOS native resolver.

Note: This may no longer be needed now that services use *.ops.eblu.me URLs (routed via Caddy) instead of *.tail8d86e.ts.net. Should be tested in the future.

prometheus - Metrics storage
loki - Log storage
grafana - Visualization

2.4 KiB Raw Blame History