Add Fly.io proxy observability via embedded Alloy #123

Merged
eblume merged 7 commits from feature/flyio-observability into main 2026-02-08 10:05:38 -08:00

7 commits

Author SHA1 Message Date
34fceff627 Update Alloy, Prometheus, Loki, Grafana reference cards for flyio-proxy
Add Fly.io proxy as a third Alloy deployment, document the new
remote_write source in Prometheus, new log source in Loki, and
two new dashboards in Grafana.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 10:05:13 -08:00
176d38be68 Fix metric names: strip loki_process_custom_ prefix, drop internal labels
Alloy's stage.metrics prefixes all metric names with
loki_process_custom_. Add a relabel rule to strip the prefix so
dashboards can query clean names (flyio_nginx_http_requests_total
etc). Also drop component_id/component_path/filename labels.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 09:50:07 -08:00
80e7d11058 Remove nginx stub_status exporter — not available in Alloy
Alloy has no built-in prometheus.exporter.nginx component. Remove
the stub_status scraping and connection panels from the Fly.io
dashboard. Replace with error rate and cache hit ratio stats.
All key signals are still covered by log-derived metrics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 09:45:39 -08:00
6e4d7f5991 Fix Alloy binary on Alpine: add libc6-compat
The grafana/alloy image is Ubuntu-based (glibc), but our container
uses nginx:alpine (musl). The binary exists but fails with "not found"
because the glibc dynamic linker is missing. libc6-compat provides
the compatibility shim.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 09:40:31 -08:00
977f63a951 Document security implications of flyio-proxy → homelab ACL
The new ACL grant lets the Fly.io proxy reach all Caddy-proxied
services, not just Loki/Prometheus. Document the expanded attack
surface and trust boundary (requires RCE on gilbert or 1Password
access) in both the flyio-proxy and caddy reference cards.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 09:33:05 -08:00
1e1e513b4a Route Alloy through Caddy for proper TLS, update ACLs
Switch Alloy endpoints from *.tail8d86e.ts.net (with insecure_skip_verify)
to *.ops.eblu.me via Caddy reverse proxy with valid TLS certificates.
Add tag:homelab:443 to flyio-proxy ACL grant so the proxy can reach Caddy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 08:45:57 -08:00
a0dc7ec511 Add observability to Fly.io proxy via embedded Alloy
Instrument the flyio-proxy container with Grafana Alloy to collect
nginx JSON access logs (→ Loki) and derive request/latency/cache
metrics (→ Prometheus). Adds stub_status for connection-level metrics.
Includes two Grafana dashboards: Docs APM (per-service) and Fly.io
Proxy Health (aggregate).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 08:04:29 -08:00