Deploy Fly.io Proxy / deploy (push) Successful in 1m16s

Details

Add Fly.io proxy observability via embedded Alloy (#123 )

## Summary

- Embed Grafana Alloy in the Fly.io proxy container to collect nginx JSON access logs (→ Loki) and derive request rate, latency histogram, cache status, and bandwidth metrics (→ Prometheus)
- Add nginx `stub_status` endpoint for connection-level metrics (active/reading/writing/waiting)
- Create two Grafana dashboards: **Docs APM** (per-service view filtered by `host="docs.eblu.me"`) and **Fly.io Proxy Health** (aggregate proxy health across all upstream services)

## Changed Files

| File | Change |
|------|--------|
| `fly/nginx.conf` | Add JSON `log_format` + `access_log`, add `stub_status` endpoint |
| `fly/Dockerfile` | COPY Alloy binary from `grafana/alloy:v1.5.1`, COPY `alloy.river` config |
| `fly/alloy.river` | **New** — Alloy config: log tailing, metric extraction, remote_write |
| `fly/start.sh` | Start Alloy after Tailscale, before nginx |
| `argocd/manifests/grafana-config/dashboards/configmap-docs-apm.yaml` | **New** — Docs APM dashboard |
| `argocd/manifests/grafana-config/dashboards/configmap-flyio.yaml` | **New** — Fly.io Proxy Health dashboard |
| `argocd/manifests/grafana-config/kustomization.yaml` | Register new dashboard configmaps |
| `docs/reference/services/flyio-proxy.md` | Document observability setup |

## Deployment and Testing

- [ ] `mise run fly-deploy` — rebuild container with Alloy
- [ ] `curl https://docs.eblu.me/` — generate traffic
- [ ] `fly logs -a blumeops-proxy` — verify Alloy startup
- [ ] Query Prometheus: `flyio_nginx_http_requests_total{instance="flyio-proxy"}`
- [ ] Query Loki: `{instance="flyio-proxy", job="flyio-nginx"}`
- [ ] `argocd app sync grafana-config` — deploy dashboards
- [ ] Verify dashboards show data in Grafana
- [ ] `mise run services-check` — no regressions

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/123

2026-02-08 10:05:38 -08:00

2.3 KiB

Raw Blame History

title

Grafana Alloy

Unified observability collector for metrics and logs with three deployments:

Indri (host) - System metrics and service logs from macOS host
Kubernetes (DaemonSet) - Automatic pod log collection and service health probes
Fly.io proxy (embedded) - nginx access log metrics and log forwarding from flyio-proxy

Quick Reference

Property	Value
Indri Binary	`~/.local/bin/alloy`
Indri Config	`~/.config/grafana-alloy/config.alloy`
K8s Namespace	`alloy`
K8s Image	`grafana/alloy:v1.8.2`
ArgoCD App	`alloy-k8s`
Fly.io Config	`fly/alloy.river`
Fly.io Image	`grafana/alloy:v1.5.1` (binary copied into nginx container)

Metrics Collected

From Indri

System metrics via prometheus.exporter.unix
Textfile collector: minikube.prom, borgmatic.prom, zot.prom, jellyfin.prom
Zot registry metrics from http://localhost:5050/metrics
Pushed to prometheus via remote_write

From Kubernetes

All pod logs via loki.source.kubernetes
Service health probes: miniflux, kiwix, transmission, devpi, argocd

From Fly.io Proxy

flyio_nginx_http_requests_total — request rate by status/method/host
flyio_nginx_http_request_duration_seconds — latency histogram
flyio_nginx_http_response_bytes_total — response bandwidth
flyio_nginx_cache_requests_total — cache HIT/MISS/EXPIRED counts
Pushed to prometheus via remote_write through caddy

Logs Collected

Brew services: forgejo, tailscale

mcquack LaunchAgents: alloy, borgmatic, zot, jellyfin

Logs pushed to loki at https://loki.tail8d86e.ts.net/loki/api/v1/push.

Fly.io proxy: nginx JSON access logs pushed to loki at https://loki.ops.eblu.me/loki/api/v1/push (via caddy).

Why Built from Source

The Homebrew bottle uses CGO_ENABLED=0, which breaks Tailscale MagicDNS. Building with CGO_ENABLED=1 uses the macOS native resolver.

Note: This may no longer be needed now that services use *.ops.eblu.me URLs (routed via Caddy) instead of *.tail8d86e.ts.net. Should be tested in the future.

prometheus - Metrics storage
loki - Log storage
grafana - Visualization

2.3 KiB Raw Blame History