eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/argocd/manifests/grafana-config
History
Erich Blume cc54b4f565
All checks were successful
Deploy Fly.io Proxy / deploy (push) Successful in 1m16s
Details
Add Fly.io proxy observability via embedded Alloy (#123) 
## Summary

- Embed Grafana Alloy in the Fly.io proxy container to collect nginx JSON access logs (→ Loki) and derive request rate, latency histogram, cache status, and bandwidth metrics (→ Prometheus)
- Add nginx `stub_status` endpoint for connection-level metrics (active/reading/writing/waiting)
- Create two Grafana dashboards: **Docs APM** (per-service view filtered by `host="docs.eblu.me"`) and **Fly.io Proxy Health** (aggregate proxy health across all upstream services)

## Changed Files

| File | Change |
|------|--------|
| `fly/nginx.conf` | Add JSON `log_format` + `access_log`, add `stub_status` endpoint |
| `fly/Dockerfile` | COPY Alloy binary from `grafana/alloy:v1.5.1`, COPY `alloy.river` config |
| `fly/alloy.river` | **New** — Alloy config: log tailing, metric extraction, remote_write |
| `fly/start.sh` | Start Alloy after Tailscale, before nginx |
| `argocd/manifests/grafana-config/dashboards/configmap-docs-apm.yaml` | **New** — Docs APM dashboard |
| `argocd/manifests/grafana-config/dashboards/configmap-flyio.yaml` | **New** — Fly.io Proxy Health dashboard |
| `argocd/manifests/grafana-config/kustomization.yaml` | Register new dashboard configmaps |
| `docs/reference/services/flyio-proxy.md` | Document observability setup |

## Deployment and Testing

- [ ] `mise run fly-deploy` — rebuild container with Alloy
- [ ] `curl https://docs.eblu.me/` — generate traffic
- [ ] `fly logs -a blumeops-proxy` — verify Alloy startup
- [ ] Query Prometheus: `flyio_nginx_http_requests_total{instance="flyio-proxy"}`
- [ ] Query Loki: `{instance="flyio-proxy", job="flyio-nginx"}`
- [ ] `argocd app sync grafana-config` — deploy dashboards
- [ ] Verify dashboards show data in Grafana
- [ ] `mise run services-check` — no regressions

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/123
2026-02-08 10:05:38 -08:00
..
dashboards Add Fly.io proxy observability via embedded Alloy (#123) 2026-02-08 10:05:38 -08:00
external-secret-admin.yaml Switch all ExternalSecrets to creationPolicy: Owner 2026-01-28 20:27:16 -08:00
external-secret-teslamate-datasource.yaml Switch all ExternalSecrets to creationPolicy: Owner 2026-01-28 20:27:16 -08:00
ingress-tailscale.yaml Expand homepage widgets and info panels (#81) 2026-02-02 16:11:20 -08:00
kustomization.yaml Add Fly.io proxy observability via embedded Alloy (#123) 2026-02-08 10:05:38 -08:00
README.md K8s Migration Phase 2: Grafana to Kubernetes (#30) 2026-01-19 14:40:25 -08:00

README.md

Grafana Configuration

This directory contains Kubernetes manifests for Grafana configuration:

  • Tailscale Ingress for external access
  • Dashboard ConfigMaps for provisioning

Secrets Management

Current approach: Secrets are manually injected using 1Password CLI.

Before deploying Grafana, create the admin password secret:

kubectl create namespace monitoring
op inject -i secret-admin.yaml.tpl | kubectl apply -f -

The secret template (secret-admin.yaml.tpl) references 1Password:

  • Vault: vg6xf6vvfmoh5hqjjhlhbeoaie (blumeops)
  • Item: oxkcr3xtxnewy7noep2izvyr6y
  • Field: password

Future improvement: Migrate to External Secrets Operator or similar for automated secret synchronization from 1Password to Kubernetes.

Dashboards

Dashboard JSON files are stored as ConfigMaps in the dashboards/ directory. The Grafana sidecar automatically discovers ConfigMaps with label grafana_dashboard: "1" and provisions them.

To add a new dashboard:

  1. Export the dashboard JSON from Grafana UI
  2. Create a ConfigMap with the JSON content
  3. Add the grafana_dashboard: "1" label
  4. Add the ConfigMap to kustomization.yaml