Switch Fly proxy to upstream keepalive pools (#337)

## Summary

- Replace per-request DNS resolution (variable-based `proxy_pass`) with static `upstream` blocks and `keepalive` connection pools
- Reuses TLS connections through the Tailscale tunnel instead of handshaking per request
- Add `mise run fly-reload` for nginx config reload without full redeploy (re-resolves upstream DNS)

## Trade-off

DNS is resolved at config load, not per-request. If Tailscale Ingress pods get new IPs (restart, reschedule), `mise run fly-reload` is needed. A Grafana alert will be added to detect this.

## Still TODO on this branch

- [ ] Grafana alert for upstream unreachable (triggers fly-reload reminder)
- [ ] Docs pass
- [ ] Deploy from branch and verify latency improvement
- [ ] Changelog fragment

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #337

docs/reference/infrastructure/routing.md

@@ -1,6 +1,6 @@
 ---
 title: Routing
-modified: 2026-03-03
+modified: 2026-04-17
 tags:
   - infrastructure
   - networking
@@ -51,6 +51,7 @@ DNS CNAMEs point to `blumeops-proxy.fly.dev`. TLS via Fly.io-managed Let's Encry
 | Service | URL | Description |
 |---------|-----|-------------|
 | [[docs]] | https://docs.eblu.me | Documentation site |
+| [[cv]] | https://cv.eblu.me | CV / resume |
 | [[forgejo]] | https://forge.eblu.me | Git hosting (public) |
 
 ## Tailscale-Only Services

docs/reference/services/flyio-proxy.md

@@ -1,6 +1,6 @@
 ---
 title: Fly.io Proxy
-modified: 2026-02-08
+modified: 2026-04-17
 tags:
   - service
   - networking
@@ -26,11 +26,21 @@ Public reverse proxy on [Fly.io](https://fly.io) that exposes selected BlumeOps
 | Public domain | Backend | Service |
 |---------------|---------|---------|
 | `docs.eblu.me` | `docs.tail8d86e.ts.net` | [[docs]] |
+| `cv.eblu.me` | `cv.tail8d86e.ts.net` | [[cv]] |
+| `forge.eblu.me` | `forge.tail8d86e.ts.net` | [[forgejo]] |
 
 ## Architecture
 
 Internet traffic hits Fly.io's Anycast edge, terminates TLS with a Let's Encrypt certificate, and is proxied by nginx to the backend service over a Tailscale WireGuard tunnel. See [[expose-service-publicly]] for the full architecture diagram.
 
+### Upstream Keepalive
+
+Nginx uses `upstream` blocks with `keepalive` connection pools to reuse TLS connections through the WireGuard tunnel. This avoids a per-request TLS handshake, which was previously the dominant source of latency (35s+ p50 before keepalive, sub-second after).
+
+**Trade-off:** DNS for upstream hostnames is resolved once at config load, not per-request. If Tailscale Ingress pods get new IPs (restart, reschedule, minikube restart), run `mise run fly-reload` to re-resolve without a full redeploy. A Grafana alert fires when upstreams are unreachable.
+
+Each upstream requires `proxy_ssl_name` set to the actual Tailscale hostname — nginx sends the upstream block name as SNI by default, which the Tailscale Ingress proxy won't recognize.
+
 ## Key Files
 
 | File | Purpose |
@@ -39,7 +49,7 @@ Internet traffic hits Fly.io's Anycast edge, terminates TLS with a Let's Encrypt
 | `fly/Dockerfile` | nginx + Tailscale + Alloy container |
 | `fly/nginx.conf` | Reverse proxy, caching, rate limiting, JSON logging |
 | `fly/alloy.river` | Alloy config: log tailing, metric extraction, remote_write |
-| `fly/start.sh` | Entrypoint: start Tailscale, Alloy, then nginx |
+| `fly/start.sh` | Entrypoint: start Tailscale, wait for MagicDNS, then nginx + Alloy |
 | `pulumi/tailscale/__main__.py` | Auth key (`tag:flyio-proxy`) |
 | `pulumi/tailscale/policy.hujson` | ACL grants for proxy |
 | `pulumi/gandi/__main__.py` | DNS CNAMEs |
@@ -57,7 +67,8 @@ The Tailscale auth key is `preauthorized=True` to avoid device approval hangs on
 - **Logs**: nginx JSON access logs tailed and pushed to [[loki|Loki]] (`{instance="flyio-proxy", job="flyio-nginx"}`)
 - **Metrics**: Derived from access logs, pushed to [[prometheus|Prometheus]] via `remote_write`
   - `flyio_nginx_http_requests_total` — request rate by status/method/host
-  - `flyio_nginx_http_request_duration_seconds` — latency histogram
+  - `flyio_nginx_http_request_duration_seconds` — total request latency histogram (includes proxy overhead)
+  - `flyio_nginx_upstream_response_time_seconds` — backend response time histogram (Forgejo processing only)
   - `flyio_nginx_http_response_bytes_total` — response bandwidth
   - `flyio_nginx_cache_requests_total` — cache HIT/MISS/EXPIRED counts
 
@@ -74,7 +85,21 @@ Alloy listens on `127.0.0.1:12345` for self-scraping its `/metrics` endpoint. Al
 
 The `tag:flyio-proxy` ACL grants access only to `tag:flyio-target:443`. Services must explicitly opt in by adding a `tailscale.com/tags: "tag:k8s,tag:flyio-target"` annotation to their Tailscale Ingress. This means the proxy can only reach endpoints that have been individually tagged — a compromised nginx config cannot route to arbitrary services on the tailnet.
 
-Currently tagged as `tag:flyio-target`: [[docs]], [[loki]], [[prometheus]]. Loki and Prometheus are tagged so that [[alloy|Alloy]] (running inside the container) can push logs and metrics directly via their Tailscale Ingress endpoints — the restricted ACL means Caddy on indri (`tag:homelab`) is not reachable from the proxy.
+Currently tagged as `tag:flyio-target`: [[docs]], [[cv]], [[forgejo]], [[loki]], [[prometheus]]. Loki and Prometheus are tagged so that [[alloy|Alloy]] (running inside the container) can push logs and metrics directly via their Tailscale Ingress endpoints — the restricted ACL means Caddy on indri (`tag:homelab`) is not reachable from the proxy.
+
+### Crawler Mitigation
+
+The proxy serves a `robots.txt` blocking crawlers from expensive endpoints:
+
+- `/mirrors/` — large mirrored repos
+- `/user/` — auth endpoints (crawlers follow redirect loops)
+- `/users/` — user profile pages
+- `/*/archive/` — git bundle generation (DoS vector, see below)
+- `/*/releases/download/` — release artifacts
+
+Archive requests (`/<owner>/<repo>/archive/*`) are 302-redirected to `forge.ops.eblu.me` (tailnet-only), preventing unauthenticated archive generation. This mitigates a known Forgejo DoS vector where crawlers requesting unique commit SHAs trigger unbounded git bundle generation.
+
+Release downloads are cached at the proxy layer (7-day TTL, keyed by URI) to absorb repeated downloads of the same artifact.
 
 To expose an additional service through the proxy, add the `tag:flyio-target` annotation to its Tailscale Ingress. See [[expose-service-publicly]] for the full workflow.
 

docs/reference/services/forgejo.md

@@ -1,6 +1,6 @@
 ---
 title: Forgejo
-modified: 2026-03-28
+modified: 2026-04-17
 tags:
   - service
   - git
@@ -148,12 +148,24 @@ The UI shows `forge.eblu.me` for HTTPS clone URLs and `forge.ops.eblu.me` for SS
 - **Rate limiting:** nginx rate limits login/signup/forgot-password endpoints (3r/s per client IP via `Fly-Client-IP` header)
 - **fail2ban:** Runs in the Fly.io container; bans IPs after 5 failed logins in 10 minutes via nginx deny list (ephemeral across deploys)
 - **Swagger:** Blocked at the proxy (`/swagger` returns 403); use forge.ops.eblu.me for API access
+- **Archive redirect:** Archive endpoints (`/*/archive/*`) are 302-redirected to `forge.ops.eblu.me` — prevents unauthenticated crawlers from triggering unbounded git bundle generation (known DoS vector, see [[flyio-proxy#Crawler Mitigation]])
+- **robots.txt:** Blocks crawlers from `/mirrors/`, `/user/`, `/users/`, `/*/archive/`, `/*/releases/download/`
 - **OAuth dead-end:** "Sign in with Authentik" redirects to the (tailnet-only) Authentik URL — SSO only works from the tailnet
 
 ### Break-glass
 
 `mise run fly-shutoff` stops all public traffic immediately. forge.ops.eblu.me continues to work from the tailnet. See [[expose-service-publicly#Break-glass shutoff]].
 
+## Monitoring
+
+Forgejo exposes a Prometheus `/metrics` endpoint (enabled via `[metrics]` in `app.ini`). Alloy on indri scrapes it at `localhost:3001/metrics`. Metrics are mostly Go runtime stats and repo counters (no per-request latency histogram).
+
+Request latency is measured at the Fly.io proxy layer via the `flyio_nginx_upstream_response_time_seconds` histogram, visible on the Forgejo Grafana dashboard under "Forgejo: Upstream Response Time".
+
+### Archive Cleanup
+
+The `[cron.archive_cleanup]` section is enabled with `OLDER_THAN = 2h` and `RUN_AT_START = true`. This prevents the `repo-archive/` directory from growing unboundedly when crawlers or users trigger archive downloads. Without this, the directory grew to 54GB in 2 days during a crawler incident in April 2026.
+
 ## Mirrors
 
 Forgejo hosts pull mirrors of external repositories (GitHub, etc.) for supply chain control. Mirrors live in the `mirrors/` org and sync on a configurable interval. See [[manage-forgejo-mirrors]] for operations.

docs/reference/tools/mise-tasks.md

@@ -33,7 +33,8 @@ Run `mise tasks --sort name` for the live list with descriptions.
 | `provision-indri` | Run Ansible playbook for [[indri]] |
 | `provision-ringtail` | Run Ansible playbook for [[ringtail]] (NixOS) |
 | `provision-sifaka` | Run Ansible playbook for [[sifaka]] |
-| `fly-deploy` | Deploy Fly.io public proxy |
+| `fly-deploy` | Deploy Fly.io public proxy (uses op for auth) |
+| `fly-reload` | Reload nginx config, re-resolve upstream DNS (no redeploy) |
 | `fly-setup` | One-time Fly.io secrets and certs setup |
 | `fly-shutoff` | Emergency shutoff: stop all Fly.io proxy machines |
 | `dns-preview` | Preview DNS changes with [[pulumi]] |