Fix 502 errors during Fly.io proxy deploys (#131)
All checks were successful
Deploy Fly.io Proxy / deploy (push) Successful in 1m20s

## Summary
- Health check (`/healthz`) now returns 503 until Tailscale is connected
- `start.sh` creates `/tmp/tailscale-ready` sentinel after `tailscale up` succeeds
- Fly.io keeps the old machine serving traffic during the ~7s startup window

Previously, nginx passed the health check immediately, Fly.io routed traffic to the new machine, but MagicDNS wasn't available yet — causing upstream DNS timeouts and 502s on every request until Tailscale connected.

## Deployment and Testing
- [ ] Merge and `fly deploy` from `fly/` directory
- [ ] Verify deploy completes with zero 502s (check Grafana docs-apm dashboard)
- [ ] Confirm health check transitions from 503 → 200 in `fly logs`

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/131
This commit is contained in:
Erich Blume 2026-02-09 11:07:36 -08:00
commit bd61da4f85
3 changed files with 9 additions and 4 deletions

View file

@ -0,0 +1 @@
Fix 502 errors during Fly.io proxy deploys by deferring health check until Tailscale is connected.

View file

@ -76,6 +76,9 @@ http {
listen 8080 default_server;
location /healthz {
if (!-f /tmp/tailscale-ready) {
return 503 "starting\n";
}
return 200 "ok\n";
}

View file

@ -1,9 +1,9 @@
#!/bin/sh
set -e
# Start nginx immediately so port 8080 is bound before Fly's deploy checks.
# Upstream DNS resolution is deferred via resolver + variable in nginx.conf,
# so nginx starts cleanly even before Tailscale connects.
# Start nginx immediately so port 8080 is bound (avoids connection refused).
# Health check returns 503 until /tmp/tailscale-ready exists, so Fly.io
# keeps the old machine serving traffic until Tailscale connects.
nginx -g "daemon off;" &
NGINX_PID=$!
echo "Nginx started (waiting for Tailscale before proxying)"
@ -16,8 +16,9 @@ sleep 2
# Authenticate and join tailnet
tailscale up --authkey="${TS_AUTHKEY}" --hostname=flyio-proxy
# Wait for tailscale to be ready
# Wait for tailscale to be ready, then signal nginx health check
until tailscale status > /dev/null 2>&1; do sleep 1; done
touch /tmp/tailscale-ready
echo "Tailscale connected"
# Start Alloy for observability (logs → Loki, metrics → Prometheus)