Fix 502 errors during Fly.io proxy deploys (#131)
All checks were successful
Deploy Fly.io Proxy / deploy (push) Successful in 1m20s
All checks were successful
Deploy Fly.io Proxy / deploy (push) Successful in 1m20s
## Summary - Health check (`/healthz`) now returns 503 until Tailscale is connected - `start.sh` creates `/tmp/tailscale-ready` sentinel after `tailscale up` succeeds - Fly.io keeps the old machine serving traffic during the ~7s startup window Previously, nginx passed the health check immediately, Fly.io routed traffic to the new machine, but MagicDNS wasn't available yet — causing upstream DNS timeouts and 502s on every request until Tailscale connected. ## Deployment and Testing - [ ] Merge and `fly deploy` from `fly/` directory - [ ] Verify deploy completes with zero 502s (check Grafana docs-apm dashboard) - [ ] Confirm health check transitions from 503 → 200 in `fly logs` Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/131
This commit is contained in:
parent
3415cad38c
commit
bd61da4f85
3 changed files with 9 additions and 4 deletions
1
docs/changelog.d/fix-deploy-healthcheck-race.bugfix.md
Normal file
1
docs/changelog.d/fix-deploy-healthcheck-race.bugfix.md
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
Fix 502 errors during Fly.io proxy deploys by deferring health check until Tailscale is connected.
|
||||||
|
|
@ -76,6 +76,9 @@ http {
|
||||||
listen 8080 default_server;
|
listen 8080 default_server;
|
||||||
|
|
||||||
location /healthz {
|
location /healthz {
|
||||||
|
if (!-f /tmp/tailscale-ready) {
|
||||||
|
return 503 "starting\n";
|
||||||
|
}
|
||||||
return 200 "ok\n";
|
return 200 "ok\n";
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,9 @@
|
||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
set -e
|
set -e
|
||||||
|
|
||||||
# Start nginx immediately so port 8080 is bound before Fly's deploy checks.
|
# Start nginx immediately so port 8080 is bound (avoids connection refused).
|
||||||
# Upstream DNS resolution is deferred via resolver + variable in nginx.conf,
|
# Health check returns 503 until /tmp/tailscale-ready exists, so Fly.io
|
||||||
# so nginx starts cleanly even before Tailscale connects.
|
# keeps the old machine serving traffic until Tailscale connects.
|
||||||
nginx -g "daemon off;" &
|
nginx -g "daemon off;" &
|
||||||
NGINX_PID=$!
|
NGINX_PID=$!
|
||||||
echo "Nginx started (waiting for Tailscale before proxying)"
|
echo "Nginx started (waiting for Tailscale before proxying)"
|
||||||
|
|
@ -16,8 +16,9 @@ sleep 2
|
||||||
# Authenticate and join tailnet
|
# Authenticate and join tailnet
|
||||||
tailscale up --authkey="${TS_AUTHKEY}" --hostname=flyio-proxy
|
tailscale up --authkey="${TS_AUTHKEY}" --hostname=flyio-proxy
|
||||||
|
|
||||||
# Wait for tailscale to be ready
|
# Wait for tailscale to be ready, then signal nginx health check
|
||||||
until tailscale status > /dev/null 2>&1; do sleep 1; done
|
until tailscale status > /dev/null 2>&1; do sleep 1; done
|
||||||
|
touch /tmp/tailscale-ready
|
||||||
echo "Tailscale connected"
|
echo "Tailscale connected"
|
||||||
|
|
||||||
# Start Alloy for observability (logs → Loki, metrics → Prometheus)
|
# Start Alloy for observability (logs → Loki, metrics → Prometheus)
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue