Deploy Fly.io Proxy / deploy (push) Failing after 9s

Details

Add Fly.io public reverse proxy for docs.eblu.me (#120 )

## Summary

- Adds a Fly.io reverse proxy (`blumeops-proxy`) that tunnels public traffic to homelab services over Tailscale
- First service exposed: `docs.eblu.me` — the Quartz static docs site
- Includes Pulumi IaC for Tailscale auth key/ACLs and Gandi DNS CNAME
- Adds mise tasks (`fly-deploy`, `fly-setup`, `fly-shutoff`) and Forgejo CI workflow

## Key details

- Fly.io Firecracker VMs support TUN devices natively — no userspace networking needed
- Tailscale auth key is `preauthorized=True` to avoid device approval hangs on container restarts
- nginx caches aggressively for the static site; health check is on the default_server block
- ACLs restrict `tag:flyio-proxy` to `tag:k8s` on port 443 only
- DNS CNAME deployed and verified: `docs.eblu.me` → `blumeops-proxy.fly.dev`

## Test plan

- [x] `curl -sf https://blumeops-proxy.fly.dev/healthz` returns `ok`
- [x] `curl -I -H "Host: docs.eblu.me" https://blumeops-proxy.fly.dev/` returns 200 with `X-Cache-Status`
- [x] `curl -I https://docs.eblu.me/` returns 200 with valid Let's Encrypt cert
- [x] `dig forge.ops.eblu.me` still resolves to 100.98.163.89 (private services unaffected)
- [x] Set `FLY_DEPLOY_TOKEN` Forgejo Actions secret for CI auto-deploy

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/120

2026-02-08 02:36:19 -08:00

2.8 KiB

Raw Blame History

title

Manage Fly.io Proxy

Operational tasks for the flyio-proxy public reverse proxy.

Deploy Changes

After modifying files in fly/:

mise run fly-deploy

Pushes to fly/ on main also trigger automatic deployment via the Forgejo CI workflow.

Add a New Public Service

See expose-service-publicly#Per-service setup for the full walkthrough. In short:

Add a server block to fly/nginx.conf
Add a Fly.io certificate: fly certs add <domain> -a blumeops-proxy
Deploy: mise run fly-deploy
Verify against blumeops-proxy.fly.dev with a Host header
Add DNS CNAME via Pulumi: mise run dns-preview then mise run dns-up

Emergency Shutoff

If the proxy is causing issues (DDoS, unexpected traffic, bandwidth consumption on the home network):

Level 1 — Stop the container (seconds, reversible):

mise run fly-shutoff
# or: fly scale count 0 -a blumeops-proxy --yes

All public services go offline immediately. Tailscale tunnel drops. Zero traffic reaches indri. Restore with fly scale count 1 -a blumeops-proxy.

Level 2 — Revoke Tailscale access (seconds): Remove the flyio-proxy node in the Tailscale admin console. Even if the container is running, it cannot reach the tailnet. Use this if the container itself may be compromised.

Level 3 — Remove DNS (minutes to hours): Delete the CNAME records at Gandi. Takes time for DNS propagation but is the permanent shutoff.

Level 1 is the primary response. It is a single command, takes effect in seconds, and is trivially reversible. Keep mise run fly-shutoff somewhere easily accessible (e.g., pinned in a notes app) so it can be run quickly under stress.

Check Status

# App and machine status
fly status -a blumeops-proxy

# Live logs
fly logs -a blumeops-proxy

# Health check
curl -sf https://blumeops-proxy.fly.dev/healthz

# Certificate status
fly certs list -a blumeops-proxy

Rotate Tailscale Auth Key

The auth key expires every 90 days. To rotate:

Re-apply Pulumi to generate a new key: mise run tailnet-up
Re-run setup to stage the new secret: mise run fly-setup
Deploy to pick up the new secret: mise run fly-deploy

Troubleshooting

502 Bad Gateway: Check fly logs for nginx upstream errors. Verify the backend Tailscale service is running (tailscale status from inside the container via fly ssh console).

Health check failing: fly ssh console -a blumeops-proxy then curl localhost:8080/healthz to test locally.

TLS errors on custom domain: Check cert status with fly certs show <domain> -a blumeops-proxy. Certs auto-provision via Let's Encrypt and may take a few minutes.

flyio-proxy - Service reference card
expose-service-publicly - Full setup guide and architecture

2.8 KiB Raw Blame History