PR review caught that we didn't need an admin login surface on WAN.
App v1.0.1 adds DJANGO_PUBLIC_URL_BASE so QR codes generated from
/host/ (now tailnet-only) still point at shower.eblu.me for guest
phones — that closes the loop and lets us strip the WAN admin surface
entirely.
Container:
- bump version to 1.0.1
- outputHash → fakeHash (build will print the real one)
- entrypoint still does migrate + collectstatic before gunicorn —
the app is small enough that auto-migration is fine
Manifests:
- configmap adds DJANGO_PUBLIC_URL_BASE=https://shower.eblu.me
Fly nginx (shower.eblu.me):
- drop the /admin/(login|logout) carveout
- 403 anything under /admin/ AND /host/ with a "tailnet only" pointer
- drop the shower_auth limit_req zone and \$shower_banned geo
- drop the shower-admin-login fail2ban filter + jail
- drop the shower-deny.conf touch from start.sh
Docs:
- rename how-to docs/how-to/operations/shower-app.md →
shower-on-ringtail.md (mirrors cv-on-indri / docs-on-indri)
- new reference card docs/reference/services/shower-app.md per PR
review comment 2 (≈30s read; quick facts + cross-links)
- rewrite Defense layers section: collapses to general rate limit +
django-axes on the tailnet-side login (the only credential surface)
- rewrite the .infra.md changelog fragment to match
- add a 'Create the admin user' step (kubectl exec createsuperuser)
so first-time deploys aren't locked out
The nginx-deny action's per-jail \`nginx_deny_file\` generalization
stays — harmless future-proofing for the next public service.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the Adelaide / Heidi / Addie baby shower app — a Django guest
splash, raffle picker, and prize-assignment console — on ringtail k3s.
Public landing at shower.eblu.me (via fly proxy), tailnet admin at
shower.ops.eblu.me. App source: forge.eblu.me/eblume/adelaide-baby-shower-app,
wheel-published to the Forgejo Packages PyPI index.
Manifests under argocd/manifests/shower/: NFS-backed PVC for /app/media,
local-path PVC for SQLite, ExternalSecret pulling DJANGO_SECRET_KEY from
1Password (item "Shower (blumeops)"), Tailscale ProxyGroup ingress.
Defense-in-depth for the public surface:
- /admin/ blocked at the fly edge except /admin/login/ and /admin/logout/
- shower_auth rate limit on the login path
- new fail2ban filter+jail with a per-service shower-deny.conf
(nginx-deny action generalized to accept nginx_deny_file)
- django-axes (5 / 1h) keyed on (username, ip_address)
Plus: Caddy route on indri, Pulumi gandi CNAME, Grafana APM dashboard
mirroring docs-apm.json, runbook at how-to/operations/shower-app.md,
and a service-versions entry. X-Clacks-Overhead set on the new server
block — GNU Terry Pratchett.
Build: containers/shower/default.nix uses dockerTools to ship a
nixpkgs Python plus a startup wrapper that installs the wheel into
/app/data/.venv on first boot and execs gunicorn. Lets the wheel come
from forge PyPI without pinning hashes for every transitive dep.
Prerequisites tracked in the runbook (not yet executed):
- NFS share sifaka:/volume1/shower (manual Synology step)
- 1Password item "Shower (blumeops)" with secret-key field
- container build via `mise run container-build-and-release shower`
- Pulumi dns-up after merge
- fly certs add shower.eblu.me
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enable direct peer-to-peer WireGuard connections by pinning tailscaled
to port 41641 and exposing it as a UDP service. Without this, all
traffic routes through Tailscale DERP relays causing 20+ second
latency. Requires dedicated IPv4 (allocated: 168.220.82.221).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Replace per-request DNS resolution (variable-based `proxy_pass`) with static `upstream` blocks and `keepalive` connection pools
- Reuses TLS connections through the Tailscale tunnel instead of handshaking per request
- Add `mise run fly-reload` for nginx config reload without full redeploy (re-resolves upstream DNS)
## Trade-off
DNS is resolved at config load, not per-request. If Tailscale Ingress pods get new IPs (restart, reschedule), `mise run fly-reload` is needed. A Grafana alert will be added to detect this.
## Still TODO on this branch
- [ ] Grafana alert for upstream unreachable (triggers fly-reload reminder)
- [ ] Docs pass
- [ ] Deploy from branch and verify latency improvement
- [ ] Changelog fragment
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #337
The Fly container pulls from tailscale/tailscale:stable which is still
v1.94.2. The `tailscale wait` command doesn't exist until v1.96.2.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Bump Tailscale operator, proxy containers, and init containers from v1.94.2 to v1.96.3 across both clusters (indri + ringtail via shared base kustomization)
- Replace hand-rolled `until tailscale status` polling loop in `fly/start.sh` with `tailscale wait --timeout 60s` (new in v1.96.2)
- Stamp kube-state-metrics review date (already current at v2.18.0)
## Notable upstream changes (v1.94.2 → v1.96.3)
- Go upgraded from 1.25 to 1.26
- `tailscale wait` command — blocks until daemon is running + interface has IP
- AuthKey policy now applies only when users are not logged in (behavioral change)
- Peer Relay improvements (metrics, EC2 IMDS, UDP socket scaling)
- UPnP stability fixes
## Deploy plan
1. Merge PR
2. Sync tailscale-operator on indri: `argocd app sync tailscale-operator`
3. Sync tailscale-operator on ringtail: `argocd app sync tailscale-operator-ringtail --server ringtail...`
4. Verify proxy pods roll with new image: `kubectl --context=minikube-indri -n tailscale get pods`
5. Verify ingress connectivity (spot-check a few `*.tail8d86e.ts.net` services)
6. Rebuild + deploy Fly proxy container (separate step, picks up `tailscale wait` change)
## Test plan
- [ ] ArgoCD diff looks clean for both apps before sync
- [ ] Proxy pods on indri come up healthy with v1.96.3 images
- [ ] Proxy pods on ringtail come up healthy with v1.96.3 images
- [ ] Tailscale ingress services remain reachable (e.g., grafana, prometheus)
- [ ] Fly proxy rebuild deploys successfully with `tailscale wait`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #304
## Summary
- Start nginx after Tailscale connects (community best practice for Tailscale sidecars)
- Switch to `bluegreen` deploy strategy — old machine serves until new one is healthy
- Replace top-level `[checks]` with `[[http_service.checks]]` — only service-level checks gate traffic routing ([confirmed by Fly.io staff](https://community.fly.io/t/clarifying-the-types-of-health-checks/20379))
- Remove sentinel file and nginx if-check (no longer needed)
Supersedes the approach in #131 — that helped (502 window dropped from ~30s to ~3s) but couldn't fully eliminate it because top-level checks don't gate routing and Fly.io's proxy sends traffic as soon as the port is reachable.
## Deployment and Testing
- [ ] Merge and `fly deploy` from `fly/` directory
- [ ] Verify deploy completes with zero 502s (watch `fly logs` and Grafana docs-apm)
- [ ] Confirm `fly checks list` shows the new service-level check passing
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/132
## Summary
- Health check (`/healthz`) now returns 503 until Tailscale is connected
- `start.sh` creates `/tmp/tailscale-ready` sentinel after `tailscale up` succeeds
- Fly.io keeps the old machine serving traffic during the ~7s startup window
Previously, nginx passed the health check immediately, Fly.io routed traffic to the new machine, but MagicDNS wasn't available yet — causing upstream DNS timeouts and 502s on every request until Tailscale connected.
## Deployment and Testing
- [ ] Merge and `fly deploy` from `fly/` directory
- [ ] Verify deploy completes with zero 502s (check Grafana docs-apm dashboard)
- [ ] Confirm health check transitions from 503 → 200 in `fly logs`
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/131
## Summary
- Start nginx before Tailscale in `start.sh` so port 8080 is bound immediately, eliminating the "app is not listening on the expected address" WARNING during `fly deploy`
- Switch `proxy_pass` to use a variable with `resolver 100.100.100.100 valid=30s` so nginx can start without resolving MagicDNS names at config load time
- DNS results cached 30s per worker — no per-request lookup overhead
## Context
The WARNING was a race condition: Fly checks for listeners right after the machine starts, but `start.sh` ran ~5-10s of Tailscale setup before starting nginx. The health check always passed later, but the warning was noisy.
## Test plan
- [ ] Merge and let the deploy-fly workflow trigger
- [ ] Check runner logs for absence of the WARNING
- [ ] Verify `docs.eblu.me` still serves correctly
- [ ] Verify `/healthz` still passes
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/128
## Summary
- Adds a Fly.io reverse proxy (`blumeops-proxy`) that tunnels public traffic to homelab services over Tailscale
- First service exposed: `docs.eblu.me` — the Quartz static docs site
- Includes Pulumi IaC for Tailscale auth key/ACLs and Gandi DNS CNAME
- Adds mise tasks (`fly-deploy`, `fly-setup`, `fly-shutoff`) and Forgejo CI workflow
## Key details
- Fly.io Firecracker VMs support TUN devices natively — no userspace networking needed
- Tailscale auth key is `preauthorized=True` to avoid device approval hangs on container restarts
- nginx caches aggressively for the static site; health check is on the default_server block
- ACLs restrict `tag:flyio-proxy` to `tag:k8s` on port 443 only
- DNS CNAME deployed and verified: `docs.eblu.me` → `blumeops-proxy.fly.dev`
## Test plan
- [x] `curl -sf https://blumeops-proxy.fly.dev/healthz` returns `ok`
- [x] `curl -I -H "Host: docs.eblu.me" https://blumeops-proxy.fly.dev/` returns 200 with `X-Cache-Status`
- [x] `curl -I https://docs.eblu.me/` returns 200 with valid Let's Encrypt cert
- [x] `dig forge.ops.eblu.me` still resolves to 100.98.163.89 (private services unaffected)
- [x] Set `FLY_DEPLOY_TOKEN` Forgejo Actions secret for CI auto-deploy
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/120
