2026-01-25 08:15:46 -08:00
|
|
|
"""Pulumi program to manage eblu.me DNS via Gandi LiveDNS.
|
|
|
|
|
|
|
|
|
|
This program manages DNS records for blumeops infrastructure:
|
|
|
|
|
- Wildcard record for *.ops.eblu.me pointing to indri's Tailscale IP
|
|
|
|
|
- indri hosts Caddy as the reverse proxy for all services
|
|
|
|
|
- This allows services to be accessed via real DNS names while remaining
|
|
|
|
|
tailnet-only (Tailscale IPs are not publicly routable)
|
|
|
|
|
|
|
|
|
|
Authentication:
|
|
|
|
|
Set GANDI_PERSONAL_ACCESS_TOKEN environment variable.
|
2026-04-27 09:48:46 -07:00
|
|
|
See docs/how-to/configuration/rotate-gandi-pat.md for PAT management.
|
2026-01-25 08:15:46 -08:00
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
import os
|
|
|
|
|
import socket
|
|
|
|
|
|
|
|
|
|
import pulumi
|
|
|
|
|
import pulumiverse_gandi as gandi
|
|
|
|
|
|
|
|
|
|
# Get configuration
|
|
|
|
|
config = pulumi.Config()
|
|
|
|
|
domain = config.require("domain") # eblu.me
|
|
|
|
|
subdomain = config.require("subdomain") # ops
|
|
|
|
|
|
|
|
|
|
# Resolve indri's Tailscale IP dynamically via MagicDNS
|
|
|
|
|
# This script runs on the tailnet, so we can resolve the hostname directly.
|
|
|
|
|
# indri hosts Caddy, which reverse-proxies all services.
|
|
|
|
|
# Break-glass: set BLUMEOPS_REVERSE_PROXY_IP env var to override DNS resolution
|
|
|
|
|
REVERSE_PROXY_HOST = "indri.tail8d86e.ts.net"
|
|
|
|
|
tailscale_ip = os.environ.get("BLUMEOPS_REVERSE_PROXY_IP") or socket.gethostbyname(
|
|
|
|
|
REVERSE_PROXY_HOST
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
# Wildcard A record for *.ops.eblu.me
|
|
|
|
|
# Points to indri's Tailscale IP, which is only routable within the tailnet.
|
|
|
|
|
# This allows containers and other systems to resolve real DNS names
|
|
|
|
|
# while keeping services private to the tailnet.
|
|
|
|
|
wildcard_record = gandi.livedns.Record(
|
|
|
|
|
"ops-wildcard",
|
|
|
|
|
zone=domain,
|
|
|
|
|
name=f"*.{subdomain}",
|
|
|
|
|
type="A",
|
|
|
|
|
ttl=300,
|
|
|
|
|
values=[tailscale_ip],
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
# Base subdomain record (ops.eblu.me) - same IP
|
|
|
|
|
base_record = gandi.livedns.Record(
|
|
|
|
|
"ops-base",
|
|
|
|
|
zone=domain,
|
|
|
|
|
name=subdomain,
|
|
|
|
|
type="A",
|
|
|
|
|
ttl=300,
|
|
|
|
|
values=[tailscale_ip],
|
|
|
|
|
)
|
|
|
|
|
|
2026-02-08 02:36:19 -08:00
|
|
|
# ============== Public Services (Fly.io proxy) ==============
|
|
|
|
|
# CNAME records pointing public subdomains to Fly.io for reverse proxying
|
|
|
|
|
# back to the tailnet. See docs/how-to/expose-service-publicly.md
|
|
|
|
|
|
|
|
|
|
docs_public = gandi.livedns.Record(
|
|
|
|
|
"docs-public",
|
|
|
|
|
zone=domain,
|
|
|
|
|
name="docs",
|
|
|
|
|
type="CNAME",
|
|
|
|
|
ttl=300,
|
|
|
|
|
values=["blumeops-proxy.fly.dev."],
|
|
|
|
|
)
|
|
|
|
|
|
2026-02-12 14:05:00 -08:00
|
|
|
cv_public = gandi.livedns.Record(
|
|
|
|
|
"cv-public",
|
|
|
|
|
zone=domain,
|
|
|
|
|
name="cv",
|
|
|
|
|
type="CNAME",
|
|
|
|
|
ttl=300,
|
|
|
|
|
values=["blumeops-proxy.fly.dev."],
|
|
|
|
|
)
|
|
|
|
|
|
Expose Forgejo publicly at forge.eblu.me (#278)
## Summary
Expose Forgejo publicly at `forge.eblu.me` via the Fly.io reverse proxy — the first dynamic, authenticated public-facing service.
- **Forgejo hardening:** Domain changed to forge.eblu.me, SSH stays on forge.ops.eblu.me, reverse proxy trust headers configured, local registration locked to external-only (Authentik SSO)
- **Tailscale Ingress:** ExternalName Service + Ingress in tailscale-operator creates forge.tail8d86e.ts.net endpoint
- **Fly.io proxy:** nginx server block with rate-limited auth endpoints (3r/s), fail2ban with custom nginx-deny action, security headers, /swagger blocked, WebSocket support, 512m body limit
- **Authentik:** OAuth callback updated to forge.eblu.me
- **DNS/TLS:** CNAME record in Pulumi, cert in fly-setup
- **Rename:** ~29 files updated from forge.ops.eblu.me to forge.eblu.me (HTTPS refs only; SSH, container builds, and Caddy table kept as-is)
## Deployment Order
1. `mise run provision-indri -- --tags forgejo` (config changes)
2. Verify forge.ops.eblu.me still works
3. `argocd app set tailscale-operator --revision feature/forge-public && argocd app sync tailscale-operator`
4. Verify `curl https://forge.tail8d86e.ts.net`
5. `cd fly && fly deploy`
6. Verify pre-DNS: `curl -H "Host: forge.eblu.me" https://blumeops-proxy.fly.dev/`
7. `fly certs add forge.eblu.me -a blumeops-proxy`
8. `argocd app set authentik --revision feature/forge-public && argocd app sync authentik`
9. `mise run dns-preview && mise run dns-up`
10. Full verification (see below)
11. Rehearse `mise run fly-shutoff`
12. After merge: reset ArgoCD revisions to main, re-sync
## Verification Checklist
- [ ] forge.eblu.me loads, shows public repos
- [ ] forge.ops.eblu.me still works from tailnet
- [ ] SSH clone via forge.ops.eblu.me:2222 works
- [ ] HTTPS clone via forge.eblu.me works
- [ ] UI shows forge.eblu.me for HTTPS clone, forge.ops.eblu.me for SSH
- [ ] /swagger returns 403
- [ ] Rapid login attempts trigger 429 rate limit
- [ ] fail2ban bans after 5 failed logins in 10 minutes
- [ ] ArgoCD can still sync (SSH unaffected)
- [ ] `mise run fly-shutoff` stops all public traffic
- [ ] `mise run services-check` passes
Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/278
2026-03-03 08:40:41 -08:00
|
|
|
forge_public = gandi.livedns.Record(
|
|
|
|
|
"forge-public",
|
|
|
|
|
zone=domain,
|
|
|
|
|
name="forge",
|
|
|
|
|
type="CNAME",
|
|
|
|
|
ttl=300,
|
|
|
|
|
values=["blumeops-proxy.fly.dev."],
|
|
|
|
|
)
|
|
|
|
|
|
C1: deploy adelaide-baby-shower-app to ringtail k3s (#349)
## Summary
Brings up the Adelaide / Heidi / Addie baby shower app on ringtail k3s with the public/private split that the app's hosting contract calls for: `shower.eblu.me` (public, via Fly proxy) and `shower.ops.eblu.me` (tailnet). App is consumed as a wheel from the Forgejo PyPI index — source lives at [`adelaide-baby-shower-app`](https://forge.eblu.me/eblume/adelaide-baby-shower-app).
### What's included
- **ArgoCD app + manifests** under `argocd/manifests/shower/` (deployment, service, ProxyGroup ingress, ConfigMap for `DJANGO_DEBUG`/`DJANGO_ADMIN_URL`, ExternalSecret for `DJANGO_SECRET_KEY` from 1Password item `Shower (blumeops)`, NFS PV on sifaka, RWX media PVC, RWO local-path data PVC for SQLite). Recreate rollout because SQLite is single-writer.
- **Public surface** (`fly/`): new `shower.eblu.me` server block proxying to `shower.ops.eblu.me`. `/admin/` returns 403 at the edge except `/admin/login/` and `/admin/logout/`, which are rate-limited via a new `shower_auth` zone. `X-Clacks-Overhead` on. GNU Terry Pratchett.
- **fail2ban** filter (`shower-admin-login.conf`) matching 401/403/429 on `/admin/login/` and jail (`shower.conf`) with `maxretry=5/findtime=600/bantime=3600`. The `nginx-deny` action was generalized to take a per-jail `nginx_deny_file` so the shower has its own deny list (forge keeps using the legacy default).
- **Caddy** route on indri (`shower.ops.eblu.me` → `https://shower.tail8d86e.ts.net`).
- **Pulumi** Gandi CNAME `shower.eblu.me → blumeops-proxy.fly.dev.`.
- **Grafana** APM dashboard `configmap-shower-apm.yaml` (request rate, error rate, failed admin login count, latency percentiles, bandwidth, access logs) mirroring `docs-apm.json` with a `host="shower.eblu.me"` filter.
- **Container** `containers/shower/default.nix` — `dockerTools.buildLayeredImage` with a nixpkgs Python and a startup wrapper that creates `/app/data/.venv`, pip-installs `adelaide-baby-shower-app==1.0.0` from the forge PyPI index on first boot, runs migrations + collectstatic, and execs gunicorn. A `local_settings.py` shim pins `DATABASES.NAME`/`MEDIA_ROOT`/`STATIC_ROOT` to absolute paths so they don't end up in site-packages.
- **Docs** runbook at `docs/how-to/operations/shower-app.md` linked from the apps registry, plus changelog fragments.
### Defense layers on the public surface
1. fly nginx geo+fail2ban `$shower_banned` (per-service deny list)
2. fly nginx `limit_req zone=shower_auth` (3 r/s per Fly-Client-IP)
3. django-axes (5 fails / 1h, keyed on username+ip_address)
4. edge `/admin/` block (returns 403 for anything that isn't login/logout)
## Prerequisites for the user to do (NOT in this PR)
Halted on these per request — they touch shared/manual systems:
- [x] **NFS share** on sifaka: `/volume1/shower`, NFS rule for ringtail RW, `chown 1000:1000`
- [ ] **1Password item** `Shower (blumeops)` in the blumeops vault with a freshly minted `secret-key` field (`openssl rand -base64 48`) — do NOT reuse anything that has lived in git
- [ ] **Container build**: `mise run container-build-and-release shower`, then update `images[].newTag` in `argocd/manifests/shower/kustomization.yaml` to the resulting `v1.0.0-<sha>-nix`
- [x] **DNS**: `mise run dns-up` after merge
- [x] **Fly cert**: `fly certs add shower.eblu.me -a blumeops-proxy`
- [ ] **Caddy push**: `mise run provision-indri -- --tags caddy`
- [ ] **Fly redeploy** to pick up the new nginx block + fail2ban jail: `mise run fly-deploy`
- [ ] **ArgoCD sync**: `argocd app set shower --revision shower-app-deploy && argocd app sync shower` to test from this branch before merging
## Test plan
- [ ] Container builds successfully on nix-container-builder runner
- [ ] Pod starts, migrations run, gunicorn answers on :8000
- [ ] `kubectl --context=k3s-ringtail -n shower logs deploy/shower` clean
- [ ] `curl -sf https://shower.ops.eblu.me/` returns the splash page (tailnet)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/` returns 200 (pre-DNS verification)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/admin/users/` returns 403 (edge block)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/admin/login/` returns a Django login response
- [ ] After DNS is up: `curl -I https://shower.eblu.me/` returns 200 with `X-Clacks-Overhead`
- [ ] Grafana dashboard "Shower APM" appears and starts showing traffic
- [ ] `mise run services-check` passes
Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/349
2026-05-11 13:47:18 -07:00
|
|
|
shower_public = gandi.livedns.Record(
|
|
|
|
|
"shower-public",
|
|
|
|
|
zone=domain,
|
|
|
|
|
name="shower",
|
|
|
|
|
type="CNAME",
|
|
|
|
|
ttl=300,
|
|
|
|
|
values=["blumeops-proxy.fly.dev."],
|
|
|
|
|
)
|
|
|
|
|
|
2026-01-25 08:15:46 -08:00
|
|
|
# ============== Exports ==============
|
|
|
|
|
pulumi.export("domain", domain)
|
|
|
|
|
pulumi.export("wildcard_fqdn", f"*.{subdomain}.{domain}")
|
|
|
|
|
pulumi.export("base_fqdn", f"{subdomain}.{domain}")
|
|
|
|
|
pulumi.export("target_ip", tailscale_ip)
|
2026-02-08 02:36:19 -08:00
|
|
|
pulumi.export("docs_public_fqdn", f"docs.{domain}")
|
2026-02-12 14:05:00 -08:00
|
|
|
pulumi.export("cv_public_fqdn", f"cv.{domain}")
|
Expose Forgejo publicly at forge.eblu.me (#278)
## Summary
Expose Forgejo publicly at `forge.eblu.me` via the Fly.io reverse proxy — the first dynamic, authenticated public-facing service.
- **Forgejo hardening:** Domain changed to forge.eblu.me, SSH stays on forge.ops.eblu.me, reverse proxy trust headers configured, local registration locked to external-only (Authentik SSO)
- **Tailscale Ingress:** ExternalName Service + Ingress in tailscale-operator creates forge.tail8d86e.ts.net endpoint
- **Fly.io proxy:** nginx server block with rate-limited auth endpoints (3r/s), fail2ban with custom nginx-deny action, security headers, /swagger blocked, WebSocket support, 512m body limit
- **Authentik:** OAuth callback updated to forge.eblu.me
- **DNS/TLS:** CNAME record in Pulumi, cert in fly-setup
- **Rename:** ~29 files updated from forge.ops.eblu.me to forge.eblu.me (HTTPS refs only; SSH, container builds, and Caddy table kept as-is)
## Deployment Order
1. `mise run provision-indri -- --tags forgejo` (config changes)
2. Verify forge.ops.eblu.me still works
3. `argocd app set tailscale-operator --revision feature/forge-public && argocd app sync tailscale-operator`
4. Verify `curl https://forge.tail8d86e.ts.net`
5. `cd fly && fly deploy`
6. Verify pre-DNS: `curl -H "Host: forge.eblu.me" https://blumeops-proxy.fly.dev/`
7. `fly certs add forge.eblu.me -a blumeops-proxy`
8. `argocd app set authentik --revision feature/forge-public && argocd app sync authentik`
9. `mise run dns-preview && mise run dns-up`
10. Full verification (see below)
11. Rehearse `mise run fly-shutoff`
12. After merge: reset ArgoCD revisions to main, re-sync
## Verification Checklist
- [ ] forge.eblu.me loads, shows public repos
- [ ] forge.ops.eblu.me still works from tailnet
- [ ] SSH clone via forge.ops.eblu.me:2222 works
- [ ] HTTPS clone via forge.eblu.me works
- [ ] UI shows forge.eblu.me for HTTPS clone, forge.ops.eblu.me for SSH
- [ ] /swagger returns 403
- [ ] Rapid login attempts trigger 429 rate limit
- [ ] fail2ban bans after 5 failed logins in 10 minutes
- [ ] ArgoCD can still sync (SSH unaffected)
- [ ] `mise run fly-shutoff` stops all public traffic
- [ ] `mise run services-check` passes
Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/278
2026-03-03 08:40:41 -08:00
|
|
|
pulumi.export("forge_public_fqdn", f"forge.{domain}")
|
C1: deploy adelaide-baby-shower-app to ringtail k3s (#349)
## Summary
Brings up the Adelaide / Heidi / Addie baby shower app on ringtail k3s with the public/private split that the app's hosting contract calls for: `shower.eblu.me` (public, via Fly proxy) and `shower.ops.eblu.me` (tailnet). App is consumed as a wheel from the Forgejo PyPI index — source lives at [`adelaide-baby-shower-app`](https://forge.eblu.me/eblume/adelaide-baby-shower-app).
### What's included
- **ArgoCD app + manifests** under `argocd/manifests/shower/` (deployment, service, ProxyGroup ingress, ConfigMap for `DJANGO_DEBUG`/`DJANGO_ADMIN_URL`, ExternalSecret for `DJANGO_SECRET_KEY` from 1Password item `Shower (blumeops)`, NFS PV on sifaka, RWX media PVC, RWO local-path data PVC for SQLite). Recreate rollout because SQLite is single-writer.
- **Public surface** (`fly/`): new `shower.eblu.me` server block proxying to `shower.ops.eblu.me`. `/admin/` returns 403 at the edge except `/admin/login/` and `/admin/logout/`, which are rate-limited via a new `shower_auth` zone. `X-Clacks-Overhead` on. GNU Terry Pratchett.
- **fail2ban** filter (`shower-admin-login.conf`) matching 401/403/429 on `/admin/login/` and jail (`shower.conf`) with `maxretry=5/findtime=600/bantime=3600`. The `nginx-deny` action was generalized to take a per-jail `nginx_deny_file` so the shower has its own deny list (forge keeps using the legacy default).
- **Caddy** route on indri (`shower.ops.eblu.me` → `https://shower.tail8d86e.ts.net`).
- **Pulumi** Gandi CNAME `shower.eblu.me → blumeops-proxy.fly.dev.`.
- **Grafana** APM dashboard `configmap-shower-apm.yaml` (request rate, error rate, failed admin login count, latency percentiles, bandwidth, access logs) mirroring `docs-apm.json` with a `host="shower.eblu.me"` filter.
- **Container** `containers/shower/default.nix` — `dockerTools.buildLayeredImage` with a nixpkgs Python and a startup wrapper that creates `/app/data/.venv`, pip-installs `adelaide-baby-shower-app==1.0.0` from the forge PyPI index on first boot, runs migrations + collectstatic, and execs gunicorn. A `local_settings.py` shim pins `DATABASES.NAME`/`MEDIA_ROOT`/`STATIC_ROOT` to absolute paths so they don't end up in site-packages.
- **Docs** runbook at `docs/how-to/operations/shower-app.md` linked from the apps registry, plus changelog fragments.
### Defense layers on the public surface
1. fly nginx geo+fail2ban `$shower_banned` (per-service deny list)
2. fly nginx `limit_req zone=shower_auth` (3 r/s per Fly-Client-IP)
3. django-axes (5 fails / 1h, keyed on username+ip_address)
4. edge `/admin/` block (returns 403 for anything that isn't login/logout)
## Prerequisites for the user to do (NOT in this PR)
Halted on these per request — they touch shared/manual systems:
- [x] **NFS share** on sifaka: `/volume1/shower`, NFS rule for ringtail RW, `chown 1000:1000`
- [ ] **1Password item** `Shower (blumeops)` in the blumeops vault with a freshly minted `secret-key` field (`openssl rand -base64 48`) — do NOT reuse anything that has lived in git
- [ ] **Container build**: `mise run container-build-and-release shower`, then update `images[].newTag` in `argocd/manifests/shower/kustomization.yaml` to the resulting `v1.0.0-<sha>-nix`
- [x] **DNS**: `mise run dns-up` after merge
- [x] **Fly cert**: `fly certs add shower.eblu.me -a blumeops-proxy`
- [ ] **Caddy push**: `mise run provision-indri -- --tags caddy`
- [ ] **Fly redeploy** to pick up the new nginx block + fail2ban jail: `mise run fly-deploy`
- [ ] **ArgoCD sync**: `argocd app set shower --revision shower-app-deploy && argocd app sync shower` to test from this branch before merging
## Test plan
- [ ] Container builds successfully on nix-container-builder runner
- [ ] Pod starts, migrations run, gunicorn answers on :8000
- [ ] `kubectl --context=k3s-ringtail -n shower logs deploy/shower` clean
- [ ] `curl -sf https://shower.ops.eblu.me/` returns the splash page (tailnet)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/` returns 200 (pre-DNS verification)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/admin/users/` returns 403 (edge block)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/admin/login/` returns a Django login response
- [ ] After DNS is up: `curl -I https://shower.eblu.me/` returns 200 with `X-Clacks-Overhead`
- [ ] Grafana dashboard "Shower APM" appears and starts showing traffic
- [ ] `mise run services-check` passes
Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/349
2026-05-11 13:47:18 -07:00
|
|
|
pulumi.export("shower_public_fqdn", f"shower.{domain}")
|