C1: deploy adelaide-baby-shower-app to ringtail k3s (#349)
All checks were successful
Deploy Fly.io Proxy / deploy (push) Successful in 1m12s

## Summary

Brings up the Adelaide / Heidi / Addie baby shower app on ringtail k3s with the public/private split that the app's hosting contract calls for: `shower.eblu.me` (public, via Fly proxy) and `shower.ops.eblu.me` (tailnet). App is consumed as a wheel from the Forgejo PyPI index — source lives at [`adelaide-baby-shower-app`](https://forge.eblu.me/eblume/adelaide-baby-shower-app).

### What's included

- **ArgoCD app + manifests** under `argocd/manifests/shower/` (deployment, service, ProxyGroup ingress, ConfigMap for `DJANGO_DEBUG`/`DJANGO_ADMIN_URL`, ExternalSecret for `DJANGO_SECRET_KEY` from 1Password item `Shower (blumeops)`, NFS PV on sifaka, RWX media PVC, RWO local-path data PVC for SQLite). Recreate rollout because SQLite is single-writer.
- **Public surface** (`fly/`): new `shower.eblu.me` server block proxying to `shower.ops.eblu.me`. `/admin/` returns 403 at the edge except `/admin/login/` and `/admin/logout/`, which are rate-limited via a new `shower_auth` zone. `X-Clacks-Overhead` on. GNU Terry Pratchett.
- **fail2ban** filter (`shower-admin-login.conf`) matching 401/403/429 on `/admin/login/` and jail (`shower.conf`) with `maxretry=5/findtime=600/bantime=3600`. The `nginx-deny` action was generalized to take a per-jail `nginx_deny_file` so the shower has its own deny list (forge keeps using the legacy default).
- **Caddy** route on indri (`shower.ops.eblu.me` → `https://shower.tail8d86e.ts.net`).
- **Pulumi** Gandi CNAME `shower.eblu.me → blumeops-proxy.fly.dev.`.
- **Grafana** APM dashboard `configmap-shower-apm.yaml` (request rate, error rate, failed admin login count, latency percentiles, bandwidth, access logs) mirroring `docs-apm.json` with a `host="shower.eblu.me"` filter.
- **Container** `containers/shower/default.nix` — `dockerTools.buildLayeredImage` with a nixpkgs Python and a startup wrapper that creates `/app/data/.venv`, pip-installs `adelaide-baby-shower-app==1.0.0` from the forge PyPI index on first boot, runs migrations + collectstatic, and execs gunicorn. A `local_settings.py` shim pins `DATABASES.NAME`/`MEDIA_ROOT`/`STATIC_ROOT` to absolute paths so they don't end up in site-packages.
- **Docs** runbook at `docs/how-to/operations/shower-app.md` linked from the apps registry, plus changelog fragments.

### Defense layers on the public surface

1. fly nginx geo+fail2ban `$shower_banned` (per-service deny list)
2. fly nginx `limit_req zone=shower_auth` (3 r/s per Fly-Client-IP)
3. django-axes (5 fails / 1h, keyed on username+ip_address)
4. edge `/admin/` block (returns 403 for anything that isn't login/logout)

## Prerequisites for the user to do (NOT in this PR)

Halted on these per request — they touch shared/manual systems:

- [x] **NFS share** on sifaka: `/volume1/shower`, NFS rule for ringtail RW, `chown 1000:1000`
- [ ] **1Password item** `Shower (blumeops)` in the blumeops vault with a freshly minted `secret-key` field (`openssl rand -base64 48`) — do NOT reuse anything that has lived in git
- [ ] **Container build**: `mise run container-build-and-release shower`, then update `images[].newTag` in `argocd/manifests/shower/kustomization.yaml` to the resulting `v1.0.0-<sha>-nix`
- [x] **DNS**: `mise run dns-up` after merge
- [x] **Fly cert**: `fly certs add shower.eblu.me -a blumeops-proxy`
- [ ] **Caddy push**: `mise run provision-indri -- --tags caddy`
- [ ] **Fly redeploy** to pick up the new nginx block + fail2ban jail: `mise run fly-deploy`
- [ ] **ArgoCD sync**: `argocd app set shower --revision shower-app-deploy && argocd app sync shower` to test from this branch before merging

## Test plan

- [ ] Container builds successfully on nix-container-builder runner
- [ ] Pod starts, migrations run, gunicorn answers on :8000
- [ ] `kubectl --context=k3s-ringtail -n shower logs deploy/shower` clean
- [ ] `curl -sf https://shower.ops.eblu.me/` returns the splash page (tailnet)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/` returns 200 (pre-DNS verification)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/admin/users/` returns 403 (edge block)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/admin/login/` returns a Django login response
- [ ] After DNS is up: `curl -I https://shower.eblu.me/` returns 200 with `X-Clacks-Overhead`
- [ ] Grafana dashboard "Shower APM" appears and starts showing traffic
- [ ] `mise run services-check` passes

Reviewed-on: #349
This commit is contained in:
Erich Blume 2026-05-11 13:47:18 -07:00
commit 292d354902
28 changed files with 1313 additions and 8 deletions

20
argocd/apps/shower.yaml Normal file
View file

@ -0,0 +1,20 @@
# Adelaide / Heidi / Addie baby shower app — Django guest/raffle/prize system.
# Public landing page at shower.eblu.me (via fly proxy), staff console + admin
# at shower.ops.eblu.me (tailnet only). Built from forge PyPI wheel.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: shower
namespace: argocd
spec:
project: default
source:
repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
targetRevision: main
path: argocd/manifests/shower
destination:
server: https://ringtail.tail8d86e.ts.net:6443
namespace: shower
syncPolicy:
syncOptions:
- CreateNamespace=true

View file

@ -0,0 +1,229 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-dashboard-shower-apm
namespace: monitoring
labels:
grafana_dashboard: "1"
data:
shower-apm.json: |
{
"annotations": { "list": [] },
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisLabel": "req/s",
"drawStyle": "line",
"fillOpacity": 20,
"lineInterpolation": "linear",
"lineWidth": 1,
"showPoints": "never",
"spanNulls": false,
"stacking": { "group": "A", "mode": "normal" }
},
"mappings": [],
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
"unit": "reqps"
},
"overrides": []
},
"gridPos": { "h": 8, "w": 16, "x": 0, "y": 0 },
"id": 1,
"options": {
"legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "right", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{ "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "sum by (status) (rate(flyio_nginx_http_requests_total{host=\"shower.eblu.me\"}[5m]))", "legendFormat": "{{status}}", "refId": "A" }
],
"title": "Request Rate by Status",
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.01 }, { "color": "red", "value": 0.05 }] },
"unit": "percentunit"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 8, "x": 16, "y": 0 },
"id": 2,
"options": {
"colorMode": "background",
"graphMode": "area",
"justifyMode": "center",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"targets": [
{ "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "sum(rate(flyio_nginx_http_requests_total{host=\"shower.eblu.me\",status=~\"5..\"}[5m])) / sum(rate(flyio_nginx_http_requests_total{host=\"shower.eblu.me\"}[5m]))", "refId": "A" }
],
"title": "Error Rate (5xx)",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 5 }] },
"unit": "short"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 4, "x": 16, "y": 4 },
"id": 3,
"options": {
"colorMode": "background",
"graphMode": "area",
"justifyMode": "center",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"targets": [
{ "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "sum(increase(flyio_nginx_http_requests_total{host=\"shower.eblu.me\",request_uri=~\"/admin/login.*\",status=~\"4..\"}[$__range]))", "refId": "A" }
],
"title": "Failed admin logins (range)",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
"unit": "reqps"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 4, "x": 20, "y": 4 },
"id": 4,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "center",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"targets": [
{ "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "sum(rate(flyio_nginx_http_requests_total{host=\"shower.eblu.me\"}[5m]))", "refId": "A" }
],
"title": "Current RPS",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisLabel": "seconds",
"drawStyle": "line",
"fillOpacity": 10,
"lineInterpolation": "linear",
"lineWidth": 1,
"showPoints": "never",
"spanNulls": false,
"stacking": { "group": "A", "mode": "none" }
},
"mappings": [],
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
"unit": "s"
},
"overrides": []
},
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
"id": 5,
"options": {
"legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "right", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{ "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "histogram_quantile(0.50, sum by (le) (rate(flyio_nginx_http_request_duration_seconds_bucket{host=\"shower.eblu.me\"}[5m])))", "legendFormat": "p50", "refId": "A" },
{ "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "histogram_quantile(0.90, sum by (le) (rate(flyio_nginx_http_request_duration_seconds_bucket{host=\"shower.eblu.me\"}[5m])))", "legendFormat": "p90", "refId": "B" },
{ "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "histogram_quantile(0.99, sum by (le) (rate(flyio_nginx_http_request_duration_seconds_bucket{host=\"shower.eblu.me\"}[5m])))", "legendFormat": "p99", "refId": "C" }
],
"title": "Latency Percentiles",
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisLabel": "",
"drawStyle": "line",
"fillOpacity": 20,
"lineInterpolation": "linear",
"lineWidth": 1,
"showPoints": "never",
"spanNulls": false,
"stacking": { "group": "A", "mode": "none" }
},
"mappings": [],
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
"unit": "Bps"
},
"overrides": []
},
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
"id": 6,
"options": {
"legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "right", "showLegend": true },
"tooltip": { "mode": "single", "sort": "none" }
},
"targets": [
{ "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "sum(rate(flyio_nginx_http_response_bytes_total{host=\"shower.eblu.me\"}[5m]))", "legendFormat": "Bandwidth", "refId": "A" }
],
"title": "Bandwidth",
"type": "timeseries"
},
{
"datasource": { "type": "loki", "uid": "loki" },
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 16 },
"id": 7,
"options": {
"dedupStrategy": "none",
"enableLogDetails": true,
"prettifyLogMessage": false,
"showCommonLabels": false,
"showLabels": false,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": false
},
"targets": [
{ "datasource": { "type": "loki", "uid": "loki" }, "expr": "{instance=\"flyio-proxy\", job=\"flyio-nginx\"} |= \"shower.eblu.me\" | json | line_format \"{{.client_ip}} {{.request_method}} {{.request_uri}} {{.status}} {{.request_time}}s\"", "refId": "A" }
],
"title": "Recent Access Logs",
"type": "logs"
}
],
"refresh": "30s",
"schemaVersion": 38,
"tags": ["shower", "flyio", "apm"],
"templating": { "list": [] },
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "",
"title": "Shower APM",
"uid": "shower-apm",
"version": 1,
"weekStart": ""
}

View file

@ -22,6 +22,7 @@ resources:
- dashboards/configmap-transmission.yaml
- dashboards/configmap-cv-apm.yaml
- dashboards/configmap-docs-apm.yaml
- dashboards/configmap-shower-apm.yaml
- dashboards/configmap-flyio.yaml
- dashboards/configmap-sifaka-disks.yaml
- dashboards/configmap-forgejo.yaml

View file

@ -0,0 +1,22 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: shower-app-config
namespace: shower
data:
DJANGO_DEBUG: "0"
# The app's settings.py hardcodes ALLOWED_HOSTS = ["shower.eblu.me",
# "localhost", "127.0.0.1"] and exposes this env var as a comma-separated
# extras list. shower.ops.eblu.me is what Caddy on indri and the
# Tailscale ProxyGroup both send as the Host header, so the app needs to
# accept it.
DJANGO_ALLOWED_HOSTS: "shower.ops.eblu.me"
# /host/, /admin/, and Django's login surface are all tailnet-only — the
# public proxy 403s everything outside of `/` and `/prizes/<token>/`.
# /host/'s "Django admin" link follows DJANGO_ADMIN_URL.
DJANGO_ADMIN_URL: "https://shower.ops.eblu.me/admin/"
# /host/ is served on shower.ops.eblu.me (tailnet), but the QR codes it
# generates need to point at the public WAN hostname so guest phones can
# reach them. PUBLIC_URL_BASE overrides Django's request.build_absolute_uri()
# in the QR views — see shower/views.py:_public_url. Added in app v1.0.1.
DJANGO_PUBLIC_URL_BASE: "https://shower.eblu.me"

View file

@ -0,0 +1,81 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: shower
namespace: shower
spec:
replicas: 1
# SQLite + RWO data PVC: only one writer at a time. Recreate ensures the
# old pod's lock on the local-path volume is released before the new one
# mounts it.
strategy:
type: Recreate
selector:
matchLabels:
app: shower
template:
metadata:
labels:
app: shower
spec:
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: shower
image: registry.ops.eblu.me/blumeops/shower:kustomized
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
ports:
- containerPort: 8000
name: http
envFrom:
- configMapRef:
name: shower-app-config
- secretRef:
name: shower-app-secrets
volumeMounts:
- name: media
mountPath: /app/media
- name: data
mountPath: /app/data
resources:
requests:
memory: "128Mi"
cpu: "50m"
limits:
memory: "512Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /
port: 8000
httpHeaders:
- name: Host
value: shower.ops.eblu.me
- name: X-Forwarded-Proto
value: https
initialDelaySeconds: 30
periodSeconds: 30
readinessProbe:
httpGet:
path: /
port: 8000
httpHeaders:
- name: Host
value: shower.ops.eblu.me
- name: X-Forwarded-Proto
value: https
initialDelaySeconds: 10
periodSeconds: 10
volumes:
- name: media
persistentVolumeClaim:
claimName: shower-media
- name: data
persistentVolumeClaim:
claimName: shower-data

View file

@ -0,0 +1,19 @@
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shower-app-secrets
namespace: shower
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: onepassword-blumeops
target:
name: shower-app-secrets
creationPolicy: Owner
data:
- secretKey: DJANGO_SECRET_KEY
remoteRef:
key: "Shower (blumeops)"
property: secret-key

View file

@ -0,0 +1,30 @@
# Tailscale Ingress for shower app.
# Exposes at shower.tail8d86e.ts.net.
# Caddy on indri proxies shower.ops.eblu.me here. The fly proxy then proxies
# shower.eblu.me through Caddy to this same endpoint (fly does not contact
# the k8s service directly — all traffic routes through indri's Caddy).
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shower-tailscale
namespace: shower
annotations:
tailscale.com/proxy-class: "default"
tailscale.com/proxy-group: "ingress"
gethomepage.dev/enabled: "true"
gethomepage.dev/name: "Shower"
gethomepage.dev/group: "Home"
gethomepage.dev/icon: "mdi-baby"
gethomepage.dev/description: "Adelaide baby shower"
gethomepage.dev/href: "https://shower.ops.eblu.me"
gethomepage.dev/pod-selector: "app=shower"
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: shower
port:
number: 8000
tls:
- hosts:
- shower

View file

@ -0,0 +1,17 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shower
resources:
- configmap.yaml
- external-secret.yaml
- pv-nfs.yaml
- pvc.yaml
- service.yaml
- ingress-tailscale.yaml
- deployment.yaml
images:
- name: registry.ops.eblu.me/blumeops/shower
newTag: v1.0.2-039d9b9-nix

View file

@ -0,0 +1,24 @@
# NFS PersistentVolume for shower app media uploads (prize photos).
#
# Requires the `shower` share on sifaka with NFS exports matching the
# blumeops standard (192.168.1.0/24 + 100.64.0.0/10, all_squash → admin).
# See docs/how-to/operations/shower-app.md for the Synology web-UI walk
# and docs/reference/storage/sifaka.md for the exports table.
#
# Because all_squash rewrites every NFS write to admin:users (1024:100),
# the in-pod runAsUser does NOT have to match an on-disk uid. Mode 0777
# on /volume1/shower lets the pod read back what it wrote.
apiVersion: v1
kind: PersistentVolume
metadata:
name: shower-media-nfs-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ""
nfs:
server: sifaka
path: /volume1/shower

View file

@ -0,0 +1,30 @@
# Media PVC — RWX NFS share for /app/media (prize photo uploads).
# SQLite DB lives in a separate local-path PVC; NFS file locking is not
# reliable enough for SQLite's WAL/journal.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shower-media
namespace: shower
spec:
accessModes:
- ReadWriteMany
storageClassName: ""
volumeName: shower-media-nfs-pv
resources:
requests:
storage: 10Gi
---
# Database PVC — k3s local-path (default storage class) for SQLite.
# RWO is fine: the deployment runs with a single replica.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shower-data
namespace: shower
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi

View file

@ -0,0 +1,13 @@
apiVersion: v1
kind: Service
metadata:
name: shower
namespace: shower
spec:
selector:
app: shower
ports:
- name: http
port: 8000
targetPort: 8000
protocol: TCP