2026-01-13 22:50:28 -08:00
|
|
|
---
|
|
|
|
|
borgmatic_config: /Users/erichblume/.config/borgmatic/config.yaml
|
2026-01-16 12:30:20 -08:00
|
|
|
borgmatic_config_dir: /Users/erichblume/.config/borgmatic
|
2026-01-13 22:50:28 -08:00
|
|
|
borgmatic_log_dir: /Users/erichblume/Library/Logs
|
|
|
|
|
|
2026-01-19 18:00:32 -08:00
|
|
|
# Full path to borg binary since LaunchAgent doesn't have homebrew in PATH
|
|
|
|
|
borgmatic_local_path: /opt/homebrew/bin/borg
|
|
|
|
|
|
2026-04-15 07:23:46 -07:00
|
|
|
# Borgmatic version — keep in sync with mise.toml in the repo root.
|
|
|
|
|
# Ansible installs this via `mise install` so indri doesn't need the repo cloned.
|
|
|
|
|
borgmatic_version: "2.1.4"
|
|
|
|
|
|
|
|
|
|
# Full path to borgmatic binary — called directly by LaunchAgents to avoid
|
|
|
|
|
# routing through mise, which triggers macOS TCC permission dialogs for
|
|
|
|
|
# protected folders (e.g. ~/Documents) that hang headless LaunchAgent sessions.
|
|
|
|
|
# Uses mise's "latest" symlink so version bumps don't break the LaunchAgent path.
|
|
|
|
|
borgmatic_bin: /Users/erichblume/.local/share/mise/installs/pipx-borgmatic/latest/bin/borgmatic
|
|
|
|
|
|
2026-01-13 22:50:28 -08:00
|
|
|
# Schedule: runs daily at 2:00 AM
|
|
|
|
|
borgmatic_schedule_hour: 2
|
|
|
|
|
borgmatic_schedule_minute: 0
|
2026-01-16 12:30:20 -08:00
|
|
|
|
|
|
|
|
# Source directories to back up
|
|
|
|
|
borgmatic_source_directories:
|
|
|
|
|
- /Users/erichblume/code/personal/zk
|
|
|
|
|
- /opt/homebrew/var/forgejo
|
|
|
|
|
- /Users/erichblume/.config/borgmatic
|
|
|
|
|
- /Users/erichblume/Documents
|
2026-03-16 21:59:10 -07:00
|
|
|
- /Users/erichblume/.local/share/borgmatic/k8s-dumps
|
C1: deploy adelaide-baby-shower-app to ringtail k3s (#349)
## Summary
Brings up the Adelaide / Heidi / Addie baby shower app on ringtail k3s with the public/private split that the app's hosting contract calls for: `shower.eblu.me` (public, via Fly proxy) and `shower.ops.eblu.me` (tailnet). App is consumed as a wheel from the Forgejo PyPI index — source lives at [`adelaide-baby-shower-app`](https://forge.eblu.me/eblume/adelaide-baby-shower-app).
### What's included
- **ArgoCD app + manifests** under `argocd/manifests/shower/` (deployment, service, ProxyGroup ingress, ConfigMap for `DJANGO_DEBUG`/`DJANGO_ADMIN_URL`, ExternalSecret for `DJANGO_SECRET_KEY` from 1Password item `Shower (blumeops)`, NFS PV on sifaka, RWX media PVC, RWO local-path data PVC for SQLite). Recreate rollout because SQLite is single-writer.
- **Public surface** (`fly/`): new `shower.eblu.me` server block proxying to `shower.ops.eblu.me`. `/admin/` returns 403 at the edge except `/admin/login/` and `/admin/logout/`, which are rate-limited via a new `shower_auth` zone. `X-Clacks-Overhead` on. GNU Terry Pratchett.
- **fail2ban** filter (`shower-admin-login.conf`) matching 401/403/429 on `/admin/login/` and jail (`shower.conf`) with `maxretry=5/findtime=600/bantime=3600`. The `nginx-deny` action was generalized to take a per-jail `nginx_deny_file` so the shower has its own deny list (forge keeps using the legacy default).
- **Caddy** route on indri (`shower.ops.eblu.me` → `https://shower.tail8d86e.ts.net`).
- **Pulumi** Gandi CNAME `shower.eblu.me → blumeops-proxy.fly.dev.`.
- **Grafana** APM dashboard `configmap-shower-apm.yaml` (request rate, error rate, failed admin login count, latency percentiles, bandwidth, access logs) mirroring `docs-apm.json` with a `host="shower.eblu.me"` filter.
- **Container** `containers/shower/default.nix` — `dockerTools.buildLayeredImage` with a nixpkgs Python and a startup wrapper that creates `/app/data/.venv`, pip-installs `adelaide-baby-shower-app==1.0.0` from the forge PyPI index on first boot, runs migrations + collectstatic, and execs gunicorn. A `local_settings.py` shim pins `DATABASES.NAME`/`MEDIA_ROOT`/`STATIC_ROOT` to absolute paths so they don't end up in site-packages.
- **Docs** runbook at `docs/how-to/operations/shower-app.md` linked from the apps registry, plus changelog fragments.
### Defense layers on the public surface
1. fly nginx geo+fail2ban `$shower_banned` (per-service deny list)
2. fly nginx `limit_req zone=shower_auth` (3 r/s per Fly-Client-IP)
3. django-axes (5 fails / 1h, keyed on username+ip_address)
4. edge `/admin/` block (returns 403 for anything that isn't login/logout)
## Prerequisites for the user to do (NOT in this PR)
Halted on these per request — they touch shared/manual systems:
- [x] **NFS share** on sifaka: `/volume1/shower`, NFS rule for ringtail RW, `chown 1000:1000`
- [ ] **1Password item** `Shower (blumeops)` in the blumeops vault with a freshly minted `secret-key` field (`openssl rand -base64 48`) — do NOT reuse anything that has lived in git
- [ ] **Container build**: `mise run container-build-and-release shower`, then update `images[].newTag` in `argocd/manifests/shower/kustomization.yaml` to the resulting `v1.0.0-<sha>-nix`
- [x] **DNS**: `mise run dns-up` after merge
- [x] **Fly cert**: `fly certs add shower.eblu.me -a blumeops-proxy`
- [ ] **Caddy push**: `mise run provision-indri -- --tags caddy`
- [ ] **Fly redeploy** to pick up the new nginx block + fail2ban jail: `mise run fly-deploy`
- [ ] **ArgoCD sync**: `argocd app set shower --revision shower-app-deploy && argocd app sync shower` to test from this branch before merging
## Test plan
- [ ] Container builds successfully on nix-container-builder runner
- [ ] Pod starts, migrations run, gunicorn answers on :8000
- [ ] `kubectl --context=k3s-ringtail -n shower logs deploy/shower` clean
- [ ] `curl -sf https://shower.ops.eblu.me/` returns the splash page (tailnet)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/` returns 200 (pre-DNS verification)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/admin/users/` returns 403 (edge block)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/admin/login/` returns a Django login response
- [ ] After DNS is up: `curl -I https://shower.eblu.me/` returns 200 with `X-Clacks-Overhead`
- [ ] Grafana dashboard "Shower APM" appears and starts showing traffic
- [ ] `mise run services-check` passes
Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/349
2026-05-11 13:47:18 -07:00
|
|
|
# Shower app prize-photo uploads (sifaka SMB mount). Mounted manually
|
|
|
|
|
# on indri via Finder — see docs/how-to/operations/shower-app.md.
|
|
|
|
|
- /Volumes/shower
|
2026-01-16 12:30:20 -08:00
|
|
|
|
2026-02-10 12:47:02 -08:00
|
|
|
# Backup repositories
|
2026-01-16 12:30:20 -08:00
|
|
|
borgmatic_repositories:
|
|
|
|
|
- path: /Volumes/backups/borg/
|
|
|
|
|
label: sifaka-borg-backups
|
|
|
|
|
encryption: repokey
|
|
|
|
|
append_only: true
|
2026-02-10 13:19:15 -08:00
|
|
|
- path: ssh://u3ugi1x1@u3ugi1x1.repo.borgbase.com/./repo
|
2026-02-10 12:47:02 -08:00
|
|
|
label: borgbase-offsite
|
|
|
|
|
encryption: repokey
|
|
|
|
|
append_only: true
|
|
|
|
|
|
|
|
|
|
# BorgBase SSH key (fetched from 1Password in playbook pre_tasks)
|
|
|
|
|
borgmatic_borgbase_ssh_key_path: /Users/erichblume/.ssh/borgbase_ed25519
|
2026-01-16 12:30:20 -08:00
|
|
|
|
2026-03-16 21:59:10 -07:00
|
|
|
# Directory for pre-backup database dumps from k8s pods
|
|
|
|
|
borgmatic_k8s_dump_dir: /Users/erichblume/.local/share/borgmatic/k8s-dumps
|
|
|
|
|
|
|
|
|
|
# K8s SQLite databases to dump before backup via kubectl exec
|
|
|
|
|
# Each entry runs: kubectl exec <pod-selector> -- sqlite3 <path> ".backup /tmp/backup.db"
|
|
|
|
|
# then copies the dump to borgmatic_k8s_dump_dir/<name>.db
|
|
|
|
|
borgmatic_k8s_sqlite_dumps:
|
|
|
|
|
- name: mealie
|
|
|
|
|
namespace: mealie
|
|
|
|
|
label_selector: app=mealie
|
|
|
|
|
db_path: /app/data/mealie.db
|
2026-03-18 06:07:44 -07:00
|
|
|
context: minikube
|
C1: deploy adelaide-baby-shower-app to ringtail k3s (#349)
## Summary
Brings up the Adelaide / Heidi / Addie baby shower app on ringtail k3s with the public/private split that the app's hosting contract calls for: `shower.eblu.me` (public, via Fly proxy) and `shower.ops.eblu.me` (tailnet). App is consumed as a wheel from the Forgejo PyPI index — source lives at [`adelaide-baby-shower-app`](https://forge.eblu.me/eblume/adelaide-baby-shower-app).
### What's included
- **ArgoCD app + manifests** under `argocd/manifests/shower/` (deployment, service, ProxyGroup ingress, ConfigMap for `DJANGO_DEBUG`/`DJANGO_ADMIN_URL`, ExternalSecret for `DJANGO_SECRET_KEY` from 1Password item `Shower (blumeops)`, NFS PV on sifaka, RWX media PVC, RWO local-path data PVC for SQLite). Recreate rollout because SQLite is single-writer.
- **Public surface** (`fly/`): new `shower.eblu.me` server block proxying to `shower.ops.eblu.me`. `/admin/` returns 403 at the edge except `/admin/login/` and `/admin/logout/`, which are rate-limited via a new `shower_auth` zone. `X-Clacks-Overhead` on. GNU Terry Pratchett.
- **fail2ban** filter (`shower-admin-login.conf`) matching 401/403/429 on `/admin/login/` and jail (`shower.conf`) with `maxretry=5/findtime=600/bantime=3600`. The `nginx-deny` action was generalized to take a per-jail `nginx_deny_file` so the shower has its own deny list (forge keeps using the legacy default).
- **Caddy** route on indri (`shower.ops.eblu.me` → `https://shower.tail8d86e.ts.net`).
- **Pulumi** Gandi CNAME `shower.eblu.me → blumeops-proxy.fly.dev.`.
- **Grafana** APM dashboard `configmap-shower-apm.yaml` (request rate, error rate, failed admin login count, latency percentiles, bandwidth, access logs) mirroring `docs-apm.json` with a `host="shower.eblu.me"` filter.
- **Container** `containers/shower/default.nix` — `dockerTools.buildLayeredImage` with a nixpkgs Python and a startup wrapper that creates `/app/data/.venv`, pip-installs `adelaide-baby-shower-app==1.0.0` from the forge PyPI index on first boot, runs migrations + collectstatic, and execs gunicorn. A `local_settings.py` shim pins `DATABASES.NAME`/`MEDIA_ROOT`/`STATIC_ROOT` to absolute paths so they don't end up in site-packages.
- **Docs** runbook at `docs/how-to/operations/shower-app.md` linked from the apps registry, plus changelog fragments.
### Defense layers on the public surface
1. fly nginx geo+fail2ban `$shower_banned` (per-service deny list)
2. fly nginx `limit_req zone=shower_auth` (3 r/s per Fly-Client-IP)
3. django-axes (5 fails / 1h, keyed on username+ip_address)
4. edge `/admin/` block (returns 403 for anything that isn't login/logout)
## Prerequisites for the user to do (NOT in this PR)
Halted on these per request — they touch shared/manual systems:
- [x] **NFS share** on sifaka: `/volume1/shower`, NFS rule for ringtail RW, `chown 1000:1000`
- [ ] **1Password item** `Shower (blumeops)` in the blumeops vault with a freshly minted `secret-key` field (`openssl rand -base64 48`) — do NOT reuse anything that has lived in git
- [ ] **Container build**: `mise run container-build-and-release shower`, then update `images[].newTag` in `argocd/manifests/shower/kustomization.yaml` to the resulting `v1.0.0-<sha>-nix`
- [x] **DNS**: `mise run dns-up` after merge
- [x] **Fly cert**: `fly certs add shower.eblu.me -a blumeops-proxy`
- [ ] **Caddy push**: `mise run provision-indri -- --tags caddy`
- [ ] **Fly redeploy** to pick up the new nginx block + fail2ban jail: `mise run fly-deploy`
- [ ] **ArgoCD sync**: `argocd app set shower --revision shower-app-deploy && argocd app sync shower` to test from this branch before merging
## Test plan
- [ ] Container builds successfully on nix-container-builder runner
- [ ] Pod starts, migrations run, gunicorn answers on :8000
- [ ] `kubectl --context=k3s-ringtail -n shower logs deploy/shower` clean
- [ ] `curl -sf https://shower.ops.eblu.me/` returns the splash page (tailnet)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/` returns 200 (pre-DNS verification)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/admin/users/` returns 403 (edge block)
- [ ] `curl -I -H "Host: shower.eblu.me" https://blumeops-proxy.fly.dev/admin/login/` returns a Django login response
- [ ] After DNS is up: `curl -I https://shower.eblu.me/` returns 200 with `X-Clacks-Overhead`
- [ ] Grafana dashboard "Shower APM" appears and starts showing traffic
- [ ] `mise run services-check` passes
Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/349
2026-05-11 13:47:18 -07:00
|
|
|
- name: shower
|
|
|
|
|
namespace: shower
|
|
|
|
|
label_selector: app=shower
|
|
|
|
|
db_path: /app/data/db.sqlite3
|
|
|
|
|
context: k3s-ringtail
|
2026-03-16 21:59:10 -07:00
|
|
|
|
2026-01-16 12:30:20 -08:00
|
|
|
# Exclude patterns
|
2026-01-20 14:55:37 -08:00
|
|
|
borgmatic_exclude_patterns: []
|
2026-01-16 12:30:20 -08:00
|
|
|
|
|
|
|
|
# Encryption passcommand (reads borg passphrase)
|
|
|
|
|
borgmatic_encryption_passcommand: cat /Users/erichblume/.borg/config.yaml
|
|
|
|
|
|
|
|
|
|
# Retention policy
|
|
|
|
|
borgmatic_keep_daily: 7
|
|
|
|
|
borgmatic_keep_monthly: 12
|
|
|
|
|
borgmatic_keep_yearly: 1000
|
|
|
|
|
|
|
|
|
|
# PostgreSQL databases to backup (streamed via pg_dump)
|
2026-01-20 09:04:47 -08:00
|
|
|
# Password is read from ~/.pgpass (managed by this role)
|
2026-01-17 09:22:01 -08:00
|
|
|
# pg_dump_command must be full path since LaunchAgent doesn't have homebrew in PATH
|
2026-03-27 19:43:05 -07:00
|
|
|
# --- Immich photo library backup (BorgBase offsite only) ---
|
|
|
|
|
borgmatic_photos_config: /Users/erichblume/.config/borgmatic/photos.yaml
|
2026-03-30 10:30:02 -07:00
|
|
|
borgmatic_photos_source_directories:
|
|
|
|
|
- /Volumes/photos/library
|
|
|
|
|
- /Volumes/photos/upload
|
2026-03-27 19:43:05 -07:00
|
|
|
borgmatic_photos_borgbase_repo: ssh://xcrtl5tg@xcrtl5tg.repo.borgbase.com/./repo
|
|
|
|
|
# Schedule: runs daily at 4:00 AM (offset from main backup at 2:00 AM)
|
|
|
|
|
borgmatic_photos_schedule_hour: 4
|
|
|
|
|
borgmatic_photos_schedule_minute: 0
|
|
|
|
|
# Retention: photos are precious, keep more history
|
|
|
|
|
borgmatic_photos_keep_daily: 7
|
|
|
|
|
borgmatic_photos_keep_monthly: 12
|
|
|
|
|
borgmatic_photos_keep_yearly: 1000
|
|
|
|
|
|
2026-01-17 09:22:01 -08:00
|
|
|
borgmatic_pg_dump_command: /opt/homebrew/opt/postgresql@18/bin/pg_dump
|
2026-01-16 12:30:20 -08:00
|
|
|
borgmatic_postgresql_databases:
|
2026-01-25 12:56:31 -08:00
|
|
|
# k8s PostgreSQL (CloudNativePG) via Caddy L4 proxy
|
2026-01-16 12:30:20 -08:00
|
|
|
- name: miniflux
|
2026-01-25 12:56:31 -08:00
|
|
|
hostname: pg.ops.eblu.me
|
2026-01-19 18:00:32 -08:00
|
|
|
port: 5432
|
|
|
|
|
username: borgmatic
|
2026-01-22 21:25:44 -08:00
|
|
|
- name: teslamate
|
2026-01-25 12:56:31 -08:00
|
|
|
hostname: pg.ops.eblu.me
|
2026-01-22 21:25:44 -08:00
|
|
|
port: 5432
|
|
|
|
|
username: borgmatic
|
2026-03-27 16:59:58 -07:00
|
|
|
- name: authentik
|
|
|
|
|
hostname: pg.ops.eblu.me
|
|
|
|
|
port: 5432
|
|
|
|
|
username: borgmatic
|
2026-04-13 17:58:06 -07:00
|
|
|
- name: paperless
|
|
|
|
|
hostname: pg.ops.eblu.me
|
|
|
|
|
port: 5432
|
|
|
|
|
username: borgmatic
|
2026-03-27 16:59:58 -07:00
|
|
|
# immich-pg cluster (VectorChord) via Caddy L4 on port 5433
|
|
|
|
|
- name: immich
|
|
|
|
|
hostname: pg.ops.eblu.me
|
|
|
|
|
port: 5433
|
|
|
|
|
username: borgmatic
|