Merge pull request 'heph Authentik: grant offline_access scope (fixes spoke sync refresh-token 400)' (#371) from heph-offline-access into main

The heph CLI requests scope "openid offline_access", but the Authentik
heph OAuth2 provider only mapped openid/email/profile. Without the
offline_access mapping the issued refresh token is bound to the login
session rather than the 30-day refresh-token window; once the session
lapses, hephd's refresh_token grant returns 400 Bad Request and spoke
sync silently degrades (heph sync --status -> auth_failure: true).

Add the built-in offline_access scope mapping to the provider's
property_mappings and document the requirement in the service reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.gitignore

@@ -1,5 +1,6 @@
 .claude/settings.local.json
 .claude/agent-memory/
+.claude/scheduled_tasks.lock
 
 # Python
 __pycache__/

AGENTS.md

@@ -65,7 +65,7 @@ See [[agent-change-process]] for the full methodology.
 ./pulumi/               # Pulumi IaC (tailnet ACLs, dns, cloud)
 ~/.config/{nvim,fish}   # user's shell config, managed by chezmoi
 ~/code/personal/        # user's projects
-~/code/personal/zk      # user's Obsidian-sync managed zettelkasten. Potential source for reference data.
+~/code/personal/zk      # user's zettelkasten (Obsidian-sync). Reference-data source; migrating into heph docs (hephaestus).
 ~/code/3rd/             # mirrored external projects
 ~/code/work             # FORBIDDEN
 ```
@@ -147,10 +147,16 @@ Create a new spork: `mise run spork-create <mirror-name>`
 
 ## Task Discovery
 
+BlumeOps tasks live in [hephaestus](https://github.com/eblume/hephaestus) (`heph`),
+the user's self-hosted context/task system. Fetch them with the CLI:
+
 ```fish
-mise run blumeops-tasks  # fetch from Todoist, sorted by priority
+heph list --project Blumeops --json  # outstanding Blumeops tasks as JSON
 ```
-Most tasks are stored in `./mise-tasks/`. For scripts with any logic or
+
+(This replaced the retired `blumeops-tasks` mise task, which read from Todoist.)
+
+Most operational scripts are stored in `./mise-tasks/`. For scripts with any logic or
 complexity, use uv run --script 's with explicit dependencies. Complex
 workflows with artifacts should become dagger pipelines. Mise tasks are for
 development processes and operations - tools for the user or the agent.

CHANGELOG.md

@@ -12,6 +12,259 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 <!-- towncrier release notes start -->
 
+## [v1.17.0] - 2026-06-03
+
+### Features
+
+- Deploy the Adelaide / Heidi / Addie baby shower app — guest splash, raffle
+  picker, and prize assignment console — on ringtail k3s with `shower.eblu.me`
+  as the public entry and `shower.ops.eblu.me` as the tailnet admin host. App
+  source: [`adelaide-baby-shower-app`](https://forge.eblu.me/eblume/adelaide-baby-shower-app).
+- Deploy adelaide-baby-shower-app v1.1.0 to ringtail k3s. Replaces the
+  boolean lock with a four-phase `ShowerState` (`pre_event` → `party` →
+  `prizes_locked` → `event_locked`), adds an append-only "guest memories"
+  panel where guests can leave photos and comments for the baby, and
+  polishes the admin and QR views. Three Django migrations
+  (`0009_shower_phase`, `0010_guest_memories`, `0011_book_description`)
+  run automatically in the entrypoint against the SQLite PV. No config
+  or env-var changes.
+
+  Container build also gains a Forgejo-PyPI workaround: Forgejo's simple
+  index returns absolute file URLs hardcoded to the public ROOT_URL
+  (`forge.eblu.me`), which the Fly edge 403s on `/api/packages/*`. The
+  wheel and sdist are now both pulled via direct `fetchurl` against
+  `forge.ops.eblu.me` (tailnet-only) and the wheel is handed to pip as
+  a local path.
+- `review-compliance-reports` now also fetches and summarizes the weekly Prowler container-image and IaC scans (previously only the K8s CIS in-cluster scan was processed). For each scan it shows status counts, severity breakdown, week-over-week delta, and — for the high-volume image/IaC scans — top-N tables grouped by check ID and resource instead of per-finding listings.
+- runner-logs now authenticates with Forgejo API token and auto-detects the repo from git remote. Job logs are fetched via SSH to indri (reading Forgejo's on-disk zstd log files) instead of the web endpoint, which doesn't support token auth for private repos.
+
+### Bug Fixes
+
+- Fix nightly borgmatic backups failing for 2 days. The shower SQLite
+  dump hook referenced `kubectl --context=k3s-ringtail`, but indri's
+  kubeconfig deliberately doesn't carry the ringtail credentials. The
+  `before_backup` hook's failure aborted the entire run, taking out
+  *both* the local sifaka repo and the BorgBase offsite. Replaced
+  the inline-shell dump with a `~/bin/borgmatic-k8s-sqlite-dump`
+  helper deployed by the ansible role. Each dump entry now declares a
+  `target` of either `local:<context>` (mealie — kubectl uses indri's
+  kubeconfig) or `ssh:<user@host>` (shower — ssh into ringtail and
+  run `k3s kubectl` there, no indri-side kubeconfig needed; k3s.yaml
+  on ringtail is mode 644 so no sudo required). Bytes stream back via
+  `kubectl exec ... -- cat` rather than `kubectl cp`, since `kubectl
+  cp` requires `tar` inside the pod and nix-built images like shower
+  don't bundle it.
+- Shower app container now bakes the wheel + Python deps into the image
+  at build time via `buildPythonPackage` instead of pip-installing on
+  first boot. Boots are deterministic and don't depend on forge PyPI
+  being reachable from the pod. The `wheelHash` in
+  `containers/shower/default.nix` is the sha256 sourced from the
+  [forge PyPI simple index](https://forge.eblu.me/api/packages/eblume/pypi/simple/adelaide-baby-shower-app/);
+  bumping the version means bumping that hash too.
+
+  Borgmatic now covers the shower app: SQLite is dumped from the live
+  pod via `kubectl exec` (mirroring the existing mealie entry, with
+  `context: k3s-ringtail`), and the prize-photo media share is picked up
+  through `/Volumes/shower` (sifaka SMB mount on indri, same pattern as
+  `/Volumes/photos`).
+- Disabled adaptive sync (VRR) on ringtail's DP-1 output. The OMEN 27i IPS panel pumps brightness when its refresh rate swings into the low VRR range during low-framerate content (e.g. game cutscenes), producing a flicker that worsened over a session until a reboot. Pinning the panel to a fixed 165Hz eliminates it.
+- Fixed forge.eblu.me static assets (CSS, JS, images, fonts) not loading — the proxy's static asset cache block was missing the `Host` header, so Caddy couldn't route the requests.
+- Fixed homepage container EACCES on cold start: the nix-built image now chowns
+  `/app/config` to uid 1000 at build time via `fakeRootCommands`, matching the
+  behavior of the old Dockerfile. Without this, homepage couldn't seed missing
+  skeleton configs (proxmox.yaml etc.) or create `/app/config/logs`, crashing on
+  its first uncached request. Caught during the ringtail cutover.
+- Fixed sway keybindings on ringtail — the home-manager `keybindings` block was replacing the module's defaults entirely, leaving only explicit overrides (no workspace switching, focus, move, splits, resize mode, etc). Switched to `lib.mkOptionDefault` with `lib.mkForce` on the conflicting custom binds (`Mod+Return`, `Mod+d`, `Mod+space`, `Mod+l`) so defaults merge back in. Also added `Mod+F1` to show a filterable fuzzel list of current keybindings.
+
+  Fixed fuzzel config errors on launch — `border-radius` and `border-width` were under `[main]`, but fuzzel expects them as `radius`/`width` under a `[border]` section.
+- Pin the Quartz docs build to v4.5.2. The Dagger `build_docs` pipeline cloned Quartz from the default branch unpinned; Quartz v5.0.0 restructured its config layout (`.quartz/plugins`, `../quartz` imports) and broke the docs build against our existing `quartz.config.ts`/`quartz.layout.ts`.
+
+### Infrastructure
+
+- Wire the ringtail `blumeops-pg` cluster (which holds the wave-1-migrated
+  paperless + teslamate databases) into backups and Grafana. Adds a Tailscale
+  LoadBalancer Service (`blumeops-pg-ringtail.tail8d86e.ts.net`) and a Caddy L4
+  route (`pg.ops.eblu.me:5434`), then repoints borgmatic's `teslamate` +
+  `paperless` postgres dumps and the `mealie` SQLite dump at ringtail, and the
+  Grafana TeslaMate datasource at the ringtail DB. Closes the backup gap that
+  opened at cutover (the migrated live data was still being backed up from the
+  now-frozen minikube copies) and unblocks the wave-1 decommission.
+- Migrated homepage dashboard from minikube (indri/arm64) to k3s (ringtail/amd64).
+  The container is now built via nix (`containers/homepage/default.nix`), adapted
+  from nixpkgs `homepage-dashboard` with the upstream Next.js cache patches and
+  wrapped with `dockerTools.buildLayeredImage`. Autodiscovery shifts: services on
+  minikube (ArgoCD, Immich, Kiwix, Mealie, Miniflux, Grafana, Prometheus,
+  Navidrome, Paperless, TeslaMate, Transmission) become explicit static entries
+  in `services.yaml`; ringtail services (Authentik, Frigate/NVR, Ntfy, Ollama)
+  auto-populate via Ingress annotations.
+- Migrated CV (`cv.eblu.me`) and Docs (`docs.eblu.me`) from minikube Deployments to indri-native ansible roles. Caddy now serves the extracted release tarballs directly via a new `kind: static` service-block in the Caddy template — no daemon, no container — replacing the prior nginx-in-a-pod layer. Removes a network hop on every request and shrinks minikube's footprint. See [[cv-on-indri]] and [[docs-on-indri]]. Part of the broader minikube wind-down.
+- Migrated devpi (PyPI mirror at `pypi.ops.eblu.me`) from a minikube StatefulSet to a launchd-managed service on indri. devpi-server now runs in a uv-managed venv with pinned `devpi-server` and `devpi-web` versions, listens on `127.0.0.1:3141`, and is fronted by Caddy. The minikube StatefulSet was crash-looping under memory pressure (and breaking the Python toolchain everywhere); the new layout removes a layer of dependency on cluster health for critical-path tooling. See [[devpi-on-indri]].
+- Move the entire Immich stack — server, machine-learning, valkey,
+  and the PostgreSQL+VectorChord cluster — off `minikube-indri` and
+  onto `k3s-ringtail`. Postgres data migrated zero-loss via CNPG
+  `pg_basebackup` (replica catch-up then promote); row counts on
+  `asset`, `user`, `album`, `smart_search`, `activity`, `asset_face`
+  verified equal between source and replica before cutover. The ML
+  pod now uses ringtail's RTX 4080 via the nvidia-device-plugin
+  (time-slicing bumped 2 → 4 to share with frigate + ollama). Caddy
+  routing at `photos.ops.eblu.me` is unchanged (still
+  `photos.tail8d86e.ts.net`, the device just lives on ringtail now).
+  Borgmatic backups continue against the same `immich-pg` tailnet
+  hostname. First concrete chain in the broader indri-k8s
+  decommission effort.
+- Add local nix container build for `tailscale` (`containers/tailscale/default.nix`) so ringtail's tailscale-operator ProxyClass proxy pods pull from the forge mirror instead of `docker.io/tailscale/tailscale`. Pinned at v1.94.2 to match `service-versions.yaml`. Indri's tailscale-operator continues to use upstream during the k8s-to-ringtail migration.
+- Address the 6 critical Prowler IaC findings against `argocd/manifests/`. Prowler's IaC provider hardcodes `self._mutelist = None` and delegates filtering to Trivy, but doesn't plumb `--ignorefile` through — so the documented "use Trivy filtering" path is actually broken. Added a shim around `trivy` in the Prowler image that injects `--ignorefile $TRIVY_IGNOREFILE` for `trivy fs` invocations when the env var points at a real file. The IaC cronjob now mounts `mutelist/trivyignore.yaml` (Trivy's per-path schema) and sets the env var, muting the `external-secrets` and `kube-state-metrics` Secret-access findings (KSV-0041, KSV-0114). Separately, `grafana-clusterrole` is tightened to remove `secrets` access entirely: the dashboard sidecar already only consumes ConfigMap-labeled dashboards, so its `RESOURCE` env var is now `configmap` instead of `both`.
+- Pin ringtail's wired IP to `192.168.1.21` via NixOS scripted networking; NetworkManager no longer manages `enp5s0`. Removes DHCP lease renewal as a failure mode after a silent lease teardown took ringtail offline. Also explicitly enables `net.ipv4.ip_forward` (previously set implicitly by scripted-DHCP) so k3s pod networking and Tailscale routing continue to work with static networking.
+- Ripped out the compensating-controls (CC) framework: deleted `compensating-controls.yaml`, the `review-compensating-controls` mise task, and the associated how-to / explanation docs. Prowler and Kingfisher continue to run weekly and produce reports; the Prowler mutelist YAML files remain in place but no longer carry `CC: <id>` prefixes — each entry just keeps a free-form `Description` of why the finding is muted. The CC review cadence proved to be more overhead than this single-operator homelab needed.
+- Wire shower app for public exposure: fly nginx `shower.eblu.me` server
+  block as a guest-only surface — splash page, `/prizes/<token>/`, static
+  assets, media. Everything authenticated (`/admin/`, `/host/`,
+  `/accounts/`) returns 403 with a "tailnet only" pointer. Staff hit
+  `shower.ops.eblu.me` for the operator console + admin; the app's
+  v1.0.1 `DJANGO_PUBLIC_URL_BASE` setting makes QR codes generated on
+  the tailnet point back at the WAN host for guests. Plus a Caddy route
+  on indri, Pulumi Gandi CNAME, and a Grafana APM dashboard tracking
+  request rate, error rate, latency, bandwidth, and access logs.
+- Mirror Valkey 8.1 locally as `registry.ops.eblu.me/blumeops/valkey`. Replaces direct pulls of `docker.io/valkey/valkey:8.1-alpine` for paperless and immich sidecars. Built via native Dagger pipeline on Alpine 3.22. Stateless swap — no data migration. Authentik's nix-built Redis remains separate.
+- Add nix-built amd64 valkey for ringtail (`containers/valkey/default.nix`) so immich-ringtail can stop pulling the upstream multi-arch `docker.io/valkey/valkey` image. Existing `container.py` continues to build Alpine arm64 for paperless on indri. Both bump to valkey 8.1.7 (Alpine 3.22 8.1.7-r0 / nixpkgs 8.1.7).
+- Upgrade Grafana Alloy v1.14.0 → v1.16.0 across all four service deployments
+  (alloy-k8s, alloy-ringtail, alloy-tracing-ringtail on k8s; alloy native on
+  indri). Pulls in stable database observability (v1.15) and the OTel Collector
+  v0.147.0 bump. Container build also migrated from Dockerfile to native Dagger
+  `container.py` per the build-container-image migration playbook.
+- Upgraded Dagger from v0.20.1 to v0.20.6 (engine, CLI pin, and SDK regen) and migrated `runner-job-image` from a Debian-based Dockerfile to a native Dagger `container.py` on Alpine 3.23, reusing the shared `alpine_runtime` helper.
+- Decommission the wave-1 services on minikube-indri now that paperless,
+  teslamate, and mealie run on ringtail with their data backed up. Removes the
+  minikube `paperless`/`teslamate`/`mealie` manifest dirs + ArgoCD app
+  definitions (pruning the parked Deployments, Services, and the redundant
+  minikube mealie/paperless PVCs), and drops the `paperless`/`teslamate` roles
+  from the minikube `blumeops-pg` cluster. The `paperless` and `teslamate`
+  databases are dropped from indri's blumeops-pg as the finalization step.
+  miniflux + authentik remain on the minikube cluster (later waves).
+- Upgraded the k8s Forgejo runner to the v12.8 line, switched it from first-boot registration to declarative `server.connections` credentials from 1Password, and consolidated the supporting runner how-to documentation.
+- Move paperless, teslamate, and mealie off `minikube-indri` onto
+  `k3s-ringtail`, shedding ~1.1 GiB of resident load from the
+  OOM-thrashing 8 GiB minikube node (the kernel OOM killer had been
+  killing `kube-apiserver`/`dockerd`/argocd, flapping every
+  minikube-hosted service at once). paperless + teslamate databases
+  move into a fresh CNPG `blumeops-pg` cluster on ringtail via a cold
+  `pg_dump`/`pg_restore` from the quiesced source — row counts verified
+  equal before any routing flip; source DBs dropped only after the
+  ringtail side serves traffic. mealie's SQLite PVC is copied as-is.
+  paperless media stays on sifaka NFS. Downtime-tolerant cold cutover
+  (no streaming replication); rollback is repoint-and-scale-up with the
+  source untouched. Second chain in the indri-k8s decommission after
+  [[migrate-immich-to-ringtail]].
+- Recurring maintenance batch:
+
+  - Ringtail flake inputs refreshed (`disko`, `home-manager`, `nixpkgs`).
+  - Tooling deps bumped: prek hooks (trufflehog v3.95.3, kingfisher v1.101.0, ruff v0.15.14, `ansible-core` 2.21.0); fly proxy base images (nginx 1.30.1-alpine, alloy v1.16.1); `typer==0.26.2` in mise tasks.
+- Updated `nixos/ringtail/flake.lock` (weekly cadence): `disko`, `home-manager`, and `nixpkgs` inputs refreshed. `nixpkgs-services` skipped per overlay convention.
+- Reviewed `mealie` service version freshness; upstream is 5 minor versions ahead (v3.17.0 vs deployed v3.12.0). Marked reviewed; upgrade deferred.
+- Deploy shower v1.1.2 — bump container build to new app release.
+- Upgrade unpoller v2.34.0 → v3.2.0 and migrate container build from Dockerfile to native Dagger (container.py). v3.0.0 carries breaking UniFi API changes; v3.2.0 introduces a 60s background poll (cached scrapes) by default — set `interval = 0` in `up.conf` to restore on-demand polling.
+- Monthly tooling dependency refresh: prek hooks (trufflehog, kingfisher, ruff, shfmt, prettier, actionlint, ansible-lint), fly proxy base images (nginx 1.30.0, tailscale v1.94.2, alloy v1.16.0), normalize pyyaml lower bound in mise-tasks.
+- Add GE-Proton (`pkgs.proton-ge-bin`) to `programs.steam.extraCompatPackages`
+  on ringtail. Subnautica 2 hangs at Mercuna plugin init under Proton
+  Experimental + DXVK D3D12; GE-Proton is available as a Steam per-game
+  compatibility option to work around it.
+- Add `sn2-prelaunch` Steam launch wrapper on ringtail that removes
+  Subnautica 2's stale `Saved/running.dat` and `Saved/beforelobby.dat`
+  lockfiles before each launch. SN2 pops up an invisible (0×0-sized)
+  Error dialog when it detects an unclean exit, blocking GameThread
+  forever; this is observable only as a black screen with a spinning
+  loader. Use via Steam launch option: `sn2-prelaunch %command%`.
+- Add local nix container build for `frigate-notify` (`containers/frigate-notify/default.nix`) so the Frigate→ntfy bridge is rebuilt on ringtail from the forge mirror instead of pulled from `ghcr.io/0x2142/frigate-notify`.
+- Add resource limits to all ArgoCD pods to prevent unbounded resource consumption during node-wide pressure events.
+- Black-hole the `/mirrors/*` repositories at the Fly proxy edge (`return 403` → `forge.ops.eblu.me`). A surprise $29.60 Fly bill traced to ~1.24 TB/30d of egress on `forge.eblu.me`, 99.95% of all proxy egress — of which ~71% was AI scrapers (Meta `meta-externalagent`, OpenAI `GPTBot`, Amazonbot) crawling the near-infinite git-history URL space of the public mirror repos and timing out Forgejo in the process. Mirrors exist for supply-chain control and are consumed over the tailnet, so their public web UI had no legitimate audience. `robots.txt` already disallowed `/mirrors/`, but the offending agents ignore it. Tier-2 mitigations (user-agent denylist, Anubis proof-of-work gateway) are documented in `docs/explanation/ai-scraper-mitigation.md`.
+- Bump paperless and immich kustomizations to the main-SHA-built valkey tag (`v8.1.6-r0-fabca04`). Routine post-merge follow-up to keep production manifests pointing at images built from a commit on main.
+- Bump shower container to v1.1.1 (probe FOD hash).
+- Bumped shower app to v1.1.3 (wheel/sdist + FOD hashes probed on ringtail).
+- Cap systemd-coredump on ringtail (ProcessSizeMax/ExternalSizeMax 1G, MaxUse 2G) so multi-GB Wine/Proton game crash dumps no longer thrash the disk and lock up the desktop.
+- Deploy shower v1.1.1 to ringtail (kustomize newTag bump).
+- Deployed shower v1.1.3 to ringtail (image built and pushed from ringtail; runner bypassed due to indri overload).
+- Fix three follow-ups from the wave-1 decommission: grant the local
+  break-glass `admin` account ArgoCD admin rights (`g, admin, role:admin` —
+  previously only the Authentik `admins` group had access, so admin was
+  locked out whenever its token expired), and repoint the alloy blackbox
+  probe for teslamate from the deleted minikube service to
+  `https://tesla.ops.eblu.me/` (through Caddy over Tailscale). The orphaned
+  paperless/teslamate roles + ExternalSecrets left on the minikube
+  blumeops-pg are also cleaned up.
+- Moved the Immich blackbox health probe from indri's alloy to ringtail's alloy. After the immich migration to ringtail, the probe still targeted `immich-server.immich.svc.cluster.local` on indri's cluster where the service no longer exists, causing a persistent `ServiceProbeFailure` alert.
+- Pin shower v1.1.1 FOD outputHash (probed locally on ringtail).
+- Rebuild Prowler container against main HEAD (v5.23.0-495e45d) after merging the IaC mutelist Dockerfile changes.
+- Rebuild and retag alloy v1.16.0 container images from the main-branch SHA
+  following the squash-merge of #345, per the build-container-image
+  squash-merge convention. Both images (`registry.ops.eblu.me/blumeops/alloy`)
+  now reference `9564435` rather than the branch SHA `26a3ab5`, restoring
+  source traceability after branch cleanup.
+- Rebuild shower from the post-merge commit on main so the container's
+  SHA tag points at a commit that will still exist after the 30-day
+  branch-cleanup window. Functionally identical to the branch-tag image
+  already deployed, just preserves source traceability per
+  [[build-container-image#Squash-merge and container tags]].
+- Rebuild unpoller container from squashed main commit so the image SHA tag matches a commit in main's history (was tagged with the pre-squash branch SHA).
+- Rebuild valkey container from squashed main commit (both arm64 dagger and amd64 nix variants), and update paperless + immich-ringtail kustomizations to the main-SHA tags `v8.1.7-ecded30` and `v8.1.7-ecded30-nix`.
+- Retired the `blumeops-tasks` mise task (Todoist API) in favor of `heph list --project Blumeops --json` from the self-hosted [hephaestus](https://github.com/eblume/hephaestus) system. Updated docs to point task discovery and rotation reminders at heph, and noted that the `~/code/personal/zk` zettelkasten is migrating into heph docs.
+- Switch the Fly proxy deploy strategy from `bluegreen` to `immediate` in `fly/fly.toml`. With a single proxy machine, bluegreen offers little benefit — the green machine routinely failed to reach "started" inside Fly's default 5-minute deploy timeout (the cold-start sequence of `tailscaled` → `tailscale up` → wait-for-MagicDNS → nginx startup eats most of the budget), and the failed deploys would roll back. `immediate` replaces the machine in place with a brief downtime (~5–10s) but actually completes.
+- Switch the ringtail provisioning playbook's blumeops clone URL from `forge.eblu.me` (public, via Fly proxy) to `forge.ops.eblu.me` (tailnet, direct via Caddy on indri). Ringtail is always on the tailnet, so the WAN round-trip is pure overhead — it also made `provision-ringtail` brittle whenever the Fly proxy was slow or down.
+- Switched Grafana's deployment strategy from `RollingUpdate` to `Recreate`. With an RWO PVC holding the SQLite database and Bleve search index, `RollingUpdate` reliably crashloops the new pod on the index lock until rollout timeout. `Recreate` terminates the old pod first so the new one acquires the lock cleanly.
+- Update `tailscale-operator-ringtail` ProxyClass to reference the `0108b68` main-SHA build of the tailscale container. Routine post-merge cleanup so the deployed image traces to a commit that survives PR branch cleanup.
+- Update the ringtail NixOS flake lockfile (`nixos/ringtail/flake.lock`): bump
+  `nixpkgs` (b77b3de → 25f5383) and `disko` (5ba0c95 → 115e521) to latest.
+  `nixpkgs-services` was intentionally left pinned (skipped by the
+  `flake-update` pipeline). Routine recurring maintenance per [[manage-lockfile]].
+- Upgrade native macOS Alloy on indri to v1.16.0. Built on gilbert with Go
+  1.26.2 + CGO (required for the macOS native DNS resolver, which Tailscale
+  MagicDNS depends on), scp'd to `~/.local/bin/alloy` on indri, codesigned,
+  and the LaunchAgent reloaded. Completes the v1.16.0 fleet upgrade started
+  in #345 — all four Alloy services (alloy-k8s, alloy-ringtail,
+  alloy-tracing-ringtail, alloy ansible) now run v1.16.0.
+- Upgraded zot on indri from v2.1.15 to v2.1.16 (security fixes: TLS verification on metrics client, CORS Allow-Credentials suppression on wildcard origins, manifest/API-key body size limits).
+
+### Documentation
+
+- Reviewed `replicating-blumeops` tutorial: fixed "BluemeOps" typos (also in `contributing.md`) and added `last-reviewed` frontmatter.
+- Reviewed [[indri]] reference card: added `devpi`, `cv`, and `docs` to the native-services list; widened the k8s note to reflect the growing set of apps now on ringtail and the planned indri-minikube decommission; added CPU/RAM specs.
+- New how-to: rotate-fly-deploy-token. Documents the 75-day rotation cadence, why we use `org`-scoped tokens (silences the cosmetic metrics-token warning on `fly status` with marginal blast-radius cost given the single-app personal org), and the procedure for rotation + Forgejo Actions secret sync.
+- Add `docs/explanation/ai-scraper-mitigation.md` — the egress-cost / AI-crawler threat model for the public Fly proxy, the tiered mitigation plan (Tier 1: mirror black-hole, shipped; Tier 2: user-agent denylist + Anubis; Tier 3: Cloudflare, rejected on principle), and the data behind it.
+- Fix manage-forgejo-mirrors verify step — sync button is on the repo settings page ("Synchronize now"), not the main repo page.
+- Fixed the `op item edit` invocation in the [[zot]] API-key rotation procedure: the previous `pbpaste | op item edit ... "field[password]=-"` stdin syntax is rejected by op 2.34 as "invalid JSON" (recent op versions treat piped input as a full JSON template, not a single field value). Procedure now reads the clipboard into a local fish variable and passes it as an inline assignment.
+- Fixed the export-filename step in [[run-1password-backup]]: 1Password's desktop app names the export `1PasswordExport-<account-uuid>-<timestamp>.1pux` automatically rather than letting you save to a fixed name, so the procedure now points the task at that glob instead of pretending the default name is `1Password-export.1pux`.
+- Refresh the contributing tutorial: add `last-reviewed`, include the `.ai.md` changelog fragment type, and clarify that `prek` is pinned via `mise`.
+- Review and refresh the Navidrome reference card: add `last-reviewed`, correct the scanner env var name, document the current image/version, and record routing and runtime details from the manifests.
+- Review and refresh the Ollama reference card: add `last-reviewed`, bump the documented image tag to 0.20.4, and add the two `qwen3.5` models now declared in `models.txt`.
+- Reviewed [[1password]] reference card: added the `blumeops` vs `Personal` vault split, noted that `onepassword-connect` runs on both indri and ringtail (not just one cluster), and pulled the `op read` vs `op item get --fields` guidance up from agent memory into the card.
+- Reviewed `index.md`; added ringtail to the infrastructure overview and stamped `last-reviewed`.
+- Reviewed transmission card: corrected storage layout (`/config/` is emptyDir, watch dir disabled) and noted the Prometheus exporter sidecar.
+- rotate-fly-deploy-token: combine mint+store into one command with both fish and bash forms; document the `op item edit` "Password item requires ps value" validator gotcha and the placeholder-password workaround.
+
+### AI Assistance
+
+- Adopt `AGENTS.md` as the canonical agent instruction file, keep `CLAUDE.md` as a compatibility shim, and update docs to reference the neutral file and the correct agent-change-process path.
+- CLAUDE.md now imports AGENTS.md via `@AGENTS.md` instead of telling agents to go read it. Claude Code only auto-loads CLAUDE.md, so the prose shim was easy to skip; the import inlines AGENTS.md into the session prompt unconditionally.
+
+### Miscellaneous
+
+- Removed the dead minikube manifests, container builds, and tooling shims left behind after the cv + docs migration to indri-native (#342). Deletes `argocd/{apps,manifests}/{cv,docs}/`, `containers/{cv,quartz}/`, and the `quartz`→`docs` mapping in `mise-tasks/container-version-check`. Bumps `docs.current-version` to `v1.16.0` (the blumeops release tag) now that the legacy nginx-base version pin is gone.
+- Rebuild shower v1.1.0 container from main HEAD (`3c7967e`) and bump the
+  kustomization tag to `v1.1.0-3c7967e-nix`. The PR was squash-merged, so
+  the branch commit `444ff91` baked into the prior tag isn't reachable
+  from main's history. The new tag points at a commit that exists on
+  main; image content is byte-identical because the FOD output is content
+  addressed and the inputs didn't change.
+- Rebuild shower v1.1.2 from main HEAD (a33fa47) and retag — PR #358 was squash-merged so the branch SHA baked into the prior image tag isn't reachable from main. FOD is content-addressed, so image bytes are identical; only provenance changes.
+- Remove the duplicate Homepage tiles for Mealie, Paperless, Immich, and
+  TeslaMate. Homepage runs on ringtail and autodiscovers ringtail Ingresses via
+  `gethomepage.dev/*` annotations; once these services migrated to ringtail they
+  were discovered automatically, making their leftover static `services.yaml`
+  entries (needed only while they lived on minikube) redundant.
+- Removed the now-unused `containers/devpi/` Dagger build artifact. Devpi runs natively on indri via uv venv; the container image is no longer referenced anywhere. Doc examples in `docs/reference/tools/dagger.md` updated to use `miniflux` as the example container name.
+- `container-build-and-release` now prints the specific `mise run runner-logs <N>` command after dispatching, polling the Forgejo API to resolve the run number for the commit it just triggered.
+- `mise run runner-logs <run> -j <n>` now reports a clear error when the log file doesn't exist on indri (e.g. a runner crash that left `action_task.log_in_storage = 0`). Previously it printed only the header and exited 0, because `zstdcat` exits 0 with a "can't stat … -- ignored" stderr message and ssh+fish on indri swallows the remote exit code.
+
+
 ## [v1.16.0] - 2026-04-18
 
 ### Infrastructure

ansible/playbooks/indri.yml

@@ -260,5 +260,7 @@
       tags: cv
     - role: docs
       tags: docs
+    - role: heph
+      tags: heph
     - role: caddy
       tags: caddy

ansible/playbooks/ringtail.yml

@@ -57,7 +57,7 @@
   tasks:
     - name: Ensure blumeops repo is present
       ansible.builtin.git:
-        repo: "https://forge.eblu.me/eblume/blumeops.git"
+        repo: "https://forge.ops.eblu.me/eblume/blumeops.git"
         dest: /etc/blumeops
         version: "{{ ringtail_commit | default('main') }}"
         force: true

ansible/roles/borgmatic/defaults/main.yml

@@ -27,6 +27,9 @@ borgmatic_source_directories:
   - /Users/erichblume/.config/borgmatic
   - /Users/erichblume/Documents
   - /Users/erichblume/.local/share/borgmatic/k8s-dumps
+  # Shower app prize-photo uploads (sifaka SMB mount). Mounted manually
+  # on indri via Finder — see docs/how-to/operations/shower-app.md.
+  - /Volumes/shower
 
 # Backup repositories
 borgmatic_repositories:
@@ -53,7 +56,17 @@ borgmatic_k8s_sqlite_dumps:
     namespace: mealie
     label_selector: app=mealie
     db_path: /app/data/mealie.db
-    context: minikube
+    # migrated to ringtail (wave-1); ssh to ringtail and run k3s kubectl
+    # there, same as shower below.
+    target: ssh:eblume@ringtail
+  - name: shower
+    namespace: shower
+    label_selector: app=shower
+    db_path: /app/data/db.sqlite3
+    # ssh to ringtail and run k3s kubectl there — avoids needing a
+    # ringtail kubeconfig on indri. k3s.yaml on ringtail is
+    # world-readable (mode 644), so no sudo required.
+    target: ssh:eblume@ringtail
 
 # Exclude patterns
 borgmatic_exclude_patterns: []
@@ -90,17 +103,18 @@ borgmatic_postgresql_databases:
     hostname: pg.ops.eblu.me
     port: 5432
     username: borgmatic
-  - name: teslamate
-    hostname: pg.ops.eblu.me
-    port: 5432
-    username: borgmatic
   - name: authentik
     hostname: pg.ops.eblu.me
     port: 5432
     username: borgmatic
+  # migrated to ringtail blumeops-pg (wave-1); port 5434 = Caddy L4 route
+  - name: teslamate
+    hostname: pg.ops.eblu.me
+    port: 5434
+    username: borgmatic
   - name: paperless
     hostname: pg.ops.eblu.me
-    port: 5432
+    port: 5434
     username: borgmatic
   # immich-pg cluster (VectorChord) via Caddy L4 on port 5433
   - name: immich

ansible/roles/borgmatic/tasks/main.yml

@@ -19,8 +19,10 @@
   ansible.builtin.copy:
     content: |
       # Managed by ansible (borgmatic role) - k8s PostgreSQL backup credentials
+      # 5432 = minikube blumeops-pg, 5433 = immich-pg, 5434 = ringtail blumeops-pg
       pg.ops.eblu.me:5432:*:borgmatic:{{ borgmatic_db_password }}
       pg.ops.eblu.me:5433:*:borgmatic:{{ borgmatic_db_password }}
+      pg.ops.eblu.me:5434:*:borgmatic:{{ borgmatic_db_password }}
     dest: ~/.pgpass
     mode: '0600'
   no_log: true
@@ -49,6 +51,20 @@
     mode: '0700'
   when: borgmatic_k8s_sqlite_dumps | length > 0
 
+- name: Ensure ~/bin exists
+  ansible.builtin.file:
+    path: "{{ ansible_env.HOME }}/bin"
+    state: directory
+    mode: '0755'
+  when: borgmatic_k8s_sqlite_dumps | length > 0
+
+- name: Deploy k8s SQLite dump helper script
+  ansible.builtin.template:
+    src: k8s-sqlite-dump.sh.j2
+    dest: "{{ ansible_env.HOME }}/bin/borgmatic-k8s-sqlite-dump"
+    mode: '0755'
+  when: borgmatic_k8s_sqlite_dumps | length > 0
+
 - name: Deploy borgmatic configuration
   ansible.builtin.template:
     src: config.yaml.j2

ansible/roles/borgmatic/templates/config.yaml.j2

@@ -32,12 +32,20 @@ exclude_patterns:
 encryption_passcommand: {{ borgmatic_encryption_passcommand }}
 
 {% if borgmatic_k8s_sqlite_dumps %}
-# Pre-backup: dump SQLite databases from k8s pods
-# Uses sqlite3 .backup for a safe, consistent copy (no corruption from concurrent writes)
+# Pre-backup: dump SQLite databases from k8s pods.
+# Uses sqlite3.backup() for a safe, consistent copy.
+#
+# Quoting/escaping is delegated to ~/bin/borgmatic-k8s-sqlite-dump
+# (deployed by the borgmatic ansible role). Each entry's `target`
+# is either:
+#   - local:<context>  -> local kubectl with --context (mealie etc.)
+#   - ssh:<user@host>  -> ssh + k3s kubectl on the cluster host,
+#                         used for ringtail since indri's kubeconfig
+#                         deliberately doesn't carry that context.
 before_backup:
     - mkdir -p {{ borgmatic_k8s_dump_dir }}
 {% for db in borgmatic_k8s_sqlite_dumps %}
-    - /opt/homebrew/bin/kubectl --context={{ db.context }} exec -n {{ db.namespace }} deploy/{{ db.name }} -- python3 -c "import sqlite3; sqlite3.connect('{{ db.db_path }}').backup(sqlite3.connect('/tmp/{{ db.name }}-backup.db'))" && /opt/homebrew/bin/kubectl --context={{ db.context }} cp {{ db.namespace }}/$(/opt/homebrew/bin/kubectl --context={{ db.context }} get pod -n {{ db.namespace }} -l {{ db.label_selector }} -o jsonpath='{.items[0].metadata.name}'):/tmp/{{ db.name }}-backup.db {{ borgmatic_k8s_dump_dir }}/{{ db.name }}.db
+    - {{ ansible_env.HOME }}/bin/borgmatic-k8s-sqlite-dump {{ db.target }} {{ db.namespace }} {{ db.label_selector }} {{ db.db_path }} {{ db.name }} {{ borgmatic_k8s_dump_dir }}/{{ db.name }}.db
 {% endfor %}
 {% endif %}
 

ansible/roles/borgmatic/templates/k8s-sqlite-dump.sh.j2

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+# {{ ansible_managed }}
+#
+# Helper script invoked by borgmatic's before_backup hook to capture a
+# k8s pod's SQLite database. Keeps the borgmatic config readable by
+# pulling all the quoting out of YAML.
+#
+# Usage:
+#   borgmatic-k8s-sqlite-dump <target> <namespace> <selector> \
+#                             <db_path> <name> <dump_target>
+#
+# <target> is one of:
+#   local:<context>   - run local kubectl with --context=<context>
+#   ssh:<user@host>   - ssh to host and run k3s kubectl there
+#                       (no indri-side kubeconfig needed)
+#
+# <namespace>      - k8s namespace of the pod
+# <selector>       - label selector to find the pod (e.g. app=shower)
+# <db_path>        - absolute path inside the pod to the SQLite DB
+# <name>           - short name used for temp filenames
+# <dump_target>    - file on this host to receive the dump
+set -euo pipefail
+
+target=${1:?missing target}
+namespace=${2:?missing namespace}
+selector=${3:?missing selector}
+db_path=${4:?missing db path}
+name=${5:?missing name}
+dump_target=${6:?missing dump target}
+
+# Stage the backup next to the source DB (a guaranteed-writable volume);
+# minimal nix images (e.g. mealie) have no /tmp.
+pod_tmp="$(dirname "$db_path")/.borgmatic-backup-${name}.db"
+
+python_backup='import sqlite3; sqlite3.connect("'"$db_path"'").backup(sqlite3.connect("'"$pod_tmp"'"))'
+
+mode=${target%%:*}
+ref=${target#*:}
+
+case "$mode" in
+    local)
+        # Pulls dump bytes out via "kubectl exec -- cat" rather than
+        # "kubectl cp", which would otherwise need tar inside the pod
+        # (nix-built images like shower don't bundle tar).
+        context=$ref
+        kubectl="/opt/homebrew/bin/kubectl --context=$context -n $namespace"
+        pod=$($kubectl get pod -l "$selector" \
+            -o jsonpath='{.items[0].metadata.name}')
+        $kubectl exec "$pod" -- python3 -c "$python_backup"
+        $kubectl exec "$pod" -- cat "$pod_tmp" > "$dump_target"
+        $kubectl exec "$pod" -- rm -f "$pod_tmp"
+        ;;
+    ssh)
+        host=$ref
+        # Force bash on the remote (user's login shell on ringtail is
+        # fish). Pipe the script via stdin to dodge nested quoting.
+        # The dump bytes come back over the ssh stdout stream — no
+        # intermediate scp, no tar requirement in the pod.
+        ssh "$host" bash <<EOF > "$dump_target"
+set -euo pipefail
+export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
+pod=\$(k3s kubectl -n "$namespace" get pod -l "$selector" -o jsonpath='{.items[0].metadata.name}')
+k3s kubectl -n "$namespace" exec "\$pod" -- python3 -c '$python_backup' 1>&2
+k3s kubectl -n "$namespace" exec "\$pod" -- cat "$pod_tmp"
+k3s kubectl -n "$namespace" exec "\$pod" -- rm -f "$pod_tmp" 1>&2
+EOF
+        ;;
+    *)
+        echo "borgmatic-k8s-sqlite-dump: unknown target mode: $mode" >&2
+        echo "  expected local:<context> or ssh:<user@host>" >&2
+        exit 1
+        ;;
+esac

ansible/roles/caddy/defaults/main.yml

@@ -52,6 +52,9 @@ caddy_services:
   - name: devpi
     host: "pypi.{{ caddy_domain }}"
     backend: "http://localhost:3141"
+  - name: heph
+    host: "heph.{{ caddy_domain }}"
+    backend: "http://localhost:8787"  # hephaestus hub (server mode) + PWA shell
   - name: kiwix
     host: "kiwix.{{ caddy_domain }}"
     backend: "https://kiwix.tail8d86e.ts.net"
@@ -101,6 +104,9 @@ caddy_services:
   - name: paperless
     host: "paperless.{{ caddy_domain }}"
     backend: "https://paperless.tail8d86e.ts.net"
+  - name: shower
+    host: "shower.{{ caddy_domain }}"
+    backend: "https://shower.tail8d86e.ts.net"
   - name: sifaka
     host: "nas.{{ caddy_domain }}"
     backend: "http://sifaka:5000"
@@ -114,6 +120,8 @@ caddy_tcp_services:
     backend: "pg.tail8d86e.ts.net:5432"  # PostgreSQL (blumeops-pg)
   - port: 5433
     backend: "immich-pg.tail8d86e.ts.net:5432"  # PostgreSQL (immich-pg)
+  - port: 5434
+    backend: "blumeops-pg-ringtail.tail8d86e.ts.net:5432"  # PostgreSQL (blumeops-pg on ringtail)
   - port: "{{ sifaka_node_exporter_port }}"
     backend: "sifaka:{{ sifaka_node_exporter_port }}"  # Sifaka node_exporter
   - port: "{{ sifaka_smartctl_exporter_port }}"

ansible/roles/caddy/templates/Caddyfile.j2

@@ -42,11 +42,11 @@
 		header @{{ service.name }}_dl{{ loop.index }} Content-Disposition `attachment; filename="{{ dl.filename }}"`
 {% endfor %}
 {% if service.try_html | default(false) %}
-		try_files {path} {path}/ {path}.html
-		handle_errors 404 {
-			rewrite * /404.html
-			file_server
-		}
+		# Quartz clean URLs: path → path/ → path.html → /404.html (200).
+		# Caddy's handle_errors is a top-level directive and can't live in
+		# this nested handle, so the 404 page rides as the final try_files
+		# candidate (served with 200 — acceptable for a human-facing 404).
+		try_files {path} {path}/ {path}.html /404.html
 {% endif %}
 		file_server
 {% else %}

ansible/roles/cv/defaults/main.yml

@@ -3,7 +3,7 @@
 # Caddy serves cv_content_dir directly via the static-kind service block.
 
 cv_version: "v1.0.3"
-cv_release_url: "https://forge.eblu.me/api/packages/eblume/generic/cv/{{ cv_version }}/cv-{{ cv_version }}.tar.gz"
+cv_release_url: "https://forge.ops.eblu.me/api/packages/eblume/generic/cv/{{ cv_version }}/cv-{{ cv_version }}.tar.gz"
 
 cv_home: /Users/erichblume/blumeops/cv
 cv_content_dir: "{{ cv_home }}/content"

ansible/roles/docs/defaults/main.yml

@@ -3,9 +3,8 @@
 # Caddy serves docs_content_dir directly via the static-kind service block,
 # with Quartz-style try_files (path → path/ → path.html → 404).
 
-docs_version: "v1.16.0"
+docs_version: "v1.17.0"
 docs_release_url: "https://forge.eblu.me/eblume/blumeops/releases/download/{{ docs_version }}/docs-{{ docs_version }}.tar.gz"
-
 docs_home: /Users/erichblume/blumeops/docs
 docs_content_dir: "{{ docs_home }}/content"
 docs_version_sentinel: "{{ docs_home }}/.installed-version"

ansible/roles/heph/defaults/main.yml

@@ -0,0 +1,49 @@
+---
+# hephaestus hub — the canonical heph replica (server mode) on indri.
+# Other devices (e.g. gilbert) are spokes that sync against this hub.
+# See [[set-up-sync-hub]] and [[host-heph-pwa]] in the hephaestus repo.
+
+# Pinned release used for the initial `cargo install` and the PWA shell.
+# After bootstrap, hephd's own --self-update keeps the binary current; this
+# pin only governs the first install and the bundled PWA shell version.
+heph_version: v1.2.1
+
+# Anonymous public HTTPS clone — matches hephd's INSTALL_GIT_URL so the initial
+# install and unattended self-update build from the same source (no ssh-agent).
+heph_repo_url: https://forge.eblu.me/eblume/hephaestus.git
+
+heph_bin_dir: /Users/erichblume/.cargo/bin
+heph_binary: "{{ heph_bin_dir }}/hephd"
+
+# rustc/cargo here are rustup shims. The bare (non-mise) environment that the
+# launchagent and ansible run in falls back to rustup's *default* toolchain,
+# which can lag behind heph's rust-version floor (Cargo.toml: 1.89). Pin the
+# channel explicitly so both the bootstrap build and unattended self-update
+# always use a current toolchain regardless of the host's rustup default.
+heph_rust_toolchain: stable
+
+heph_data_dir: /Users/erichblume/.local/share/heph
+heph_db: "{{ heph_data_dir }}/heph.db"
+heph_socket: "{{ heph_data_dir }}/hephd.sock"
+heph_log_dir: /Users/erichblume/Library/Logs
+
+# Version-pinned source checkout; the PWA static shell is served directly from
+# its heph-pwa/ subdir (no copy), keeping shell and hub in lockstep at heph_version.
+heph_pwa_src_dir: /Users/erichblume/.cache/heph-pwa-src
+heph_web_root: "{{ heph_pwa_src_dir }}/heph-pwa"
+
+# Hub listens on all interfaces so tailnet spokes can reach it directly
+# (http://indri.tail8d86e.ts.net:8787) and Caddy can proxy heph.ops.eblu.me.
+# Access is gated by Authentik OIDC regardless — tailnet reachability is not
+# enough (this is the owner's most sensitive data).
+heph_http_addr: 0.0.0.0:8787
+heph_port: 8787
+heph_external_url: https://heph.ops.eblu.me
+
+# Authentik OIDC — issuer + audience together turn hub auth on. The audience is
+# the device-code client id (see argocd/manifests/authentik heph blueprint).
+heph_oidc_issuer: https://authentik.ops.eblu.me/application/o/heph/
+heph_oidc_audience: heph
+
+# Self-update poll interval (seconds). 10 minutes.
+heph_self_update_interval_secs: 600

ansible/roles/heph/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: Restart heph
+  ansible.builtin.shell: |
+    launchctl unload ~/Library/LaunchAgents/mcquack.eblume.heph.plist 2>/dev/null || true
+    launchctl load ~/Library/LaunchAgents/mcquack.eblume.heph.plist
+  changed_when: true

ansible/roles/heph/tasks/main.yml

@@ -0,0 +1,82 @@
+---
+# hephaestus hub (server mode) on indri.
+#
+# DATA SEEDING (one-time, Path A — do this BEFORE the first provision so the hub
+# adopts gilbert's existing data instead of being born empty):
+#
+#   1. On the seed device (gilbert):   heph daemon stop
+#   2. Copy its store to indri:         scp ~/.local/share/heph/heph.db \
+#                                           indri:~/.local/share/heph/heph.db
+#   3. On indri, give the hub its OWN device origin (keeps gilbert's owner_id +
+#      data; hephd regenerates a fresh origin on next start when it is missing):
+#        sqlite3 ~/.local/share/heph/heph.db "DELETE FROM meta WHERE key='origin';"
+#   4. Run this role (installs hephd, stages the PWA, loads the launchagent).
+#
+# hephd auto-creates an empty store on first start if none exists, so seeding is
+# optional — skip it only if you intend a fresh, empty hub.
+
+- name: Ensure heph data directory exists
+  ansible.builtin.file:
+    path: "{{ heph_data_dir }}"
+    state: directory
+    mode: '0700'
+
+- name: Check for installed hephd binary
+  ansible.builtin.stat:
+    path: "{{ heph_binary }}"
+  register: heph_binary_stat
+
+# Bootstrap install only when hephd is absent. Thereafter hephd's own
+# --self-update keeps it current; ansible must not fight (or downgrade) it.
+# This builds from source and can take several minutes on a cold cargo cache.
+- name: Bootstrap-install heph + hephd from the forge ({{ heph_version }})
+  ansible.builtin.command:
+    cmd: >-
+      {{ heph_bin_dir }}/cargo install --locked
+      --git {{ heph_repo_url }}
+      --tag {{ heph_version }}
+      heph hephd
+  environment:
+    PATH: "{{ heph_bin_dir }}:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin"
+    RUSTUP_TOOLCHAIN: "{{ heph_rust_toolchain }}"
+  when: not heph_binary_stat.stat.exists
+  changed_when: true
+  notify: Restart heph
+
+# Checkout provides the PWA shell at {{ heph_web_root }} (heph-pwa/ subdir),
+# served directly by hephd. Static files are read from disk per request, so a
+# version bump needs no restart; the service worker (CACHE = "heph-pwa-vN")
+# evicts stale assets on next load.
+- name: Ensure heph cache parent directory exists
+  ansible.builtin.file:
+    path: "{{ heph_pwa_src_dir | dirname }}"
+    state: directory
+    mode: '0755'
+
+- name: Stage heph-pwa source at {{ heph_version }}
+  ansible.builtin.git:
+    repo: "{{ heph_repo_url }}"
+    dest: "{{ heph_pwa_src_dir }}"
+    version: "{{ heph_version }}"
+    depth: 1
+    single_branch: true
+    force: true
+
+- name: Deploy heph LaunchAgent plist
+  ansible.builtin.template:
+    src: heph.plist.j2
+    dest: ~/Library/LaunchAgents/mcquack.eblume.heph.plist
+    mode: '0644'
+  notify: Restart heph
+
+- name: Check if heph LaunchAgent is loaded
+  ansible.builtin.command: launchctl list mcquack.eblume.heph
+  register: heph_launchctl_check
+  changed_when: false
+  failed_when: false
+
+- name: Load heph LaunchAgent if not loaded
+  ansible.builtin.command: launchctl load ~/Library/LaunchAgents/mcquack.eblume.heph.plist
+  when: heph_launchctl_check.rc != 0
+  changed_when: true
+  failed_when: false

ansible/roles/heph/templates/heph.plist.j2

@@ -0,0 +1,50 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- {{ ansible_managed }} -->
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>Label</key>
+	<string>mcquack.eblume.heph</string>
+	<key>ProgramArguments</key>
+	<array>
+		<string>{{ heph_binary }}</string>
+		<string>--mode</string>
+		<string>server</string>
+		<string>--http-addr</string>
+		<string>{{ heph_http_addr }}</string>
+		<string>--db</string>
+		<string>{{ heph_db }}</string>
+		<string>--socket</string>
+		<string>{{ heph_socket }}</string>
+		<string>--web-root</string>
+		<string>{{ heph_web_root }}</string>
+		<string>--oidc-issuer</string>
+		<string>{{ heph_oidc_issuer }}</string>
+		<string>--oidc-audience</string>
+		<string>{{ heph_oidc_audience }}</string>
+		<string>--self-update</string>
+		<string>--self-update-interval-secs</string>
+		<string>{{ heph_self_update_interval_secs }}</string>
+	</array>
+	<key>RunAtLoad</key>
+	<true/>
+	<key>KeepAlive</key>
+	<true/>
+	<key>EnvironmentVariables</key>
+	<dict>
+		<!-- cargo + toolchain on PATH so --self-update can run `cargo install`. -->
+		<key>PATH</key>
+		<string>{{ heph_bin_dir }}:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
+		<key>HOME</key>
+		<string>/Users/erichblume</string>
+		<!-- Pin the rustup channel: the launchagent runs without mise, so a bare
+		     cargo shim would otherwise use rustup's (stale) default toolchain. -->
+		<key>RUSTUP_TOOLCHAIN</key>
+		<string>{{ heph_rust_toolchain }}</string>
+	</dict>
+	<key>StandardOutPath</key>
+	<string>{{ heph_log_dir }}/mcquack.heph.out.log</string>
+	<key>StandardErrorPath</key>
+	<string>{{ heph_log_dir }}/mcquack.heph.err.log</string>
+</dict>
+</plist>

argocd/apps/cloudnative-pg-ringtail.yaml

@@ -0,0 +1,27 @@
+# CloudNativePG Operator for ringtail k3s cluster
+# Deploys the operator only; PostgreSQL clusters are created separately
+#
+# Sibling of cloudnative-pg.yaml (minikube). Same mirror, same release,
+# different destination. Both apps will coexist during the immich
+# migration; the minikube one is removed at the end of the broader
+# indri-k8s decommission.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: cloudnative-pg-ringtail
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/mirrors/cloudnative-pg.git
+    targetRevision: v1.27.1
+    path: releases
+    directory:
+      include: 'cnpg-1.27.1.yaml'
+  destination:
+    server: https://ringtail.tail8d86e.ts.net:6443
+    namespace: cnpg-system
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true  # Required for large CRDs that exceed annotation size limit

argocd/apps/cv.yaml

@@ -1,18 +0,0 @@
----
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: cv
-  namespace: argocd
-spec:
-  project: default
-  source:
-    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
-    targetRevision: main
-    path: argocd/manifests/cv
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: cv
-  syncPolicy:
-    syncOptions:
-      - CreateNamespace=true

argocd/apps/databases-ringtail.yaml

@@ -0,0 +1,26 @@
+# Databases on ringtail k3s.
+#
+# Today: only immich-pg (CNPG Cluster) + its borgmatic ExternalSecret.
+# More databases may move here as the indri-k8s decommission proceeds.
+#
+# Prerequisites:
+# - cloudnative-pg-ringtail (operator must exist before the Cluster CR)
+# - external-secrets-ringtail + 1password-connect-ringtail (for the
+#   immich-pg-borgmatic ExternalSecret to sync)
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: databases-ringtail
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
+    targetRevision: main
+    path: argocd/manifests/databases-ringtail
+  destination:
+    server: https://ringtail.tail8d86e.ts.net:6443
+    namespace: databases
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

argocd/apps/docs.yaml

@@ -1,18 +0,0 @@
----
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: docs
-  namespace: argocd
-spec:
-  project: default
-  source:
-    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
-    targetRevision: main
-    path: argocd/manifests/docs
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: docs
-  syncPolicy:
-    syncOptions:
-      - CreateNamespace=true

argocd/apps/external-secrets-ringtail.yaml

@@ -15,7 +15,7 @@ spec:
   source:
     repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
     targetRevision: main
-    path: argocd/manifests/external-secrets
+    path: argocd/manifests/external-secrets-ringtail
   destination:
     server: https://ringtail.tail8d86e.ts.net:6443
     namespace: external-secrets

argocd/apps/homepage.yaml

@@ -14,7 +14,7 @@ spec:
     targetRevision: main
     path: argocd/manifests/homepage
   destination:
-    server: https://kubernetes.default.svc
+    server: https://ringtail.tail8d86e.ts.net:6443
     namespace: homepage
   syncPolicy:
     syncOptions:

argocd/apps/immich-ringtail.yaml

@@ -0,0 +1,31 @@
+# Immich on ringtail k3s.
+#
+# Staging deployment; the minikube `immich` app remains in parallel
+# until cutover. See [[immich-cutover-and-decommission]] for the
+# routing flip + minikube cleanup.
+#
+# Prerequisites:
+# - cnpg-on-ringtail + databases-ringtail (postgres)
+# - 1password-connect-ringtail + external-secrets-ringtail (not used
+#   by this app today — immich-db Secret is created manually,
+#   matching the minikube pattern)
+# - The immich-db Secret in the immich namespace, holding the
+#   password for the `immich` postgres role (copied from the source
+#   immich-pg-app Secret at migration time).
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: immich-ringtail
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
+    targetRevision: main
+    path: argocd/manifests/immich-ringtail
+  destination:
+    server: https://ringtail.tail8d86e.ts.net:6443
+    namespace: immich
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

argocd/apps/immich.yaml

@@ -1,30 +0,0 @@
-# Immich - Self-hosted photo and video management
-# High-performance Google Photos/iCloud alternative with AI features
-#
-# Kustomize manifests in argocd/manifests/immich/
-# Components: server, machine-learning, valkey (Redis)
-#
-# Prerequisites:
-# 1. Create immich namespace and secrets:
-#    kubectl create namespace immich
-#    kubectl --context=minikube-indri create secret generic immich-db -n immich \
-#      --from-literal=password="$(kubectl --context=minikube-indri -n databases get secret immich-pg-app -o jsonpath='{.data.password}' | base64 -d)"
-# 2. Create immich-pg database and user (see immich-pg app)
-# 3. NFS share on sifaka at /volume1/photos with read/write for indri
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: immich
-  namespace: argocd
-spec:
-  project: default
-  source:
-    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
-    targetRevision: main
-    path: argocd/manifests/immich
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: immich
-  syncPolicy:
-    syncOptions:
-      - CreateNamespace=true

argocd/apps/mealie-ringtail.yaml

@@ -0,0 +1,26 @@
+# Mealie on ringtail k3s.
+#
+# Wave-1 indri-k8s decommission. Staging deployment; the minikube `mealie`
+# app stays in parallel until cutover (copy SQLite PVC, drop the minikube
+# tailscale ingress, flip Caddy). See [[migrate-wave1-ringtail]].
+#
+# Prerequisites:
+# - external-secrets-ringtail (onepassword-blumeops ClusterSecretStore)
+# - mealie-data PVC contents copied from minikube at cutover
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: mealie-ringtail
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
+    targetRevision: main
+    path: argocd/manifests/mealie-ringtail
+  destination:
+    server: https://ringtail.tail8d86e.ts.net:6443
+    namespace: mealie
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

argocd/apps/mealie.yaml

@@ -1,17 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: mealie
-  namespace: argocd
-spec:
-  project: default
-  source:
-    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
-    targetRevision: main
-    path: argocd/manifests/mealie
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: mealie
-  syncPolicy:
-    syncOptions:
-      - CreateNamespace=true

argocd/apps/paperless-ringtail.yaml

@@ -0,0 +1,28 @@
+# Paperless-ngx on ringtail k3s.
+#
+# Wave-1 indri-k8s decommission. Staging deployment; the minikube
+# `paperless` app stays in parallel until cutover (drop the minikube
+# tailscale ingress to free the name, then flip Caddy). See
+# [[migrate-wave1-ringtail]].
+#
+# Prerequisites:
+# - databases-ringtail blumeops-pg (paperless database + role)
+# - external-secrets-ringtail (onepassword-blumeops ClusterSecretStore)
+# - sifaka NFS rule granting ringtail access to /volume1/paperless
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: paperless-ringtail
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
+    targetRevision: main
+    path: argocd/manifests/paperless-ringtail
+  destination:
+    server: https://ringtail.tail8d86e.ts.net:6443
+    namespace: paperless
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

argocd/apps/paperless.yaml

@@ -1,17 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: paperless
-  namespace: argocd
-spec:
-  project: default
-  source:
-    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
-    targetRevision: main
-    path: argocd/manifests/paperless
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: paperless
-  syncPolicy:
-    syncOptions:
-      - CreateNamespace=true

argocd/apps/shower.yaml

@@ -0,0 +1,20 @@
+# Adelaide / Heidi / Addie baby shower app — Django guest/raffle/prize system.
+# Public landing page at shower.eblu.me (via fly proxy), staff console + admin
+# at shower.ops.eblu.me (tailnet only). Built from forge PyPI wheel.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: shower
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
+    targetRevision: main
+    path: argocd/manifests/shower
+  destination:
+    server: https://ringtail.tail8d86e.ts.net:6443
+    namespace: shower
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

argocd/apps/teslamate-ringtail.yaml

@@ -0,0 +1,28 @@
+# TeslaMate on ringtail k3s.
+#
+# Wave-1 indri-k8s decommission. Staging deployment; the minikube
+# `teslamate` app stays in parallel until cutover (migrate the teslamate
+# database, drop the minikube tailscale ingress, flip Caddy). See
+# [[migrate-wave1-ringtail]].
+#
+# Prerequisites:
+# - databases-ringtail blumeops-pg (teslamate database + role; cube +
+#   earthdistance extensions created by superuser at cutover)
+# - external-secrets-ringtail (onepassword-blumeops ClusterSecretStore)
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: teslamate-ringtail
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
+    targetRevision: main
+    path: argocd/manifests/teslamate-ringtail
+  destination:
+    server: https://ringtail.tail8d86e.ts.net:6443
+    namespace: teslamate
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

argocd/apps/teslamate.yaml

@@ -1,32 +0,0 @@
-# TeslaMate Tesla Data Logger
-# Requires: CloudNativePG PostgreSQL cluster and manual secret setup
-#
-# Before syncing, create the namespace and secrets:
-#   kubectl create namespace teslamate
-#   op inject -i argocd/manifests/databases/secret-teslamate.yaml.tpl | kubectl apply -f -
-#   op inject -i argocd/manifests/teslamate/secret-encryption-key.yaml.tpl | kubectl apply -f -
-#   op inject -i argocd/manifests/teslamate/secret-db.yaml.tpl | kubectl apply -f -
-#
-# Then create the database:
-#   PGPASSWORD=$(op read "op://blumeops/postgres/password") \
-#     psql -h pg.ops.eblu.me -U eblume -c "CREATE DATABASE teslamate OWNER teslamate;"
-#
-# After syncing, access the TeslaMate UI at https://tesla.tail8d86e.ts.net to complete
-# Tesla API authentication via OAuth flow.
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: teslamate
-  namespace: argocd
-spec:
-  project: default
-  source:
-    repoURL: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git
-    targetRevision: main
-    path: argocd/manifests/teslamate
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: teslamate
-  syncPolicy:
-    syncOptions:
-      - CreateNamespace=true

argocd/manifests/alloy-k8s/config.alloy

@@ -191,14 +191,9 @@ prometheus.exporter.blackbox "services" {
   }
 
   target {
+    // Migrated to ringtail (wave-1); probe through Caddy over Tailscale.
     name    = "teslamate"
-    address = "http://teslamate.teslamate.svc.cluster.local:4000/"
-    module  = "http_2xx"
-  }
-
-  target {
-    name    = "immich"
-    address = "http://immich-server.immich.svc.cluster.local:2283/api/server/ping"
+    address = "https://tesla.ops.eblu.me/"
     module  = "http_2xx"
   }
 

argocd/manifests/alloy-k8s/kustomization.yaml

@@ -10,7 +10,7 @@ resources:
 
 images:
   - name: registry.ops.eblu.me/blumeops/alloy
-    newTag: v1.14.0-fd0bebb
+    newTag: v1.16.0-9564435
 
 configMapGenerator:
   - name: alloy-config

argocd/manifests/alloy-ringtail/config.alloy

@@ -45,6 +45,26 @@ prometheus.scrape "kube_state_metrics" {
   forward_to      = [prometheus.remote_write.prometheus.receiver]
 }
 
+// ============== SERVICE HEALTH PROBES ==============
+
+// Blackbox-style HTTP probes for in-cluster services on ringtail
+prometheus.exporter.blackbox "services" {
+  config = "{ modules: { http_2xx: { prober: http, timeout: 5s } } }"
+
+  target {
+    name    = "immich"
+    address = "http://immich-server.immich.svc.cluster.local:2283/api/server/ping"
+    module  = "http_2xx"
+  }
+}
+
+// Scrape blackbox probe results
+prometheus.scrape "blackbox" {
+  targets         = prometheus.exporter.blackbox.services.targets
+  scrape_interval = "30s"
+  forward_to      = [prometheus.remote_write.prometheus.receiver]
+}
+
 // Push metrics to indri Prometheus
 prometheus.remote_write "prometheus" {
   external_labels = { cluster = "ringtail" }

argocd/manifests/alloy-ringtail/kustomization.yaml

@@ -10,7 +10,7 @@ resources:
 
 images:
   - name: registry.ops.eblu.me/blumeops/alloy
-    newTag: v1.14.0-fd0bebb-nix
+    newTag: v1.16.0-9564435-nix
 
 configMapGenerator:
   - name: alloy-config

argocd/manifests/alloy-tracing-ringtail/kustomization.yaml

@@ -9,7 +9,7 @@ resources:
 
 images:
   - name: registry.ops.eblu.me/blumeops/alloy
-    newTag: v1.14.0-fd0bebb-nix
+    newTag: v1.16.0-9564435-nix
 
 configMapGenerator:
   - name: alloy-tracing-config

argocd/manifests/argocd/argocd-rbac-cm-patch.yaml

@@ -2,6 +2,9 @@
 #
 # - workflow-bot: minimal CI/CD permissions (sync, get)
 # - admins: Authentik admins group mapped to ArgoCD admin role
+# - admin: local break-glass account — keeps ArgoCD admin rights for when
+#   Authentik SSO is unavailable (without this it has no permissions, since
+#   policy.default is unset)
 #
 apiVersion: v1
 kind: ConfigMap
@@ -14,3 +17,4 @@ data:
     p, role:workflow-bot, applications, get, *, allow
     g, workflow-bot, role:workflow-bot
     g, admins, role:admin
+    g, admin, role:admin

argocd/manifests/authentik/configmap-blueprint.yaml

@@ -434,3 +434,93 @@ data:
           provider: !KeyOf mealie-provider
           meta_launch_url: https://meals.ops.eblu.me
           policy_engine_mode: all
+
+  heph.yaml: |
+    version: 1
+    metadata:
+      name: BlumeOps Heph SSO
+      labels:
+        blueprints.goauthentik.io/description: "Hephaestus hub OIDC (device-code) provider, application, and device-code flow"
+    entries:
+      # Device-code flow (RFC 8628). authentik ships no default for this, so we
+      # create one and bind it to the brand below. An empty stage_configuration
+      # flow is sufficient: the already-authenticated user just confirms the code.
+      - model: authentik_flows.flow
+        id: device-code-flow
+        identifiers:
+          slug: default-device-code-flow
+        attrs:
+          name: Device code flow
+          title: Device code flow
+          slug: default-device-code-flow
+          designation: stage_configuration
+          authentication: require_authenticated
+
+      # Enable the device-code grant globally by binding the flow to the default
+      # brand (domain authentik-default). Partial update — only sets this field.
+      - model: authentik_brands.brand
+        identifiers:
+          domain: authentik-default
+        attrs:
+          flow_device_code: !KeyOf device-code-flow
+
+      # OAuth2 provider for heph — PUBLIC client (device-code + PKCE, no secret).
+      # client_id doubles as the token audience the hub verifies (--oidc-audience heph),
+      # and the app slug 'heph' is the issuer path (/application/o/heph/).
+      - model: authentik_providers_oauth2.oauth2provider
+        id: heph-provider
+        identifiers:
+          name: Heph
+        attrs:
+          name: Heph
+          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          client_type: public
+          client_id: heph
+          # CLI/TUI use the device-code grant (no redirect). The heph-pwa browser
+          # login uses Authorization Code + PKCE, which DOES redirect back to the
+          # app's origin — register those here (Authentik also keys token-endpoint
+          # CORS off these origins). Trailing slash matters: the PWA's redirect_uri
+          # is its base dir, e.g. https://heph.ops.eblu.me/.
+          redirect_uris:
+            - matching_mode: strict
+              url: https://heph.ops.eblu.me/
+            - matching_mode: strict
+              url: http://localhost:8787/  # local dev (hephd --web-root)
+          signing_key: !Find [authentik_crypto.certificatekeypair, [name, authentik Self-signed Certificate]]
+          property_mappings:
+            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+            # offline_access: heph CLI requests "openid offline_access"; without
+            # this mapping the refresh token is session-bound and hephd's
+            # refresh_token grant 400s once the session lapses (spoke sync dies).
+            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, offline_access]]
+          sub_mode: hashed_user_id
+          include_claims_in_id_token: true
+
+      # Heph application — linked to the OAuth2 provider
+      - model: authentik_core.application
+        id: heph-app
+        identifiers:
+          slug: heph
+        attrs:
+          name: Hephaestus
+          slug: heph
+          provider: !KeyOf heph-provider
+          meta_launch_url: https://heph.ops.eblu.me
+          policy_engine_mode: any
+
+      # Policy binding — restrict heph to admins group (single-owner, sensitive data)
+      - model: authentik_policies.policybinding
+        identifiers:
+          order: 0
+          target: !KeyOf heph-app
+          group: !Find [authentik_core.group, [name, admins]]
+        attrs:
+          target: !KeyOf heph-app
+          group: !Find [authentik_core.group, [name, admins]]
+          order: 0
+          enabled: true
+          negate: false
+          timeout: 30

argocd/manifests/cv/deployment.yaml

@@ -1,51 +0,0 @@
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: cv
-  namespace: cv
-spec:
-  replicas: 2
-  strategy:
-    type: RollingUpdate
-    rollingUpdate:
-      maxUnavailable: 0
-      maxSurge: 1
-  selector:
-    matchLabels:
-      app: cv
-  template:
-    metadata:
-      labels:
-        app: cv
-    spec:
-      securityContext:
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: cv
-          image: registry.ops.eblu.me/blumeops/cv:kustomized
-          ports:
-            - containerPort: 80
-              name: http
-          env:
-            - name: CV_RELEASE_URL
-              value: "https://forge.eblu.me/api/packages/eblume/generic/cv/v1.0.3/cv-v1.0.3.tar.gz"
-          resources:
-            requests:
-              memory: "64Mi"
-              cpu: "10m"
-            limits:
-              memory: "128Mi"
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 80
-            initialDelaySeconds: 10
-            periodSeconds: 30
-          readinessProbe:
-            httpGet:
-              path: /healthz
-              port: 80
-            initialDelaySeconds: 5
-            periodSeconds: 10

argocd/manifests/cv/ingress-tailscale.yaml

@@ -1,27 +0,0 @@
----
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: cv-tailscale
-  namespace: cv
-  annotations:
-    tailscale.com/proxy-class: "default"
-    tailscale.com/proxy-group: "ingress"
-    tailscale.com/tags: "tag:k8s,tag:flyio-target"
-    gethomepage.dev/enabled: "true"
-    gethomepage.dev/name: "CV"
-    gethomepage.dev/group: "Services"
-    gethomepage.dev/icon: "mdi-file-document"
-    gethomepage.dev/description: "Resume / CV"
-    gethomepage.dev/href: "https://cv.eblu.me"
-    gethomepage.dev/pod-selector: "app=cv"
-spec:
-  ingressClassName: tailscale
-  defaultBackend:
-    service:
-      name: cv
-      port:
-        number: 80
-  tls:
-    - hosts:
-        - cv

argocd/manifests/cv/kustomization.yaml

@@ -1,12 +0,0 @@
----
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-namespace: cv
-resources:
-  - deployment.yaml
-  - service.yaml
-  - ingress-tailscale.yaml
-  - pdb.yaml
-images:
-  - name: registry.ops.eblu.me/blumeops/cv
-    newTag: v1.0.3-613f05d

argocd/manifests/cv/pdb.yaml

@@ -1,10 +0,0 @@
----
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: cv
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app: cv

argocd/manifests/cv/service.yaml

@@ -1,13 +0,0 @@
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: cv
-  namespace: cv
-spec:
-  selector:
-    app: cv
-  ports:
-    - name: http
-      port: 80
-      targetPort: 80

argocd/manifests/databases-ringtail/blumeops-pg.yaml

@@ -0,0 +1,97 @@
+# PostgreSQL Cluster for blumeops services on ringtail k3s.
+#
+# Wave-1 indri-k8s decommission target (see [[migrate-wave1-ringtail]]).
+# Holds the paperless and teslamate databases migrated off the minikube
+# blumeops-pg via cold pg_dump/pg_restore at cutover. miniflux + authentik
+# stay where they are for now (later waves), so this cluster only carries
+# the wave-1 roles.
+#
+# Apps reach this in-cluster at blumeops-pg-rw.databases.svc.cluster.local
+# — the same name they used on minikube, so teslamate's DATABASE_HOST is
+# unchanged.
+#
+# Database creation is deferred to cutover, mirroring the minikube cluster
+# (where only the bootstrap database is declared and the rest were created
+# out-of-band):
+#   - paperless: the bootstrap database below (restored into at cutover).
+#   - teslamate: created at its cutover by the eblume superuser, because the
+#     dump's `earthdistance` extension is untrusted and CREATE EXTENSION
+#     needs superuser. (cube + earthdistance ownership then transferred to
+#     the teslamate role so it can ALTER EXTENSION UPDATE.)
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: blumeops-pg
+  namespace: databases
+spec:
+  instances: 1
+  imageName: ghcr.io/cloudnative-pg/postgresql:18.3
+
+  storage:
+    size: 10Gi
+    storageClass: local-path
+
+  bootstrap:
+    initdb:
+      database: paperless
+      owner: paperless
+
+  managed:
+    roles:
+      # eblume superuser for admin + privileged restore steps (extensions)
+      - name: eblume
+        login: true
+        superuser: true
+        createdb: true
+        createrole: true
+        connectionLimit: -1
+        ensure: present
+        inherit: true
+        passwordSecret:
+          name: blumeops-pg-eblume
+      # borgmatic read-only user for backups
+      - name: borgmatic
+        login: true
+        connectionLimit: -1
+        ensure: present
+        inherit: true
+        inRoles:
+          - pg_read_all_data
+        passwordSecret:
+          name: blumeops-pg-borgmatic
+      # paperless user (also the bootstrap database owner above; the
+      # managed role sets its password from the 1Password-backed secret)
+      - name: paperless
+        login: true
+        connectionLimit: -1
+        ensure: present
+        inherit: true
+        passwordSecret:
+          name: blumeops-pg-paperless
+      # teslamate user. Extension ownership (cube, earthdistance) is
+      # transferred to this role at cutover so it can ALTER EXTENSION UPDATE.
+      - name: teslamate
+        login: true
+        connectionLimit: -1
+        ensure: present
+        inherit: true
+        passwordSecret:
+          name: blumeops-pg-teslamate
+
+  resources:
+    requests:
+      memory: "256Mi"
+      cpu: "100m"
+    limits:
+      memory: "1Gi"
+      cpu: "500m"
+
+  postgresql:
+    parameters:
+      max_connections: "50"
+      shared_buffers: "128MB"
+      password_encryption: "scram-sha-256"
+    pg_hba:
+      # Password auth from anywhere; network security is via Tailscale.
+      - host all all 0.0.0.0/0 scram-sha-256
+      - host all all ::/0 scram-sha-256

argocd/manifests/databases-ringtail/external-secret-borgmatic.yaml

@@ -1,13 +1,14 @@
-# ExternalSecret for borgmatic backup user password on immich-pg cluster
+# ExternalSecret for borgmatic backup user password
+#
+# Replaces the manual op inject workflow from secret-borgmatic.yaml.tpl
 #
-# Reuses the same 1Password item as blumeops-pg-borgmatic.
 # 1Password item: "borgmatic" in blumeops vault
 # Field: "db-password"
 #
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
-  name: immich-pg-borgmatic
+  name: blumeops-pg-borgmatic
   namespace: databases
 spec:
   refreshInterval: 1h
@@ -15,7 +16,7 @@ spec:
     kind: ClusterSecretStore
     name: onepassword-blumeops
   target:
-    name: immich-pg-borgmatic
+    name: blumeops-pg-borgmatic
     creationPolicy: Owner
     template:
       type: kubernetes.io/basic-auth

argocd/manifests/databases-ringtail/external-secret-eblume.yaml

@@ -0,0 +1,30 @@
+# ExternalSecret for eblume superuser password
+#
+# Replaces the manual op inject workflow from secret-eblume.yaml.tpl
+#
+# 1Password item: "postgres" in blumeops vault
+# Field: "password"
+#
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: blumeops-pg-eblume
+  namespace: databases
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-blumeops
+  target:
+    name: blumeops-pg-eblume
+    creationPolicy: Owner
+    template:
+      type: kubernetes.io/basic-auth
+      data:
+        username: eblume
+        password: "{{ .password }}"
+  data:
+  - secretKey: password
+    remoteRef:
+      key: postgres
+      property: password

argocd/manifests/databases-ringtail/external-secret-immich-borgmatic.yaml

@@ -0,0 +1,32 @@
+# ExternalSecret for borgmatic backup user password on immich-pg cluster
+# (ringtail k3s).
+#
+# Mirror of argocd/manifests/databases/external-secret-immich-borgmatic.yaml.
+# The onepassword-blumeops ClusterSecretStore exists on ringtail via the
+# external-secrets-ringtail app.
+#
+# 1Password item: "borgmatic" in blumeops vault
+# Field: "db-password"
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: immich-pg-borgmatic
+  namespace: databases
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-blumeops
+  target:
+    name: immich-pg-borgmatic
+    creationPolicy: Owner
+    template:
+      type: kubernetes.io/basic-auth
+      data:
+        username: borgmatic
+        password: "{{ .password }}"
+  data:
+    - secretKey: password
+      remoteRef:
+        key: borgmatic
+        property: db-password

argocd/manifests/databases-ringtail/immich-pg.yaml

@@ -0,0 +1,53 @@
+# PostgreSQL Cluster for Immich on ringtail k3s.
+#
+# Initially bootstrapped via CNPG pg_basebackup from the minikube
+# immich-pg cluster on 2026-05-13, then promoted to primary. The
+# externalClusters + bootstrap.pg_basebackup blocks have been pruned
+# from this manifest now that the migration is complete — leaving
+# them around is a footgun (re-enabling replica.enabled=true would
+# try to demote this cluster against a stale source). See
+# [[immich-pg-data-migration]] for the procedure used.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: immich-pg
+  namespace: databases
+spec:
+  instances: 1
+  imageName: ghcr.io/tensorchord/cloudnative-vectorchord:17-0.5.0
+
+  storage:
+    size: 10Gi
+    storageClass: local-path
+
+  # Managed roles
+  managed:
+    roles:
+      - name: borgmatic
+        login: true
+        connectionLimit: -1
+        ensure: present
+        inherit: true
+        inRoles:
+          - pg_read_all_data
+        passwordSecret:
+          name: immich-pg-borgmatic
+
+  resources:
+    requests:
+      memory: "256Mi"
+      cpu: "100m"
+    limits:
+      memory: "1Gi"
+      cpu: "500m"
+
+  postgresql:
+    shared_preload_libraries:
+      - "vchord.so"
+    parameters:
+      max_connections: "50"
+      shared_buffers: "128MB"
+      password_encryption: "scram-sha-256"
+    pg_hba:
+      - host all all 0.0.0.0/0 scram-sha-256
+      - host all all ::/0 scram-sha-256

argocd/manifests/databases-ringtail/kustomization.yaml

@@ -0,0 +1,16 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: databases
+
+resources:
+  - immich-pg.yaml
+  - external-secret-immich-borgmatic.yaml
+  - service-immich-pg-tailscale.yaml
+  # wave-1 indri-k8s decommission: blumeops-pg (paperless + teslamate)
+  - blumeops-pg.yaml
+  - service-blumeops-pg-tailscale.yaml
+  - external-secret-eblume.yaml
+  - external-secret-borgmatic.yaml
+  - external-secret-paperless.yaml
+  - external-secret-teslamate.yaml

argocd/manifests/databases-ringtail/service-blumeops-pg-tailscale.yaml

@@ -0,0 +1,24 @@
+# Tailscale LoadBalancer for the ringtail blumeops-pg cluster.
+# Canonical hostname: blumeops-pg-ringtail.tail8d86e.ts.net (distinct from
+# the minikube blumeops-pg, which still owns pg.tail8d86e.ts.net until the
+# wave-1 decommission). Borgmatic on indri and the Grafana TeslaMate
+# datasource reach it via the Caddy L4 route pg.ops.eblu.me:5434.
+apiVersion: v1
+kind: Service
+metadata:
+  name: blumeops-pg-tailscale
+  namespace: databases
+  annotations:
+    tailscale.com/hostname: "blumeops-pg-ringtail"
+    tailscale.com/proxy-class: "default"
+spec:
+  type: LoadBalancer
+  loadBalancerClass: tailscale
+  selector:
+    cnpg.io/cluster: blumeops-pg
+    role: primary
+  ports:
+    - name: postgresql
+      port: 5432
+      targetPort: 5432
+      protocol: TCP

argocd/manifests/databases-ringtail/service-immich-pg-tailscale.yaml

@@ -1,6 +1,8 @@
-# Tailscale LoadBalancer for immich-pg PostgreSQL access
-# Canonical hostname: immich-pg.tail8d86e.ts.net
-# Caddy L4 proxies pg.ops.eblu.me:5433 → this service for borgmatic backups
+# Tailscale LoadBalancer for immich-pg PostgreSQL access on ringtail.
+# Canonical hostname: immich-pg.tail8d86e.ts.net (claimed from the
+# minikube side after the minikube service was removed during the
+# immich-to-ringtail migration). Borgmatic on indri uses this
+# hostname for nightly backups.
 apiVersion: v1
 kind: Service
 metadata:

argocd/manifests/databases/blumeops-pg.yaml

@@ -44,18 +44,9 @@ spec:
           - pg_read_all_data
         passwordSecret:
           name: blumeops-pg-borgmatic
-      # teslamate user for TeslaMate Tesla data logger
-      # Superuser removed. Extension ownership (cube, earthdistance)
-      # transferred manually so teslamate can ALTER EXTENSION UPDATE.
-      # earthdistance is untrusted — DROP+CREATE needs temporary
-      # superuser escalation during upgrades.
-      - name: teslamate
-        login: true
-        connectionLimit: -1
-        ensure: present
-        inherit: true
-        passwordSecret:
-          name: blumeops-pg-teslamate
+      # teslamate + paperless roles removed: migrated to ringtail blumeops-pg
+      # (wave-1 decommission). Their databases were dropped from this cluster
+      # after the cutover was verified and backed up.
       # authentik user for Authentik identity provider (runs on ringtail)
       - name: authentik
         login: true
@@ -65,14 +56,6 @@ spec:
         createdb: true
         passwordSecret:
           name: blumeops-pg-authentik
-      # paperless user for Paperless-ngx document management
-      - name: paperless
-        login: true
-        connectionLimit: -1
-        ensure: present
-        inherit: true
-        passwordSecret:
-          name: blumeops-pg-paperless
 
   # Resource limits for minikube environment
   resources:

argocd/manifests/databases/immich-pg.yaml

@@ -1,69 +0,0 @@
-# PostgreSQL Cluster for Immich
-# Uses VectorChord (successor to pgvecto.rs) for AI-powered vector search
-# See: https://github.com/immich-app/immich/discussions/9060
-# Managed by CloudNativePG operator
-apiVersion: postgresql.cnpg.io/v1
-kind: Cluster
-metadata:
-  name: immich-pg
-  namespace: databases
-spec:
-  instances: 1
-  # VectorChord image for PostgreSQL 17 with VectorChord 0.5.0
-  # Immich v2.4.1 requires VectorChord >=0.3 <0.6
-  # See: https://github.com/tensorchord/VectorChord
-  imageName: ghcr.io/tensorchord/cloudnative-vectorchord:17-0.5.0
-
-  storage:
-    size: 10Gi
-    storageClass: standard
-
-  # Bootstrap creates initial database and owner
-  bootstrap:
-    initdb:
-      database: immich
-      owner: immich
-      postInitSQL:
-        # Extensions required by Immich
-        - CREATE EXTENSION IF NOT EXISTS vector;
-        - CREATE EXTENSION IF NOT EXISTS vchord CASCADE;
-        - CREATE EXTENSION IF NOT EXISTS cube CASCADE;
-        - CREATE EXTENSION IF NOT EXISTS earthdistance CASCADE;
-
-  # Managed roles
-  # Note: connectionLimit, ensure, inherit are CNPG defaults added to prevent ArgoCD drift
-  managed:
-    roles:
-      # borgmatic read-only user for backups
-      - name: borgmatic
-        login: true
-        connectionLimit: -1
-        ensure: present
-        inherit: true
-        inRoles:
-          - pg_read_all_data
-        passwordSecret:
-          name: immich-pg-borgmatic
-
-  # Resource limits for minikube environment
-  resources:
-    requests:
-      memory: "256Mi"
-      cpu: "100m"
-    limits:
-      memory: "1Gi"
-      cpu: "500m"
-
-  # PostgreSQL configuration
-  postgresql:
-    # VectorChord requires vchord.so in shared_preload_libraries
-    shared_preload_libraries:
-      - "vchord.so"
-    parameters:
-      max_connections: "50"
-      shared_buffers: "128MB"
-      password_encryption: "scram-sha-256"
-    pg_hba:
-      # Allow connections from k8s pods
-      - host all all 0.0.0.0/0 scram-sha-256
-      - host all all ::/0 scram-sha-256

argocd/manifests/databases/kustomization.yaml

@@ -5,13 +5,8 @@ namespace: databases
 
 resources:
   - blumeops-pg.yaml
-  - immich-pg.yaml
   - service-tailscale.yaml
-  - service-immich-pg-tailscale.yaml
   - service-metrics-tailscale.yaml
   - external-secret-eblume.yaml
   - external-secret-borgmatic.yaml
-  - external-secret-immich-borgmatic.yaml
-  - external-secret-teslamate.yaml
   - external-secret-authentik.yaml
-  - external-secret-paperless.yaml

argocd/manifests/docs/deployment.yaml

@@ -1,51 +0,0 @@
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: docs
-  namespace: docs
-spec:
-  replicas: 2
-  strategy:
-    type: RollingUpdate
-    rollingUpdate:
-      maxUnavailable: 0
-      maxSurge: 1
-  selector:
-    matchLabels:
-      app: docs
-  template:
-    metadata:
-      labels:
-        app: docs
-    spec:
-      securityContext:
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: docs
-          image: registry.ops.eblu.me/blumeops/quartz:kustomized
-          ports:
-            - containerPort: 80
-              name: http
-          env:
-            - name: DOCS_RELEASE_URL
-              value: "https://forge.eblu.me/eblume/blumeops/releases/download/v1.16.0/docs-v1.16.0.tar.gz"
-          resources:
-            requests:
-              memory: "64Mi"
-              cpu: "10m"
-            limits:
-              memory: "128Mi"
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 80
-            initialDelaySeconds: 10
-            periodSeconds: 30
-          readinessProbe:
-            httpGet:
-              path: /healthz
-              port: 80
-            initialDelaySeconds: 5
-            periodSeconds: 10

argocd/manifests/docs/ingress-tailscale.yaml

@@ -1,27 +0,0 @@
----
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: docs-tailscale
-  namespace: docs
-  annotations:
-    tailscale.com/proxy-class: "default"
-    tailscale.com/proxy-group: "ingress"
-    tailscale.com/tags: "tag:k8s,tag:flyio-target"
-    gethomepage.dev/enabled: "true"
-    gethomepage.dev/name: "Docs"
-    gethomepage.dev/group: "Services"
-    gethomepage.dev/icon: "mdi-book-open-page-variant"
-    gethomepage.dev/description: "BlumeOps Documentation"
-    gethomepage.dev/href: "https://docs.eblu.me"
-    gethomepage.dev/pod-selector: "app=docs"
-spec:
-  ingressClassName: tailscale
-  defaultBackend:
-    service:
-      name: docs
-      port:
-        number: 80
-  tls:
-    - hosts:
-        - docs

argocd/manifests/docs/kustomization.yaml

@@ -1,12 +0,0 @@
----
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-namespace: docs
-resources:
-  - deployment.yaml
-  - service.yaml
-  - ingress-tailscale.yaml
-  - pdb.yaml
-images:
-  - name: registry.ops.eblu.me/blumeops/quartz
-    newTag: v1.28.2-613f05d

argocd/manifests/docs/pdb.yaml

@@ -1,10 +0,0 @@
----
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: docs
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app: docs

argocd/manifests/docs/service.yaml

@@ -1,13 +0,0 @@
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: docs
-  namespace: docs
-spec:
-  selector:
-    app: docs
-  ports:
-    - name: http
-      port: 80
-      targetPort: 80

argocd/manifests/external-secrets-ringtail/kustomization.yaml

@@ -0,0 +1,16 @@
+# Ringtail (amd64) overlay for external-secrets.
+#
+# Reuses the shared indri manifest as a base and only overrides the controller
+# image to the nix-built amd64 variant (`-nix` tag). The base sets the arm64
+# image (built via containers/external-secrets/container.py on indri's Dagger
+# runner); ringtail's k3s is amd64 and needs the image built by
+# containers/external-secrets/default.nix on the nix-container-builder.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - ../external-secrets
+
+images:
+  - name: registry.ops.eblu.me/blumeops/external-secrets
+    newTag: v2.2.0-13895bb-nix

argocd/manifests/external-secrets/kustomization.yaml

@@ -12,4 +12,5 @@ resources:
 
 images:
   - name: ghcr.io/external-secrets/external-secrets
-    newTag: v2.2.0
+    newName: registry.ops.eblu.me/blumeops/external-secrets
+    newTag: v2.2.0-13895bb

argocd/manifests/grafana-config/dashboards/configmap-shower-apm.yaml

@@ -0,0 +1,229 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboard-shower-apm
+  namespace: monitoring
+  labels:
+    grafana_dashboard: "1"
+data:
+  shower-apm.json: |
+    {
+      "annotations": { "list": [] },
+      "editable": true,
+      "fiscalYearStartMonth": 0,
+      "graphTooltip": 1,
+      "id": null,
+      "links": [],
+      "panels": [
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "color": { "mode": "palette-classic" },
+              "custom": {
+                "axisLabel": "req/s",
+                "drawStyle": "line",
+                "fillOpacity": 20,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": { "group": "A", "mode": "normal" }
+              },
+              "mappings": [],
+              "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
+              "unit": "reqps"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 8, "w": 16, "x": 0, "y": 0 },
+          "id": 1,
+          "options": {
+            "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "right", "showLegend": true },
+            "tooltip": { "mode": "multi", "sort": "desc" }
+          },
+          "targets": [
+            { "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "sum by (status) (rate(flyio_nginx_http_requests_total{host=\"shower.eblu.me\"}[5m]))", "legendFormat": "{{status}}", "refId": "A" }
+          ],
+          "title": "Request Rate by Status",
+          "type": "timeseries"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "color": { "mode": "thresholds" },
+              "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.01 }, { "color": "red", "value": 0.05 }] },
+              "unit": "percentunit"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 4, "w": 8, "x": 16, "y": 0 },
+          "id": 2,
+          "options": {
+            "colorMode": "background",
+            "graphMode": "area",
+            "justifyMode": "center",
+            "orientation": "auto",
+            "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
+            "textMode": "auto"
+          },
+          "targets": [
+            { "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "sum(rate(flyio_nginx_http_requests_total{host=\"shower.eblu.me\",status=~\"5..\"}[5m])) / sum(rate(flyio_nginx_http_requests_total{host=\"shower.eblu.me\"}[5m]))", "refId": "A" }
+          ],
+          "title": "Error Rate (5xx)",
+          "type": "stat"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "color": { "mode": "thresholds" },
+              "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 5 }] },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 4, "w": 4, "x": 16, "y": 4 },
+          "id": 3,
+          "options": {
+            "colorMode": "background",
+            "graphMode": "area",
+            "justifyMode": "center",
+            "orientation": "auto",
+            "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
+            "textMode": "auto"
+          },
+          "targets": [
+            { "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "sum(increase(flyio_nginx_http_requests_total{host=\"shower.eblu.me\",request_uri=~\"/admin/login.*\",status=~\"4..\"}[$__range]))", "refId": "A" }
+          ],
+          "title": "Failed admin logins (range)",
+          "type": "stat"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "color": { "mode": "thresholds" },
+              "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
+              "unit": "reqps"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 4, "w": 4, "x": 20, "y": 4 },
+          "id": 4,
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "center",
+            "orientation": "auto",
+            "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
+            "textMode": "auto"
+          },
+          "targets": [
+            { "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "sum(rate(flyio_nginx_http_requests_total{host=\"shower.eblu.me\"}[5m]))", "refId": "A" }
+          ],
+          "title": "Current RPS",
+          "type": "stat"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "color": { "mode": "palette-classic" },
+              "custom": {
+                "axisLabel": "seconds",
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": { "group": "A", "mode": "none" }
+              },
+              "mappings": [],
+              "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
+              "unit": "s"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
+          "id": 5,
+          "options": {
+            "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "right", "showLegend": true },
+            "tooltip": { "mode": "multi", "sort": "desc" }
+          },
+          "targets": [
+            { "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "histogram_quantile(0.50, sum by (le) (rate(flyio_nginx_http_request_duration_seconds_bucket{host=\"shower.eblu.me\"}[5m])))", "legendFormat": "p50", "refId": "A" },
+            { "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "histogram_quantile(0.90, sum by (le) (rate(flyio_nginx_http_request_duration_seconds_bucket{host=\"shower.eblu.me\"}[5m])))", "legendFormat": "p90", "refId": "B" },
+            { "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "histogram_quantile(0.99, sum by (le) (rate(flyio_nginx_http_request_duration_seconds_bucket{host=\"shower.eblu.me\"}[5m])))", "legendFormat": "p99", "refId": "C" }
+          ],
+          "title": "Latency Percentiles",
+          "type": "timeseries"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "color": { "mode": "palette-classic" },
+              "custom": {
+                "axisLabel": "",
+                "drawStyle": "line",
+                "fillOpacity": 20,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": { "group": "A", "mode": "none" }
+              },
+              "mappings": [],
+              "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
+              "unit": "Bps"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
+          "id": 6,
+          "options": {
+            "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "right", "showLegend": true },
+            "tooltip": { "mode": "single", "sort": "none" }
+          },
+          "targets": [
+            { "datasource": { "type": "prometheus", "uid": "prometheus" }, "expr": "sum(rate(flyio_nginx_http_response_bytes_total{host=\"shower.eblu.me\"}[5m]))", "legendFormat": "Bandwidth", "refId": "A" }
+          ],
+          "title": "Bandwidth",
+          "type": "timeseries"
+        },
+        {
+          "datasource": { "type": "loki", "uid": "loki" },
+          "gridPos": { "h": 8, "w": 24, "x": 0, "y": 16 },
+          "id": 7,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "prettifyLogMessage": false,
+            "showCommonLabels": false,
+            "showLabels": false,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": false
+          },
+          "targets": [
+            { "datasource": { "type": "loki", "uid": "loki" }, "expr": "{instance=\"flyio-proxy\", job=\"flyio-nginx\"} |= \"shower.eblu.me\" | json | line_format \"{{.client_ip}} {{.request_method}} {{.request_uri}} {{.status}} {{.request_time}}s\"", "refId": "A" }
+          ],
+          "title": "Recent Access Logs",
+          "type": "logs"
+        }
+      ],
+      "refresh": "30s",
+      "schemaVersion": 38,
+      "tags": ["shower", "flyio", "apm"],
+      "templating": { "list": [] },
+      "time": { "from": "now-6h", "to": "now" },
+      "timepicker": {},
+      "timezone": "",
+      "title": "Shower APM",
+      "uid": "shower-apm",
+      "version": 1,
+      "weekStart": ""
+    }

argocd/manifests/grafana-config/kustomization.yaml

@@ -22,6 +22,7 @@ resources:
   - dashboards/configmap-transmission.yaml
   - dashboards/configmap-cv-apm.yaml
   - dashboards/configmap-docs-apm.yaml
+  - dashboards/configmap-shower-apm.yaml
   - dashboards/configmap-flyio.yaml
   - dashboards/configmap-sifaka-disks.yaml
   - dashboards/configmap-forgejo.yaml

argocd/manifests/grafana/datasources.yaml

@@ -63,5 +63,7 @@ datasources:
     password: $TESLAMATE_DB_PASSWORD
   type: postgres
   uid: TeslaMate
-  url: blumeops-pg-rw.databases.svc.cluster.local:5432
+  # teslamate DB migrated to ringtail blumeops-pg (wave-1); reached via the
+  # Caddy L4 route on indri (pg.ops.eblu.me:5434 -> blumeops-pg-ringtail).
+  url: pg.ops.eblu.me:5434
   user: teslamate

argocd/manifests/grafana/deployment.yaml

@@ -14,7 +14,9 @@ spec:
       app.kubernetes.io/name: grafana
       app.kubernetes.io/instance: grafana
   strategy:
-    type: RollingUpdate
+    # RWO PVC for SQLite + Bleve index — RollingUpdate spawns the new pod
+    # before the old one terminates, and it crashloops on the index lock.
+    type: Recreate
   template:
     metadata:
       labels:

argocd/manifests/homepage/kustomization.yaml

@@ -17,7 +17,7 @@ resources:
 
 images:
   - name: registry.ops.eblu.me/blumeops/homepage
-    newTag: v1.11.0-e375859
+    newTag: v1.11.0-678f26b-nix
 
 configMapGenerator:
   - name: homepage-config

argocd/manifests/homepage/services.yaml

@@ -1,3 +1,6 @@
+# Homepage runs on ringtail (k3s) — its k8s autodiscovery only sees ringtail
+# Ingresses (frigate→NVR, authentik, ntfy, ollama). Services that live on
+# minikube (and indri-native) need explicit static entries here.
 - Host Services:
     - Forgejo:
         href: https://forge.eblu.me
@@ -57,10 +60,6 @@
     #       type: caddy
     #       url: http://indri.tail8d86e.ts.net:2019
 - Home:
-    - NVR:
-        href: https://nvr.ops.eblu.me
-        icon: frigate.png
-        description: Network video recorder
     - Jellyfin:
         href: https://jellyfin.ops.eblu.me
         icon: jellyfin
@@ -72,15 +71,49 @@
           enableBlocks: true
           enableNowPlaying: false
           fields: ["movies", "series", "episodes"]
+    - DJ:
+        href: https://dj.ops.eblu.me
+        icon: navidrome.png
+        description: Music streaming server
+        widget:
+          type: navidrome
+          url: https://dj.ops.eblu.me
+          user: "{{HOMEPAGE_VAR_NAVIDROME_USER}}"
+          token: "{{HOMEPAGE_VAR_NAVIDROME_TOKEN}}"
+          salt: "{{HOMEPAGE_VAR_NAVIDROME_SALT}}"
+- Content:
+    - Kiwix:
+        href: https://kiwix.ops.eblu.me
+        icon: kiwix.png
+        description: Offline Wikipedia
+    - Miniflux:
+        href: https://feed.ops.eblu.me
+        icon: miniflux.png
+        description: RSS reader
+        widget:
+          type: miniflux
+          url: https://feed.ops.eblu.me
+          key: "{{HOMEPAGE_VAR_MINIFLUX_API_KEY}}"
+          fields: ["unread"]
 - Infrastructure:
-    - Authentik:
-        href: https://authentik.ops.eblu.me
-        icon: authentik
-        description: Identity provider
-    - Ntfy:
-        href: https://ntfy.ops.eblu.me
-        icon: ntfy.png
-        description: Push notifications
+    - ArgoCD:
+        href: https://argocd.ops.eblu.me
+        icon: argo-cd.png
+        description: GitOps CD
+    - Grafana:
+        href: https://grafana.ops.eblu.me
+        icon: grafana.png
+        description: Metrics dashboards
+        widget:
+          type: grafana
+          url: https://grafana.ops.eblu.me
+          username: "{{HOMEPAGE_VAR_GRAFANA_USERNAME}}"
+          password: "{{HOMEPAGE_VAR_GRAFANA_PASSWORD}}"
+          fields: ["dashboards", "totalalerts", "alertstriggered"]
+    - Prometheus:
+        href: https://prometheus.ops.eblu.me
+        icon: prometheus.png
+        description: Metrics storage
 - Services:
     # CV and Docs were previously auto-discovered from k8s Ingresses; after
     # the indri-native migration ([[cv-on-indri]], [[docs-on-indri]]) there
@@ -93,3 +126,7 @@
         href: https://docs.eblu.me
         icon: mdi-book-open-page-variant
         description: BlumeOps Documentation
+    - Transmission:
+        href: https://torrent.ops.eblu.me
+        icon: transmission.png
+        description: Torrent client

argocd/manifests/immich-ringtail/deployment-ml.yaml

@@ -16,11 +16,16 @@ spec:
         app: immich
         component: machine-learning
     spec:
+      runtimeClassName: nvidia
       securityContext:
         seccompProfile:
           type: RuntimeDefault
       containers:
         - name: machine-learning
+          # ringtail uses the -cuda tag (set in kustomization.yaml)
+          # to take advantage of the RTX 4080 via the nvidia
+          # device plugin. Time-slicing is configured for 4 replicas
+          # so frigate + ollama + this pod can share.
           image: ghcr.io/immich-app/immich-machine-learning:kustomized
           ports:
             - name: http
@@ -57,6 +62,7 @@ spec:
               cpu: "100m"
             limits:
               memory: "4Gi"
+              nvidia.com/gpu: "1"
       volumes:
         - name: cache
           persistentVolumeClaim:

argocd/manifests/immich-ringtail/ingress-tailscale.yaml

@@ -1,6 +1,9 @@
-# Tailscale Ingress for Immich
-# Exposes Immich at photos.tail8d86e.ts.net
-# Caddy will proxy photos.ops.eblu.me to this endpoint
+# Tailscale ProxyGroup Ingress for Immich on ringtail.
+#
+# Production hostname: photos.tail8d86e.ts.net
+# (during the cutover window this was photos-ringtail; the minikube
+# ingress was torn down before this was renamed to photos to avoid
+# the Tailscale device-name collision.)
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
@@ -16,12 +19,6 @@ metadata:
     gethomepage.dev/description: "Photo management"
     gethomepage.dev/href: "https://photos.ops.eblu.me"
     gethomepage.dev/pod-selector: "app=immich,component=server"
-    # TODO: Add Immich widget - requires API key from Account Settings > API Keys
-    # See: https://gethomepage.dev/widgets/services/immich/
-    # gethomepage.dev/widget.type: "immich"
-    # gethomepage.dev/widget.url: "https://photos.ops.eblu.me"
-    # gethomepage.dev/widget.key: "{{HOMEPAGE_VAR_IMMICH_API_KEY}}"
-    # gethomepage.dev/widget.version: "2"
 spec:
   ingressClassName: tailscale
   rules:

argocd/manifests/immich-ringtail/kustomization.yaml

@@ -1,7 +1,8 @@
----
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
+
 namespace: immich
+
 resources:
   - deployment-server.yaml
   - deployment-ml.yaml
@@ -13,10 +14,16 @@ resources:
   - pv-nfs.yaml
   - pvc.yaml
   - ingress-tailscale.yaml
+
 images:
   - name: ghcr.io/immich-app/immich-server
     newTag: v2.6.3
   - name: ghcr.io/immich-app/immich-machine-learning
-    newTag: v2.6.3
+    # CUDA variant of the same release — ringtail has an RTX 4080
+    newTag: v2.6.3-cuda
+  # amd64 valkey built via nix on the ringtail nix-container-builder
+  # (see containers/valkey/default.nix). The Alpine container.py build
+  # is arm64-only and serves paperless on indri.
   - name: docker.io/valkey/valkey
-    newTag: "8.1-alpine"
+    newName: registry.ops.eblu.me/blumeops/valkey
+    newTag: v8.1.7-ecded30-nix

argocd/manifests/immich-ringtail/pv-nfs.yaml

@@ -0,0 +1,29 @@
+# NFS PersistentVolume for Immich photo library on ringtail k3s.
+#
+# Mirror of argocd/manifests/immich/pv-nfs.yaml (minikube) but with
+# a distinct name (minikube and ringtail are separate clusters, so PV
+# names don't collide cluster-side, but using the same name in two
+# manifests is confusing).
+#
+# The sifaka NFS export for /volume1/photos already permits
+# 192.168.1.0/24 + 100.64.0.0/10. Ringtail's wired IP (192.168.1.21)
+# falls in the first CIDR, so no DSM rule changes are needed.
+#
+# Verified 2026-05-13: ringtail pod can read existing dirs, write
+# new files, and delete them. DNS resolves sifaka to 192.168.1.203
+# (LAN), so NFS traffic stays off the tailnet — avoids the known
+# sifaka-tailscale-userspace bite.
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: immich-library-nfs-pv-ringtail
+spec:
+  capacity:
+    storage: 2Ti
+  accessModes:
+    - ReadWriteMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ""
+  nfs:
+    server: sifaka
+    path: /volume1/photos

argocd/manifests/immich-ringtail/pvc.yaml

@@ -1,5 +1,5 @@
-# PersistentVolumeClaim for Immich photo library
-# Binds to the NFS PV for sifaka:/volume1/photos
+# PersistentVolumeClaim for Immich photo library on ringtail.
+# Binds to immich-library-nfs-pv-ringtail (sifaka:/volume1/photos).
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -9,7 +9,7 @@ spec:
   accessModes:
     - ReadWriteMany
   storageClassName: ""
-  volumeName: immich-library-nfs-pv
+  volumeName: immich-library-nfs-pv-ringtail
   resources:
     requests:
       storage: 2Ti

argocd/manifests/immich/README.md

@@ -1,115 +0,0 @@
-# Immich
-
-Self-hosted photo and video management solution with AI-powered search and face recognition.
-
-## Prerequisites
-
-1. **NFS Share**: Create `/volume1/photos` on sifaka with NFS permissions for indri
-2. **PostgreSQL**: The `immich-pg` cluster (with pgvecto.rs) must be healthy
-3. **Secrets**: Create the database password secret
-
-## Deployment Order
-
-1. Sync `blumeops-pg` (to get CloudNativePG operator if not already running)
-2. Wait for `immich-pg` cluster to be healthy
-3. Create secrets (see below)
-4. Sync `immich` (deploys all resources: storage, services, deployments)
-5. Run `mise run provision-indri -- --tags caddy` to update Caddy config
-
-## Components
-
-| Component | Deployment | Service | Port |
-|-----------|------------|---------|------|
-| Server (web/API) | `immich-server` | `immich-server` | 2283 |
-| Machine Learning | `immich-machine-learning` | `immich-machine-learning` | 3003 |
-| Valkey (Redis) | `immich-valkey` | `immich-valkey` | 6379 |
-
-## Secret Setup
-
-The `immich-db` secret contains the database password, which is auto-generated by CloudNativePG
-in the `immich-pg-app` secret. To create or regenerate the secret:
-
-```bash
-# Create namespace if needed
-kubectl --context=minikube-indri create namespace immich
-
-# Copy password from CNPG secret to immich namespace
-kubectl --context=minikube-indri create secret generic immich-db -n immich \
-  --from-literal=password="$(kubectl --context=minikube-indri -n databases get secret immich-pg-app -o jsonpath='{.data.password}' | base64 -d)"
-```
-
-Note: This secret is not managed by ExternalSecrets since the source of truth is the CNPG-generated secret.
-
-## Access
-
-- **URL**: https://photos.ops.eblu.me (after Caddy is updated)
-- **Tailscale**: https://photos.tail8d86e.ts.net (direct)
-
-## First-Time Setup
-
-1. Navigate to https://photos.ops.eblu.me
-2. Create an admin account
-3. Configure external library (optional - for importing existing photos)
-
-## External Library (iCloud Photos)
-
-To import existing photos from iCloud sync on indri:
-
-1. In Immich Admin > External Libraries, create a new library
-2. Set the import path to the location where iCloud photos sync
-3. Configure scan schedule or trigger manual scan
-
-## Architecture
-
-```
-┌─────────────────┐     ┌─────────────────┐
-│  immich-server  │────▶│  immich-pg      │
-│  (web/api)      │     │  (PostgreSQL    │
-└────────┬────────┘     │   + pgvecto.rs) │
-         │              └─────────────────┘
-         │
-┌────────▼────────┐     ┌─────────────────┐
-│  immich-ml      │     │  valkey         │
-│  (ML inference) │     │  (Redis cache)  │
-└─────────────────┘     └─────────────────┘
-         │
-┌────────▼────────┐
-│  sifaka NFS     │
-│  /volume1/photos│
-└─────────────────┘
-```
-
-## Version Management
-
-Image versions are controlled via `kustomization.yaml`:
-
-```yaml
-images:
-  - name: ghcr.io/immich-app/immich-server
-    newTag: v2.6.3
-  - name: ghcr.io/immich-app/immich-machine-learning
-    newTag: v2.6.3
-  - name: docker.io/valkey/valkey
-    newTag: "8.1-alpine"
-```
-
-To upgrade, update `newTag` values and sync via ArgoCD.
-
-## Troubleshooting
-
-```bash
-# Check pods
-kubectl --context=minikube-indri -n immich get pods
-
-# Check immich-pg cluster
-kubectl --context=minikube-indri -n databases get cluster immich-pg
-
-# View server logs
-kubectl --context=minikube-indri -n immich logs -l app=immich,component=server
-
-# View ML logs
-kubectl --context=minikube-indri -n immich logs -l app=immich,component=machine-learning
-
-# Check PVC binding
-kubectl --context=minikube-indri -n immich get pvc
-```

argocd/manifests/immich/pv-nfs.yaml

@@ -1,22 +0,0 @@
-# NFS PersistentVolume for Immich photo library
-# Requires: NFS share on sifaka at /volume1/photos with NFS permissions for indri
-#
-# To create on Synology:
-# 1. Control Panel > Shared Folder > Create
-# 2. Name: photos, Location: Volume 1
-# 3. Control Panel > File Services > NFS > NFS Rules
-# 4. Add rule for "photos" share: Hostname=indri, Privilege=Read/Write, Squash=No mapping
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: immich-library-nfs-pv
-spec:
-  capacity:
-    storage: 2Ti
-  accessModes:
-    - ReadWriteMany
-  persistentVolumeReclaimPolicy: Retain
-  storageClassName: ""
-  nfs:
-    server: sifaka
-    path: /volume1/photos

argocd/manifests/mealie-ringtail/deployment.yaml

@@ -1,3 +1,9 @@
+# Mealie on ringtail k3s — Nix image.
+#
+# Single gunicorn process (the Nix image's default `mealie-run` entrypoint
+# runs init_db then gunicorn), serving the prebuilt frontend. DB is SQLite
+# on the mealie-data PVC; its contents are copied from the minikube PVC at
+# cutover. See [[migrate-wave1-ringtail]].
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -5,6 +11,8 @@ metadata:
   namespace: mealie
 spec:
   replicas: 1
+  strategy:
+    type: Recreate
   selector:
     matchLabels:
       app: mealie

argocd/manifests/mealie-ringtail/kustomization.yaml

@@ -12,4 +12,4 @@ resources:
 
 images:
   - name: registry.ops.eblu.me/blumeops/mealie
-    newTag: v3.12.0-613f05d
+    newTag: v3.16.0-e0057b4-nix

argocd/manifests/mealie-ringtail/pvc.yaml

@@ -1,4 +1,5 @@
----
+# SQLite data volume for Mealie on ringtail. Contents copied from the
+# minikube mealie-data PVC at cutover (recipes, meal plans, uploaded media).
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -7,7 +8,7 @@ metadata:
 spec:
   accessModes:
     - ReadWriteOnce
-  storageClassName: standard
+  storageClassName: local-path
   resources:
     requests:
       storage: 2Gi

argocd/manifests/nvidia-device-plugin/kustomization.yaml

@@ -10,4 +10,4 @@ resources:
 
 images:
   - name: nvcr.io/nvidia/k8s-device-plugin
-    newTag: v0.19.0
+    newTag: v0.19.2

argocd/manifests/nvidia-device-plugin/time-slicing-config.yaml

@@ -11,4 +11,4 @@ data:
       timeSlicing:
         resources:
           - name: nvidia.com/gpu
-            replicas: 2
+            replicas: 4

argocd/manifests/paperless-ringtail/deployment.yaml

@@ -1,3 +1,17 @@
+# Paperless-ngx on ringtail k3s — Nix image, multi-process.
+#
+# The upstream s6 image ran web + worker + scheduler + consumer (and DB
+# migrations) in one container. The Nix image (containers/paperless/
+# default.nix) ships the binaries but no supervisor, so we run those as
+# four containers in one pod, sharing the local data/consume dirs
+# (emptyDir) and the NFS media volume; redis is colocated so
+# PAPERLESS_REDIS=localhost works for all. A migrate initContainer runs
+# DB migrations once before the app containers start.
+#
+# DB points in-cluster at the ringtail blumeops-pg (was pg.ops.eblu.me on
+# indri). PAPERLESS_{DATA_DIR,MEDIA_ROOT,CONSUMPTION_DIR} are set
+# explicitly because the Nix package does not default to the upstream
+# /usr/src/paperless paths.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -5,6 +19,8 @@ metadata:
   namespace: paperless
 spec:
   replicas: 1
+  strategy:
+    type: Recreate
   selector:
     matchLabels:
       app: paperless
@@ -16,27 +32,38 @@ spec:
       securityContext:
         seccompProfile:
           type: RuntimeDefault
-      containers:
-        - name: paperless
-          image: registry.ops.eblu.me/blumeops/paperless:kustomized
+      initContainers:
+        # redis as a native sidecar (restartPolicy: Always): starts before
+        # the migrate init and stays running for the app containers, so all
+        # of them reach PAPERLESS_REDIS=localhost:6379.
+        - name: redis
+          image: docker.io/library/redis:kustomized
+          restartPolicy: Always
           ports:
-            - containerPort: 8000
-              name: http
-          env:
+            - containerPort: 6379
+          volumeMounts:
+            - name: redis-data
+              mountPath: /data
+          resources:
+            requests:
+              memory: "32Mi"
+              cpu: "10m"
+            limits:
+              memory: "128Mi"
+        - name: migrate
+          image: registry.ops.eblu.me/blumeops/paperless:kustomized
+          command: ["paperless-ngx", "migrate", "--no-input"]
+          env: &paperless-env
             - name: PAPERLESS_URL
               value: "https://paperless.ops.eblu.me"
             - name: PAPERLESS_REDIS
               value: "redis://localhost:6379"
             - name: PAPERLESS_DBHOST
-              value: "pg.ops.eblu.me"
+              value: "blumeops-pg-rw.databases.svc.cluster.local"
             - name: PAPERLESS_DBPORT
               value: "5432"
             - name: PAPERLESS_DBNAME
               value: "paperless"
-            # Explicit port to override k8s-injected PAPERLESS_PORT env var
-            # (k8s sets PAPERLESS_PORT=tcp://... for a service named 'paperless')
-            - name: PAPERLESS_PORT
-              value: "8000"
             - name: PAPERLESS_DBUSER
               value: "paperless"
             - name: PAPERLESS_DBPASS
@@ -44,6 +71,16 @@ spec:
                 secretKeyRef:
                   name: paperless-secrets
                   key: db-password
+            # Explicit port to override the k8s-injected PAPERLESS_PORT
+            # (service named 'paperless' would set PAPERLESS_PORT=tcp://...)
+            - name: PAPERLESS_PORT
+              value: "8000"
+            - name: PAPERLESS_DATA_DIR
+              value: "/usr/src/paperless/data"
+            - name: PAPERLESS_MEDIA_ROOT
+              value: "/usr/src/paperless/media"
+            - name: PAPERLESS_CONSUMPTION_DIR
+              value: "/usr/src/paperless/consume"
             - name: PAPERLESS_SECRET_KEY
               valueFrom:
                 secretKeyRef:
@@ -55,7 +92,6 @@ spec:
               value: "eng"
             - name: PAPERLESS_TASK_WORKERS
               value: "1"
-            # Admin account (created on first startup)
             - name: PAPERLESS_ADMIN_USER
               value: "eblume"
             - name: PAPERLESS_ADMIN_PASSWORD
@@ -65,8 +101,6 @@ spec:
                   key: admin-password
             - name: PAPERLESS_ADMIN_MAIL
               value: "blume.erich@gmail.com"
-            # OIDC via Authentik
-            # Full JSON blob pulled from 1Password (includes client secret)
             - name: PAPERLESS_APPS
               value: "allauth.socialaccount.providers.openid_connect"
             - name: PAPERLESS_SOCIALACCOUNT_PROVIDERS
@@ -82,19 +116,27 @@ spec:
               value: "false"
             - name: PAPERLESS_REDIRECT_LOGIN_TO_SSO
               value: "false"
-          volumeMounts:
+          volumeMounts: &paperless-mounts
             - name: data
               mountPath: /usr/src/paperless/data
             - name: media
               mountPath: /usr/src/paperless/media
             - name: consume
               mountPath: /usr/src/paperless/consume
+      containers:
+        - name: web
+          image: registry.ops.eblu.me/blumeops/paperless:kustomized
+          ports:
+            - containerPort: 8000
+              name: http
+          env: *paperless-env
+          volumeMounts: *paperless-mounts
           resources:
             requests:
               memory: "256Mi"
               cpu: "100m"
             limits:
-              memory: "2Gi"
+              memory: "1Gi"
               cpu: "1000m"
           livenessProbe:
             httpGet:
@@ -109,16 +151,42 @@ spec:
             initialDelaySeconds: 30
             periodSeconds: 10
 
-        - name: redis
-          image: docker.io/library/redis:kustomized
-          ports:
-            - containerPort: 6379
+        - name: worker
+          image: registry.ops.eblu.me/blumeops/paperless:kustomized
+          command: ["celery", "--app", "paperless", "worker", "--loglevel", "INFO"]
+          env: *paperless-env
+          volumeMounts: *paperless-mounts
           resources:
             requests:
-              memory: "32Mi"
-              cpu: "10m"
+              memory: "256Mi"
+              cpu: "100m"
             limits:
+              memory: "1Gi"
+              cpu: "1000m"
+
+        - name: beat
+          image: registry.ops.eblu.me/blumeops/paperless:kustomized
+          command: ["celery", "--app", "paperless", "beat", "--loglevel", "INFO"]
+          env: *paperless-env
+          volumeMounts: *paperless-mounts
+          resources:
+            requests:
+              memory: "64Mi"
+              cpu: "20m"
+            limits:
+              memory: "256Mi"
+
+        - name: consumer
+          image: registry.ops.eblu.me/blumeops/paperless:kustomized
+          command: ["paperless-ngx", "document_consumer"]
+          env: *paperless-env
+          volumeMounts: *paperless-mounts
+          resources:
+            requests:
               memory: "128Mi"
+              cpu: "50m"
+            limits:
+              memory: "512Mi"
 
       volumes:
         - name: data
@@ -128,3 +196,6 @@ spec:
             claimName: paperless-media
         - name: consume
           emptyDir: {}
+        - name: redis-data
+          emptyDir:
+            sizeLimit: 1Gi

argocd/manifests/paperless-ringtail/kustomization.yaml

@@ -0,0 +1,21 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: paperless
+
+resources:
+  - deployment.yaml
+  - service.yaml
+  - pv-nfs.yaml
+  - pvc.yaml
+  - ingress-tailscale.yaml
+  - external-secret.yaml
+
+images:
+  - name: registry.ops.eblu.me/blumeops/paperless
+    newTag: v2.20.15-fcac8e5-nix
+  # amd64 valkey built via nix (the v8.1.7-ecded30 tag without -nix is the
+  # arm64 Alpine build for indri and fails on ringtail with exec format error)
+  - name: docker.io/library/redis
+    newName: registry.ops.eblu.me/blumeops/valkey
+    newTag: v8.1.7-ecded30-nix

argocd/manifests/paperless-ringtail/pv-nfs.yaml

@@ -0,0 +1,22 @@
+# NFS PersistentVolume for the Paperless document library, mounted from
+# ringtail. Same sifaka export (/volume1/paperless) as the minikube PV,
+# but a distinct PV name so both clusters can declare it during the
+# parallel-run before cutover.
+#
+# Prerequisite: sifaka must have an NFS rule granting ringtail Read/Write
+# (Squash=No mapping) on the paperless share — the same step done for
+# immich. See [[sifaka-nfs-from-ringtail]].
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: paperless-media-nfs-pv-ringtail
+spec:
+  capacity:
+    storage: 500Gi
+  accessModes:
+    - ReadWriteMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ""
+  nfs:
+    server: sifaka
+    path: /volume1/paperless

argocd/manifests/paperless-ringtail/pvc.yaml

@@ -1,5 +1,5 @@
-# PersistentVolumeClaim for Paperless document library
-# Binds to the NFS PV for sifaka:/volume1/paperless
+# PersistentVolumeClaim for the Paperless document library on ringtail.
+# Binds the NFS PV for sifaka:/volume1/paperless.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -9,7 +9,7 @@ spec:
   accessModes:
     - ReadWriteMany
   storageClassName: ""
-  volumeName: paperless-media-nfs-pv
+  volumeName: paperless-media-nfs-pv-ringtail
   resources:
     requests:
       storage: 500Gi

argocd/manifests/paperless/kustomization.yaml

@@ -1,22 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-
-namespace: paperless
-
-resources:
-  - deployment.yaml
-  - service.yaml
-  - pv-nfs.yaml
-  - pvc.yaml
-  - ingress-tailscale.yaml
-  - external-secret.yaml
-
-images:
-  - name: registry.ops.eblu.me/blumeops/paperless
-    newTag: v2.20.13-07f52e9
-  # TODO(DR-2026-04): authentik-redis is amd64-only (nix-built on ringtail).
-  # Was running under QEMU emulation before. Switched to upstream valkey
-  # during DR recovery. Build a multi-arch blumeops/redis or keep upstream.
-  - name: docker.io/library/redis
-    newName: docker.io/valkey/valkey
-    newTag: "8.1-alpine"

argocd/manifests/paperless/pv-nfs.yaml

@@ -1,22 +0,0 @@
-# NFS PersistentVolume for Paperless document library
-# Requires: NFS share on sifaka at /volume1/paperless with NFS permissions for indri
-#
-# To create on Synology:
-# 1. Control Panel > Shared Folder > Create
-# 2. Name: paperless, Location: Volume 1
-# 3. Control Panel > File Services > NFS > NFS Rules
-# 4. Add rule for "paperless" share: Hostname=indri, Privilege=Read/Write, Squash=No mapping
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: paperless-media-nfs-pv
-spec:
-  capacity:
-    storage: 500Gi
-  accessModes:
-    - ReadWriteMany
-  persistentVolumeReclaimPolicy: Retain
-  storageClassName: ""
-  nfs:
-    server: sifaka
-    path: /volume1/paperless