Erich Blume 947e4310c3 C2: migrate immich from minikube to ringtail (mikado chain) (#356 )

## Summary

C2 Mikado chain to move the entire Immich stack (server, ML, valkey,
postgres) off `minikube-indri` and onto `k3s-ringtail`. Immich is the
largest single tenant on minikube (~1.5 GiB resident) and minikube is
currently memory-saturated (97% RAM, swapping). This is the first
concrete chain in the broader indri-k8s decommission effort.

This PR contains the planning layer only — 7 cards (1 goal + 6
prerequisites). Implementation cycles follow per the Mikado Branch
Invariant.

## Goal end-state

- Immich `server`, `machine-learning`, `valkey` on ringtail.
- ML pod uses ringtail's RTX 4080 (performance win — currently
  CPU-only).
- CNPG `immich-pg` (PG17 + VectorChord) runs on ringtail.
- Library still on sifaka NFS — ringtail mounts the same path.
- `photos.ops.eblu.me` reroutes through Caddy → ringtail ingress.
- Minikube `immich` and `immich-pg` are removed.

## Cards

| Card | Depends on |
|---|---|
| `migrate-immich-to-ringtail` (goal) | all six below |
| `cnpg-on-ringtail` | — |
| `immich-pg-on-ringtail` | cnpg-on-ringtail |
| `immich-pg-data-migration` | immich-pg-on-ringtail |
| `sifaka-nfs-from-ringtail` | — |
| `immich-app-on-ringtail` | immich-pg-on-ringtail, sifaka-nfs-from-ringtail |
| `immich-cutover-and-decommission` | immich-pg-data-migration, immich-app-on-ringtail |

## Key constraints

- **No data loss.** Downtime is acceptable; data loss is not. Two
  surfaces matter: postgres (ML embeddings, face data — slow to
  re-derive) and the library files (don't move, but NFS access from
  ringtail must be verified).
- **Migration method:** Option A is a CNPG `externalCluster`
  basebackup → promote. Option B is `pg_dump`/`pg_restore` as a
  documented fallback. Either way, dry-run against a scratch
  cluster first.
- **Why pg moves too** (not cross-cluster): keeping pg on minikube
  would block the whole decommission, and Immich is chatty with pg
  so tailnet round-trips would hurt.

## Test plan

- [ ] Plan review — does the dependency graph make sense?
- [ ] `mise run docs-mikado migrate-immich-to-ringtail` shows the
      chain correctly.
- [ ] Per-card implementation cycles land separately (commit
      convention enforced by hook).

Reviewed-on: #356

2026-05-13 16:46:17 -07:00

2.5 KiB

Raw Blame History

title

modified

last-reviewed

Sifaka NFS Photos from Ringtail

The Immich library lives at sifaka:/volume1/photos and is mounted into the pod via an NFS PV (see argocd/manifests/immich/pv-nfs.yaml). That PV is currently scoped to indri. We need ringtail to mount the same path with the same RWX semantics, without breaking the existing indri mount during the transition.

What to verify / do

Check sifaka DSM NFS rules for the photos share. Per shower-on-ringtail#NFS + SMB share on sifaka convention, rules use 192.168.1.0/24 + 100.64.0.0/10 with all_squash/Map all users to admin. The existing rule may already cover ringtail (it's on 192.168.1.21 per the recent static-IP pin). If so this card is a verification card.
If the rule is locked to indri's IP: add an entry for ringtail (192.168.1.21) or widen to the subnet pattern above.
Test mount from a ringtail debug pod (busybox or alpine with nfs-utils) against the photos share. Read a file. Write a temp file. Delete it.
Watch for the known sifaka NFS-over-Tailscale gotcha: sifaka's Tailscale must be in TUN mode (not userspace) for NFS to work reliably over the tailnet. The NFS path here goes over the LAN (not tailnet), so this shouldn't bite, but worth confirming the NFS traffic is on 192.168.1.x not 100.x.

PV + PVC on ringtail

New pv-nfs.yaml mirroring the minikube one (name can be shared if the PV is cluster-scoped — but PVs are per-cluster, so just duplicate). Same server: sifaka, same path, same accessModes: [ReadWriteMany], persistentVolumeReclaimPolicy: Retain.
New pvc.yaml in the ringtail immich namespace bound to it.
The minikube PVC stays bound and active until cutover — both clusters can have the share NFS-mounted simultaneously (NFS RWX permits this). Immich itself must not be running on both sides at once.

Verification

A pod on ringtail can ls /mnt/photos/ and see the same files as the indri pod.
File written from ringtail pod is visible from indri pod and vice versa (proves there's no caching surprise).

Out of scope

Migrating photo files. Nothing moves; this is just adding a second NFS client.
The pvc-ml-cache.yaml PVC (a separate ML model cache). That's not on NFS — it's a regular PVC. Recreated empty on ringtail in immich-app-on-ringtail; the first ML pod boot will repopulate it.

2.5 KiB Raw Blame History