Erich Blume 947e4310c3 C2: migrate immich from minikube to ringtail (mikado chain) (#356 )

## Summary

C2 Mikado chain to move the entire Immich stack (server, ML, valkey,
postgres) off `minikube-indri` and onto `k3s-ringtail`. Immich is the
largest single tenant on minikube (~1.5 GiB resident) and minikube is
currently memory-saturated (97% RAM, swapping). This is the first
concrete chain in the broader indri-k8s decommission effort.

This PR contains the planning layer only — 7 cards (1 goal + 6
prerequisites). Implementation cycles follow per the Mikado Branch
Invariant.

## Goal end-state

- Immich `server`, `machine-learning`, `valkey` on ringtail.
- ML pod uses ringtail's RTX 4080 (performance win — currently
  CPU-only).
- CNPG `immich-pg` (PG17 + VectorChord) runs on ringtail.
- Library still on sifaka NFS — ringtail mounts the same path.
- `photos.ops.eblu.me` reroutes through Caddy → ringtail ingress.
- Minikube `immich` and `immich-pg` are removed.

## Cards

| Card | Depends on |
|---|---|
| `migrate-immich-to-ringtail` (goal) | all six below |
| `cnpg-on-ringtail` | — |
| `immich-pg-on-ringtail` | cnpg-on-ringtail |
| `immich-pg-data-migration` | immich-pg-on-ringtail |
| `sifaka-nfs-from-ringtail` | — |
| `immich-app-on-ringtail` | immich-pg-on-ringtail, sifaka-nfs-from-ringtail |
| `immich-cutover-and-decommission` | immich-pg-data-migration, immich-app-on-ringtail |

## Key constraints

- **No data loss.** Downtime is acceptable; data loss is not. Two
  surfaces matter: postgres (ML embeddings, face data — slow to
  re-derive) and the library files (don't move, but NFS access from
  ringtail must be verified).
- **Migration method:** Option A is a CNPG `externalCluster`
  basebackup → promote. Option B is `pg_dump`/`pg_restore` as a
  documented fallback. Either way, dry-run against a scratch
  cluster first.
- **Why pg moves too** (not cross-cluster): keeping pg on minikube
  would block the whole decommission, and Immich is chatty with pg
  so tailnet round-trips would hurt.

## Test plan

- [ ] Plan review — does the dependency graph make sense?
- [ ] `mise run docs-mikado migrate-immich-to-ringtail` shows the
      chain correctly.
- [ ] Per-card implementation cycles land separately (commit
      convention enforced by hook).

Reviewed-on: #356

2026-05-13 16:46:17 -07:00

2.8 KiB

Raw Blame History

title

modified

last-reviewed

Immich Postgres Cluster on Ringtail

Stand up a fresh immich-pg CNPG Cluster on ringtail, ready to receive data. No data import yet — that's immich-pg-data-migration.

What to do

Create argocd/manifests/databases-ringtail/ (or pick another namespace name — verify what other ringtail pg clusters will use; if none yet, databases is fine).
Port these from the minikube side:
- immich-pg.yaml — CNPG Cluster CR. Same image (ghcr.io/tensorchord/cloudnative-vectorchord:17-0.5.0), same extensions, same managed borgmatic role. Bump storage.size if the minikube 10 GiB looks tight (check actual usage first). storageClass: local-path on ringtail (default).
- external-secret-immich-borgmatic.yaml — same 1Password item, same field, but referencing the ringtail ClusterSecretStore (onepassword-blumeops already exists per the external-secrets-ringtail app).
- Service for in-cluster access (the operator creates immich-pg-rw etc. automatically; verify the app deployment uses those names).
- A Tailscale Service if we want backups to keep working via the same hostname during the transition — see "Borgmatic" below.
New ArgoCD app argocd/apps/databases-ringtail.yaml pointing at the new path, destination ringtail.

Verification

Cluster reaches Ready.
borgmatic role exists, rolcanlogin=t, and is a member of pg_read_all_data (via managed.roles[].inRoles).
ExternalSecret immich-pg-borgmatic syncs from 1Password (Ready: True) and the rendered Secret has username=borgmatic.
The vchord, vector, cube, earthdistance extensions show installed in the postgres database (\dx from psql -U postgres). They are NOT installed in the immich database at this point — postInitSQL in CNPG's initdb block runs against the postgres superuser database. The Immich app itself creates the extensions in its own immich database at startup; do not be alarmed by their absence pre-immich-deploy. The vchord.so library is preloaded via shared_preload_libraries regardless, so CREATE EXTENSION at app startup just registers it in the right database.

Borgmatic implications

borgmatic.cfg on indri targets immich-pg-tailscale over the tailnet. During migration both clusters will exist briefly. Decide upfront: backup the source pg until cutover, then flip borgmatic to the ringtail Tailscale service. Document the flip in immich-cutover-and-decommission.

Out of scope

Importing data. That is immich-pg-data-migration, which may drive a reset on this card if the migration approach (e.g. CNPG externalCluster bootstrap) requires changes to this Cluster CR.

2.8 KiB Raw Blame History