Erich Blume 947e4310c3 C2: migrate immich from minikube to ringtail (mikado chain) (#356 )

## Summary

C2 Mikado chain to move the entire Immich stack (server, ML, valkey,
postgres) off `minikube-indri` and onto `k3s-ringtail`. Immich is the
largest single tenant on minikube (~1.5 GiB resident) and minikube is
currently memory-saturated (97% RAM, swapping). This is the first
concrete chain in the broader indri-k8s decommission effort.

This PR contains the planning layer only — 7 cards (1 goal + 6
prerequisites). Implementation cycles follow per the Mikado Branch
Invariant.

## Goal end-state

- Immich `server`, `machine-learning`, `valkey` on ringtail.
- ML pod uses ringtail's RTX 4080 (performance win — currently
  CPU-only).
- CNPG `immich-pg` (PG17 + VectorChord) runs on ringtail.
- Library still on sifaka NFS — ringtail mounts the same path.
- `photos.ops.eblu.me` reroutes through Caddy → ringtail ingress.
- Minikube `immich` and `immich-pg` are removed.

## Cards

| Card | Depends on |
|---|---|
| `migrate-immich-to-ringtail` (goal) | all six below |
| `cnpg-on-ringtail` | — |
| `immich-pg-on-ringtail` | cnpg-on-ringtail |
| `immich-pg-data-migration` | immich-pg-on-ringtail |
| `sifaka-nfs-from-ringtail` | — |
| `immich-app-on-ringtail` | immich-pg-on-ringtail, sifaka-nfs-from-ringtail |
| `immich-cutover-and-decommission` | immich-pg-data-migration, immich-app-on-ringtail |

## Key constraints

- **No data loss.** Downtime is acceptable; data loss is not. Two
  surfaces matter: postgres (ML embeddings, face data — slow to
  re-derive) and the library files (don't move, but NFS access from
  ringtail must be verified).
- **Migration method:** Option A is a CNPG `externalCluster`
  basebackup → promote. Option B is `pg_dump`/`pg_restore` as a
  documented fallback. Either way, dry-run against a scratch
  cluster first.
- **Why pg moves too** (not cross-cluster): keeping pg on minikube
  would block the whole decommission, and Immich is chatty with pg
  so tailnet round-trips would hurt.

## Test plan

- [ ] Plan review — does the dependency graph make sense?
- [ ] `mise run docs-mikado migrate-immich-to-ringtail` shows the
      chain correctly.
- [ ] Per-card implementation cycles land separately (commit
      convention enforced by hook).

Reviewed-on: #356

2026-05-13 16:46:17 -07:00

1.5 KiB

Raw Permalink Blame History

title

modified

last-reviewed

CNPG Operator on Ringtail

Bring up the cloudnative-pg operator on k3s-ringtail. Today the operator only exists on minikube-indri (see argocd/apps/cloudnative-pg.yaml, destination kubernetes.default.svc).

Prerequisite of migrate-immich-to-ringtail; consumed by immich-pg-on-ringtail.

What to do

Add a sibling argocd/apps/cloudnative-pg-ringtail.yaml pointing at the same mirror (mirrors/cloudnative-pg, tag v1.27.1), destination https://ringtail.tail8d86e.ts.net:6443, namespace cnpg-system.
Mirror the ServerSideApply=true and CreateNamespace=true sync options (the CRDs exceed the annotation size limit).
Sync apps then cloudnative-pg-ringtail. Verify the operator pod is running on ringtail.

Verification

kubectl --context=k3s-ringtail -n cnpg-system get pods
kubectl --context=k3s-ringtail get crd clusters.postgresql.cnpg.io

Why a separate app

Each ArgoCD app targets a single cluster via destination.server. We could parameterize with ApplicationSets, but blumeops' convention is to duplicate the manifest with a -ringtail suffix (see alloy-ringtail, external-secrets-ringtail, etc.). Keep the convention.

Out of scope

Postgres clusters themselves (immich-pg, etc.) — those come from immich-pg-on-ringtail.
Removing the minikube cnpg operator. That happens at the very end of the indri-k8s decommission, not in this chain.

1.5 KiB Raw Permalink Blame History