## Summary
C2 Mikado chain to move the entire Immich stack (server, ML, valkey,
postgres) off `minikube-indri` and onto `k3s-ringtail`. Immich is the
largest single tenant on minikube (~1.5 GiB resident) and minikube is
currently memory-saturated (97% RAM, swapping). This is the first
concrete chain in the broader indri-k8s decommission effort.
This PR contains the planning layer only — 7 cards (1 goal + 6
prerequisites). Implementation cycles follow per the Mikado Branch
Invariant.
## Goal end-state
- Immich `server`, `machine-learning`, `valkey` on ringtail.
- ML pod uses ringtail's RTX 4080 (performance win — currently
CPU-only).
- CNPG `immich-pg` (PG17 + VectorChord) runs on ringtail.
- Library still on sifaka NFS — ringtail mounts the same path.
- `photos.ops.eblu.me` reroutes through Caddy → ringtail ingress.
- Minikube `immich` and `immich-pg` are removed.
## Cards
| Card | Depends on |
|---|---|
| `migrate-immich-to-ringtail` (goal) | all six below |
| `cnpg-on-ringtail` | — |
| `immich-pg-on-ringtail` | cnpg-on-ringtail |
| `immich-pg-data-migration` | immich-pg-on-ringtail |
| `sifaka-nfs-from-ringtail` | — |
| `immich-app-on-ringtail` | immich-pg-on-ringtail, sifaka-nfs-from-ringtail |
| `immich-cutover-and-decommission` | immich-pg-data-migration, immich-app-on-ringtail |
## Key constraints
- **No data loss.** Downtime is acceptable; data loss is not. Two
surfaces matter: postgres (ML embeddings, face data — slow to
re-derive) and the library files (don't move, but NFS access from
ringtail must be verified).
- **Migration method:** Option A is a CNPG `externalCluster`
basebackup → promote. Option B is `pg_dump`/`pg_restore` as a
documented fallback. Either way, dry-run against a scratch
cluster first.
- **Why pg moves too** (not cross-cluster): keeping pg on minikube
would block the whole decommission, and Immich is chatty with pg
so tailnet round-trips would hurt.
## Test plan
- [ ] Plan review — does the dependency graph make sense?
- [ ] `mise run docs-mikado migrate-immich-to-ringtail` shows the
chain correctly.
- [ ] Per-card implementation cycles land separately (commit
convention enforced by hook).
Reviewed-on: #356
|
||
|---|---|---|
| .. | ||
| blumeops-pg.yaml | ||
| external-secret-authentik.yaml | ||
| external-secret-borgmatic.yaml | ||
| external-secret-eblume.yaml | ||
| external-secret-paperless.yaml | ||
| external-secret-teslamate.yaml | ||
| kustomization.yaml | ||
| README.md | ||
| service-metrics-tailscale.yaml | ||
| service-tailscale.yaml | ||
Database Manifests
PostgreSQL clusters managed by CloudNativePG operator.
Clusters
| Cluster | Image | Purpose |
|---|---|---|
| blumeops-pg | cloudnative-pg/postgresql:18 | General services (miniflux, teslamate) |
| immich-pg | tensorchord/cloudnative-vectorchord:17 | Immich (requires pgvecto.rs extension) |
blumeops-pg
Single-instance PostgreSQL cluster for blumeops services.
Configuration
- Instances: 1 (single-node for minikube)
- Storage: 10Gi on
standardstorage class - Initial database:
minifluxowned byminifluxuser
Users/Roles
| User | Role | Purpose | Password Source |
|---|---|---|---|
| postgres | superuser | CNPG internal (avoid using) | blumeops-pg-superuser secret |
| miniflux | app owner | Owns miniflux database | blumeops-pg-app secret |
| eblume | superuser | Admin access (matches brew pg) | blumeops-pg-eblume secret (manual) |
| borgmatic | pg_read_all_data | Backup access for borgmatic | blumeops-pg-borgmatic secret (manual) |
Manual Secret Setup
Before deploying, create the password secrets:
# Create namespace first
kubectl create namespace databases
# Apply eblume password from 1Password
op inject -i argocd/manifests/databases/secret-eblume.yaml.tpl | kubectl apply -f -
# Apply borgmatic password from 1Password
op inject -i argocd/manifests/databases/secret-borgmatic.yaml.tpl | kubectl apply -f -
The miniflux user password is auto-generated by CloudNativePG and stored in blumeops-pg-app.
Connection Information
After the cluster is healthy:
# Connect via Tailscale (temporary hostname during migration)
psql -h k8s-pg.tail8d86e.ts.net -U eblume -W -d miniflux
# Or with password from 1Password
PGPASSWORD=$(op read "op://blumeops/guxu3j7ajhjyey6xxl2ovsl2ui/password") \
psql -h k8s-pg.tail8d86e.ts.net -U eblume -d miniflux
# Get miniflux app credentials (for applications)
kubectl -n databases get secret blumeops-pg-app -o jsonpath='{.data.uri}' | base64 -d
# Get postgres superuser credentials (emergency only)
kubectl -n databases get secret blumeops-pg-superuser -o jsonpath='{.data.password}' | base64 -d
Connecting via kubectl port-forward
Alternative if Tailscale service is unavailable:
# Terminal 1: Port-forward to the primary
kubectl -n databases port-forward svc/blumeops-pg-rw 5432:5432
# Terminal 2: Connect as eblume
PGPASSWORD=$(op read "op://blumeops/guxu3j7ajhjyey6xxl2ovsl2ui/password") \
psql -h localhost -U eblume -d miniflux
Status
# Check cluster health
kubectl -n databases get cluster blumeops-pg
# Check pods
kubectl -n databases get pods -l cnpg.io/cluster=blumeops-pg
# Check managed roles status
kubectl -n databases get cluster blumeops-pg -o jsonpath='{.status.managedRolesStatus}' | jq
# Operator logs
kubectl -n databases logs -l cnpg.io/cluster=blumeops-pg
Tailscale Exposure
Current: Temporary Service
k8s-pg.tail8d86e.ts.net - LoadBalancer service for testing during migration.
Phase 4: Production Service
After miniflux migrates to k8s, the pg.tail8d86e.ts.net Tailscale service will switch
from brew PostgreSQL (indri) to this k8s cluster. At that point:
- Delete
service-tailscale.yaml(thek8s-pgservice) - Update/create a service with
tailscale.com/hostname: "pg" - Verify the orphaned
k8s-pgdevice is removed from tailnet
immich-pg
PostgreSQL cluster for Immich with VectorChord extension for AI-powered vector search.
Configuration
- Instances: 1 (single-node for minikube)
- Storage: 10Gi on
standardstorage class - Image:
ghcr.io/tensorchord/cloudnative-vectorchord:17-0.5.0(VectorChord 0.5.0 for Immich compatibility) - Extensions:
vector,vchord,cube,earthdistance
Connection
Immich connects via immich-pg-rw.databases.svc.cluster.local:5432.
The immich user password is auto-generated by CloudNativePG and stored in immich-pg-app secret:
# Get immich app credentials
kubectl -n databases get secret immich-pg-app -o jsonpath='{.data.password}' | base64 -d
Status
# Check cluster health
kubectl -n databases get cluster immich-pg
# Check pods
kubectl -n databases get pods -l cnpg.io/cluster=immich-pg