blumeops/argocd/manifests/databases
Erich Blume 947e4310c3 C2: migrate immich from minikube to ringtail (mikado chain) (#356)
## Summary

C2 Mikado chain to move the entire Immich stack (server, ML, valkey,
postgres) off `minikube-indri` and onto `k3s-ringtail`. Immich is the
largest single tenant on minikube (~1.5 GiB resident) and minikube is
currently memory-saturated (97% RAM, swapping). This is the first
concrete chain in the broader indri-k8s decommission effort.

This PR contains the planning layer only — 7 cards (1 goal + 6
prerequisites). Implementation cycles follow per the Mikado Branch
Invariant.

## Goal end-state

- Immich `server`, `machine-learning`, `valkey` on ringtail.
- ML pod uses ringtail's RTX 4080 (performance win — currently
  CPU-only).
- CNPG `immich-pg` (PG17 + VectorChord) runs on ringtail.
- Library still on sifaka NFS — ringtail mounts the same path.
- `photos.ops.eblu.me` reroutes through Caddy → ringtail ingress.
- Minikube `immich` and `immich-pg` are removed.

## Cards

| Card | Depends on |
|---|---|
| `migrate-immich-to-ringtail` (goal) | all six below |
| `cnpg-on-ringtail` | — |
| `immich-pg-on-ringtail` | cnpg-on-ringtail |
| `immich-pg-data-migration` | immich-pg-on-ringtail |
| `sifaka-nfs-from-ringtail` | — |
| `immich-app-on-ringtail` | immich-pg-on-ringtail, sifaka-nfs-from-ringtail |
| `immich-cutover-and-decommission` | immich-pg-data-migration, immich-app-on-ringtail |

## Key constraints

- **No data loss.** Downtime is acceptable; data loss is not. Two
  surfaces matter: postgres (ML embeddings, face data — slow to
  re-derive) and the library files (don't move, but NFS access from
  ringtail must be verified).
- **Migration method:** Option A is a CNPG `externalCluster`
  basebackup → promote. Option B is `pg_dump`/`pg_restore` as a
  documented fallback. Either way, dry-run against a scratch
  cluster first.
- **Why pg moves too** (not cross-cluster): keeping pg on minikube
  would block the whole decommission, and Immich is chatty with pg
  so tailnet round-trips would hurt.

## Test plan

- [ ] Plan review — does the dependency graph make sense?
- [ ] `mise run docs-mikado migrate-immich-to-ringtail` shows the
      chain correctly.
- [ ] Per-card implementation cycles land separately (commit
      convention enforced by hook).

Reviewed-on: #356
2026-05-13 16:46:17 -07:00
..
blumeops-pg.yaml Deploy Paperless-ngx document management (#328) 2026-04-08 17:54:12 -07:00
external-secret-authentik.yaml
external-secret-borgmatic.yaml
external-secret-eblume.yaml
external-secret-paperless.yaml Deploy Paperless-ngx document management (#328) 2026-04-08 17:54:12 -07:00
external-secret-teslamate.yaml
kustomization.yaml C2: migrate immich from minikube to ringtail (mikado chain) (#356) 2026-05-13 16:46:17 -07:00
README.md
service-metrics-tailscale.yaml
service-tailscale.yaml

Database Manifests

PostgreSQL clusters managed by CloudNativePG operator.

Clusters

Cluster Image Purpose
blumeops-pg cloudnative-pg/postgresql:18 General services (miniflux, teslamate)
immich-pg tensorchord/cloudnative-vectorchord:17 Immich (requires pgvecto.rs extension)

blumeops-pg

Single-instance PostgreSQL cluster for blumeops services.

Configuration

  • Instances: 1 (single-node for minikube)
  • Storage: 10Gi on standard storage class
  • Initial database: miniflux owned by miniflux user

Users/Roles

User Role Purpose Password Source
postgres superuser CNPG internal (avoid using) blumeops-pg-superuser secret
miniflux app owner Owns miniflux database blumeops-pg-app secret
eblume superuser Admin access (matches brew pg) blumeops-pg-eblume secret (manual)
borgmatic pg_read_all_data Backup access for borgmatic blumeops-pg-borgmatic secret (manual)

Manual Secret Setup

Before deploying, create the password secrets:

# Create namespace first
kubectl create namespace databases

# Apply eblume password from 1Password
op inject -i argocd/manifests/databases/secret-eblume.yaml.tpl | kubectl apply -f -

# Apply borgmatic password from 1Password
op inject -i argocd/manifests/databases/secret-borgmatic.yaml.tpl | kubectl apply -f -

The miniflux user password is auto-generated by CloudNativePG and stored in blumeops-pg-app.

Connection Information

After the cluster is healthy:

# Connect via Tailscale (temporary hostname during migration)
psql -h k8s-pg.tail8d86e.ts.net -U eblume -W -d miniflux

# Or with password from 1Password
PGPASSWORD=$(op read "op://blumeops/guxu3j7ajhjyey6xxl2ovsl2ui/password") \
  psql -h k8s-pg.tail8d86e.ts.net -U eblume -d miniflux

# Get miniflux app credentials (for applications)
kubectl -n databases get secret blumeops-pg-app -o jsonpath='{.data.uri}' | base64 -d

# Get postgres superuser credentials (emergency only)
kubectl -n databases get secret blumeops-pg-superuser -o jsonpath='{.data.password}' | base64 -d

Connecting via kubectl port-forward

Alternative if Tailscale service is unavailable:

# Terminal 1: Port-forward to the primary
kubectl -n databases port-forward svc/blumeops-pg-rw 5432:5432

# Terminal 2: Connect as eblume
PGPASSWORD=$(op read "op://blumeops/guxu3j7ajhjyey6xxl2ovsl2ui/password") \
  psql -h localhost -U eblume -d miniflux

Status

# Check cluster health
kubectl -n databases get cluster blumeops-pg

# Check pods
kubectl -n databases get pods -l cnpg.io/cluster=blumeops-pg

# Check managed roles status
kubectl -n databases get cluster blumeops-pg -o jsonpath='{.status.managedRolesStatus}' | jq

# Operator logs
kubectl -n databases logs -l cnpg.io/cluster=blumeops-pg

Tailscale Exposure

Current: Temporary Service

k8s-pg.tail8d86e.ts.net - LoadBalancer service for testing during migration.

Phase 4: Production Service

After miniflux migrates to k8s, the pg.tail8d86e.ts.net Tailscale service will switch from brew PostgreSQL (indri) to this k8s cluster. At that point:

  1. Delete service-tailscale.yaml (the k8s-pg service)
  2. Update/create a service with tailscale.com/hostname: "pg"
  3. Verify the orphaned k8s-pg device is removed from tailnet

immich-pg

PostgreSQL cluster for Immich with VectorChord extension for AI-powered vector search.

Configuration

  • Instances: 1 (single-node for minikube)
  • Storage: 10Gi on standard storage class
  • Image: ghcr.io/tensorchord/cloudnative-vectorchord:17-0.5.0 (VectorChord 0.5.0 for Immich compatibility)
  • Extensions: vector, vchord, cube, earthdistance

Connection

Immich connects via immich-pg-rw.databases.svc.cluster.local:5432.

The immich user password is auto-generated by CloudNativePG and stored in immich-pg-app secret:

# Get immich app credentials
kubectl -n databases get secret immich-pg-app -o jsonpath='{.data.password}' | base64 -d

Status

# Check cluster health
kubectl -n databases get cluster immich-pg

# Check pods
kubectl -n databases get pods -l cnpg.io/cluster=immich-pg