blumeops/argocd/manifests/databases
Erich Blume ca0c9354ee Add borgmatic backups for authentik and immich databases (#314)
## Summary

- Add `authentik` database (blumeops-pg cluster) to borgmatic pg_dump backups
- Add `immich` database (immich-pg cluster) to borgmatic pg_dump backups
- For immich-pg: new borgmatic managed role with `pg_read_all_data`, ExternalSecret, Tailscale LoadBalancer service, and Caddy L4 TCP proxy on port 5433
- Update backup docs to reflect all four CNPG databases + mealie SQLite

## Deploy plan

Deploy order matters — k8s resources must exist before ansible can route to them:

1. **ArgoCD (databases app):** sync to pick up immich-pg borgmatic role, ExternalSecret, and Tailscale service
   ```
   argocd app set blumeops-pg --revision feature/borgmatic-all-pg-backups
   argocd app sync blumeops-pg
   ```
2. **Wait** for `immich-pg-tailscale` service to get a Tailscale IP and `immich-pg.tail8d86e.ts.net` to resolve
3. **Ansible (caddy):** deploy Caddy L4 route for port 5433
   ```
   mise run provision-indri -- --tags caddy
   ```
4. **Ansible (borgmatic):** deploy updated config and .pgpass
   ```
   mise run provision-indri -- --tags borgmatic
   ```
5. **Verify:** trigger a manual borgmatic run and check all four pg_dump streams succeed
   ```
   borgmatic --verbosity 1 2>&1 | grep -E '(Dumping|ERROR)'
   ```

## Test plan

- [x] `kubectl kustomize` builds cleanly
- [x] `ansible --check --diff` for borgmatic and caddy show expected changes
- [ ] ArgoCD sync succeeds for databases app
- [ ] `immich-pg.tail8d86e.ts.net` resolves
- [ ] `pg.ops.eblu.me:5433` accepts connections
- [ ] `borgmatic --verbosity 1` dumps all four databases without errors

Reviewed-on: #314
2026-03-27 16:59:58 -07:00
..
blumeops-pg.yaml Pin blumeops-pg to PostgreSQL 18.3 2026-02-27 16:25:32 -08:00
external-secret-authentik.yaml Deploy Authentik identity provider (C2 Mikado) (#227) 2026-02-20 12:55:59 -08:00
external-secret-borgmatic.yaml Switch all ExternalSecrets to creationPolicy: Owner 2026-01-28 20:27:16 -08:00
external-secret-eblume.yaml Switch all ExternalSecrets to creationPolicy: Owner 2026-01-28 20:27:16 -08:00
external-secret-immich-borgmatic.yaml Add borgmatic backups for authentik and immich databases (#314) 2026-03-27 16:59:58 -07:00
external-secret-teslamate.yaml Switch all ExternalSecrets to creationPolicy: Owner 2026-01-28 20:27:16 -08:00
immich-pg.yaml Add borgmatic backups for authentik and immich databases (#314) 2026-03-27 16:59:58 -07:00
kustomization.yaml Add borgmatic backups for authentik and immich databases (#314) 2026-03-27 16:59:58 -07:00
README.md Doc review: connect-to-postgres, create-release-artifact-workflow, deploy-k8s-service (#191) 2026-02-15 07:42:01 -08:00
service-immich-pg-tailscale.yaml Add borgmatic backups for authentik and immich databases (#314) 2026-03-27 16:59:58 -07:00
service-metrics-tailscale.yaml
service-tailscale.yaml

Database Manifests

PostgreSQL clusters managed by CloudNativePG operator.

Clusters

Cluster Image Purpose
blumeops-pg cloudnative-pg/postgresql:18 General services (miniflux, teslamate)
immich-pg tensorchord/cloudnative-vectorchord:17 Immich (requires pgvecto.rs extension)

blumeops-pg

Single-instance PostgreSQL cluster for blumeops services.

Configuration

  • Instances: 1 (single-node for minikube)
  • Storage: 10Gi on standard storage class
  • Initial database: miniflux owned by miniflux user

Users/Roles

User Role Purpose Password Source
postgres superuser CNPG internal (avoid using) blumeops-pg-superuser secret
miniflux app owner Owns miniflux database blumeops-pg-app secret
eblume superuser Admin access (matches brew pg) blumeops-pg-eblume secret (manual)
borgmatic pg_read_all_data Backup access for borgmatic blumeops-pg-borgmatic secret (manual)

Manual Secret Setup

Before deploying, create the password secrets:

# Create namespace first
kubectl create namespace databases

# Apply eblume password from 1Password
op inject -i argocd/manifests/databases/secret-eblume.yaml.tpl | kubectl apply -f -

# Apply borgmatic password from 1Password
op inject -i argocd/manifests/databases/secret-borgmatic.yaml.tpl | kubectl apply -f -

The miniflux user password is auto-generated by CloudNativePG and stored in blumeops-pg-app.

Connection Information

After the cluster is healthy:

# Connect via Tailscale (temporary hostname during migration)
psql -h k8s-pg.tail8d86e.ts.net -U eblume -W -d miniflux

# Or with password from 1Password
PGPASSWORD=$(op read "op://blumeops/guxu3j7ajhjyey6xxl2ovsl2ui/password") \
  psql -h k8s-pg.tail8d86e.ts.net -U eblume -d miniflux

# Get miniflux app credentials (for applications)
kubectl -n databases get secret blumeops-pg-app -o jsonpath='{.data.uri}' | base64 -d

# Get postgres superuser credentials (emergency only)
kubectl -n databases get secret blumeops-pg-superuser -o jsonpath='{.data.password}' | base64 -d

Connecting via kubectl port-forward

Alternative if Tailscale service is unavailable:

# Terminal 1: Port-forward to the primary
kubectl -n databases port-forward svc/blumeops-pg-rw 5432:5432

# Terminal 2: Connect as eblume
PGPASSWORD=$(op read "op://blumeops/guxu3j7ajhjyey6xxl2ovsl2ui/password") \
  psql -h localhost -U eblume -d miniflux

Status

# Check cluster health
kubectl -n databases get cluster blumeops-pg

# Check pods
kubectl -n databases get pods -l cnpg.io/cluster=blumeops-pg

# Check managed roles status
kubectl -n databases get cluster blumeops-pg -o jsonpath='{.status.managedRolesStatus}' | jq

# Operator logs
kubectl -n databases logs -l cnpg.io/cluster=blumeops-pg

Tailscale Exposure

Current: Temporary Service

k8s-pg.tail8d86e.ts.net - LoadBalancer service for testing during migration.

Phase 4: Production Service

After miniflux migrates to k8s, the pg.tail8d86e.ts.net Tailscale service will switch from brew PostgreSQL (indri) to this k8s cluster. At that point:

  1. Delete service-tailscale.yaml (the k8s-pg service)
  2. Update/create a service with tailscale.com/hostname: "pg"
  3. Verify the orphaned k8s-pg device is removed from tailnet

immich-pg

PostgreSQL cluster for Immich with VectorChord extension for AI-powered vector search.

Configuration

  • Instances: 1 (single-node for minikube)
  • Storage: 10Gi on standard storage class
  • Image: ghcr.io/tensorchord/cloudnative-vectorchord:17-0.5.0 (VectorChord 0.5.0 for Immich compatibility)
  • Extensions: vector, vchord, cube, earthdistance

Connection

Immich connects via immich-pg-rw.databases.svc.cluster.local:5432.

The immich user password is auto-generated by CloudNativePG and stored in immich-pg-app secret:

# Get immich app credentials
kubectl -n databases get secret immich-pg-app -o jsonpath='{.data.password}' | base64 -d

Status

# Check cluster health
kubectl -n databases get cluster immich-pg

# Check pods
kubectl -n databases get pods -l cnpg.io/cluster=immich-pg