Erich Blume eb952aae01 P3: PostgreSQL disaster recovery test and borgmatic k8s-pg backup (#32 )

## Summary
- Fixed borgmatic `borg: command not found` by adding `local_path` config option
- Successfully tested disaster recovery: restored miniflux data from borgmatic backup to k8s-pg
- Added borgmatic user to k8s-pg via CloudNativePG managed roles
- Configured borgmatic to backup both localhost and k8s-pg PostgreSQL databases
- Added Tailscale ACL grant for `tag:homelab` → `tag:k8s` on port 5432
- Disabled selfHeal on apps app to allow manual revision changes during development

## Changes
- `ansible/roles/borgmatic/` - Added `local_path` and k8s-pg database entry
- `ansible/roles/postgresql/tasks/main.yml` - Added k8s-pg to `.pgpass`
- `argocd/apps/apps.yaml` - Disabled selfHeal
- `argocd/manifests/databases/blumeops-pg.yaml` - Added borgmatic managed role
- `argocd/manifests/databases/secret-borgmatic.yaml.tpl` - New secret template
- `pulumi/policy.hujson` - Added ACL grant for backup access

## Deployment and Testing
- [x] Borgmatic backup runs successfully
- [x] Miniflux data restored to k8s-pg (2 users, 2 feeds, 44 entries verified)
- [x] borgmatic user created in k8s-pg with pg_read_all_data role
- [x] Both localhost and k8s-pg databases in backup archive
- [x] zk documentation updated (borgmatic.md, postgresql.md)
- [ ] After merge: set blumeops-pg app back to main revision

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/32

2026-01-19 18:00:32 -08:00

3.3 KiB

Raw Permalink Blame History

Database Manifests

PostgreSQL clusters managed by CloudNativePG operator.

blumeops-pg

Single-instance PostgreSQL cluster for blumeops services.

Configuration

Instances: 1 (single-node for minikube)
Storage: 10Gi on standard storage class
Initial database: miniflux owned by miniflux user

Users/Roles

User	Role	Purpose	Password Source
postgres	superuser	CNPG internal (avoid using)	`blumeops-pg-superuser` secret
miniflux	app owner	Owns miniflux database	`blumeops-pg-app` secret
eblume	superuser	Admin access (matches brew pg)	`blumeops-pg-eblume` secret (manual)
borgmatic	pg_read_all_data	Backup access for borgmatic	`blumeops-pg-borgmatic` secret (manual)

Manual Secret Setup

Before deploying, create the password secrets:

# Create namespace first
kubectl create namespace databases

# Apply eblume password from 1Password
op inject -i argocd/manifests/databases/secret-eblume.yaml.tpl | kubectl apply -f -

# Apply borgmatic password from 1Password
op inject -i argocd/manifests/databases/secret-borgmatic.yaml.tpl | kubectl apply -f -

The miniflux user password is auto-generated by CloudNativePG and stored in blumeops-pg-app.

Connection Information

After the cluster is healthy:

# Connect via Tailscale (temporary hostname during migration)
psql -h k8s-pg.tail8d86e.ts.net -U eblume -W -d miniflux

# Or with password from 1Password
PGPASSWORD=$(op --vault blumeops item get guxu3j7ajhjyey6xxl2ovsl2ui --fields password --reveal) \
  psql -h k8s-pg.tail8d86e.ts.net -U eblume -d miniflux

# Get miniflux app credentials (for applications)
kubectl -n databases get secret blumeops-pg-app -o jsonpath='{.data.uri}' | base64 -d

# Get postgres superuser credentials (emergency only)
kubectl -n databases get secret blumeops-pg-superuser -o jsonpath='{.data.password}' | base64 -d

Connecting via kubectl port-forward

Alternative if Tailscale service is unavailable:

# Terminal 1: Port-forward to the primary
kubectl -n databases port-forward svc/blumeops-pg-rw 5432:5432

# Terminal 2: Connect as eblume
PGPASSWORD=$(op --vault blumeops item get guxu3j7ajhjyey6xxl2ovsl2ui --fields password --reveal) \
  psql -h localhost -U eblume -d miniflux

Status

# Check cluster health
kubectl -n databases get cluster blumeops-pg

# Check pods
kubectl -n databases get pods -l cnpg.io/cluster=blumeops-pg

# Check managed roles status
kubectl -n databases get cluster blumeops-pg -o jsonpath='{.status.managedRolesStatus}' | jq

# Operator logs
kubectl -n databases logs -l cnpg.io/cluster=blumeops-pg

Tailscale Exposure

Current: Temporary Service

k8s-pg.tail8d86e.ts.net - LoadBalancer service for testing during migration.

Phase 4: Production Service

After miniflux migrates to k8s, the pg.tail8d86e.ts.net Tailscale service will switch from brew PostgreSQL (indri) to this k8s cluster. At that point:

Delete service-tailscale.yaml (the k8s-pg service)
Update/create a service with tailscale.com/hostname: "pg"
Verify the orphaned k8s-pg device is removed from tailnet

3.3 KiB Raw Permalink Blame History