P3: PostgreSQL disaster recovery test and borgmatic k8s-pg backup (#32)

## Summary
- Fixed borgmatic `borg: command not found` by adding `local_path` config option
- Successfully tested disaster recovery: restored miniflux data from borgmatic backup to k8s-pg
- Added borgmatic user to k8s-pg via CloudNativePG managed roles
- Configured borgmatic to backup both localhost and k8s-pg PostgreSQL databases
- Added Tailscale ACL grant for `tag:homelab` → `tag:k8s` on port 5432
- Disabled selfHeal on apps app to allow manual revision changes during development

## Changes
- `ansible/roles/borgmatic/` - Added `local_path` and k8s-pg database entry
- `ansible/roles/postgresql/tasks/main.yml` - Added k8s-pg to `.pgpass`
- `argocd/apps/apps.yaml` - Disabled selfHeal
- `argocd/manifests/databases/blumeops-pg.yaml` - Added borgmatic managed role
- `argocd/manifests/databases/secret-borgmatic.yaml.tpl` - New secret template
- `pulumi/policy.hujson` - Added ACL grant for backup access

## Deployment and Testing
- [x] Borgmatic backup runs successfully
- [x] Miniflux data restored to k8s-pg (2 users, 2 feeds, 44 entries verified)
- [x] borgmatic user created in k8s-pg with pg_read_all_data role
- [x] Both localhost and k8s-pg databases in backup archive
- [x] zk documentation updated (borgmatic.md, postgresql.md)
- [ ] After merge: set blumeops-pg app back to main revision

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/32

ansible/roles/borgmatic/defaults/main.yml

@@ -3,6 +3,9 @@ borgmatic_config: /Users/erichblume/.config/borgmatic/config.yaml
 borgmatic_config_dir: /Users/erichblume/.config/borgmatic
 borgmatic_log_dir: /Users/erichblume/Library/Logs
 
+# Full path to borg binary since LaunchAgent doesn't have homebrew in PATH
+borgmatic_local_path: /opt/homebrew/bin/borg
+
 # Schedule: runs daily at 2:00 AM
 borgmatic_schedule_hour: 2
 borgmatic_schedule_minute: 0
@@ -42,7 +45,13 @@ borgmatic_keep_yearly: 1000
 # pg_dump_command must be full path since LaunchAgent doesn't have homebrew in PATH
 borgmatic_pg_dump_command: /opt/homebrew/opt/postgresql@18/bin/pg_dump
 borgmatic_postgresql_databases:
+  # Brew PostgreSQL on indri (current production)
   - name: miniflux
     hostname: localhost
     port: 5432
     username: borgmatic
+  # k8s PostgreSQL (CloudNativePG) - backup both during migration
+  - name: miniflux
+    hostname: k8s-pg.tail8d86e.ts.net
+    port: 5432
+    username: borgmatic

ansible/roles/borgmatic/templates/config.yaml.j2

@@ -1,5 +1,8 @@
 # {{ ansible_managed }}
 
+# Path to borg binary (LaunchAgent doesn't have homebrew in PATH)
+local_path: {{ borgmatic_local_path }}
+
 source_directories:
 {% for dir in borgmatic_source_directories %}
     - {{ dir }}

ansible/roles/postgresql/tasks/main.yml

@@ -184,6 +184,7 @@
     content: |
       # Managed by ansible - only read-only roles
       localhost:{{ postgresql_port }}:*:borgmatic:{{ postgresql_user_passwords['borgmatic'] }}
+      k8s-pg.tail8d86e.ts.net:5432:*:borgmatic:{{ postgresql_user_passwords['borgmatic'] }}
     dest: ~/.pgpass
     mode: '0600'
   no_log: true

argocd/apps/apps.yaml

@@ -17,8 +17,7 @@ spec:
   syncPolicy:
     automated:
       prune: true
-      selfHeal: true
+      # selfHeal disabled: allows manual revision changes on child apps during development
+      # Sync apps app manually when adding/removing Application manifests
     syncOptions:
       - CreateNamespace=true
-  # Auto-sync enabled: new/changed Application manifests appear automatically
-  # but child apps still require manual sync (they have manual sync policy)

argocd/manifests/databases/README.md

@@ -15,14 +15,15 @@ Single-instance PostgreSQL cluster for blumeops services.
 ### Users/Roles
 
 | User      | Role           | Purpose                          | Password Source                       |
-|----------|-------------|----------------------------------|------------------------------------|
+|-----------|----------------|----------------------------------|---------------------------------------|
 | postgres  | superuser      | CNPG internal (avoid using)      | `blumeops-pg-superuser` secret        |
 | miniflux  | app owner      | Owns miniflux database           | `blumeops-pg-app` secret              |
 | eblume    | superuser      | Admin access (matches brew pg)   | `blumeops-pg-eblume` secret (manual)  |
+| borgmatic | pg_read_all_data | Backup access for borgmatic    | `blumeops-pg-borgmatic` secret (manual) |
 
 ### Manual Secret Setup
 
-Before deploying, create the eblume password secret:
+Before deploying, create the password secrets:
 
 ```bash
 # Create namespace first
@@ -30,6 +31,9 @@ kubectl create namespace databases
 
 # Apply eblume password from 1Password
 op inject -i argocd/manifests/databases/secret-eblume.yaml.tpl | kubectl apply -f -
+
+# Apply borgmatic password from 1Password
+op inject -i argocd/manifests/databases/secret-borgmatic.yaml.tpl | kubectl apply -f -
 ```
 
 The `miniflux` user password is auto-generated by CloudNativePG and stored in `blumeops-pg-app`.

argocd/manifests/databases/blumeops-pg.yaml

@@ -29,6 +29,14 @@ spec:
         createrole: true
         passwordSecret:
           name: blumeops-pg-eblume
+      # borgmatic read-only user for backups
+      - name: borgmatic
+        login: true
+        superuser: false
+        inRoles:
+          - pg_read_all_data
+        passwordSecret:
+          name: blumeops-pg-borgmatic
 
   # Resource limits for minikube environment
   resources:

argocd/manifests/databases/secret-borgmatic.yaml.tpl

@@ -0,0 +1,13 @@
+# Template for borgmatic backup user password
+# Apply with: op inject -i secret-borgmatic.yaml.tpl | kubectl apply -f -
+#
+# Uses the same borgmatic password from 1Password as the brew PostgreSQL setup
+apiVersion: v1
+kind: Secret
+metadata:
+  name: blumeops-pg-borgmatic
+  namespace: databases
+type: kubernetes.io/basic-auth
+stringData:
+  username: borgmatic
+  password: {{ op://vg6xf6vvfmoh5hqjjhlhbeoaie/mw2bv5we7woicjza7hc6s44yvy/db-password }}

pulumi/policy.hujson

@@ -74,6 +74,12 @@
 			"dst": ["tag:homelab"],
 			"ip":  ["tcp:3001", "tcp:2200"],
 		},
+		// Homelab can reach k8s PostgreSQL for borgmatic backups
+		{
+			"src": ["tag:homelab"],
+			"dst": ["tag:k8s"],
+			"ip":  ["tcp:5432"],
+		},
 	],
 
 	// ============== SSH Access ==============