eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/docs/zk/borgmatic.md
Erich Blume b8104d75ad Move zk cards to docs/zk/ for documentation restructuring (#84) 
## Summary
- Move all existing zettelkasten cards from `docs/` to `docs/zk/` as a temporary holding area
- Update `zk-docs` mise task to look in the new location
- Add `docs/README.md` explaining the Diataxis-based restructuring plan and target audiences

## Context
This is phase 1 of a multi-phase documentation restructuring effort. The goal is to reorganize docs to follow the Diataxis framework while serving multiple audiences:
1. Erich (owner) - knowledge graph/zk
2. Claude/AI agents - memory and context enrichment
3. New external readers - high-level overview
4. Potential operators/contributors - onboarding
5. Replicators - people wanting to duplicate the approach

## Testing
- [x] Verified `mise run zk-docs` still works with the new path
- [x] Updated obsidian.nvim config (in ~/.config/nvim) to point to new path

## Note
The obsidian.nvim config change is outside this repo but was made as part of this work.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/84
2026-02-03 09:13:50 -08:00

176 lines
6.7 KiB
Markdown
Raw Blame History

---
id: borgmatic
aliases:
- borgmatic
- borg-backup
tags:
- blumeops
---
# Borgmatic Management Log
Borgmatic runs daily backups from Indri to Sifaka NAS using Borg backup.
## Service Details
- Installed via: mise (pipx)
- Config: `~/.config/borgmatic/config.yaml` (ansible-managed)
- Schedule: Daily at 2:00 AM via LaunchAgent
- Repository: `/Volumes/backups/borg/` on Sifaka
## What Gets Backed Up
**Directories:**
- `~/code/personal/zk` - Zettelkasten (primary)
- `/opt/homebrew/var/forgejo` - Git forge data
- `~/.config/borgmatic` - Borgmatic config itself
- `~/Documents` - Personal documents
- `~/Pictures` - Photos (see note below)
**Note on iCloud Photos:** macOS Photos.app defaults to "Optimize Mac Storage" which keeps only thumbnails locally. Borgmatic only backs up what's on disk, so iCloud-only photos are NOT backed up. If you need full photo backups via borgmatic, either disable "Optimize Mac Storage" in Photos preferences, or use a tool like osxphotos which forces downloads. See log entry 2026-01-28.
**Databases:**
- `miniflux` PostgreSQL database on k8s CloudNativePG cluster (pg.ops.eblu.me)
- `teslamate` PostgreSQL database on k8s CloudNativePG cluster (pg.ops.eblu.me)
**Not backed up (by design):**
- ZIM archives in `~/transmission/` - re-downloadable via torrent
- Prometheus metrics - ephemeral data
- Loki logs - ephemeral (now in k8s PVC)
- devpi data - in k8s PVC, backup strategy TBD
## PostgreSQL Backup
Borgmatic uses native `postgresql_databases` support to stream `pg_dump` directly to Borg:
- No intermediate files needed
- Database keeps running (no downtime)
- Consistent transactional snapshots
- Uses `borgmatic` user with `pg_read_all_data` role
- Password read from `~/.pgpass` (managed by borgmatic ansible role)
- Uses explicit `pg_dump_command` path (`/opt/homebrew/opt/postgresql@18/bin/pg_dump`) since LaunchAgent doesn't have homebrew in PATH
- Uses explicit `local_path` (`/opt/homebrew/bin/borg`) for same reason
**Databases backed up:**
- `pg.ops.eblu.me:5432/miniflux` - CloudNativePG cluster in k8s
- `pg.ops.eblu.me:5432/teslamate` - CloudNativePG cluster in k8s
## Ansible Management
Borgmatic is fully managed via ansible in [[1767747119-YCPO|blumeops]]:
```bash
mise run provision-indri -- --tags borgmatic
```
The role deploys:
- `~/.config/borgmatic/config.yaml` - Main configuration
- LaunchAgent plist for scheduled runs
## Useful Commands
```bash
# List archives
ssh indri 'mise x -- borgmatic list'
# Extract from latest archive
ssh indri 'mise x -- borgmatic extract --archive latest --path /some/path'
# Run backup manually
ssh indri 'mise x -- borgmatic create --verbosity 1'
# Check repository health
ssh indri 'mise x -- borgmatic check'
```
## Retention Policy
- 7 daily backups
- 12 monthly backups
- 1000 yearly backups (effectively forever)
## Monitoring
Borgmatic metrics are collected hourly via a script at `~/bin/borgmatic-metrics` and exposed to Prometheus via the node_exporter textfile collector.
View the Grafana dashboard at: https://grafana.tail8d86e.ts.net (select "Borgmatic Backups" dashboard)
Metrics include:
- `borgmatic_up` - repository accessibility
- `borgmatic_repo_deduplicated_size_bytes` - actual disk usage
- `borgmatic_last_archive_original_size_bytes` - size of data being backed up
- `borgmatic_last_archive_deduplicated_size_bytes` - new data added per backup
- `borgmatic_archive_count` - number of archives
- `borgmatic_last_archive_timestamp` - when last backup ran
```bash
# Check metrics file
ssh indri 'cat /opt/homebrew/var/node_exporter/textfile/borgmatic.prom'
# Check metrics LaunchAgent status
ssh indri 'launchctl list | grep borgmatic-metrics'
```
## Log
### Tue Jan 28 2026
- Investigated massive backup size increase (~69GB deduplicated, ~94GB per archive)
- Root cause: immich-sync role (added Jan 26, removed Jan 28) used osxphotos to export photos
- **Lesson learned:** osxphotos forces Photos.app to download all iCloud originals locally
- Photos.app defaults to "Optimize Mac Storage" which keeps only thumbnails locally
- Before immich-sync: borgmatic was backing up thumbnails (~few GB)
- After immich-sync: borgmatic now has full 42GB of photo originals
- This is actually a bonus - provides redundant photo backup alongside iCloud and Immich
- Retention policy means these photos will be kept in yearly archives essentially forever
- **Future plan:** Once Immich (on sifaka "photos" volume with Synology offsite backup) is fully set up, Pictures may be removed from borgmatic as redundant
### Thu Jan 23 2026
- Note: Forgejo `app.ini` is now managed by ansible (secrets in 1Password)
- `/opt/homebrew/var/forgejo` still backed up for git repositories and data
- But `app.ini` recovery no longer depends on borgmatic (can be regenerated via ansible)
### Wed Jan 22 2026
- Removed `/opt/homebrew/var/loki` from backup sources (stale data from pre-k8s migration)
- Loki now runs in k8s with ephemeral storage - logs are not backed up by design
- Verified backup integrity after cleanup
### Mon Jan 20 2026 (P5)
- Removed `~/devpi` from backup sources (devpi migrated to k8s)
- devpi data now in k8s PVC - backup strategy TBD
### Sun Jan 19 2026 (P4)
- Removed localhost PostgreSQL backup (brew pg retired)
- Updated to backup only `pg.tail8d86e.ts.net` (k8s CloudNativePG)
- Moved .pgpass management from postgresql role to borgmatic role
### Sun Jan 19 2026 (P3)
- Fixed borgmatic failing to find `borg` binary by adding `local_path` option to config
- Added k8s-pg (CloudNativePG cluster) backup alongside brew PostgreSQL
- Added ACL grant for `tag:homelab` → `tag:k8s` on port 5432 for backup access
- Successfully tested disaster recovery: restored miniflux data from borgmatic dump to k8s-pg
- Created `borgmatic` user in k8s-pg via CloudNativePG managed roles
- Both localhost and k8s-pg databases backed up during migration period
### Sat Jan 18 2026
- Fixed borgmatic-metrics script failing in LaunchAgent context by using absolute paths (`/opt/homebrew/bin/borg`, `/opt/homebrew/bin/jq`) instead of `mise x -- borg`
- This was causing the Grafana dashboard to show "Repository Status: DOWN" and missing time series data
### Fri Jan 17 2026
- Fixed PostgreSQL backup failure by adding explicit `pg_dump_command` path (was failing with "pg_dump: command not found")
- Removed `~/code/3rd/kiwix-tools` from backups (was just symlinks, ZIM archives are re-downloadable)
- Enabled Loki log backup (removed from exclude_patterns)
- Added borgmatic_metrics role for Prometheus metrics collection
- Added Grafana dashboard for backup monitoring (size trends, dedup ratio, time since last backup)
### Thu Jan 16 2026
- Moved config from manual management to ansible-managed template
- Added `postgresql_databases` backup for miniflux database
- Config now deployed via `ansible/roles/borgmatic/templates/config.yaml.j2`
Reference in a new issue View git blame Copy permalink