eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/docs/reference/storage/backups.md

97 lines
3.2 KiB
Markdown
Raw Normal View History

Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
---
Update all docs titles to human-readable (#117) ## Summary - Updated frontmatter `title:` in all 63 doc cards from slug-case to human-readable (e.g. `borgmatic` → `Borgmatic`, `ai-assistance-guide` → `AI Assistance Guide`) - Titles now closely match file stems so `[[wiki-links]]` render naturally without alternate anchor text - Corrected titles that diverged from stems (e.g. `host-inventory` → `Hosts`, `grafana-alloy` → `Alloy`, `argocd-applications` → `Apps`) - Deleted `title-test-alpha.md` and `title-test-beta.md` test cards and removed their reference index entry ## Deployment and Testing - [x] `docs-check-links` passes — all wiki-links valid - [x] `docs-check-index` passes - [x] `docs-check-filenames` passes - [ ] Verify titles render correctly on docs site after deploy Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/117
2026-02-07 21:44:57 -08:00
title: Backups
Add borgmatic backups for authentik and immich databases (#314) ## Summary - Add `authentik` database (blumeops-pg cluster) to borgmatic pg_dump backups - Add `immich` database (immich-pg cluster) to borgmatic pg_dump backups - For immich-pg: new borgmatic managed role with `pg_read_all_data`, ExternalSecret, Tailscale LoadBalancer service, and Caddy L4 TCP proxy on port 5433 - Update backup docs to reflect all four CNPG databases + mealie SQLite ## Deploy plan Deploy order matters — k8s resources must exist before ansible can route to them: 1. **ArgoCD (databases app):** sync to pick up immich-pg borgmatic role, ExternalSecret, and Tailscale service ``` argocd app set blumeops-pg --revision feature/borgmatic-all-pg-backups argocd app sync blumeops-pg ``` 2. **Wait** for `immich-pg-tailscale` service to get a Tailscale IP and `immich-pg.tail8d86e.ts.net` to resolve 3. **Ansible (caddy):** deploy Caddy L4 route for port 5433 ``` mise run provision-indri -- --tags caddy ``` 4. **Ansible (borgmatic):** deploy updated config and .pgpass ``` mise run provision-indri -- --tags borgmatic ``` 5. **Verify:** trigger a manual borgmatic run and check all four pg_dump streams succeed ``` borgmatic --verbosity 1 2>&1 | grep -E '(Dumping|ERROR)' ``` ## Test plan - [x] `kubectl kustomize` builds cleanly - [x] `ansible --check --diff` for borgmatic and caddy show expected changes - [ ] ArgoCD sync succeeds for databases app - [ ] `immich-pg.tail8d86e.ts.net` resolves - [ ] `pg.ops.eblu.me:5433` accepts connections - [ ] `borgmatic --verbosity 1` dumps all four databases without errors Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/314
2026-03-27 16:59:58 -07:00
modified: 2026-03-27
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
tags:
- storage
- backup
---
# Backup Policy
Enforce unique doc filenames and simple wiki-links (#109) ## Summary - Rename section index files to match their titles (tutorials.md, reference.md, how-to.md, explanation.md) so all filenames are unique - Convert all ~47 path-based wiki-links to simple filename format across 15 files - Update doc-filenames task to no longer skip index.md files - Update doc-links task to reject path-based links containing '/' This ensures all wiki-links work correctly in obsidian.nvim by making links resolvable by filename alone. ## Testing - `mise run doc-filenames` - all unique - `mise run doc-links` - no broken or path-based links - `mise run doc-titles` - no duplicates Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/109
2026-02-04 17:21:34 -08:00
Daily automated backups from [[indri]] to [[sifaka|Sifaka]] NAS.
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
## Schedule
| Time | Frequency | System |
|------|-----------|--------|
Convert wiki-link titles to lowercase slugs (#92) ## Summary - Convert all frontmatter titles to lowercase-hyphenated format (e.g., `grafana-alloy` instead of `Grafana Alloy`) - Update all wiki-links to use the new slug format - Update `doc-titles` task to validate slug format (lowercase, hyphens only) Quartz appears to require titles without spaces for wiki-link resolution. ## Deployment and Testing - [x] Pre-commit hooks pass (`doc-titles` and `doc-links`) - [ ] Build docs v1.0.8 and deploy - [ ] Verify wiki-links resolve correctly (e.g., `[[grafana-alloy]]`) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/92
2026-02-03 16:06:35 -08:00
| 2:00 AM | Daily | [[borgmatic]] |
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
## What Gets Backed Up
### Directories
| Path | Description | Priority |
|------|-------------|----------|
| `~/code/personal/zk` | Zettelkasten notes | Critical |
| `/opt/homebrew/var/forgejo` | Git repositories | Critical |
| `~/.config/borgmatic` | Backup config | High |
Add how-to guide for restoring 1Password backup from borgmatic (#141) ## Summary - New how-to guide at `docs/how-to/restore-1password-backup.md` with step-by-step procedure for extracting and decrypting a 1Password `.1pux` export from borgmatic backup - **End-to-end verified**: extracted from today's borg archive, decrypted age key with openssl, decrypted .1pux with age → valid 31MB zip with vault data - Cross-links added from: disaster-recovery, 1password, borgmatic, backups policy, and how-to index - Updated disaster-recovery.md from TBD stub to include a procedures table ## Deployment and Testing - [x] Verified full extraction + decryption flow against live borgmatic archive - [x] `docs-check-links` passes — all wiki-links valid - [ ] Review guide for clarity and completeness Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/141
2026-02-10 10:55:00 -08:00
| `~/Documents` | Personal documents (includes [[1password]] encrypted export) | High |
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
### Databases
Add borgmatic backups for authentik and immich databases (#314) ## Summary - Add `authentik` database (blumeops-pg cluster) to borgmatic pg_dump backups - Add `immich` database (immich-pg cluster) to borgmatic pg_dump backups - For immich-pg: new borgmatic managed role with `pg_read_all_data`, ExternalSecret, Tailscale LoadBalancer service, and Caddy L4 TCP proxy on port 5433 - Update backup docs to reflect all four CNPG databases + mealie SQLite ## Deploy plan Deploy order matters — k8s resources must exist before ansible can route to them: 1. **ArgoCD (databases app):** sync to pick up immich-pg borgmatic role, ExternalSecret, and Tailscale service ``` argocd app set blumeops-pg --revision feature/borgmatic-all-pg-backups argocd app sync blumeops-pg ``` 2. **Wait** for `immich-pg-tailscale` service to get a Tailscale IP and `immich-pg.tail8d86e.ts.net` to resolve 3. **Ansible (caddy):** deploy Caddy L4 route for port 5433 ``` mise run provision-indri -- --tags caddy ``` 4. **Ansible (borgmatic):** deploy updated config and .pgpass ``` mise run provision-indri -- --tags borgmatic ``` 5. **Verify:** trigger a manual borgmatic run and check all four pg_dump streams succeed ``` borgmatic --verbosity 1 2>&1 | grep -E '(Dumping|ERROR)' ``` ## Test plan - [x] `kubectl kustomize` builds cleanly - [x] `ansible --check --diff` for borgmatic and caddy show expected changes - [ ] ArgoCD sync succeeds for databases app - [ ] `immich-pg.tail8d86e.ts.net` resolves - [ ] `pg.ops.eblu.me:5433` accepts connections - [ ] `borgmatic --verbosity 1` dumps all four databases without errors Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/314
2026-03-27 16:59:58 -07:00
| Database | Cluster | Host | Method |
|----------|---------|------|--------|
| miniflux | blumeops-pg | [[postgresql|pg.ops.eblu.me:5432]] | pg_dump stream |
| teslamate | blumeops-pg | [[postgresql|pg.ops.eblu.me:5432]] | pg_dump stream |
| authentik | blumeops-pg | [[postgresql|pg.ops.eblu.me:5432]] | pg_dump stream |
| immich | immich-pg | [[postgresql|pg.ops.eblu.me:5433]] | pg_dump stream |
| mealie | — (SQLite) | k8s pod | kubectl exec sqlite3 .backup |
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
Add offsite backup for immich photo library to BorgBase (#315) ## Summary - Adds a second borgmatic config (`photos.yaml`) that backs up `/Volumes/photos` (sifaka SMB mount, ~128 GB) to a dedicated BorgBase repo (`immich-photos`), running daily at 4 AM - Separate launchd agent (`mcquack.eblume.borgmatic-photos`) so photo backups run independently from the main backup - Refactors `borgmatic_metrics` script to support multiple repos with a `repo` Prometheus label - Updates Grafana "Borg Backups" dashboard with a `repo` template variable so you can filter/compare repos - Docs updated: `backups.md`, `borgmatic.md` ## Prerequisites (manual) - [x] Create `immich-photos` repo on BorgBase with same SSH key - [ ] Upgrade BorgBase plan to Small ($24/yr) if currently on free tier (128 GB exceeds 10 GB limit) - [ ] After deploy: `borg init` the new repo (borgmatic does this automatically on first run) ## Test plan - [ ] Dry run: `mise run provision-indri -- --check --diff --tags borgmatic,borgmatic_metrics` - [ ] Deploy borgmatic role and verify both configs deployed - [ ] Run `borgmatic --config ~/.config/borgmatic/photos.yaml create --verbosity 1` manually for first backup (will take hours) - [ ] Verify metrics script collects from both repos: `~/.local/bin/borgmatic-metrics && cat /opt/homebrew/var/node_exporter/textfile/borgmatic.prom` - [ ] Sync grafana-config in ArgoCD and verify dashboard repo selector works 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/315
2026-03-27 19:43:05 -07:00
## Immich Photo Library (Offsite Only)
The [[immich]] photo library lives on [[sifaka]] at `/volume1/photos` (SMB-mounted on [[indri]] as `/Volumes/photos`). Since sifaka is already the local backup target, photos are backed up to BorgBase offsite only — not back to sifaka.
| Property | Value |
|----------|-------|
| **Config** | `~/.config/borgmatic/photos.yaml` |
| **Schedule** | Daily at 4:00 AM (offset from main backup) |
| **Source** | `/Volumes/photos` (sifaka SMB mount) |
| **Target** | BorgBase `borgbase-immich-photos` repo |
| **Size** | ~128 GB |
Uses the same encryption passphrase and SSH key as the main borgmatic config.
Remove iCloud Photos from borgmatic backup (#100) ## Summary - Remove ~/Pictures from borgmatic source directories - Update borgmatic and backup policy documentation - Add Sifaka-Native Data section to clarify that photos (via Immich), music (via Navidrome), and video (via Jellyfin) are stored directly on Sifaka ## Deployment and Testing - [ ] Run `mise run provision-indri -- --tags borgmatic --check --diff` to preview changes - [ ] Run `mise run provision-indri -- --tags borgmatic` to apply - [ ] Verify borgmatic config no longer includes ~/Pictures 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/100
2026-02-04 07:09:28 -08:00
## Sifaka-Native Data
Add offsite backup for immich photo library to BorgBase (#315) ## Summary - Adds a second borgmatic config (`photos.yaml`) that backs up `/Volumes/photos` (sifaka SMB mount, ~128 GB) to a dedicated BorgBase repo (`immich-photos`), running daily at 4 AM - Separate launchd agent (`mcquack.eblume.borgmatic-photos`) so photo backups run independently from the main backup - Refactors `borgmatic_metrics` script to support multiple repos with a `repo` Prometheus label - Updates Grafana "Borg Backups" dashboard with a `repo` template variable so you can filter/compare repos - Docs updated: `backups.md`, `borgmatic.md` ## Prerequisites (manual) - [x] Create `immich-photos` repo on BorgBase with same SSH key - [ ] Upgrade BorgBase plan to Small ($24/yr) if currently on free tier (128 GB exceeds 10 GB limit) - [ ] After deploy: `borg init` the new repo (borgmatic does this automatically on first run) ## Test plan - [ ] Dry run: `mise run provision-indri -- --check --diff --tags borgmatic,borgmatic_metrics` - [ ] Deploy borgmatic role and verify both configs deployed - [ ] Run `borgmatic --config ~/.config/borgmatic/photos.yaml create --verbosity 1` manually for first backup (will take hours) - [ ] Verify metrics script collects from both repos: `~/.local/bin/borgmatic-metrics && cat /opt/homebrew/var/node_exporter/textfile/borgmatic.prom` - [ ] Sync grafana-config in ArgoCD and verify dashboard repo selector works 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/315
2026-03-27 19:43:05 -07:00
Other data lives directly on [[sifaka]] (music via [[navidrome]], video via [[jellyfin]]). See [[sifaka]] for data protection details.
Remove iCloud Photos from borgmatic backup (#100) ## Summary - Remove ~/Pictures from borgmatic source directories - Update borgmatic and backup policy documentation - Add Sifaka-Native Data section to clarify that photos (via Immich), music (via Navidrome), and video (via Jellyfin) are stored directly on Sifaka ## Deployment and Testing - [ ] Run `mise run provision-indri -- --tags borgmatic --check --diff` to preview changes - [ ] Run `mise run provision-indri -- --tags borgmatic` to apply - [ ] Verify borgmatic config no longer includes ~/Pictures 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/100
2026-02-04 07:09:28 -08:00
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
## What Is NOT Backed Up
| Data | Reason |
|------|--------|
| ZIM archives (`~/transmission/`) | Re-downloadable via torrent |
| Prometheus metrics | Ephemeral, in k8s PVC |
| Loki logs | Ephemeral, in k8s PVC |
Migrate devpi from minikube to indri (launchd) (#341) ## Summary Devpi was crash-looping under memory pressure on the minikube StatefulSet, breaking the Python toolchain across the repo (`mise run docs-mikado`, `prek`, every `uv pip install`). It moves to indri as a native LaunchAgent. ## What changed - **New ansible role** `ansible/roles/devpi/`: installs `devpi-server` + `devpi-web` into a uv-managed venv, initializes the server-dir on first run via 1Password root password, runs as a LaunchAgent (`mcquack.eblume.devpi`) bound to `127.0.0.1:3141`. Bootstraps from upstream PyPI (so devpi can install itself on a fresh box). - **Caddy**: `pypi.ops.eblu.me` now proxies to `http://localhost:3141`. - **Playbook**: `indri.yml` gains pre_tasks for the root password and the new role. - **service-versions.yaml**: devpi flipped from `type: argocd` to `type: ansible`. - **ArgoCD**: removed `apps/devpi.yaml` and `manifests/devpi/`. The in-cluster Application, namespace, and PVC have been deleted. - **Docs**: new how-to `docs/how-to/operations/devpi-on-indri.md`; `restart-indri.md` lists devpi in the LaunchAgent stop list. ## Already deployed (live on indri) - Service running: `launchctl list mcquack.eblume.devpi` → PID 53888 - `curl https://pypi.ops.eblu.me/+api` returns 200 ✅ - `mise run docs-mikado` works again ✅ - 1.0G of cached PyPI data was migrated from the PVC to `~erichblume/devpi/server-dir/` - Minikube namespace and PVC fully reclaimed ## Test plan - [ ] `mise run services-check` (after merge) - [ ] CI workflows that use devpi succeed - [ ] No regressions in tools that depend on `pypi.ops.eblu.me` (prek, uv-script tasks, dagger pipelines) ## Context This is the C1 prelude to a planned C2 chain (`mikado/retire-minikube-indri`) to retire minikube on indri entirely. Doing devpi as a standalone C1 was the right call because (a) it was urgent — it was breaking the toolchain — and (b) it shakes out the migration recipe before we commit to a multi-leaf chain. Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/341
2026-04-29 13:38:36 -07:00
| devpi cache (`~/devpi/server-dir/` on indri) | Re-fetchable from PyPI on first request |
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
## Retention Policy
| Period | Retention |
|--------|-----------|
| Daily | 7 backups |
| Monthly | 12 backups |
| Yearly | 1000 backups |
Review restore-1password-backup doc: fix offsite TBD, clarify archive name, add BorgBase to backups Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 10:12:53 -07:00
## Backup Targets
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
Add offsite backup for immich photo library to BorgBase (#315) ## Summary - Adds a second borgmatic config (`photos.yaml`) that backs up `/Volumes/photos` (sifaka SMB mount, ~128 GB) to a dedicated BorgBase repo (`immich-photos`), running daily at 4 AM - Separate launchd agent (`mcquack.eblume.borgmatic-photos`) so photo backups run independently from the main backup - Refactors `borgmatic_metrics` script to support multiple repos with a `repo` Prometheus label - Updates Grafana "Borg Backups" dashboard with a `repo` template variable so you can filter/compare repos - Docs updated: `backups.md`, `borgmatic.md` ## Prerequisites (manual) - [x] Create `immich-photos` repo on BorgBase with same SSH key - [ ] Upgrade BorgBase plan to Small ($24/yr) if currently on free tier (128 GB exceeds 10 GB limit) - [ ] After deploy: `borg init` the new repo (borgmatic does this automatically on first run) ## Test plan - [ ] Dry run: `mise run provision-indri -- --check --diff --tags borgmatic,borgmatic_metrics` - [ ] Deploy borgmatic role and verify both configs deployed - [ ] Run `borgmatic --config ~/.config/borgmatic/photos.yaml create --verbosity 1` manually for first backup (will take hours) - [ ] Verify metrics script collects from both repos: `~/.local/bin/borgmatic-metrics && cat /opt/homebrew/var/node_exporter/textfile/borgmatic.prom` - [ ] Sync grafana-config in ArgoCD and verify dashboard repo selector works 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/315
2026-03-27 19:43:05 -07:00
| Repository | Location | Label | Backs up |
|------------|----------|-------|----------|
| `/Volumes/backups/borg/` | [[sifaka]] (local NAS) | `sifaka-borg-backups` | indri data |
| `ssh://u3ugi1x1@...repo.borgbase.com/./repo` | BorgBase (offsite) | `borgbase-offsite` | indri data |
| `ssh://xcrtl5tg@...repo.borgbase.com/./repo` | BorgBase (offsite) | `borgbase-immich-photos` | immich photos |
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
## Monitoring
Convert wiki-link titles to lowercase slugs (#92) ## Summary - Convert all frontmatter titles to lowercase-hyphenated format (e.g., `grafana-alloy` instead of `Grafana Alloy`) - Update all wiki-links to use the new slug format - Update `doc-titles` task to validate slug format (lowercase, hyphens only) Quartz appears to require titles without spaces for wiki-link resolution. ## Deployment and Testing - [x] Pre-commit hooks pass (`doc-titles` and `doc-links`) - [ ] Build docs v1.0.8 and deploy - [ ] Verify wiki-links resolve correctly (e.g., `[[grafana-alloy]]`) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/92
2026-02-03 16:06:35 -08:00
Metrics exposed to [[prometheus]]:
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
- `borgmatic_up` - Repository accessible
- `borgmatic_last_archive_timestamp` - Last backup time
- `borgmatic_repo_deduplicated_size_bytes` - Disk usage
Convert wiki-link titles to lowercase slugs (#92) ## Summary - Convert all frontmatter titles to lowercase-hyphenated format (e.g., `grafana-alloy` instead of `Grafana Alloy`) - Update all wiki-links to use the new slug format - Update `doc-titles` task to validate slug format (lowercase, hyphens only) Quartz appears to require titles without spaces for wiki-link resolution. ## Deployment and Testing - [x] Pre-commit hooks pass (`doc-titles` and `doc-links`) - [ ] Build docs v1.0.8 and deploy - [ ] Verify wiki-links resolve correctly (e.g., `[[grafana-alloy]]`) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/92
2026-02-03 16:06:35 -08:00
Dashboard: "Borgmatic Backups" in [[grafana]]
Phase 2: Add Reference section with 24 technical reference cards (#88) ## Summary - Create `docs/reference/` section with 24 technical reference cards - Services (16): alloy, argocd, borgmatic, 1password, forgejo, grafana, jellyfin, kiwix, loki, miniflux, navidrome, postgresql, prometheus, teslamate, transmission, zot - Infrastructure (3): hosts, tailscale, routing - Kubernetes (2): cluster, apps - Storage (2): sifaka, backups - Update README to mark Phase 2 as complete - Add towncrier changelog fragment ## Deployment and Testing - [ ] Build docs locally to verify wiki-links resolve - [ ] Deploy via ArgoCD and verify at docs.ops.eblu.me/reference/ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/88
2026-02-03 14:27:37 -08:00
## Related
Convert wiki-link titles to lowercase slugs (#92) ## Summary - Convert all frontmatter titles to lowercase-hyphenated format (e.g., `grafana-alloy` instead of `Grafana Alloy`) - Update all wiki-links to use the new slug format - Update `doc-titles` task to validate slug format (lowercase, hyphens only) Quartz appears to require titles without spaces for wiki-link resolution. ## Deployment and Testing - [x] Pre-commit hooks pass (`doc-titles` and `doc-links`) - [ ] Build docs v1.0.8 and deploy - [ ] Verify wiki-links resolve correctly (e.g., `[[grafana-alloy]]`) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/92
2026-02-03 16:06:35 -08:00
- [[borgmatic]] - Backup system details
Enforce unique doc filenames and simple wiki-links (#109) ## Summary - Rename section index files to match their titles (tutorials.md, reference.md, how-to.md, explanation.md) so all filenames are unique - Convert all ~47 path-based wiki-links to simple filename format across 15 files - Update doc-filenames task to no longer skip index.md files - Update doc-links task to reject path-based links containing '/' This ensures all wiki-links work correctly in obsidian.nvim by making links resolvable by filename alone. ## Testing - `mise run doc-filenames` - all unique - `mise run doc-links` - no broken or path-based links - `mise run doc-titles` - no duplicates Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/109
2026-02-04 17:21:34 -08:00
- [[sifaka|Sifaka]] - Backup storage
Convert wiki-link titles to lowercase slugs (#92) ## Summary - Convert all frontmatter titles to lowercase-hyphenated format (e.g., `grafana-alloy` instead of `Grafana Alloy`) - Update all wiki-links to use the new slug format - Update `doc-titles` task to validate slug format (lowercase, hyphens only) Quartz appears to require titles without spaces for wiki-link resolution. ## Deployment and Testing - [x] Pre-commit hooks pass (`doc-titles` and `doc-links`) - [ ] Build docs v1.0.8 and deploy - [ ] Verify wiki-links resolve correctly (e.g., `[[grafana-alloy]]`) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/92
2026-02-03 16:06:35 -08:00
- [[postgresql]] - Database backups
Add how-to guide for restoring 1Password backup from borgmatic (#141) ## Summary - New how-to guide at `docs/how-to/restore-1password-backup.md` with step-by-step procedure for extracting and decrypting a 1Password `.1pux` export from borgmatic backup - **End-to-end verified**: extracted from today's borg archive, decrypted age key with openssl, decrypted .1pux with age → valid 31MB zip with vault data - Cross-links added from: disaster-recovery, 1password, borgmatic, backups policy, and how-to index - Updated disaster-recovery.md from TBD stub to include a procedures table ## Deployment and Testing - [x] Verified full extraction + decryption flow against live borgmatic archive - [x] `docs-check-links` passes — all wiki-links valid - [ ] Review guide for clarity and completeness Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/141
2026-02-10 10:55:00 -08:00
- [[restore-1password-backup]] - Recover 1Password from backup
Reference in a new issue Copy permalink