Erich Blume
14ca0160ba
## Summary Devpi was crash-looping under memory pressure on the minikube StatefulSet, breaking the Python toolchain across the repo (`mise run docs-mikado`, `prek`, every `uv pip install`). It moves to indri as a native LaunchAgent. ## What changed - **New ansible role** `ansible/roles/devpi/`: installs `devpi-server` + `devpi-web` into a uv-managed venv, initializes the server-dir on first run via 1Password root password, runs as a LaunchAgent (`mcquack.eblume.devpi`) bound to `127.0.0.1:3141`. Bootstraps from upstream PyPI (so devpi can install itself on a fresh box). - **Caddy**: `pypi.ops.eblu.me` now proxies to `http://localhost:3141`. - **Playbook**: `indri.yml` gains pre_tasks for the root password and the new role. - **service-versions.yaml**: devpi flipped from `type: argocd` to `type: ansible`. - **ArgoCD**: removed `apps/devpi.yaml` and `manifests/devpi/`. The in-cluster Application, namespace, and PVC have been deleted. - **Docs**: new how-to `docs/how-to/operations/devpi-on-indri.md`; `restart-indri.md` lists devpi in the LaunchAgent stop list. ## Already deployed (live on indri) - Service running: `launchctl list mcquack.eblume.devpi` → PID 53888 - `curl https://pypi.ops.eblu.me/+api` returns 200 ✅ - `mise run docs-mikado` works again ✅ - 1.0G of cached PyPI data was migrated from the PVC to `~erichblume/devpi/server-dir/` - Minikube namespace and PVC fully reclaimed ## Test plan - [ ] `mise run services-check` (after merge) - [ ] CI workflows that use devpi succeed - [ ] No regressions in tools that depend on `pypi.ops.eblu.me` (prek, uv-script tasks, dagger pipelines) ## Context This is the C1 prelude to a planned C2 chain (`mikado/retire-minikube-indri`) to retire minikube on indri entirely. Doing devpi as a standalone C1 was the right call because (a) it was urgent — it was breaking the toolchain — and (b) it shakes out the migration recipe before we commit to a multi-leaf chain. Reviewed-on: #341
97 lines
3.2 KiB
Markdown
97 lines
3.2 KiB
Markdown
---
|
|
title: Backups
|
|
modified: 2026-03-27
|
|
tags:
|
|
- storage
|
|
- backup
|
|
---
|
|
|
|
# Backup Policy
|
|
|
|
Daily automated backups from [[indri]] to [[sifaka|Sifaka]] NAS.
|
|
|
|
## Schedule
|
|
|
|
| Time | Frequency | System |
|
|
|------|-----------|--------|
|
|
| 2:00 AM | Daily | [[borgmatic]] |
|
|
|
|
## What Gets Backed Up
|
|
|
|
### Directories
|
|
|
|
| Path | Description | Priority |
|
|
|------|-------------|----------|
|
|
| `~/code/personal/zk` | Zettelkasten notes | Critical |
|
|
| `/opt/homebrew/var/forgejo` | Git repositories | Critical |
|
|
| `~/.config/borgmatic` | Backup config | High |
|
|
| `~/Documents` | Personal documents (includes [[1password]] encrypted export) | High |
|
|
|
|
### Databases
|
|
|
|
| Database | Cluster | Host | Method |
|
|
|----------|---------|------|--------|
|
|
| miniflux | blumeops-pg | [[postgresql|pg.ops.eblu.me:5432]] | pg_dump stream |
|
|
| teslamate | blumeops-pg | [[postgresql|pg.ops.eblu.me:5432]] | pg_dump stream |
|
|
| authentik | blumeops-pg | [[postgresql|pg.ops.eblu.me:5432]] | pg_dump stream |
|
|
| immich | immich-pg | [[postgresql|pg.ops.eblu.me:5433]] | pg_dump stream |
|
|
| mealie | — (SQLite) | k8s pod | kubectl exec sqlite3 .backup |
|
|
|
|
## Immich Photo Library (Offsite Only)
|
|
|
|
The [[immich]] photo library lives on [[sifaka]] at `/volume1/photos` (SMB-mounted on [[indri]] as `/Volumes/photos`). Since sifaka is already the local backup target, photos are backed up to BorgBase offsite only — not back to sifaka.
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| **Config** | `~/.config/borgmatic/photos.yaml` |
|
|
| **Schedule** | Daily at 4:00 AM (offset from main backup) |
|
|
| **Source** | `/Volumes/photos` (sifaka SMB mount) |
|
|
| **Target** | BorgBase `borgbase-immich-photos` repo |
|
|
| **Size** | ~128 GB |
|
|
|
|
Uses the same encryption passphrase and SSH key as the main borgmatic config.
|
|
|
|
## Sifaka-Native Data
|
|
|
|
Other data lives directly on [[sifaka]] (music via [[navidrome]], video via [[jellyfin]]). See [[sifaka]] for data protection details.
|
|
|
|
## What Is NOT Backed Up
|
|
|
|
| Data | Reason |
|
|
|------|--------|
|
|
| ZIM archives (`~/transmission/`) | Re-downloadable via torrent |
|
|
| Prometheus metrics | Ephemeral, in k8s PVC |
|
|
| Loki logs | Ephemeral, in k8s PVC |
|
|
| devpi cache (`~/devpi/server-dir/` on indri) | Re-fetchable from PyPI on first request |
|
|
|
|
## Retention Policy
|
|
|
|
| Period | Retention |
|
|
|--------|-----------|
|
|
| Daily | 7 backups |
|
|
| Monthly | 12 backups |
|
|
| Yearly | 1000 backups |
|
|
|
|
## Backup Targets
|
|
|
|
| Repository | Location | Label | Backs up |
|
|
|------------|----------|-------|----------|
|
|
| `/Volumes/backups/borg/` | [[sifaka]] (local NAS) | `sifaka-borg-backups` | indri data |
|
|
| `ssh://u3ugi1x1@...repo.borgbase.com/./repo` | BorgBase (offsite) | `borgbase-offsite` | indri data |
|
|
| `ssh://xcrtl5tg@...repo.borgbase.com/./repo` | BorgBase (offsite) | `borgbase-immich-photos` | immich photos |
|
|
|
|
## Monitoring
|
|
|
|
Metrics exposed to [[prometheus]]:
|
|
- `borgmatic_up` - Repository accessible
|
|
- `borgmatic_last_archive_timestamp` - Last backup time
|
|
- `borgmatic_repo_deduplicated_size_bytes` - Disk usage
|
|
|
|
Dashboard: "Borgmatic Backups" in [[grafana]]
|
|
|
|
## Related
|
|
|
|
- [[borgmatic]] - Backup system details
|
|
- [[sifaka|Sifaka]] - Backup storage
|
|
- [[postgresql]] - Database backups
|
|
- [[restore-1password-backup]] - Recover 1Password from backup
|