Erich Blume
dc69b8c68b
## Summary Nightly borgmatic backups have been failing for 2 days. Root cause: the shower SQLite dump `before_backup` hook (added in PR #349) referenced `kubectl --context=k3s-ringtail`, but indri's kubeconfig deliberately doesn't carry the ringtail credentials. The hook's failure aborted the entire run, taking out *both* the local sifaka repo and the BorgBase offsite. Verified the last good archive was `indri-2026-05-11T02:00`. ## Approach ssh into ringtail and run `k3s kubectl` there — no indri-side kubeconfig needed. `/etc/rancher/k3s/k3s.yaml` is mode 644 so no sudo required, and the existing ssh access from indri to ringtail works. Inline-shell quoting got hairy fast (fish on ringtail rejected `POD=...` bash syntax; the nix shower image lacks `tar` so `kubectl cp` fails). Pulled the dump logic into `~/bin/borgmatic-k8s-sqlite-dump`, deployed by the ansible role. Each dump entry now declares a `target`: - `local:<context>` — local kubectl with explicit context (mealie) - `ssh:<user@host>` — ssh + `k3s kubectl` on the cluster host (shower) Bytes come back via `kubectl exec ... -- cat` instead of `kubectl cp` since `cp` needs `tar` in the pod (nix-built containers don't bundle it). ## Test plan - [x] `mise run provision-indri -- --tags borgmatic --check --diff` shows expected diff - [x] Apply, helper script deployed at `~/bin/borgmatic-k8s-sqlite-dump` - [x] Helper invoked directly with `ssh:eblume@ringtail` produces a valid 288 KB SQLite file - [x] Full `borgmatic create` completes without errors — both mealie.db (1.7 MB) and shower.db (288 KB) appear in `~/.local/share/borgmatic/k8s-dumps/`, archive `indri-2026-05-13T17:31:02` written to sifaka borg repo 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #357
14 lines
887 B
Markdown
14 lines
887 B
Markdown
Fix nightly borgmatic backups failing for 2 days. The shower SQLite
|
|
dump hook referenced `kubectl --context=k3s-ringtail`, but indri's
|
|
kubeconfig deliberately doesn't carry the ringtail credentials. The
|
|
`before_backup` hook's failure aborted the entire run, taking out
|
|
*both* the local sifaka repo and the BorgBase offsite. Replaced
|
|
the inline-shell dump with a `~/bin/borgmatic-k8s-sqlite-dump`
|
|
helper deployed by the ansible role. Each dump entry now declares a
|
|
`target` of either `local:<context>` (mealie — kubectl uses indri's
|
|
kubeconfig) or `ssh:<user@host>` (shower — ssh into ringtail and
|
|
run `k3s kubectl` there, no indri-side kubeconfig needed; k3s.yaml
|
|
on ringtail is mode 644 so no sudo required). Bytes stream back via
|
|
`kubectl exec ... -- cat` rather than `kubectl cp`, since `kubectl
|
|
cp` requires `tar` inside the pod and nix-built images like shower
|
|
don't bundle it.
|