C1: fix borgmatic shower SQLite dump (ssh to ringtail) #357

Merged

eblume merged 1 commit from fix-borgmatic-shower-via-ssh into main

2026-05-13 18:55:51 -07:00

eblume commented

2026-05-13 17:38:49 -07:00

Owner

Summary

Nightly borgmatic backups have been failing for 2 days. Root cause: the
shower SQLite dump before_backup hook (added in PR #349) referenced
kubectl --context=k3s-ringtail, but indri's kubeconfig deliberately
doesn't carry the ringtail credentials. The hook's failure aborted the
entire run, taking out both the local sifaka repo and the BorgBase
offsite. Verified the last good archive was indri-2026-05-11T02:00.

Approach

ssh into ringtail and run k3s kubectl there — no indri-side
kubeconfig needed. /etc/rancher/k3s/k3s.yaml is mode 644 so no sudo
required, and the existing ssh access from indri to ringtail works.

Inline-shell quoting got hairy fast (fish on ringtail rejected POD=...
bash syntax; the nix shower image lacks tar so kubectl cp fails).
Pulled the dump logic into ~/bin/borgmatic-k8s-sqlite-dump, deployed
by the ansible role. Each dump entry now declares a target:

local:<context> — local kubectl with explicit context (mealie)
ssh:<user@host> — ssh + k3s kubectl on the cluster host (shower)

Bytes come back via kubectl exec ... -- cat instead of kubectl cp
since cp needs tar in the pod (nix-built containers don't bundle it).

Test plan

mise run provision-indri -- --tags borgmatic --check --diff shows expected diff
Apply, helper script deployed at ~/bin/borgmatic-k8s-sqlite-dump
Helper invoked directly with ssh:eblume@ringtail produces a valid 288 KB SQLite file
Full borgmatic create completes without errors — both mealie.db (1.7 MB) and shower.db (288 KB) appear in ~/.local/share/borgmatic/k8s-dumps/, archive indri-2026-05-13T17:31:02 written to sifaka borg repo

🤖 Generated with Claude Code

## Summary Nightly borgmatic backups have been failing for 2 days. Root cause: the shower SQLite dump `before_backup` hook (added in PR #349) referenced `kubectl --context=k3s-ringtail`, but indri's kubeconfig deliberately doesn't carry the ringtail credentials. The hook's failure aborted the entire run, taking out *both* the local sifaka repo and the BorgBase offsite. Verified the last good archive was `indri-2026-05-11T02:00`. ## Approach ssh into ringtail and run `k3s kubectl` there — no indri-side kubeconfig needed. `/etc/rancher/k3s/k3s.yaml` is mode 644 so no sudo required, and the existing ssh access from indri to ringtail works. Inline-shell quoting got hairy fast (fish on ringtail rejected `POD=...` bash syntax; the nix shower image lacks `tar` so `kubectl cp` fails). Pulled the dump logic into `~/bin/borgmatic-k8s-sqlite-dump`, deployed by the ansible role. Each dump entry now declares a `target`: - `local:<context>` — local kubectl with explicit context (mealie) - `ssh:<user@host>` — ssh + `k3s kubectl` on the cluster host (shower) Bytes come back via `kubectl exec ... -- cat` instead of `kubectl cp` since `cp` needs `tar` in the pod (nix-built containers don't bundle it). ## Test plan - [x] `mise run provision-indri -- --tags borgmatic --check --diff` shows expected diff - [x] Apply, helper script deployed at `~/bin/borgmatic-k8s-sqlite-dump` - [x] Helper invoked directly with `ssh:eblume@ringtail` produces a valid 288 KB SQLite file - [x] Full `borgmatic create` completes without errors — both mealie.db (1.7 MB) and shower.db (288 KB) appear in `~/.local/share/borgmatic/k8s-dumps/`, archive `indri-2026-05-13T17:31:02` written to sifaka borg repo 🤖 Generated with [Claude Code](https://claude.com/claude-code)

eblume added 1 commit

2026-05-13 17:38:50 -07:00

C1: borgmatic shower SQLite dump via ssh to ringtail f26105dea9

The shower dump hook referenced kubectl --context=k3s-ringtail, but
indri's kubeconfig deliberately doesn't carry the ringtail
credentials. Since PR #349 (2026-05-11), nightly borgmatic runs have
failed at the before_backup hook, aborting both sifaka-borg-backups
and borgbase-offsite.

Rewrite the dump to ssh into ringtail and run k3s kubectl there.
/etc/rancher/k3s/k3s.yaml on ringtail is mode 644, so no sudo is
needed; the ssh user (eblume) reads it directly. Dump file is
created in the pod via sqlite3.backup, copied to ringtail's host
filesystem via k3s kubectl cp, then scp'd back to indri.

Template gains a `ssh_host` field on dump entries — when set, uses
the ssh path; when absent (as for mealie), uses local kubectl with
the existing `context` field.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>