Erich Blume
dc69b8c68b
## Summary Nightly borgmatic backups have been failing for 2 days. Root cause: the shower SQLite dump `before_backup` hook (added in PR #349) referenced `kubectl --context=k3s-ringtail`, but indri's kubeconfig deliberately doesn't carry the ringtail credentials. The hook's failure aborted the entire run, taking out *both* the local sifaka repo and the BorgBase offsite. Verified the last good archive was `indri-2026-05-11T02:00`. ## Approach ssh into ringtail and run `k3s kubectl` there — no indri-side kubeconfig needed. `/etc/rancher/k3s/k3s.yaml` is mode 644 so no sudo required, and the existing ssh access from indri to ringtail works. Inline-shell quoting got hairy fast (fish on ringtail rejected `POD=...` bash syntax; the nix shower image lacks `tar` so `kubectl cp` fails). Pulled the dump logic into `~/bin/borgmatic-k8s-sqlite-dump`, deployed by the ansible role. Each dump entry now declares a `target`: - `local:<context>` — local kubectl with explicit context (mealie) - `ssh:<user@host>` — ssh + `k3s kubectl` on the cluster host (shower) Bytes come back via `kubectl exec ... -- cat` instead of `kubectl cp` since `cp` needs `tar` in the pod (nix-built containers don't bundle it). ## Test plan - [x] `mise run provision-indri -- --tags borgmatic --check --diff` shows expected diff - [x] Apply, helper script deployed at `~/bin/borgmatic-k8s-sqlite-dump` - [x] Helper invoked directly with `ssh:eblume@ringtail` produces a valid 288 KB SQLite file - [x] Full `borgmatic create` completes without errors — both mealie.db (1.7 MB) and shower.db (288 KB) appear in `~/.local/share/borgmatic/k8s-dumps/`, archive `indri-2026-05-13T17:31:02` written to sifaka borg repo 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #357
71 lines
2.7 KiB
Django/Jinja
71 lines
2.7 KiB
Django/Jinja
#!/usr/bin/env bash
|
|
# {{ ansible_managed }}
|
|
#
|
|
# Helper script invoked by borgmatic's before_backup hook to capture a
|
|
# k8s pod's SQLite database. Keeps the borgmatic config readable by
|
|
# pulling all the quoting out of YAML.
|
|
#
|
|
# Usage:
|
|
# borgmatic-k8s-sqlite-dump <target> <namespace> <selector> \
|
|
# <db_path> <name> <dump_target>
|
|
#
|
|
# <target> is one of:
|
|
# local:<context> - run local kubectl with --context=<context>
|
|
# ssh:<user@host> - ssh to host and run k3s kubectl there
|
|
# (no indri-side kubeconfig needed)
|
|
#
|
|
# <namespace> - k8s namespace of the pod
|
|
# <selector> - label selector to find the pod (e.g. app=shower)
|
|
# <db_path> - absolute path inside the pod to the SQLite DB
|
|
# <name> - short name used for temp filenames
|
|
# <dump_target> - file on this host to receive the dump
|
|
set -euo pipefail
|
|
|
|
target=${1:?missing target}
|
|
namespace=${2:?missing namespace}
|
|
selector=${3:?missing selector}
|
|
db_path=${4:?missing db path}
|
|
name=${5:?missing name}
|
|
dump_target=${6:?missing dump target}
|
|
|
|
pod_tmp="/tmp/${name}-backup.db"
|
|
|
|
python_backup='import sqlite3; sqlite3.connect("'"$db_path"'").backup(sqlite3.connect("'"$pod_tmp"'"))'
|
|
|
|
mode=${target%%:*}
|
|
ref=${target#*:}
|
|
|
|
case "$mode" in
|
|
local)
|
|
# Pulls dump bytes out via "kubectl exec -- cat" rather than
|
|
# "kubectl cp", which would otherwise need tar inside the pod
|
|
# (nix-built images like shower don't bundle tar).
|
|
context=$ref
|
|
kubectl="/opt/homebrew/bin/kubectl --context=$context -n $namespace"
|
|
pod=$($kubectl get pod -l "$selector" \
|
|
-o jsonpath='{.items[0].metadata.name}')
|
|
$kubectl exec "$pod" -- python3 -c "$python_backup"
|
|
$kubectl exec "$pod" -- cat "$pod_tmp" > "$dump_target"
|
|
$kubectl exec "$pod" -- rm -f "$pod_tmp"
|
|
;;
|
|
ssh)
|
|
host=$ref
|
|
# Force bash on the remote (user's login shell on ringtail is
|
|
# fish). Pipe the script via stdin to dodge nested quoting.
|
|
# The dump bytes come back over the ssh stdout stream — no
|
|
# intermediate scp, no tar requirement in the pod.
|
|
ssh "$host" bash <<EOF > "$dump_target"
|
|
set -euo pipefail
|
|
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
|
|
pod=\$(k3s kubectl -n "$namespace" get pod -l "$selector" -o jsonpath='{.items[0].metadata.name}')
|
|
k3s kubectl -n "$namespace" exec "\$pod" -- python3 -c '$python_backup' 1>&2
|
|
k3s kubectl -n "$namespace" exec "\$pod" -- cat "$pod_tmp"
|
|
k3s kubectl -n "$namespace" exec "\$pod" -- rm -f "$pod_tmp" 1>&2
|
|
EOF
|
|
;;
|
|
*)
|
|
echo "borgmatic-k8s-sqlite-dump: unknown target mode: $mode" >&2
|
|
echo " expected local:<context> or ssh:<user@host>" >&2
|
|
exit 1
|
|
;;
|
|
esac
|