blumeops/ansible/roles/borgmatic/templates/k8s-sqlite-dump.sh.j2

73 lines
2.8 KiB
Text
Raw Normal View History

C1: fix borgmatic shower SQLite dump (ssh to ringtail) (#357) ## Summary Nightly borgmatic backups have been failing for 2 days. Root cause: the shower SQLite dump `before_backup` hook (added in PR #349) referenced `kubectl --context=k3s-ringtail`, but indri's kubeconfig deliberately doesn't carry the ringtail credentials. The hook's failure aborted the entire run, taking out *both* the local sifaka repo and the BorgBase offsite. Verified the last good archive was `indri-2026-05-11T02:00`. ## Approach ssh into ringtail and run `k3s kubectl` there — no indri-side kubeconfig needed. `/etc/rancher/k3s/k3s.yaml` is mode 644 so no sudo required, and the existing ssh access from indri to ringtail works. Inline-shell quoting got hairy fast (fish on ringtail rejected `POD=...` bash syntax; the nix shower image lacks `tar` so `kubectl cp` fails). Pulled the dump logic into `~/bin/borgmatic-k8s-sqlite-dump`, deployed by the ansible role. Each dump entry now declares a `target`: - `local:<context>` — local kubectl with explicit context (mealie) - `ssh:<user@host>` — ssh + `k3s kubectl` on the cluster host (shower) Bytes come back via `kubectl exec ... -- cat` instead of `kubectl cp` since `cp` needs `tar` in the pod (nix-built containers don't bundle it). ## Test plan - [x] `mise run provision-indri -- --tags borgmatic --check --diff` shows expected diff - [x] Apply, helper script deployed at `~/bin/borgmatic-k8s-sqlite-dump` - [x] Helper invoked directly with `ssh:eblume@ringtail` produces a valid 288 KB SQLite file - [x] Full `borgmatic create` completes without errors — both mealie.db (1.7 MB) and shower.db (288 KB) appear in `~/.local/share/borgmatic/k8s-dumps/`, archive `indri-2026-05-13T17:31:02` written to sifaka borg repo 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/357
2026-05-13 18:55:50 -07:00
#!/usr/bin/env bash
# {{ ansible_managed }}
#
# Helper script invoked by borgmatic's before_backup hook to capture a
# k8s pod's SQLite database. Keeps the borgmatic config readable by
# pulling all the quoting out of YAML.
#
# Usage:
# borgmatic-k8s-sqlite-dump <target> <namespace> <selector> \
# <db_path> <name> <dump_target>
#
# <target> is one of:
# local:<context> - run local kubectl with --context=<context>
# ssh:<user@host> - ssh to host and run k3s kubectl there
# (no indri-side kubeconfig needed)
#
# <namespace> - k8s namespace of the pod
# <selector> - label selector to find the pod (e.g. app=shower)
# <db_path> - absolute path inside the pod to the SQLite DB
# <name> - short name used for temp filenames
# <dump_target> - file on this host to receive the dump
set -euo pipefail
target=${1:?missing target}
namespace=${2:?missing namespace}
selector=${3:?missing selector}
db_path=${4:?missing db path}
name=${5:?missing name}
dump_target=${6:?missing dump target}
# Stage the backup next to the source DB (a guaranteed-writable volume);
# minimal nix images (e.g. mealie) have no /tmp.
pod_tmp="$(dirname "$db_path")/.borgmatic-backup-${name}.db"
C1: fix borgmatic shower SQLite dump (ssh to ringtail) (#357) ## Summary Nightly borgmatic backups have been failing for 2 days. Root cause: the shower SQLite dump `before_backup` hook (added in PR #349) referenced `kubectl --context=k3s-ringtail`, but indri's kubeconfig deliberately doesn't carry the ringtail credentials. The hook's failure aborted the entire run, taking out *both* the local sifaka repo and the BorgBase offsite. Verified the last good archive was `indri-2026-05-11T02:00`. ## Approach ssh into ringtail and run `k3s kubectl` there — no indri-side kubeconfig needed. `/etc/rancher/k3s/k3s.yaml` is mode 644 so no sudo required, and the existing ssh access from indri to ringtail works. Inline-shell quoting got hairy fast (fish on ringtail rejected `POD=...` bash syntax; the nix shower image lacks `tar` so `kubectl cp` fails). Pulled the dump logic into `~/bin/borgmatic-k8s-sqlite-dump`, deployed by the ansible role. Each dump entry now declares a `target`: - `local:<context>` — local kubectl with explicit context (mealie) - `ssh:<user@host>` — ssh + `k3s kubectl` on the cluster host (shower) Bytes come back via `kubectl exec ... -- cat` instead of `kubectl cp` since `cp` needs `tar` in the pod (nix-built containers don't bundle it). ## Test plan - [x] `mise run provision-indri -- --tags borgmatic --check --diff` shows expected diff - [x] Apply, helper script deployed at `~/bin/borgmatic-k8s-sqlite-dump` - [x] Helper invoked directly with `ssh:eblume@ringtail` produces a valid 288 KB SQLite file - [x] Full `borgmatic create` completes without errors — both mealie.db (1.7 MB) and shower.db (288 KB) appear in `~/.local/share/borgmatic/k8s-dumps/`, archive `indri-2026-05-13T17:31:02` written to sifaka borg repo 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.eblu.me/eblume/blumeops/pulls/357
2026-05-13 18:55:50 -07:00
python_backup='import sqlite3; sqlite3.connect("'"$db_path"'").backup(sqlite3.connect("'"$pod_tmp"'"))'
mode=${target%%:*}
ref=${target#*:}
case "$mode" in
local)
# Pulls dump bytes out via "kubectl exec -- cat" rather than
# "kubectl cp", which would otherwise need tar inside the pod
# (nix-built images like shower don't bundle tar).
context=$ref
kubectl="/opt/homebrew/bin/kubectl --context=$context -n $namespace"
pod=$($kubectl get pod -l "$selector" \
-o jsonpath='{.items[0].metadata.name}')
$kubectl exec "$pod" -- python3 -c "$python_backup"
$kubectl exec "$pod" -- cat "$pod_tmp" > "$dump_target"
$kubectl exec "$pod" -- rm -f "$pod_tmp"
;;
ssh)
host=$ref
# Force bash on the remote (user's login shell on ringtail is
# fish). Pipe the script via stdin to dodge nested quoting.
# The dump bytes come back over the ssh stdout stream — no
# intermediate scp, no tar requirement in the pod.
ssh "$host" bash <<EOF > "$dump_target"
set -euo pipefail
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
pod=\$(k3s kubectl -n "$namespace" get pod -l "$selector" -o jsonpath='{.items[0].metadata.name}')
k3s kubectl -n "$namespace" exec "\$pod" -- python3 -c '$python_backup' 1>&2
k3s kubectl -n "$namespace" exec "\$pod" -- cat "$pod_tmp"
k3s kubectl -n "$namespace" exec "\$pod" -- rm -f "$pod_tmp" 1>&2
EOF
;;
*)
echo "borgmatic-k8s-sqlite-dump: unknown target mode: $mode" >&2
echo " expected local:<context> or ssh:<user@host>" >&2
exit 1
;;
esac