Erich Blume
dc69b8c68b
## Summary Nightly borgmatic backups have been failing for 2 days. Root cause: the shower SQLite dump `before_backup` hook (added in PR #349) referenced `kubectl --context=k3s-ringtail`, but indri's kubeconfig deliberately doesn't carry the ringtail credentials. The hook's failure aborted the entire run, taking out *both* the local sifaka repo and the BorgBase offsite. Verified the last good archive was `indri-2026-05-11T02:00`. ## Approach ssh into ringtail and run `k3s kubectl` there — no indri-side kubeconfig needed. `/etc/rancher/k3s/k3s.yaml` is mode 644 so no sudo required, and the existing ssh access from indri to ringtail works. Inline-shell quoting got hairy fast (fish on ringtail rejected `POD=...` bash syntax; the nix shower image lacks `tar` so `kubectl cp` fails). Pulled the dump logic into `~/bin/borgmatic-k8s-sqlite-dump`, deployed by the ansible role. Each dump entry now declares a `target`: - `local:<context>` — local kubectl with explicit context (mealie) - `ssh:<user@host>` — ssh + `k3s kubectl` on the cluster host (shower) Bytes come back via `kubectl exec ... -- cat` instead of `kubectl cp` since `cp` needs `tar` in the pod (nix-built containers don't bundle it). ## Test plan - [x] `mise run provision-indri -- --tags borgmatic --check --diff` shows expected diff - [x] Apply, helper script deployed at `~/bin/borgmatic-k8s-sqlite-dump` - [x] Helper invoked directly with `ssh:eblume@ringtail` produces a valid 288 KB SQLite file - [x] Full `borgmatic create` completes without errors — both mealie.db (1.7 MB) and shower.db (288 KB) appear in `~/.local/share/borgmatic/k8s-dumps/`, archive `indri-2026-05-13T17:31:02` written to sifaka borg repo 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #357
71 lines
2.3 KiB
Django/Jinja
71 lines
2.3 KiB
Django/Jinja
# {{ ansible_managed }}
|
|
|
|
# Path to borg binary (LaunchAgent doesn't have homebrew in PATH)
|
|
local_path: {{ borgmatic_local_path }}
|
|
|
|
source_directories:
|
|
{% for dir in borgmatic_source_directories %}
|
|
- {{ dir }}
|
|
{% endfor %}
|
|
|
|
source_directories_must_exist: true
|
|
|
|
repositories:
|
|
{% for repo in borgmatic_repositories %}
|
|
- path: {{ repo.path }}
|
|
label: {{ repo.label }}
|
|
{% if repo.encryption is defined %}
|
|
encryption: {{ repo.encryption }}
|
|
{% endif %}
|
|
{% if repo.append_only is defined %}
|
|
append_only: {{ repo.append_only | lower }}
|
|
{% endif %}
|
|
{% endfor %}
|
|
|
|
{% if borgmatic_exclude_patterns %}
|
|
exclude_patterns:
|
|
{% for pattern in borgmatic_exclude_patterns %}
|
|
- {{ pattern }}
|
|
{% endfor %}
|
|
{% endif %}
|
|
|
|
encryption_passcommand: {{ borgmatic_encryption_passcommand }}
|
|
|
|
{% if borgmatic_k8s_sqlite_dumps %}
|
|
# Pre-backup: dump SQLite databases from k8s pods.
|
|
# Uses sqlite3.backup() for a safe, consistent copy.
|
|
#
|
|
# Quoting/escaping is delegated to ~/bin/borgmatic-k8s-sqlite-dump
|
|
# (deployed by the borgmatic ansible role). Each entry's `target`
|
|
# is either:
|
|
# - local:<context> -> local kubectl with --context (mealie etc.)
|
|
# - ssh:<user@host> -> ssh + k3s kubectl on the cluster host,
|
|
# used for ringtail since indri's kubeconfig
|
|
# deliberately doesn't carry that context.
|
|
before_backup:
|
|
- mkdir -p {{ borgmatic_k8s_dump_dir }}
|
|
{% for db in borgmatic_k8s_sqlite_dumps %}
|
|
- {{ ansible_env.HOME }}/bin/borgmatic-k8s-sqlite-dump {{ db.target }} {{ db.namespace }} {{ db.label_selector }} {{ db.db_path }} {{ db.name }} {{ borgmatic_k8s_dump_dir }}/{{ db.name }}.db
|
|
{% endfor %}
|
|
{% endif %}
|
|
|
|
ssh_command: ssh -o IdentitiesOnly=yes -i {{ borgmatic_borgbase_ssh_key_path }}
|
|
|
|
# Retention policy
|
|
keep_daily: {{ borgmatic_keep_daily }}
|
|
keep_monthly: {{ borgmatic_keep_monthly }}
|
|
keep_yearly: {{ borgmatic_keep_yearly }}
|
|
|
|
{% if borgmatic_postgresql_databases %}
|
|
# PostgreSQL database backups (streamed via pg_dump)
|
|
postgresql_databases:
|
|
{% for db in borgmatic_postgresql_databases %}
|
|
- name: {{ db.name }}
|
|
hostname: {{ db.hostname | default('localhost') }}
|
|
port: {{ db.port | default(5432) }}
|
|
username: {{ db.username }}
|
|
{% if borgmatic_pg_dump_command is defined %}
|
|
pg_dump_command: {{ borgmatic_pg_dump_command }}
|
|
{% endif %}
|
|
{% endfor %}
|
|
{% endif %}
|