- Change driver from qemu2 to docker
- Remove socket_vmnet and qemu dependencies
- Remove NFS mount and minikube mount LaunchAgent/LaunchDaemon
- Remove old podman zot-mirror.conf
- Update containerd registry mirror config for docker driver
- Uses host.minikube.internal:5050 to reach zot
- Configures pull-through cache for docker.io, ghcr.io, quay.io
- Add dynamic tailscale serve configuration for k8s API
(port is dynamic with docker driver, not fixed at 6443)
- Remove svc:k8s from tailscale_serve defaults (minikube role handles it)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The k8s API server is now at 192.168.105.2:6443 (inside qemu2 VM)
instead of localhost:44491 (old podman port mapping).
Note: TCP passthrough via tailscale svc:k8s is configured but
connection times out - may need admin console approval or debugging.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Check containerd registry config before writing to avoid unnecessary changes
- Fix ansible_env deprecation warnings (use ansible_facts['env'])
- Document macOS network permission popup for minikube mount
- Document passwordless sudo configuration for indri
- Add checks to skip sudo tasks when state already matches
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- LaunchDaemon: mounts sifaka:/volume1/torrents to /Volumes/torrents-nfs at boot
- LaunchAgent: runs minikube mount to pass through to /mnt/torrents in VM
- Handlers to load both services when plist files change
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add hosts file entry for registry.tail8d86e.ts.net in VM
- Configure containerd registry mirror to use local zot
- Update P5.1 doc with implementation notes and manual steps
- Mark P5.1 as complete
Manual steps still required after cluster creation:
1. sudo brew services start socket_vmnet (once per reboot)
2. sudo mount -t nfs sifaka:/volume1/torrents /Volumes/torrents-nfs
3. minikube mount /Volumes/torrents-nfs:/mnt/torrents (GUI session)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Install socket_vmnet via homebrew
- Start socket_vmnet service (requires sudo)
- Add --network=socket_vmnet to minikube start
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Change minikube driver from podman to qemu2
- Change container runtime from cri-o to containerd
- Add qemu installation to minikube role
- Remove podman role from indri.yml playbook
- Update handlers for containerd instead of cri-o
- Temporarily disable registry mirror config (needs containerd format)
- Add k8s-storage synology user creation steps to P5.1 doc
- Add post-migration tasks for zot registry mirror reconfiguration
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Fixed borgmatic `borg: command not found` by adding `local_path` config option
- Successfully tested disaster recovery: restored miniflux data from borgmatic backup to k8s-pg
- Added borgmatic user to k8s-pg via CloudNativePG managed roles
- Configured borgmatic to backup both localhost and k8s-pg PostgreSQL databases
- Added Tailscale ACL grant for `tag:homelab` → `tag:k8s` on port 5432
- Disabled selfHeal on apps app to allow manual revision changes during development
## Changes
- `ansible/roles/borgmatic/` - Added `local_path` and k8s-pg database entry
- `ansible/roles/postgresql/tasks/main.yml` - Added k8s-pg to `.pgpass`
- `argocd/apps/apps.yaml` - Disabled selfHeal
- `argocd/manifests/databases/blumeops-pg.yaml` - Added borgmatic managed role
- `argocd/manifests/databases/secret-borgmatic.yaml.tpl` - New secret template
- `pulumi/policy.hujson` - Added ACL grant for backup access
## Deployment and Testing
- [x] Borgmatic backup runs successfully
- [x] Miniflux data restored to k8s-pg (2 users, 2 feeds, 44 entries verified)
- [x] borgmatic user created in k8s-pg with pg_read_all_data role
- [x] Both localhost and k8s-pg databases in backup archive
- [x] zk documentation updated (borgmatic.md, postgresql.md)
- [ ] After merge: set blumeops-pg app back to main revision
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/32
## Summary
- Fixed borgmatic-metrics script failing in LaunchAgent context
- Changed from `mise x -- borg` to absolute paths (`/opt/homebrew/bin/borg`, `/opt/homebrew/bin/jq`)
- This fixes the Grafana dashboard showing "DOWN" for Repository Status and missing time series data
## Deployment and Testing
- [ ] Run `mise run provision-indri -- --tags borgmatic-metrics` to deploy the fix
- [ ] Wait for the hourly metrics collection (or manually run `ssh indri '~/bin/borgmatic-metrics'`)
- [ ] Verify Grafana dashboard shows "UP" status and populated graphs
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/28
## Summary
- Add `tag:k8s-api` to Pulumi ACLs and indri device tags
- Configure Tailscale serve with TCP passthrough for k8s API at `k8s.tail8d86e.ts.net`
- Update minikube role to include `k8s.tail8d86e.ts.net` in certificate SANs
- Add `apiserver_port` config option (internal port 6443, dynamic host port with podman driver)
- Document Step 0.14 in k8s-migration plan (added post-Phase 0 completion)
The Kubernetes API is now accessible at `https://k8s.tail8d86e.ts.net` using TCP passthrough to preserve mTLS authentication.
## Deployment and Testing
- [x] Pulumi ACLs applied
- [x] Tailscale service created and approved in admin console
- [x] Minikube cluster recreated with new cert SANs
- [x] tailscale serve configured with TCP passthrough
- [x] 1Password credentials updated with new certs
- [x] Kubeconfig updated on gilbert
- [x] `mise run indri-services-check` passes
- [x] `kubectl --context=minikube-indri get nodes` works via Tailscale
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/27
## Summary
- Remove all `meta/main.yml` dependencies from ansible roles
- Role ordering is now controlled entirely by `indri.yml` playbook
- Fix incorrect roles path in CLAUDE.md (`playbooks/roles` → `roles`)
## Why
Ansible's tag accumulation behavior prevents proper role deduplication when using meta dependencies. When a role is pulled in as a dependency, the parent role's tags are added to the dependency's tags (e.g., `[loki]` becomes `[alloy, loki]`), making them appear as different invocations to Ansible and causing roles to run multiple times.
## Deployment and Testing
- [x] Verified with `ansible-playbook --list-tasks` that each role now appears exactly once
- [x] Run full provision to verify no regressions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/20
## Summary
- Simplify kiwix role from 213 lines to 151 lines (-30%)
- Replace per-archive torrent status loops with single shell command
- Decouple kiwix startup from declared inventory - now serves whatever completed ZIM files exist
- Fix tailscale_serve role to handle empty JSON in check mode
## Performance improvement
- **Before**: ~132 operations (44 archives × 3 loops for status check, recheck, symlink)
- **After**: ~5 operations (1 shell script + 1 find + conditional symlinks)
- Expected reduction: ~3 minutes per ansible run
## Test plan
- [x] Ran `mise run provision-indri -- --check --diff` to preview changes
- [x] Ran `mise run provision-indri` to apply changes
- [x] Ran `mise run indri-services-check` - all services healthy
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/18
## Summary
- Add `postgresql_superuser` variable (`eblume`) to prevent PostgreSQL from inheriting OS username during initdb
- Update all psql/createdb commands to use explicit `-U` flag
- Add `check_mode: false` to op commands so 1Password fetches run during `--check` mode
- Add PostgreSQL and Miniflux health checks to indri-services-check
## Test plan
- [x] Renamed existing superuser from `erichblume` to `eblume`
- [x] Ran `mise run provision-indri -- --tags postgresql --check --diff` successfully
- [x] Verified connection as `eblume` superuser via Tailscale
- [x] Ran `mise run indri-services-check` - all services healthy
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/17
## Summary
- Manage tail8d86e.ts.net ACLs, tags, and DNS via Pulumi + Python
- State stored in Pulumi Cloud (free tier) to avoid circular dependency
- OAuth authentication via 1Password for secure credential management
- New mise tasks: `tailnet-preview`, `tailnet-up`
## Architecture
Two-layer approach:
- **Layer 1 (Pulumi)**: Tailnet-wide config (ACLs, tags, DNS)
- **Layer 2 (Ansible)**: Node-local `tailscale serve` config (unchanged)
## Test plan
- [x] Exported current ACL from Tailscale API
- [x] Imported existing ACL into Pulumi state
- [x] Verified `mise run tailnet-preview` shows no changes
- [x] Verified `mise run tailnet-up` applies successfully
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/15
## Summary
- Use async with poll: 0 for alloy and loki restart handlers
- Fire-and-forget approach prevents ansible from hanging on graceful shutdown
## Test plan
- [x] Manually verified `brew services restart grafana-alloy` works
- [x] Run full ansible playbook and verify it completes without timeout
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/12
Use async with poll: 0 to fire-and-forget service restarts.
These services have graceful shutdown periods that can exceed
ansible's default command timeout.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Add ansible role for devpi-server as a transparent PyPI caching proxy
- LaunchAgent with KeepAlive runs via `mise x -- devpi-server`
- Listens on port 3141, data stored in `~/devpi`
- Health checks added to `indri-services-check` script
## Manual Setup Required (on indri, before provisioning)
1. Add to `~/.config/mise/config.toml`:
```toml
[tools]
"pipx:devpi-server" = "latest"
"pipx:devpi-web" = "latest"
"pipx:devpi-client" = "latest"
```
2. Run `mise install`
3. Initialize: `mise x -- devpi-init --serverdir ~/devpi`
## Post-Provisioning
- Set up Tailscale service `pypi` on port 443 → 3141
- Configure client pip.conf with index-url
## Test plan
- [x] Ansible syntax check passes
- [x] Dry-run: `mise run provision-indri -- --check --diff`
- [x] Apply: `mise run provision-indri`
- [x] Health check: `mise run indri-services-check`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/9
## Summary
- Adds a new Grafana dashboard for Node Exporter metrics on macOS hosts
- Uses macOS-native memory metrics (node_memory_total_bytes, node_memory_active_bytes, etc.) instead of Linux-specific ones
- Includes dropdown selectors for instance, disk, and network device filtering
## Details
The standard Node Exporter dashboards show "No Data" for memory panels on macOS because they query Linux-specific metrics like `node_memory_MemTotal_bytes`. macOS node_exporter exports different metrics:
| Linux | macOS |
|-------|-------|
| node_memory_MemTotal_bytes | node_memory_total_bytes |
| node_memory_MemFree_bytes | node_memory_free_bytes |
| node_memory_Buffers_bytes | (not available) |
| node_memory_Cached_bytes | (not available) |
macOS has unique memory categories: Wired, Active, Compressed, Inactive, Free.
## Test plan
- [x] Dashboard deployed to indri via ansible
- [x] All panels showing data for indri
- [x] Instance selector works to switch between hosts
- [x] Disk and network device filters work
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/8
## Summary
- Adds Upload/Download Ratio stat panel with color thresholds (red < 0.5, yellow < 1, green >= 1)
- Adds Downloaded (Period) stat panel showing bytes downloaded in selected time range
- Adds Uploaded (Period) stat panel showing bytes uploaded in selected time range
Uses PromQL `increase()` on existing counter metrics - no new metrics collection needed.
## Test plan
- [x] Deployed to indri via `mise run provision-indri`
- [x] Grafana restarted successfully
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/6
- Add mise-tasks/provision-indri script to run ansible playbook
- Fix transmission_metrics launchctl load to be idempotent
- Update CLAUDE.md to reference mise run provision-indri
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Query torrent-get RPC to sum totalSize of all torrents
- Add transmission_torrents_size_bytes gauge metric
- Add "Total Torrent Size" timeseries panel to dashboard
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Transmission doesn't support HEAD requests, so use -i flag with sed to
parse only the HTTP headers (stopping at the blank line before body).
Also anchor grep pattern to line start to avoid matching HTML content.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add node_exporter ansible role to enable textfile collector
- Add transmission_metrics role with script and LaunchAgent
- Collects metrics every 60s via transmission RPC
- Writes to /opt/homebrew/var/node_exporter/textfile/transmission.prom
- Update grafana role to provision dashboards from files
- Add transmission.json dashboard with:
- Status indicator, torrent counts
- Transfer speeds, cumulative stats
- Time series graphs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The LaunchAgent plist now dynamically includes only ZIM files that
actually exist in the kiwix directory, rather than all configured
archives. This prevents kiwix-serve from crashing when torrents are
still downloading.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Expand settings.json template to include all transmission defaults
- Use static pre-hashed rpc-password so transmission doesn't regenerate
- Change file mode from 0644 to 0600 to match transmission's default
- Add Jinja comment explaining the RPC password workaround
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Homebrew's transmission-cli service uses /opt/homebrew/var/transmission/
not ~/.config/transmission-daemon/
- Add task to clean up old config directory
- Update zettelkasten with correct paths
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add new transmission ansible role using homebrew + brew services
- Configure transmission to download to ~/transmission with localhost-only RPC
- Modify kiwix role to use transmission for downloading ZIM archives via BitTorrent
- Add role dependency so running --tags kiwix auto-runs transmission
- Keep fallback to direct HTTP download when kiwix_use_transmission: false
- Symlink completed downloads from transmission dir to kiwix-tools dir
This reduces load on kiwix.org servers and allows downloads to continue
in the background without blocking ansible runs.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The LaunchAgent was failing because launchd runs with a minimal PATH
that doesn't include mise-installed binaries or homebrew. This adds:
- Use `mise x` wrapper to run borgmatic (survives version updates)
- Add /opt/homebrew/bin to PATH for borg dependency
- Add ansible tags to indri playbook for targeted role runs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Configure grafana to use provisioned datasources instead of UI config
- Add prometheus datasource template managed by ansible
- Create minimal grafana.ini with custom provisioning path
- Move ansible_managed to group_vars (fixes deprecation warning)
- Add Remote Hosts and Git Workflow sections to CLAUDE.md
- Document feature branch workflow with tea CLI for PRs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Manages installation and service via homebrew. Config at
/opt/homebrew/var/forgejo/custom/conf/app.ini contains secrets
and is not templated - backed up by borgmatic instead.
Includes check that fails with restore instructions if config missing.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Manages scheduled LaunchAgent for daily backups at 2:00 AM.
Borgmatic itself is installed via mise (pipx), not managed by ansible.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Configure ZIM archives as a variable list with download URLs
- Auto-download missing archives from download.kiwix.org
- Template plist to serve all configured archives
- Skip checksum calculation on stat for performance
- Add commented options for Gutenberg, iFixit, Stack Exchange, LibreTexts
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
