Alloy labels scraped metrics with job="prometheus.scrape.zot",
not just "zot". Updated all dashboard queries to match.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The {{.Host}} Go template syntax was being interpreted by Jinja2.
Added {% raw %} block to escape it.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
All Phase 0 steps completed:
- Zot registry with pull-through cache
- Podman container runtime
- Minikube Kubernetes cluster
- Remote kubectl access with 1Password credentials
- Health checks in indri-services-check
- Zettelkasten documentation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Step 0.11 implementation:
- Check minikube status via SSH to indri
- Check k8s API server accessible from indri
- Check k8s API server accessible remotely from gilbert (via 1Password creds)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Step 0.12 update:
- Document ki, k9i, k9 abbreviations for quick kubectl/k9s access
- These avoid accidentally triggering work SAML flow
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Step 0.10 implementation:
- Recreate minikube with --apiserver-names=indri --listen-address=0.0.0.0
- Add kubectl-credential-1password exec plugin for 1Password integration
- Client certs fetched from 1Password on-demand (no private keys on disk)
- CA cert stored locally (not secret - public key for server verification)
Minikube role updates:
- Add minikube_apiserver_names and minikube_listen_address variables
- Update tasks to include remote access flags
This mirrors the 1Password SSH agent pattern - biometric auth required
for each kubectl command that needs credentials.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Step 0.10 (kubeconfig on gilbert):
- Document research on kubectl remote access options
- Choose --apiserver-names + --listen-address approach
- Add references to sources
Step 0.12 (zettelkasten):
- Add instructions to update main blumeops card
- Fix zot port from 5000 to 5050
- Add minikube.md template with remote access docs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create ansible/roles/minikube for minikube cluster setup
- Use podman driver with cri-o runtime
- Set memory to 7800MB (vs 8192 podman) to account for VM overhead
- Add minikube role to indri playbook
- Update k8s-migration plan with implementation details
Deployed with Kubernetes v1.34.0 and CRI-O 1.24.6.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create ansible/roles/podman for podman machine setup on indri
- Document known reliability issue with podman machine init/start via SSH
(race condition from containers/podman#16945)
- Role attempts init/start but doesn't fail if start hangs
- Workaround: manual init/start on indri if needed
- Update k8s-migration plan with implementation details
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- zot LaunchAgent running
- zot-metrics LaunchAgent running
- Zot registry HTTP endpoint (via Tailscale service URL)
- Zot metrics file exists
Also updated Grafana, Kiwix, Forgejo, Devpi to use Tailscale
service URLs for consistency with Miniflux and Zot.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Collects zot_up metric via textfile collector every 60 seconds.
Additional metrics can be added later if zot exposes them.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Required for testing zot registry push from workstation.
Podman uses a Linux VM under the hood on macOS.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Change zot port from 5000 to 5050 (macOS ControlCenter uses 5000)
- Fix sync config: use destination for namespacing, prefix ** for matching
- Update tailscale_serve to use port 5050
- Add zot role to main playbook
- Document implementation details in plan
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 0: Creates zot role with:
- Config for pull-through cache (Docker Hub, GHCR, Quay)
- mcquack LaunchAgent for service management
- Sync registries configured for on-demand caching
Binary is built from source at ~/code/3rd/zot (not homebrew).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add tag:registry to indri DeviceTags in __main__.py
- Update plan with implementation details noting the tag is
managed via Pulumi, not manually in admin console
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 0 of k8s migration: Add registry tag to ACLs.
- Admins get full access via wildcard grant
- Members denied access (infrastructure only)
- Enables tailscale serve for registry.tail8d86e.ts.net
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Replace permissive wildcard ACL (`*` -> `*`) with specific service grants
- Admin: full access to all services including NAS
- Member: user-facing services only (no Grafana/Loki/NAS)
- Add device tagging for gilbert (workstation) and sifaka (NAS) via Pulumi
- SSH hardening: remove root access, use "check" action with MFA
- Add ACL tests to validate policy behavior
## Deployment and Testing
- [x] Pulumi preview passes
- [x] HuJSON syntax validated
- [x] ACL tests defined and passing
- [ ] Deploy with `mise run tailnet-up`
- [ ] Verify SSH access from gilbert to indri
- [ ] Verify Allison cannot access Grafana/Loki/NAS
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/23
## Summary
- Remove all `meta/main.yml` dependencies from ansible roles
- Role ordering is now controlled entirely by `indri.yml` playbook
- Fix incorrect roles path in CLAUDE.md (`playbooks/roles` → `roles`)
## Why
Ansible's tag accumulation behavior prevents proper role deduplication when using meta dependencies. When a role is pulled in as a dependency, the parent role's tags are added to the dependency's tags (e.g., `[loki]` becomes `[alloy, loki]`), making them appear as different invocations to Ansible and causing roles to run multiple times.
## Deployment and Testing
- [x] Verified with `ansible-playbook --list-tasks` that each role now appears exactly once
- [x] Run full provision to verify no regressions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/20
## Summary
- Simplify kiwix role from 213 lines to 151 lines (-30%)
- Replace per-archive torrent status loops with single shell command
- Decouple kiwix startup from declared inventory - now serves whatever completed ZIM files exist
- Fix tailscale_serve role to handle empty JSON in check mode
## Performance improvement
- **Before**: ~132 operations (44 archives × 3 loops for status check, recheck, symlink)
- **After**: ~5 operations (1 shell script + 1 find + conditional symlinks)
- Expected reduction: ~3 minutes per ansible run
## Test plan
- [x] Ran `mise run provision-indri -- --check --diff` to preview changes
- [x] Ran `mise run provision-indri` to apply changes
- [x] Ran `mise run indri-services-check` - all services healthy
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/18
## Summary
- Add `postgresql_superuser` variable (`eblume`) to prevent PostgreSQL from inheriting OS username during initdb
- Update all psql/createdb commands to use explicit `-U` flag
- Add `check_mode: false` to op commands so 1Password fetches run during `--check` mode
- Add PostgreSQL and Miniflux health checks to indri-services-check
## Test plan
- [x] Renamed existing superuser from `erichblume` to `eblume`
- [x] Ran `mise run provision-indri -- --tags postgresql --check --diff` successfully
- [x] Verified connection as `eblume` superuser via Tailscale
- [x] Ran `mise run indri-services-check` - all services healthy
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/17
## Summary
- Manage tail8d86e.ts.net ACLs, tags, and DNS via Pulumi + Python
- State stored in Pulumi Cloud (free tier) to avoid circular dependency
- OAuth authentication via 1Password for secure credential management
- New mise tasks: `tailnet-preview`, `tailnet-up`
## Architecture
Two-layer approach:
- **Layer 1 (Pulumi)**: Tailnet-wide config (ACLs, tags, DNS)
- **Layer 2 (Ansible)**: Node-local `tailscale serve` config (unchanged)
## Test plan
- [x] Exported current ACL from Tailscale API
- [x] Imported existing ACL into Pulumi state
- [x] Verified `mise run tailnet-preview` shows no changes
- [x] Verified `mise run tailnet-up` applies successfully
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/15
## Summary
- Add `mise run blumeops-tasks` to fetch and display tasks from Todoist
- Uses uv run script with inline dependencies (httpx, rich)
- Fetches API credential securely via 1Password CLI
- Sorts tasks by custom priority order: p1, p2, p4, p3 (backlog last)
- Documents the task discovery workflow in CLAUDE.md
## Test plan
- [x] Verified `mise run blumeops-tasks` fetches and displays tasks correctly
- [x] Confirmed priority sorting works as expected
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/14
## Summary
- Use async with poll: 0 for alloy and loki restart handlers
- Fire-and-forget approach prevents ansible from hanging on graceful shutdown
## Test plan
- [x] Manually verified `brew services restart grafana-alloy` works
- [x] Run full ansible playbook and verify it completes without timeout
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/12
Use async with poll: 0 to fire-and-forget service restarts.
These services have graceful shutdown periods that can exceed
ansible's default command timeout.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Add `mise run zk-docs` task to concatenate all blumeops-tagged zettelkasten cards
- Main project card is shown first, followed by service management logs
- Uses `bat` for output (added to Brewfile)
- Args are passed through to bat for custom formatting
- Update CLAUDE.md to use zk-docs command with plain output options
- Update README.md to note zettelkasten is private with contact email
## Test plan
- [x] `mise run zk-docs` displays all 6 blumeops cards
- [x] `mise run zk-docs -- --style=header --color=never --decorations=always` shows filenames without decoration
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/10
## Summary
- Add ansible role for devpi-server as a transparent PyPI caching proxy
- LaunchAgent with KeepAlive runs via `mise x -- devpi-server`
- Listens on port 3141, data stored in `~/devpi`
- Health checks added to `indri-services-check` script
## Manual Setup Required (on indri, before provisioning)
1. Add to `~/.config/mise/config.toml`:
```toml
[tools]
"pipx:devpi-server" = "latest"
"pipx:devpi-web" = "latest"
"pipx:devpi-client" = "latest"
```
2. Run `mise install`
3. Initialize: `mise x -- devpi-init --serverdir ~/devpi`
## Post-Provisioning
- Set up Tailscale service `pypi` on port 443 → 3141
- Configure client pip.conf with index-url
## Test plan
- [x] Ansible syntax check passes
- [x] Dry-run: `mise run provision-indri -- --check --diff`
- [x] Apply: `mise run provision-indri`
- [x] Health check: `mise run indri-services-check`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/9
## Summary
- Adds a new Grafana dashboard for Node Exporter metrics on macOS hosts
- Uses macOS-native memory metrics (node_memory_total_bytes, node_memory_active_bytes, etc.) instead of Linux-specific ones
- Includes dropdown selectors for instance, disk, and network device filtering
## Details
The standard Node Exporter dashboards show "No Data" for memory panels on macOS because they query Linux-specific metrics like `node_memory_MemTotal_bytes`. macOS node_exporter exports different metrics:
| Linux | macOS |
|-------|-------|
| node_memory_MemTotal_bytes | node_memory_total_bytes |
| node_memory_MemFree_bytes | node_memory_free_bytes |
| node_memory_Buffers_bytes | (not available) |
| node_memory_Cached_bytes | (not available) |
macOS has unique memory categories: Wired, Active, Compressed, Inactive, Free.
## Test plan
- [x] Dashboard deployed to indri via ansible
- [x] All panels showing data for indri
- [x] Instance selector works to switch between hosts
- [x] Disk and network device filters work
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/8
## Summary
- Adds Upload/Download Ratio stat panel with color thresholds (red < 0.5, yellow < 1, green >= 1)
- Adds Downloaded (Period) stat panel showing bytes downloaded in selected time range
- Adds Uploaded (Period) stat panel showing bytes uploaded in selected time range
Uses PromQL `increase()` on existing counter metrics - no new metrics collection needed.
## Test plan
- [x] Deployed to indri via `mise run provision-indri`
- [x] Grafana restarted successfully
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/6
- Add mise-tasks/provision-indri script to run ansible playbook
- Fix transmission_metrics launchctl load to be idempotent
- Update CLAUDE.md to reference mise run provision-indri
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Query torrent-get RPC to sum totalSize of all torrents
- Add transmission_torrents_size_bytes gauge metric
- Add "Total Torrent Size" timeseries panel to dashboard
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
