Commit graph

43 commits

Author SHA1 Message Date
acff29f554 Fix ansible handler timeouts for alloy and loki restarts
Use async with poll: 0 to fire-and-forget service restarts.
These services have graceful shutdown periods that can exceed
ansible's default command timeout.

Also update transmission_metrics and devpi_metrics to depend on
alloy instead of node_exporter, since alloy now handles the
textfile collector.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-15 13:06:17 -08:00
242c1880de Add Grafana Alloy and Loki for unified observability (#11)
## Summary
- Add Grafana Alloy to replace node_exporter for metrics collection
- Add Loki for log aggregation and storage
- Configure Alloy to collect logs from all services (grafana, forgejo, prometheus, tailscale, transmission, devpi, kiwix, borgmatic)
- Update Prometheus to accept metrics via remote_write
- Add Loki datasource to Grafana

## Test plan
- [ ] Run \`mise run provision-indri -- --check --diff\` to verify changes
- [ ] Apply with \`mise run provision-indri\`
- [ ] Verify services: \`mise run indri-services-check\`
- [ ] Check Grafana Explore with Loki datasource
- [ ] Query logs: \`{service="grafana"}\`
- [ ] Verify metrics still flowing to Prometheus dashboards

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/11
2026-01-15 12:24:13 -08:00
070f26dc6d Add zk-docs mise task for zettelkasten documentation (#10)
## Summary
- Add `mise run zk-docs` task to concatenate all blumeops-tagged zettelkasten cards
- Main project card is shown first, followed by service management logs
- Uses `bat` for output (added to Brewfile)
- Args are passed through to bat for custom formatting
- Update CLAUDE.md to use zk-docs command with plain output options
- Update README.md to note zettelkasten is private with contact email

## Test plan
- [x] `mise run zk-docs` displays all 6 blumeops cards
- [x] `mise run zk-docs -- --style=header --color=never --decorations=always` shows filenames without decoration

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/10
2026-01-15 11:25:02 -08:00
2e326eb30d Critical security note for claude 2026-01-15 09:02:27 -08:00
c660674891 Remove settings.local.json from repo and add to gitignore
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-15 08:59:24 -08:00
d8a0ef6482 Add devpi PyPI caching proxy role for indri (#9)
## Summary
- Add ansible role for devpi-server as a transparent PyPI caching proxy
- LaunchAgent with KeepAlive runs via `mise x -- devpi-server`
- Listens on port 3141, data stored in `~/devpi`
- Health checks added to `indri-services-check` script

## Manual Setup Required (on indri, before provisioning)
1. Add to `~/.config/mise/config.toml`:
   ```toml
   [tools]
   "pipx:devpi-server" = "latest"
   "pipx:devpi-web" = "latest"
   "pipx:devpi-client" = "latest"
   ```
2. Run `mise install`
3. Initialize: `mise x -- devpi-init --serverdir ~/devpi`

## Post-Provisioning
- Set up Tailscale service `pypi` on port 443 → 3141
- Configure client pip.conf with index-url

## Test plan
- [x] Ansible syntax check passes
- [x] Dry-run: `mise run provision-indri -- --check --diff`
- [x] Apply: `mise run provision-indri`
- [x] Health check: `mise run indri-services-check`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/9
2026-01-15 08:31:09 -08:00
50c713b5de Add macOS-compatible Node Exporter Grafana dashboard (#8)
## Summary
- Adds a new Grafana dashboard for Node Exporter metrics on macOS hosts
- Uses macOS-native memory metrics (node_memory_total_bytes, node_memory_active_bytes, etc.) instead of Linux-specific ones
- Includes dropdown selectors for instance, disk, and network device filtering

## Details
The standard Node Exporter dashboards show "No Data" for memory panels on macOS because they query Linux-specific metrics like `node_memory_MemTotal_bytes`. macOS node_exporter exports different metrics:

| Linux | macOS |
|-------|-------|
| node_memory_MemTotal_bytes | node_memory_total_bytes |
| node_memory_MemFree_bytes | node_memory_free_bytes |
| node_memory_Buffers_bytes | (not available) |
| node_memory_Cached_bytes | (not available) |

macOS has unique memory categories: Wired, Active, Compressed, Inactive, Free.

## Test plan
- [x] Dashboard deployed to indri via ansible
- [x] All panels showing data for indri
- [x] Instance selector works to switch between hosts
- [x] Disk and network device filters work

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/8
2026-01-14 20:53:57 -08:00
d9be8c27bc Add 32 devdocs ZIM archives for programming documentation (#7)
## Summary
- Adds offline documentation for: bash, c, click, cmake, cpp, css, django-rest-framework, django, docker, duckdb, fish, gcc, git, go, godot, hammerspoon, homebrew, javascript, kubectl, kubernetes, latex, lua, markdown, nginx, nix, postgresql, python, redis, sqlite, typescript, werkzeug, zig
- All January 2026 versions from download.kiwix.org/zim/devdocs/
- Downloads via BitTorrent through transmission

## Test plan
- [x] Deployed to indri via `mise run provision-indri`
- [x] All 32 torrents added and downloaded (small files, completed instantly)
- [x] 43 ZIM files now available in kiwix directory

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/7
2026-01-14 18:28:34 -08:00
10012a4cf2 Add upload/download ratio and period transfer panels to Transmission dashboard (#6)
## Summary
- Adds Upload/Download Ratio stat panel with color thresholds (red < 0.5, yellow < 1, green >= 1)
- Adds Downloaded (Period) stat panel showing bytes downloaded in selected time range
- Adds Uploaded (Period) stat panel showing bytes uploaded in selected time range

Uses PromQL `increase()` on existing counter metrics - no new metrics collection needed.

## Test plan
- [x] Deployed to indri via `mise run provision-indri`
- [x] Grafana restarted successfully

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/6
2026-01-14 18:08:39 -08:00
ba03af15eb Set MISE_TASK_OUTPUT=interleave in provision-indri
Shows ansible output in real-time instead of buffered.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 14:15:11 -08:00
2f28b151f5 Fix launchctl idempotency in kiwix and borgmatic roles
Check if LaunchAgent is already loaded before attempting to load it.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 14:14:52 -08:00
e534e59556 Add provision-indri mise task and fix idempotency
- Add mise-tasks/provision-indri script to run ansible playbook
- Fix transmission_metrics launchctl load to be idempotent
- Update CLAUDE.md to reference mise run provision-indri

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 14:10:30 -08:00
e264b39cd6 Add total torrent size metric and dashboard panel
- Query torrent-get RPC to sum totalSize of all torrents
- Add transmission_torrents_size_bytes gauge metric
- Add "Total Torrent Size" timeseries panel to dashboard

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 14:00:52 -08:00
d18c3a6f3c Add node_exporter and transmission-metrics to service checks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 13:57:08 -08:00
0afd34590d Fix transmission-metrics session ID parsing
Transmission doesn't support HEAD requests, so use -i flag with sed to
parse only the HTTP headers (stopping at the blank line before body).
Also anchor grep pattern to line start to avoid matching HTML content.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 13:56:08 -08:00
7468023cd2 Add transmission dashboard to grafana
- Add node_exporter ansible role to enable textfile collector
- Add transmission_metrics role with script and LaunchAgent
  - Collects metrics every 60s via transmission RPC
  - Writes to /opt/homebrew/var/node_exporter/textfile/transmission.prom
- Update grafana role to provision dashboards from files
- Add transmission.json dashboard with:
  - Status indicator, torrent counts
  - Transfer speeds, cumulative stats
  - Time series graphs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 13:46:51 -08:00
9c0ff8bb9b Add mise task for indri service health checks
- Create mise-tasks/indri-services-check script
- Checks all indri services (prometheus, grafana, kiwix, transmission, forgejo)
- Verifies both local service status and HTTP endpoints
- Transmission RPC checked via SSH since it's localhost-only (secure)
- Update CLAUDE.md with instructions to run after service changes

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 13:23:05 -08:00
eb2f5b44cd Fix kiwix plist to only include available ZIM archives
The LaunchAgent plist now dynamically includes only ZIM files that
actually exist in the kiwix directory, rather than all configured
archives. This prevents kiwix-serve from crashing when torrents are
still downloading.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 13:17:55 -08:00
3847c12b42 Fix transmission config to prevent perpetual ansible diffs
- Expand settings.json template to include all transmission defaults
- Use static pre-hashed rpc-password so transmission doesn't regenerate
- Change file mode from 0644 to 0600 to match transmission's default
- Add Jinja comment explaining the RPC password workaround

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 13:03:46 -08:00
4add1684c3 Enable additional ZIM archives for kiwix
New archives (~95G total):
- Project Gutenberg 2023 (72G) - 60,000+ public domain books
- iFixit (3.3G) - Repair guides
- Stack Exchange: SuperUser (3.7G), Math (6.9G)
- LibreTexts: Biology, Chemistry, Engineering, Mathematics, Physics, Humanities

Also:
- Fix transmission to only restart when config changes
- Update CLAUDE.md to use full ansible paths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 12:47:54 -08:00
2a6fcd915c Add reminder to always branch from main
Prevents accidentally including commits from other feature branches.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 12:00:00 -08:00
6bb5f323d6 Fix transmission config path to use homebrew location
- Homebrew's transmission-cli service uses /opt/homebrew/var/transmission/
  not ~/.config/transmission-daemon/
- Add task to clean up old config directory
- Update zettelkasten with correct paths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 11:59:24 -08:00
b865d70456 Add transmission role for torrent-based ZIM downloads
- Add new transmission ansible role using homebrew + brew services
- Configure transmission to download to ~/transmission with localhost-only RPC
- Modify kiwix role to use transmission for downloading ZIM archives via BitTorrent
- Add role dependency so running --tags kiwix auto-runs transmission
- Keep fallback to direct HTTP download when kiwix_use_transmission: false
- Symlink completed downloads from transmission dir to kiwix-tools dir

This reduces load on kiwix.org servers and allows downloads to continue
in the background without blocking ansible runs.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 11:59:24 -08:00
852329dd77 Merge pull request 'Fix borgmatic LaunchAgent to work with mise-installed binaries' (#2) from feature/fix-borgmatic-launchagent into main
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/2
2026-01-14 10:36:16 -08:00
763899e856 Document tea pr create syntax in CLAUDE.md
Add full example with heredoc for multi-line descriptions and note
the difference from gh CLI (--description vs --body).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 10:35:27 -08:00
4b365a1294 Fix borgmatic LaunchAgent to work with mise-installed binaries
The LaunchAgent was failing because launchd runs with a minimal PATH
that doesn't include mise-installed binaries or homebrew. This adds:

- Use `mise x` wrapper to run borgmatic (survives version updates)
- Add /opt/homebrew/bin to PATH for borg dependency
- Add ansible tags to indri playbook for targeted role runs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 10:33:48 -08:00
fbf7a89035 Personal note 2026-01-14 07:34:31 -08:00
8544dfec1a Merge pull request 'Add grafana datasource provisioning and update workflow docs' (#1) from feature/grafana-provisioning into main
Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/1
2026-01-14 07:32:12 -08:00
9fcf27a4c5 Add note to commit often while working 2026-01-14 07:25:15 -08:00
4283f2237d Add grafana datasource provisioning and update workflow docs
- Configure grafana to use provisioned datasources instead of UI config
- Add prometheus datasource template managed by ansible
- Create minimal grafana.ini with custom provisioning path
- Move ansible_managed to group_vars (fixes deprecation warning)
- Add Remote Hosts and Git Workflow sections to CLAUDE.md
- Document feature branch workflow with tea CLI for PRs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 07:23:10 -08:00
fb5d6314e8 Fix claude settings 2026-01-14 06:52:59 -08:00
6995a2d27d Manual update to CLAUDE and moved mise.toml 2026-01-14 06:45:19 -08:00
d1396b1cfb Add forgejo role to ansible playbook
Manages installation and service via homebrew. Config at
/opt/homebrew/var/forgejo/custom/conf/app.ini contains secrets
and is not templated - backed up by borgmatic instead.

Includes check that fails with restore instructions if config missing.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 23:00:46 -08:00
d761e61809 Add borgmatic role to ansible playbook
Manages scheduled LaunchAgent for daily backups at 2:00 AM.
Borgmatic itself is installed via mise (pipx), not managed by ansible.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 22:50:28 -08:00
6977197c98 Add ZIM archive management to kiwix role
- Configure ZIM archives as a variable list with download URLs
- Auto-download missing archives from download.kiwix.org
- Template plist to serve all configured archives
- Skip checksum calculation on stat for performance
- Add commented options for Gutenberg, iFixit, Stack Exchange, LibreTexts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 22:36:53 -08:00
0d2978259d Add kiwix role to ansible playbook
Manages kiwix-serve LaunchAgent configuration for the wikipedia
mirror on indri (port 5501).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 21:58:12 -08:00
c182f4903d Add ansible_managed header to prometheus.yml template
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 21:50:55 -08:00
e26b830865 Update ASCII art with broom and mop image
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 21:46:58 -08:00
dd6d024b3a Add Brewfile and document dependency tracking
- Add Brewfile with tea (Forgejo CLI) for forge interaction
- Update CLAUDE.md with dependency tracking guidance

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 21:38:58 -08:00
6f182caade Suppress Python interpreter discovery warning in ansible
Sets interpreter_python = auto_silent to keep auto-detection
while silencing the warning about future interpreter changes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 21:26:41 -08:00
5da6155bb5 Add mise.toml for ansible-core@latest
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 21:18:27 -08:00
6823b43745 Add ansible playbook for indri with prometheus and grafana roles
- Inventory with tailnet hosts (indri, gilbert, sifaka)
- Prometheus role: installs via homebrew, templates config from current indri setup
- Grafana role: installs via homebrew, starts service
- ansible.cfg and requirements.yml for community.general collection
- Updated CLAUDE.md with ansible dry-run instructions

Tested: ansible-playbook playbooks/indri.yml --check --diff (all ok)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 21:12:24 -08:00
4bb4e37c8f Initial blumeops setup
- CLAUDE.md with project guidance and zk reference
- README.md with crossed mops ascii art
- Project settings requiring review for markdown edits

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 21:07:14 -08:00