Add Grafana Alloy and Loki for unified observability #11

Merged

eblume merged 4 commits from feature/alloy-loki-logging into main

2026-01-15 12:24:14 -08:00

eblume commented

2026-01-15 11:59:37 -08:00

Owner

Summary

Add Grafana Alloy to replace node_exporter for metrics collection
Add Loki for log aggregation and storage
Configure Alloy to collect logs from all services (grafana, forgejo, prometheus, tailscale, transmission, devpi, kiwix, borgmatic)
Update Prometheus to accept metrics via remote_write
Add Loki datasource to Grafana

Test plan

Run `mise run provision-indri -- --check --diff` to verify changes
Apply with `mise run provision-indri`
Verify services: `mise run indri-services-check`
Check Grafana Explore with Loki datasource
Query logs: `{service="grafana"}`
Verify metrics still flowing to Prometheus dashboards

🤖 Generated with Claude Code

## Summary - Add Grafana Alloy to replace node_exporter for metrics collection - Add Loki for log aggregation and storage - Configure Alloy to collect logs from all services (grafana, forgejo, prometheus, tailscale, transmission, devpi, kiwix, borgmatic) - Update Prometheus to accept metrics via remote_write - Add Loki datasource to Grafana ## Test plan - [ ] Run \`mise run provision-indri -- --check --diff\` to verify changes - [ ] Apply with \`mise run provision-indri\` - [ ] Verify services: \`mise run indri-services-check\` - [ ] Check Grafana Explore with Loki datasource - [ ] Query logs: \`{service="grafana"}\` - [ ] Verify metrics still flowing to Prometheus dashboards 🤖 Generated with [Claude Code](https://claude.com/claude-code)

eblume added 1 commit

2026-01-15 11:59:37 -08:00

Add Grafana Alloy and Loki for unified observability 09c432e0c1

- Add ansible/roles/alloy/ - replaces node_exporter for metrics collection
  - Uses prometheus.exporter.unix with textfile collector
  - Pushes metrics to Prometheus via remote_write
  - Collects logs from brew services and mcquack LaunchAgents
  - Forwards logs to Loki

- Add ansible/roles/loki/ - log storage and query engine
  - Single-node filesystem-based deployment
  - TSDB storage with 31-day retention
  - Integrated with Grafana as datasource

- Update Prometheus to enable remote_write receiver
  - Remove node-exporter-indri scrape job (Alloy pushes instead)
  - Keep sifaka scraping via traditional node_exporter

- Update Grafana datasources to include Loki

- Update indri-services-check to verify Loki and Alloy

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

eblume added 1 commit

2026-01-15 12:01:27 -08:00

Add role dependencies via meta/main.yml 7fd601d864

- alloy depends on prometheus and loki
- grafana depends on prometheus and loki
- loki has no dependencies

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

eblume added 1 commit

2026-01-15 12:02:54 -08:00

Remove ordering comments from playbook (handled by meta dependencies) 8291a8cb22

eblume added 1 commit

2026-01-15 12:14:34 -08:00

Add Loki metrics scraping and Grafana dashboard 47640511c4

- Add Loki scrape job to Prometheus config
- Add Loki dashboard with storage/ingestion metrics:
  - Chunks in memory
  - Active chunks/streams
  - Heap usage
  - Ingestion rate (bytes/sec)
  - Log lines rate
  - Memory usage over time
  - Storage growth (24h rolling)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>