Add Grafana Alloy and Loki for unified observability (#11)

## Summary
- Add Grafana Alloy to replace node_exporter for metrics collection
- Add Loki for log aggregation and storage
- Configure Alloy to collect logs from all services (grafana, forgejo, prometheus, tailscale, transmission, devpi, kiwix, borgmatic)
- Update Prometheus to accept metrics via remote_write
- Add Loki datasource to Grafana

## Test plan
- [ ] Run \`mise run provision-indri -- --check --diff\` to verify changes
- [ ] Apply with \`mise run provision-indri\`
- [ ] Verify services: \`mise run indri-services-check\`
- [ ] Check Grafana Explore with Loki datasource
- [ ] Query logs: \`{service="grafana"}\`
- [ ] Verify metrics still flowing to Prometheus dashboards

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/11
This commit is contained in:
Erich Blume 2026-01-15 12:24:13 -08:00
commit 242c1880de
17 changed files with 799 additions and 10 deletions

View file

@ -0,0 +1,45 @@
---
# Grafana Alloy installation and configuration
# Replaces node_exporter for metrics, adds log collection
- name: Install grafana-alloy via homebrew
community.general.homebrew:
name: grafana-alloy
state: present
- name: Ensure alloy config directory exists
ansible.builtin.file:
path: "{{ alloy_config_dir }}"
state: directory
mode: '0755'
- name: Ensure alloy data directory exists
ansible.builtin.file:
path: "{{ alloy_data_dir }}"
state: directory
mode: '0755'
- name: Ensure textfile collector directory exists
ansible.builtin.file:
path: "{{ alloy_textfile_dir }}"
state: directory
mode: '0755'
- name: Deploy alloy configuration
ansible.builtin.template:
src: config.alloy.j2
dest: "{{ alloy_config_dir }}/config.alloy"
mode: '0644'
notify: restart alloy
- name: Stop node_exporter service (replaced by alloy)
ansible.builtin.command: brew services stop node_exporter
register: node_exporter_stop
changed_when: "'Stopping' in node_exporter_stop.stdout or 'Successfully stopped' in node_exporter_stop.stdout"
failed_when: false
- name: Ensure alloy service is started
ansible.builtin.command: brew services start grafana-alloy
register: brew_start
changed_when: "'Successfully started' in brew_start.stdout"
failed_when: false