GitOps repository for personal infrastructure management
  • Nix 32.5%
  • Jinja 21.5%
  • Python 17.9%
  • Shell 11.8%
  • Go 8.1%
  • Other 8.2%
Find a file
Erich Blume e4a8405de7 Observability cleanup and k8s service monitoring (#43) (#43)
## Summary
- Remove stale `/opt/homebrew/var/loki` from borgmatic backup (Loki migrated to k8s)
- Add Alloy k8s DaemonSet for automatic pod log collection with auto-discovery
- Add blackbox probes for miniflux, kiwix, transmission, devpi, argocd
- Add transmission-exporter sidecar for full metrics (speed, torrent counts, ratios)
- Replace stale devpi dashboard with probe-based metrics (status, response time, uptime)
- Add unified "K8s Services Health" dashboard for service uptime/response monitoring

## Manual cleanup already performed
- Deleted stale textfile metrics on indri: `devpi.prom`, `transmission.prom`
- Deleted stale data directories on indri: `/opt/homebrew/var/loki/`, `/opt/homebrew/var/prometheus/`

## Deployment and Testing
- [x] Sync `apps` application to pick up new alloy-k8s app
- [x] Deploy alloy-k8s on feature branch: `argocd app set alloy-k8s --revision feature/observability-cleanup && argocd app sync alloy-k8s`
- [x] Deploy torrent on feature branch (for transmission exporter): `argocd app set torrent --revision feature/observability-cleanup && argocd app sync torrent`
- [x] Deploy prometheus on feature branch (for new scrape config): `argocd app set prometheus --revision feature/observability-cleanup && argocd app sync prometheus`
- [x] Deploy grafana-config on feature branch (for dashboards): `argocd app set grafana-config --revision feature/observability-cleanup && argocd app sync grafana-config`
- [x] Verify pod logs appear in Loki/Grafana
- [x] Verify transmission metrics appear in Prometheus
- [x] Verify service probe metrics appear in Prometheus
- [x] Run `mise run provision-indri -- --tags borgmatic` to update borgmatic config
- [ ] After merge, reset apps to main and resync

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/43
2026-01-22 13:51:01 -08:00
.claude Add pre-commit hooks for code quality (#19) 2026-01-16 19:33:02 -08:00
ansible Observability cleanup and k8s service monitoring (#43) (#43) 2026-01-22 13:51:01 -08:00
argocd Observability cleanup and k8s service monitoring (#43) (#43) 2026-01-22 13:51:01 -08:00
mise-tasks Migrate observability stack to Kubernetes (#42) 2026-01-22 12:06:02 -08:00
plans/k8s-migration P7 forgejo plan updated 2026-01-21 20:04:18 -08:00
pulumi Migrate observability stack to Kubernetes (#42) 2026-01-22 12:06:02 -08:00
.ansible-lint Add pre-commit hooks for code quality (#19) 2026-01-16 19:33:02 -08:00
.gitignore Add pre-commit hooks for code quality (#19) 2026-01-16 19:33:02 -08:00
.pre-commit-config.yaml Scan full repo history in trufflehog 2026-01-19 10:12:56 -08:00
.yamllint.yaml K8s Migration Phase 1: Infrastructure Setup (#29) 2026-01-19 09:49:52 -08:00
Brewfile K8s Migration Phase 1: Infrastructure Setup (#29) 2026-01-19 09:49:52 -08:00
CLAUDE.md Add mise task to list unresolved PR comments (#40) 2026-01-21 19:14:27 -08:00
mise.toml Add Pulumi for tailnet IaC management (#15) 2026-01-15 20:55:25 -08:00
README.md Add pre-commit hooks for code quality (#19) 2026-01-16 19:33:02 -08:00

blumeops

                    l0K                                k..:k.
                  .:...c.                            ;c....
                    ....'o                          x.....
                      ....k                        x....
                       ... l'                    'c....
                        ....,l                  o'....
                         .....x                k....
                          .....d.             c....
                            ... l            x....
                              .,.d         ;c.c'
                               'c':;      x',c.
                                .:,'o   .x.::.
                                 .;:.k ,:.c'
                                   ,c.c';:.
                                    .,.:;.
                                   ;'.c, l
                                  d',c..:.d.
                                 O.:;.  'c';c
                               ;c.c'     .:;.x
                              o',c.       .;:.k
                             x.::.          'c.l.
                         dOKl.c,             .c,'o
                   0l'...... ..'              .::.ocx.
                 'o ............              o .... :olx;
                x,ox;. ....... .k             ....,dKKo;..x
              'd,OXXXXk:. ...... ;            ;:dXOl;',';l;o;
             x,oXXXXXXXXXkc. ...              .lc,',':dKNNNx;x;
           ;o;0KXXXXXXXXXXXX0l.                .',ckNNNNNNNNNxco0d
          l,d0oOXKOKXXXXKXXXX0.                  kNNNNNNNNNNNNNXxloo::
             .OXxdXKOX0kXXXX0.                   .KNNNNNNNNNNXONX0o.
                ,OdxKldXXXXx.                     ,NNNNNNNNNNNKoc
                   :.OXXkKo                       .kNNNNNNNNXx.
                      ':0c                         .NdNkXkc

Blue Mops — GitOps for Erich Blume's personal computing environment.

What is this?

Infrastructure-as-code for my tailnet (tail8d86e.ts.net). This repo contains ansible playbooks, configuration, and automation for managing my personal infrastructure.

This codebase was heavily co-authored by Claude Code, as an experiment in LLM-assisted development. I want to include a personal note here that I don't know entirely how I feel about LLMs in our current era, but it felt important to learn.

Development

Pre-commit Hooks

This repo uses pre-commit for code quality and consistency. Install hooks with:

uvx pre-commit install

Run all hooks manually:

uvx pre-commit run --all-files

Hooks include:

  • General: trailing whitespace, end-of-file fixer, large files, merge conflicts
  • Secrets: TruffleHog for secret detection
  • YAML: yamllint, ansible-lint
  • Python: ruff (linting + formatting)
  • Shell: shellcheck, shfmt
  • TOML: taplo
  • JSON: prettier

Documentation

Detailed documentation lives in my personal zettelkasten, which is not included in this repository. You can view the docs with:

mise run zk-docs

The zettelkasten is private at time of writing. If you're interested in the documentation or have questions about this project, please reach out to blume.erich@gmail.com.