Log filtering cleanup and observability improvements #45

Merged

eblume merged 7 commits from feature/log-filtering-cleanup into main

2026-01-22 17:30:08 -08:00

eblume commented

2026-01-22 17:29:47 -08:00

Owner

Summary

Suppress noisy storage-provisioner Endpoints deprecation warning (upstream minikube issue)
Disable thermal collector on indri Alloy (not supported on macOS M1)
Add macOS power/thermal metrics collection via powermetrics LaunchDaemon
Add Power & Thermal section to macOS Grafana dashboard
Add logfmt parser for k8s log level extraction (Loki, Prometheus, etc.)
Extract more fields from JSON logs (zot compatibility - uses "message" not "msg")
Silence logfmt parse errors for non-logfmt logs
Fix JSON escaping in devpi dashboard

Deployment and Testing

Deployed Alloy config changes to indri via ansible
Synced alloy-k8s and grafana-config via ArgoCD
Verified power metrics appearing in Prometheus
Verified thermal collector errors stopped
Verified logfmt parse errors silenced
Verified devpi dashboard loads correctly

🤖 Generated with Claude Code

## Summary - Suppress noisy storage-provisioner Endpoints deprecation warning (upstream minikube issue) - Disable thermal collector on indri Alloy (not supported on macOS M1) - Add macOS power/thermal metrics collection via powermetrics LaunchDaemon - Add Power & Thermal section to macOS Grafana dashboard - Add logfmt parser for k8s log level extraction (Loki, Prometheus, etc.) - Extract more fields from JSON logs (zot compatibility - uses "message" not "msg") - Silence logfmt parse errors for non-logfmt logs - Fix JSON escaping in devpi dashboard ## Deployment and Testing - [x] Deployed Alloy config changes to indri via ansible - [x] Synced alloy-k8s and grafana-config via ArgoCD - [x] Verified power metrics appearing in Prometheus - [x] Verified thermal collector errors stopped - [x] Verified logfmt parse errors silenced - [x] Verified devpi dashboard loads correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code)

eblume added 7 commits

2026-01-22 17:29:48 -08:00

Suppress storage-provisioner Endpoints deprecation warning f0201c06d9

Minikube's storage-provisioner uses the deprecated v1 Endpoints API which
spams warnings every 2 seconds on K8s 1.33+. This is an upstream issue
(kubernetes/minikube#21009) with no fix yet.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Disable thermal collector on indri Alloy 3886c43de8

The thermal collector fails on macOS M1 with "no CPU power status has
been recorded" - this hardware doesn't expose power metrics in a way
the collector understands.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add macOS power/thermal metrics collection and dashboard 358bbcdffb

- Add powermetrics collector to Alloy role (via LaunchDaemon, requires root)
- Collect CPU, GPU, ANE power (watts) and thermal pressure level
- Add "Power & Thermal" section to macOS Grafana dashboard with:
  - Total power stat
  - Thermal pressure indicator (Nominal/Moderate/Heavy/Critical)
  - Stacked power consumption graph (CPU/GPU/ANE)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add logfmt parser for k8s log level extraction aa2f562120

The JSON parser only works for JSON-formatted logs. Many Go services
(Loki, Prometheus, etc.) use logfmt format. Add logfmt parser to extract
level, caller, and component labels for better log filtering in Grafana.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Extract more fields from JSON logs (zot compatibility) d887d4cdcb

Zot uses "message" instead of "msg" for the message field. Also extract
caller and repository from JSON logs for better filtering.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Silence logfmt parse errors for non-logfmt logs 848fea9e1f

The logfmt stage tries to parse all logs, including JSON. When it fails,
it adds __error__ labels. Drop these labels since the logs are still
processed correctly (JSON parser already extracted fields).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix JSON escaping in devpi dashboard 9de2e961cc

Was using \\" (double backslash) but YAML literal blocks only need \"
to produce a backslash-quote in the output.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>