eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/docs/reference/services/grafana.md
Erich Blume a2bb9abbdb
All checks were successful
Build Container (Nix) / detect (push) Successful in 2s
Details
Build Container / detect (push) Successful in 2s
Details
Build Container (Nix) / build (grafana-sidecar) (push) Successful in 2s
Details
Build Container / build (grafana-sidecar) (push) Successful in 6s
Details
Home-build grafana-sidecar container (#281) 
## Summary
- Home-build the k8s-sidecar container (`grafana-sidecar`) from forge mirror, replacing upstream `quay.io/kiwigrid/k8s-sidecar:1.28.0`
- Pinned to v1.28.0 — v2.x deferred due to 135% memory regression and readOnlyRootFilesystem crashloop
- Adds Dockerfile, service-versions entry, docs, and changelog fragment
- Manifest switch to home-built image pending container build

## Deployment and Testing
- [ ] `mise run container-build-and-release grafana-sidecar`
- [ ] Update kustomization.yaml with built image tag
- [ ] `argocd app set grafana --revision feature/grafana-sidecar && argocd app sync grafana`
- [ ] Verify sidecar logs and dashboards at https://grafana.ops.eblu.me
- [ ] Post-merge: `argocd app set grafana --revision main && argocd app sync grafana`

Reviewed-on: #281
2026-03-03 13:48:24 -08:00

2.2 KiB
Raw Blame History

title modified tags
Grafana 2026-02-28
service
observability

Grafana

Dashboards and visualization for BlumeOps observability.

Quick Reference

Property Value
URL https://grafana.ops.eblu.me
Tailscale URL https://grafana.tail8d86e.ts.net
Namespace monitoring
Deployment Kustomize (argocd/manifests/grafana/)
Image registry.ops.eblu.me/blumeops/grafana
Sidecar Image registry.ops.eblu.me/blumeops/grafana-sidecar

Authentication

Grafana supports two login methods:

  • SSO via authentik — OIDC login through Authentik (auth.generic_oauth). Users click "Sign in with Authentik", authenticate at Authentik, and are redirected back as Admin.
  • Local admin — break-glass login using the password from 1Password ("Grafana (blumeops)"). Always available if Authentik is down.

The OIDC client secret is injected via external-secrets (grafana-authentik-oauth secret in monitoring namespace).

Datasources

Name Type Target
Prometheus prometheus prometheus.monitoring.svc.cluster.local:9090
Loki loki loki.monitoring.svc.cluster.local:3100
TeslaMate postgres blumeops-pg-rw.databases.svc.cluster.local:5432

Dashboard Provisioning

Dashboards are ConfigMaps with label grafana_dashboard: "1".

Location: argocd/manifests/grafana-config/dashboards/

Optional annotation: grafana_folder: "FolderName"

Key Dashboards

  • macOS System - Host metrics for indri
  • Minikube - Kubernetes cluster overview
  • Borgmatic Backups - Backup status and trends
  • Services Health - HTTP probe results
  • Docs APM - Request rate, latency, cache for docs.eblu.me
  • Fly.io Proxy Health - Aggregate proxy health across all upstream services
  • TeslaMate (18 dashboards) - Vehicle data

Related