eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/docs/reference/services/grafana.md
Erich Blume e7d3871144 Deploy Grafana Pyroscope for continuous eBPF profiling on ringtail 
Add Pyroscope server (StatefulSet on ringtail k3s) and Alloy profiling
DaemonSet with pyroscope.ebpf collection. Grafana datasource with
traces-to-profiles cross-linking. Docs updated across observability
reference card, Alloy, Grafana, apps registry, and README.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 14:05:57 -07:00

2.4 KiB
Raw Blame History

title modified tags
Grafana 2026-03-26
service
observability

Grafana

Dashboards and visualization for BlumeOps observability.

Quick Reference

Property Value
URL https://grafana.ops.eblu.me
Tailscale URL https://grafana.tail8d86e.ts.net
Namespace monitoring
Deployment Kustomize (argocd/manifests/grafana/)
Image registry.ops.eblu.me/blumeops/grafana
Sidecar Image registry.ops.eblu.me/blumeops/grafana-sidecar

Authentication

Grafana supports two login methods:

  • SSO via authentik — OIDC login through Authentik (auth.generic_oauth). Users click "Sign in with Authentik", authenticate at Authentik, and are redirected back as Admin.
  • Local admin — break-glass login using the password from 1Password ("Grafana (blumeops)"). Always available if Authentik is down.

The OIDC client secret is injected via external-secrets (grafana-authentik-oauth secret in monitoring namespace).

Datasources

Name Type Target
Prometheus prometheus prometheus.monitoring.svc.cluster.local:9090
Loki loki loki.monitoring.svc.cluster.local:3100
Tempo tempo tempo.monitoring.svc.cluster.local:3200
Pyroscope grafana-pyroscope-datasource pyroscope.tail8d86e.ts.net (ringtail, via Tailscale)
TeslaMate postgres blumeops-pg-rw.databases.svc.cluster.local:5432

Dashboard Provisioning

Dashboards are ConfigMaps with label grafana_dashboard: "1".

Location: argocd/manifests/grafana-config/dashboards/

Optional annotation: grafana_folder: "FolderName"

Key Dashboards

  • macOS System - Host metrics for indri
  • Minikube - Kubernetes cluster overview
  • Borgmatic Backups - Backup status and trends
  • Services Health - HTTP probe results
  • Docs APM - Request rate, latency, cache for docs.eblu.me
  • Fly.io Proxy Health - Aggregate proxy health across all upstream services
  • TeslaMate (18 dashboards) - Vehicle data

Related