eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions
blumeops/argocd/manifests/grafana-config
History
Erich Blume 17023085cb Migrate observability stack to Kubernetes (#42) 
Note: the name of this branch was chosen before the scope widened to encompass the entire observability stack.

Summary

  - Fix Grafana data source URLs (docker driver uses host.minikube.internal, not host.containers.internal)
  - Migrate Prometheus and Loki from indri to Kubernetes with Tailscale Ingresses
  - Expose CNPG PostgreSQL metrics via Tailscale and update dashboard to use cnpg_* metrics
  - Update Alloy to push metrics/logs to k8s endpoints (prometheus.tail8d86e.ts.net, loki.tail8d86e.ts.net)
  - Add ACL rule for port 9187 (CNPG metrics)
  - Delete obsolete ansible roles for prometheus and loki

Changes

  - argocd/manifests/prometheus/ - New Prometheus StatefulSet with 20Gi PVC and Tailscale Ingress
  - argocd/manifests/loki/ - New Loki StatefulSet with 20Gi PVC and Tailscale Ingress
  - argocd/apps/prometheus.yaml, argocd/apps/loki.yaml - ArgoCD Applications
  - argocd/manifests/grafana/values.yaml - Data sources now use k8s internal DNS
  - argocd/manifests/databases/service-metrics-tailscale.yaml - CNPG metrics endpoint
  - argocd/manifests/grafana-config/dashboards/configmap-postgresql.yaml - Updated to cnpg_* metrics
  - ansible/roles/alloy/defaults/main.yml - Push to k8s Tailscale endpoints
  - pulumi/policy.hujson - ACL for port 9187
  - Deleted ansible/roles/prometheus/ and ansible/roles/loki/

Deployment and Testing

  - Stop prometheus and loki on indri
  - Sync ArgoCD apps (apps, prometheus, loki, grafana)
  - Run mise run provision-indri -- --tags alloy
  - Verify Grafana dashboards show data

🤖 Generated with https://claude.ai/claude-code

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/42
2026-01-22 12:06:02 -08:00
..
dashboards Migrate observability stack to Kubernetes (#42) 2026-01-22 12:06:02 -08:00
ingress-tailscale.yaml P5.1: Migrate minikube from podman to QEMU2 driver (#38) 2026-01-21 16:03:37 -08:00
kustomization.yaml K8s Migration Phase 2: Grafana to Kubernetes (#30) 2026-01-19 14:40:25 -08:00
README.md K8s Migration Phase 2: Grafana to Kubernetes (#30) 2026-01-19 14:40:25 -08:00
secret-admin.yaml.tpl K8s Migration Phase 2: Grafana to Kubernetes (#30) 2026-01-19 14:40:25 -08:00

README.md

Grafana Configuration

This directory contains Kubernetes manifests for Grafana configuration:

  • Tailscale Ingress for external access
  • Dashboard ConfigMaps for provisioning

Secrets Management

Current approach: Secrets are manually injected using 1Password CLI.

Before deploying Grafana, create the admin password secret:

kubectl create namespace monitoring
op inject -i secret-admin.yaml.tpl | kubectl apply -f -

The secret template (secret-admin.yaml.tpl) references 1Password:

  • Vault: vg6xf6vvfmoh5hqjjhlhbeoaie (blumeops)
  • Item: oxkcr3xtxnewy7noep2izvyr6y
  • Field: password

Future improvement: Migrate to External Secrets Operator or similar for automated secret synchronization from 1Password to Kubernetes.

Dashboards

Dashboard JSON files are stored as ConfigMaps in the dashboards/ directory. The Grafana sidecar automatically discovers ConfigMaps with label grafana_dashboard: "1" and provisions them.

To add a new dashboard:

  1. Export the dashboard JSON from Grafana UI
  2. Create a ConfigMap with the JSON content
  3. Add the grafana_dashboard: "1" label
  4. Add the ConfigMap to kustomization.yaml