diff --git a/docs/changelog.d/upgrade-grafana.feature.md b/docs/changelog.d/upgrade-grafana.feature.md new file mode 100644 index 0000000..cbd7e7c --- /dev/null +++ b/docs/changelog.d/upgrade-grafana.feature.md @@ -0,0 +1 @@ +Upgrade Grafana from 11.4.0 to 12.3.3 with home-built container image and Kustomize manifests, replacing the Helm chart deployment. diff --git a/docs/how-to/grafana/build-grafana-container.md b/docs/how-to/grafana/build-grafana-container.md index d1d9d7c..0c073ce 100644 --- a/docs/how-to/grafana/build-grafana-container.md +++ b/docs/how-to/grafana/build-grafana-container.md @@ -9,30 +9,32 @@ tags: # Build Grafana Container -Build a home-built Grafana 12.x container image and publish to the forge registry. +Home-built Grafana container image published to `registry.ops.eblu.me/blumeops/grafana`. -## Context +## How It Works -Grafana currently uses the upstream `docker.io/grafana/grafana:11.4.0` image via the Helm chart. Per supply-chain policy, this should be replaced with a locally built image pushed to `forge.ops.eblu.me/eblume/grafana`. +The Dockerfile at `containers/grafana/Dockerfile` downloads the official Grafana OSS tarball for the target architecture (arm64/amd64), installs it into Alpine, and sets up standard paths. -## Steps +To build and push a new version: -1. Add a Grafana container build to Dagger (or Nix, following existing patterns) -2. Base on the official Grafana source or use a Nix derivation -3. Tag and push to `forge.ops.eblu.me/eblume/grafana:` -4. Add to `mise run container-list` inventory +```fish +# Update version in Dockerfile +# ARG CONTAINER_APP_VERSION=12.3.3 -## Lessons +mise run container-build-and-release grafana +``` -- **Tarball directory name:** The Grafana OSS tarball extracts to `grafana-` (e.g. `grafana-12.3.3`), *not* `grafana-v`. The `mv` command in the Dockerfile must match this. -- **Binary PATH:** The Grafana binary lives at `bin/grafana` inside the extracted directory. The Dockerfile must add the bin directory to `$PATH` (e.g. `ENV PATH="/usr/share/grafana/bin:$PATH"`) or use the full path in CMD. +## Gotchas -## Reference +- **Tarball directory name:** Extracts to `grafana-` (e.g. `grafana-12.3.3`), *not* `grafana-v`. +- **Binary PATH:** The binary lives at `bin/grafana` inside the extracted directory. The Dockerfile sets `ENV PATH="/usr/share/grafana/bin:$PATH"`. +- **UID 472:** Matches the official Grafana image for PVC ownership compatibility. -- Follow [[build-container-image]] for the standard container build workflow -- See existing container builds in `.dagger/` for patterns -- The k8s-sidecar image (`quay.io/kiwigrid/k8s-sidecar`) is a secondary concern — address after the main Grafana image +## Future Work + +The k8s-sidecar image (`quay.io/kiwigrid/k8s-sidecar`) is still pulled from upstream. Replace with a home-built image when prioritized. ## Related -- [[upgrade-grafana]] — Goal card +- [[upgrade-grafana]] — Migration context +- [[build-container-image]] — Standard container build workflow diff --git a/docs/how-to/grafana/kustomize-grafana-deployment.md b/docs/how-to/grafana/kustomize-grafana-deployment.md index d1ec4e7..eb30ae9 100644 --- a/docs/how-to/grafana/kustomize-grafana-deployment.md +++ b/docs/how-to/grafana/kustomize-grafana-deployment.md @@ -8,35 +8,27 @@ tags: # Kustomize Grafana Deployment -Convert Grafana from a Helm chart deployment to plain Kustomize manifests. +Grafana is deployed via plain Kustomize manifests in `argocd/manifests/grafana/`, replacing the previous Helm chart. -## Context +## Manifest Structure -Grafana is currently deployed via ArgoCD using a Helm chart (`grafana-8.8.2`) from a forge mirror. The chart produces: Deployment, Service, PVC, ConfigMaps (grafana.ini, datasources), RBAC resources, and a sidecar container for dashboard provisioning. +| File | Purpose | +|------|---------| +| `kustomization.yaml` | Resource list | +| `deployment.yaml` | Grafana container + k8s-sidecar for dashboards | +| `service.yaml` | ClusterIP on port 80 → 3000 | +| `pvc.yaml` | 1Gi SQLite storage | +| `configmap.yaml` | `grafana.ini` and datasource provisioning | +| `serviceaccount.yaml` | Service account | +| `rbac.yaml` | ClusterRole/RoleBinding for sidecar ConfigMap access | -## Steps +## Key Details -1. Template the current Helm chart to see what it produces: - ```fish - # From the forge mirror, or use argocd app manifests - argocd app manifests grafana > /tmp/grafana-helm-output.yaml - ``` -2. Create Kustomize equivalents in `argocd/manifests/grafana/`: - - `kustomization.yaml` - - `deployment.yaml` — Grafana container + k8s-sidecar container - - `service.yaml` - - `pvc.yaml` — Reuse existing 1Gi PVC - - `configmap.yaml` — grafana.ini and datasource provisioning - - `rbac.yaml` — ClusterRole, ClusterRoleBinding, Role, RoleBinding -3. Update `argocd/apps/grafana.yaml` to use a single kustomize source instead of the Helm multi-source -4. Remove the Helm values.yaml (replaced by the kustomize manifests) - -## Notes - -- The existing PVC must not be deleted during the transition — ensure the kustomize PVC matches the existing one's name -- The sidecar (`quay.io/kiwigrid/k8s-sidecar`) should also be replaced with a home-built image eventually, but is lower priority — focus on Grafana itself first -- Preserve all existing config: Authentik OIDC, datasources, dashboard sidecar labels +- **PVC name must remain `grafana`** — changing it would create a new volume and lose the SQLite DB +- **Sidecar** watches ConfigMaps with label `grafana_dashboard=1` and reloads dashboards via the Grafana API +- **Secrets** come from ExternalSecrets (`grafana-admin`, `grafana-authentik-oauth`, `grafana-teslamate-datasource`) managed by the `grafana-config` ArgoCD app ## Related -- [[upgrade-grafana]] — Goal card +- [[upgrade-grafana]] — Migration context +- [[grafana]] — Service reference card diff --git a/docs/how-to/grafana/upgrade-grafana.md b/docs/how-to/grafana/upgrade-grafana.md index ec42114..6371ee1 100644 --- a/docs/how-to/grafana/upgrade-grafana.md +++ b/docs/how-to/grafana/upgrade-grafana.md @@ -1,6 +1,5 @@ --- title: Upgrade Grafana -branch: mikado/upgrade-grafana requires: - kustomize-grafana-deployment - build-grafana-container @@ -13,60 +12,38 @@ tags: # Upgrade Grafana -Upgrade Grafana from 11.4.0 (Helm chart 8.8.2) to 12.x, converting from Helm to Kustomize with a home-built container image. +Upgraded Grafana from 11.4.0 (Helm chart) to 12.3.3, converting from Helm to Kustomize with a home-built container image. -## Current State +## What Changed -| Property | Value | -|----------|-------| -| **Helm chart** | `grafana-8.8.2` (from forge mirror of `grafana/helm-charts`) | -| **Grafana app** | 11.4.0 | -| **Deployment** | Helm via ArgoCD multi-source | -| **Namespace** | `monitoring` | -| **Storage** | SQLite on 1Gi PVC | +- **Image:** `docker.io/grafana/grafana:11.4.0` → `registry.ops.eblu.me/blumeops/grafana:v12.3.3` +- **Deployment:** Helm multi-source (chart + values) → single Kustomize directory +- **ArgoCD app:** Simplified to one source pointing at `argocd/manifests/grafana/` -Datasources: [[prometheus]], [[loki]], PostgreSQL (TeslaMate). Dashboard ConfigMaps provisioned via sidecar. - -## Target State - -- Grafana 12.x running from a home-built container (`forge.ops.eblu.me/eblume/grafana`) -- Kustomize manifests in `argocd/manifests/grafana/` (no Helm chart dependency) -- ArgoCD app simplified to a single source (kustomize directory) -- All existing datasources, dashboards, and Authentik OIDC intact +All existing datasources ([[prometheus]], [[loki]], TeslaMate), dashboard ConfigMaps, and Authentik OIDC were preserved without changes. ## Grafana 12 Breaking Changes -- **Angular plugin removal:** All AngularJS panels force-migrated to React. Our dashboards already use only React panels — no action needed. -- **Datasource UID format enforcement:** UIDs must be alphanumeric + dash/underscore, ≤40 chars. Our UIDs (`prometheus`, `loki`, `TeslaMate`) are compliant. -- **Annotation table migration:** Full rewrite of the `annotation` table (adds `dashboard_uid` column). Small SQLite DB — should be fast. PVC is disposable if anything goes wrong. +None affected us: -Overall risk: **Low.** +- **Angular plugin removal** — our dashboards already used React panels +- **Datasource UID format enforcement** — our UIDs were already compliant +- **Annotation table migration** — completed automatically on the small SQLite DB -## Execution +## How to Repeat -Once both prerequisites are complete: +To upgrade Grafana again in the future: -1. Update `argocd/apps/grafana.yaml` to point at the kustomize directory (single source, remove Helm multi-source) -2. Update `argocd/manifests/grafana/` with the kustomize manifests using the home-built image -3. Deploy on branch, verify with checklist below -4. Update `service-versions.yaml` to the new version and today's date +1. Update `CONTAINER_APP_VERSION` in `containers/grafana/Dockerfile` +2. Build and push via `mise run container-build-and-release grafana` +3. Update the image tag in `argocd/manifests/grafana/deployment.yaml` +4. Update `service-versions.yaml` +5. Sync: `argocd app sync grafana` -The SQLite PVC is disposable — dashboards are provisioned from ConfigMaps and datasources from config. No backup needed. - -## Verification Checklist - -- [ ] Pod running: `kubectl --context=minikube-indri -n monitoring get pods -l app.kubernetes.io/name=grafana` -- [ ] UI loads at `https://grafana.ops.eblu.me` -- [ ] Admin login works -- [ ] Authentik OIDC login works -- [ ] Datasources healthy: Prometheus, Loki, TeslaMate (Settings → Datasources → Test each) -- [ ] Key dashboards render: macOS System, Services Health, TeslaMate Overview -- [ ] Sidecar loaded all dashboard ConfigMaps -- [ ] `mise run services-check` passes -- [ ] No errors in pod logs +The SQLite PVC is disposable — dashboards come from ConfigMaps and datasources from config. ## Related - [[grafana]] — Service reference card -- [[build-grafana-container]] — Prereq: build the container image -- [[kustomize-grafana-deployment]] — Prereq: create kustomize manifests +- [[build-grafana-container]] — Building the container image +- [[kustomize-grafana-deployment]] — Kustomize manifest structure