Upgrade grafana-sidecar 1.28.0 → 2.6.0 + container.py port (#332)

## Summary

- Upgrade grafana-sidecar from 1.28.0 to 2.6.0 (the 2.x memory regression #462 is resolved; ~35MB static overhead is acceptable)
- Port build from Dockerfile to native Dagger container.py
- Add liveness/readiness probes using the new /healthz endpoint on port 8080
- Update docs to reflect container.py migration and remove stale pin note

## Test plan

- [ ] Build container: `mise run container-build-and-release grafana-sidecar`
- [ ] Update kustomization tag with new image tag
- [ ] Deploy from branch: `argocd app set grafana --revision grafana-sidecar-2.6.0 && argocd app sync grafana`
- [ ] Verify sidecar health endpoint: `kubectl exec -n monitoring <pod> -c grafana-sc-dashboard -- wget -qO- http://localhost:8080/healthz`
- [ ] Verify dashboards load in Grafana UI
- [ ] `mise run services-check`

Reviewed-on: #332

argocd/manifests/grafana/deployment.yaml

@@ -200,6 +200,18 @@ spec:
               value: http://localhost:3000/api/admin/provisioning/dashboards/reload
             - name: REQ_METHOD
               value: POST
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 8080
+            initialDelaySeconds: 10
+            periodSeconds: 30
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: 8080
+            initialDelaySeconds: 5
+            periodSeconds: 10
           securityContext:
             allowPrivilegeEscalation: false
             capabilities:

argocd/manifests/grafana/kustomization.yaml

@@ -16,7 +16,7 @@ images:
   - name: docker.io/library/busybox
     newTag: 1.31.1
   - name: registry.ops.eblu.me/blumeops/grafana-sidecar
-    newTag: v1.28.0-613f05d
+    newTag: v2.6.0-b75c4f9
   - name: registry.ops.eblu.me/blumeops/grafana
     newTag: v12.4.2-4c54774
 

containers/grafana-sidecar/Dockerfile

@@ -1,36 +0,0 @@
-# Grafana dashboard sidecar - watches ConfigMaps and syncs into Grafana
-# Two-stage build: Python venv (builder), runtime (Alpine)
-
-ARG CONTAINER_APP_VERSION=1.28.0
-
-FROM python:3.12-alpine3.22 AS base
-
-FROM base AS builder
-ARG CONTAINER_APP_VERSION
-WORKDIR /app
-RUN apk add --no-cache git gcc musl-dev
-RUN git clone --depth 1 --branch ${CONTAINER_APP_VERSION} \
-    https://forge.ops.eblu.me/mirrors/kiwigrid-grafana-sidecar.git /tmp/k8s-sidecar
-RUN python -m venv .venv && \
-    .venv/bin/pip install --no-cache-dir -U pip setuptools && \
-    .venv/bin/pip install --no-cache-dir -r /tmp/k8s-sidecar/src/requirements.txt && \
-    cp /tmp/k8s-sidecar/src/*.py /app/ && \
-    find /app/.venv \( -type d -a -name test -o -name tests \) \
-      -o \( -type f -a -name '*.pyc' -o -name '*.pyo' \) -exec rm -rf '{}' \+
-
-FROM base
-
-ARG CONTAINER_APP_VERSION
-LABEL org.opencontainers.image.title="Grafana Sidecar"
-LABEL org.opencontainers.image.description="K8s sidecar to sync ConfigMap dashboards into Grafana"
-LABEL org.opencontainers.image.version="${CONTAINER_APP_VERSION}"
-LABEL org.opencontainers.image.source="https://forge.eblu.me/eblume/blumeops"
-LABEL org.opencontainers.image.vendor="blumeops"
-
-ENV PYTHONUNBUFFERED=1
-WORKDIR /app
-COPY --from=builder /app /app
-ENV PATH="/app/.venv/bin:$PATH"
-
-USER 65534:65534
-CMD ["python", "-u", "/app/sidecar.py"]

containers/grafana-sidecar/container.py

@@ -0,0 +1,63 @@
+"""Grafana dashboard sidecar — native Dagger build.
+
+Two-stage build: Python venv (builder), Python Alpine (runtime).
+Source cloned from forge mirror.
+"""
+
+import dagger
+from dagger import dag
+
+from blumeops.containers import clone_from_forge, oci_labels
+
+VERSION = "2.6.0"
+
+PYTHON_BASE = "python:3.14-alpine3.23"
+
+
+async def build(src: dagger.Directory) -> dagger.Container:
+    source = clone_from_forge("kiwigrid-grafana-sidecar", VERSION)
+
+    # Stage 1: Build Python venv with dependencies
+    builder = (
+        dag.container()
+        .from_(PYTHON_BASE)
+        .with_exec(["apk", "add", "--no-cache", "gcc", "musl-dev"])
+        .with_workdir("/app")
+        .with_exec(
+            ["python", "-m", "venv", ".venv"],
+        )
+        .with_exec(
+            [".venv/bin/pip", "install", "--no-cache-dir", "-U", "pip", "setuptools"],
+        )
+        .with_file("/app/pyproject.toml", source.file("pyproject.toml"))
+        .with_directory("/app/src", source.directory("src"))
+        .with_exec([".venv/bin/pip", "install", "--no-cache-dir", "."])
+        # Strip test dirs and bytecode from venv to shrink the image
+        .with_exec(
+            [
+                "sh",
+                "-c",
+                "find /app/.venv"
+                " \\( -type d -a -name test -o -name tests \\)"
+                " -o \\( -type f -a -name '*.pyc' -o -name '*.pyo' \\)"
+                " -exec rm -rf {} +",
+            ]
+        )
+    )
+
+    # Stage 2: Runtime
+    runtime = dag.container().from_(PYTHON_BASE)
+    runtime = oci_labels(
+        runtime,
+        title="Grafana Sidecar",
+        description="K8s sidecar to sync ConfigMap dashboards into Grafana",
+        version=VERSION,
+    )
+    return (
+        runtime.with_env_variable("PYTHONUNBUFFERED", "1")
+        .with_workdir("/app")
+        .with_directory("/app/.venv", builder.directory("/app/.venv"))
+        .with_env_variable("PATH", "/app/.venv/bin:$PATH")
+        .with_user("65534:65534")
+        .with_default_args(args=["python", "-u", "-m", "sidecar"])
+    )

docs/changelog.d/grafana-sidecar-2.6.0.feature.md

@@ -0,0 +1 @@
+Upgrade grafana-sidecar from 1.28.0 to 2.6.0, adding health probes and porting build to native Dagger container.py.

docs/how-to/grafana/build-grafana-images.md

@@ -34,23 +34,22 @@ mise run container-build-and-release grafana
 
 ## Grafana Sidecar
 
-**Dockerfile:** `containers/grafana-sidecar/Dockerfile`
+**Build:** `containers/grafana-sidecar/container.py` (native Dagger)
 **Image:** `registry.ops.eblu.me/blumeops/grafana-sidecar`
 
-Clones the [kiwigrid/k8s-sidecar](https://github.com/kiwigrid/k8s-sidecar) source from the forge mirror, installs Python dependencies into a venv, and copies the application into a minimal Alpine runtime image.
+Clones the [kiwigrid/k8s-sidecar](https://github.com/kiwigrid/k8s-sidecar) source from the forge mirror, installs the Python package into a venv, and copies it into a Python Alpine runtime image.
 
 ```fish
-# Update version in Dockerfile
+# Update VERSION in container.py
-# ARG CONTAINER_APP_VERSION=1.28.0
 
 mise run container-build-and-release grafana-sidecar
 ```
 
 **Gotchas:**
 
-- **Pinned to v1.28.0:** v2.x has a 135% memory regression ([#462](https://github.com/kiwigrid/k8s-sidecar/issues/462)) and `readOnlyRootFilesystem` crashloop ([#3936](https://github.com/grafana/helm-charts/issues/3936)). Upgrade separately after upstream fixes land.
 - **UID 65534:** Matches upstream's `nobody` user convention for non-root execution.
 - **Forge mirror name:** `mirrors/kiwigrid-grafana-sidecar` (not `k8s-sidecar`).
+- **Health endpoint:** 2.x exposes `/healthz` on port 8080 (liveness + readiness probes configured in deployment).
 
 ## Related
 

service-versions.yaml

@@ -106,8 +106,8 @@ services:
   - name: grafana-sidecar
     type: argocd
     parent: grafana
-    last-reviewed: "2026-03-03"
+    last-reviewed: "2026-04-13"
-    current-version: "1.28.0"
+    current-version: "2.6.0"
     upstream-source: https://github.com/kiwigrid/k8s-sidecar/releases
     notes: Dashboard ConfigMap watcher sidecar in grafana deployment