Home-build grafana-sidecar container (#281)

## Summary
- Home-build the k8s-sidecar container (`grafana-sidecar`) from forge mirror, replacing upstream `quay.io/kiwigrid/k8s-sidecar:1.28.0`
- Pinned to v1.28.0 — v2.x deferred due to 135% memory regression and readOnlyRootFilesystem crashloop
- Adds Dockerfile, service-versions entry, docs, and changelog fragment
- Manifest switch to home-built image pending container build

## Deployment and Testing
- [ ] `mise run container-build-and-release grafana-sidecar`
- [ ] Update kustomization.yaml with built image tag
- [ ] `argocd app set grafana --revision feature/grafana-sidecar && argocd app sync grafana`
- [ ] Verify sidecar logs and dashboards at https://grafana.ops.eblu.me
- [ ] Post-merge: `argocd app set grafana --revision main && argocd app sync grafana`

Reviewed-on: #281

argocd/manifests/grafana/deployment.yaml

@@ -48,7 +48,7 @@ spec:
       containers:
         # Dashboard sidecar - watches ConfigMaps with grafana_dashboard=1
         - name: grafana-sc-dashboard
-          image: quay.io/kiwigrid/k8s-sidecar
+          image: registry.ops.eblu.me/blumeops/grafana-sidecar
           imagePullPolicy: IfNotPresent
           env:
             - name: METHOD

argocd/manifests/grafana/kustomization.yaml

@@ -13,8 +13,8 @@ resources:
 images:
   - name: docker.io/library/busybox
     newTag: 1.31.1
-  - name: quay.io/kiwigrid/k8s-sidecar
-    newTag: 1.28.0
+  - name: registry.ops.eblu.me/blumeops/grafana-sidecar
+    newTag: v1.28.0-49c8045
   - name: registry.ops.eblu.me/blumeops/grafana
     newTag: v12.3.3-d05d2fb
 

containers/grafana-sidecar/Dockerfile

@@ -0,0 +1,33 @@
+# Grafana dashboard sidecar - watches ConfigMaps and syncs into Grafana
+# Two-stage build: Python venv (builder), runtime (Alpine)
+
+ARG CONTAINER_APP_VERSION=1.28.0
+
+FROM python:3.12-alpine3.22 AS base
+
+FROM base AS builder
+ARG CONTAINER_APP_VERSION
+WORKDIR /app
+RUN apk add --no-cache git gcc musl-dev
+RUN git clone --depth 1 --branch ${CONTAINER_APP_VERSION} \
+    https://forge.ops.eblu.me/mirrors/kiwigrid-grafana-sidecar.git /tmp/k8s-sidecar
+RUN python -m venv .venv && \
+    .venv/bin/pip install --no-cache-dir -U pip setuptools && \
+    .venv/bin/pip install --no-cache-dir -r /tmp/k8s-sidecar/src/requirements.txt && \
+    cp /tmp/k8s-sidecar/src/*.py /app/ && \
+    find /app/.venv \( -type d -a -name test -o -name tests \) \
+      -o \( -type f -a -name '*.pyc' -o -name '*.pyo' \) -exec rm -rf '{}' \+
+
+FROM base
+
+LABEL org.opencontainers.image.title="Grafana Sidecar"
+LABEL org.opencontainers.image.description="K8s sidecar to sync ConfigMap dashboards into Grafana"
+LABEL org.opencontainers.image.source="https://github.com/kiwigrid/k8s-sidecar"
+
+ENV PYTHONUNBUFFERED=1
+WORKDIR /app
+COPY --from=builder /app /app
+ENV PATH="/app/.venv/bin:$PATH"
+
+USER 65534:65534
+CMD ["python", "-u", "/app/sidecar.py"]

docs/changelog.d/feature-grafana-sidecar.infra.md

@@ -0,0 +1 @@
+Home-build grafana-sidecar container image, replacing upstream `quay.io/kiwigrid/k8s-sidecar` for supply chain control.

docs/how-to/grafana/build-grafana-container.md

@@ -31,13 +31,10 @@ mise run container-build-and-release grafana
 - **Binary PATH:** The binary lives at `bin/grafana` inside the extracted directory. The Dockerfile sets `ENV PATH="/usr/share/grafana/bin:$PATH"`.
 - **UID 472:** Matches the official Grafana image for PVC ownership compatibility.
 
-## Future Work
-
-The k8s-sidecar image (`quay.io/kiwigrid/k8s-sidecar`) is still pulled from upstream. Replace with a home-built image when prioritized.
-
 ## Related
 
 - [[grafana]] — Service reference card
 - [[upgrade-grafana]] — Migration context
 - [[kustomize-grafana-deployment]] — Kustomize manifest structure
+- [[build-grafana-sidecar]] — Home-built sidecar container
 - [[build-container-image]] — Standard container build workflow

docs/how-to/grafana/build-grafana-sidecar.md

@@ -0,0 +1,39 @@
+---
+title: Build Grafana Sidecar
+modified: 2026-03-03
+last-reviewed: 2026-03-03
+tags:
+  - how-to
+  - grafana
+  - containers
+---
+
+# Build Grafana Sidecar
+
+Home-built k8s-sidecar container image published to `registry.ops.eblu.me/blumeops/grafana-sidecar`.
+
+## How It Works
+
+The Dockerfile at `containers/grafana-sidecar/Dockerfile` clones the [kiwigrid/k8s-sidecar](https://github.com/kiwigrid/k8s-sidecar) source from the forge mirror, installs Python dependencies into a venv, and copies the application into a minimal Alpine runtime image.
+
+To build and push a new version:
+
+```fish
+# Update version in Dockerfile
+# ARG CONTAINER_APP_VERSION=1.28.0
+
+mise run container-build-and-release grafana-sidecar
+```
+
+## Gotchas
+
+- **Pinned to v1.28.0:** v2.x has a 135% memory regression ([#462](https://github.com/kiwigrid/k8s-sidecar/issues/462)) and `readOnlyRootFilesystem` crashloop ([#3936](https://github.com/grafana/helm-charts/issues/3936)). Upgrade separately after upstream fixes land.
+- **UID 65534:** Matches upstream's `nobody` user convention for non-root execution.
+- **Forge mirror name:** `mirrors/kiwigrid-grafana-sidecar` (not `k8s-sidecar`).
+
+## Related
+
+- [[grafana]] — Service reference card
+- [[build-grafana-container]] — Home-built Grafana container
+- [[kustomize-grafana-deployment]] — Kustomize manifest structure
+- [[build-container-image]] — Standard container build workflow

docs/how-to/grafana/kustomize-grafana-deployment.md

@@ -16,7 +16,7 @@ Grafana is deployed via plain Kustomize manifests in `argocd/manifests/grafana/`
 | File | Purpose |
 |------|---------|
 | `kustomization.yaml` | Resource list + configMapGenerator for config files |
-| `deployment.yaml` | Grafana container + k8s-sidecar for dashboards |
+| `deployment.yaml` | Grafana container + home-built k8s-sidecar for dashboards |
 | `service.yaml` | ClusterIP on port 80 → 3000 |
 | `pvc.yaml` | 1Gi SQLite storage |
 | `grafana.ini` | Grafana server configuration (fed to configMapGenerator) |
@@ -34,4 +34,5 @@ Grafana is deployed via plain Kustomize manifests in `argocd/manifests/grafana/`
 ## Related
 
 - [[upgrade-grafana]] — Migration context
+- [[build-grafana-sidecar]] — Home-built sidecar container
 - [[grafana]] — Service reference card

docs/how-to/how-to.md

@@ -114,6 +114,7 @@ Mikado chain for upgrading Grafana to 12.x with kustomize and home-built contain
 - [[upgrade-grafana]]
 - [[kustomize-grafana-deployment]]
 - [[build-grafana-container]]
+- [[build-grafana-sidecar]]
 
 ## Forgejo Runner
 

docs/reference/services/grafana.md

@@ -19,6 +19,7 @@ Dashboards and visualization for BlumeOps observability.
 | **Namespace** | `monitoring` |
 | **Deployment** | Kustomize (`argocd/manifests/grafana/`) |
 | **Image** | `registry.ops.eblu.me/blumeops/grafana` |
+| **Sidecar Image** | `registry.ops.eblu.me/blumeops/grafana-sidecar` |
 
 ## Authentication
 
@@ -58,6 +59,7 @@ Optional annotation: `grafana_folder: "FolderName"`
 ## Related
 
 - [[build-grafana-container]] - Home-built container image
+- [[build-grafana-sidecar]] - Home-built sidecar container
 - [[kustomize-grafana-deployment]] - Kustomize manifest structure
 - [[authentik]] - OIDC identity provider for SSO
 - [[prometheus]] - Metrics datasource

service-versions.yaml

@@ -87,6 +87,13 @@ services:
     upstream-source: https://github.com/grafana/grafana/releases
     notes: Home-built container; upgrading to 12.x via Mikado chain
 
+  - name: grafana-sidecar
+    type: argocd
+    last-reviewed: "2026-03-03"
+    current-version: "1.28.0"
+    upstream-source: https://github.com/kiwigrid/k8s-sidecar/releases
+    notes: Dashboard ConfigMap watcher sidecar in grafana deployment
+
   - name: cloudnative-pg
     type: argocd
     last-reviewed: 2026-02-24