Home-build grafana-sidecar container (#281)

## Summary
- Home-build the k8s-sidecar container (`grafana-sidecar`) from forge mirror, replacing upstream `quay.io/kiwigrid/k8s-sidecar:1.28.0`
- Pinned to v1.28.0 — v2.x deferred due to 135% memory regression and readOnlyRootFilesystem crashloop
- Adds Dockerfile, service-versions entry, docs, and changelog fragment
- Manifest switch to home-built image pending container build

## Deployment and Testing
- [ ] `mise run container-build-and-release grafana-sidecar`
- [ ] Update kustomization.yaml with built image tag
- [ ] `argocd app set grafana --revision feature/grafana-sidecar && argocd app sync grafana`
- [ ] Verify sidecar logs and dashboards at https://grafana.ops.eblu.me
- [ ] Post-merge: `argocd app set grafana --revision main && argocd app sync grafana`

Reviewed-on: #281

docs/changelog.d/feature-grafana-sidecar.infra.md

@@ -0,0 +1 @@
+Home-build grafana-sidecar container image, replacing upstream `quay.io/kiwigrid/k8s-sidecar` for supply chain control.

docs/how-to/grafana/build-grafana-container.md

@@ -31,13 +31,10 @@ mise run container-build-and-release grafana
 - **Binary PATH:** The binary lives at `bin/grafana` inside the extracted directory. The Dockerfile sets `ENV PATH="/usr/share/grafana/bin:$PATH"`.
 - **UID 472:** Matches the official Grafana image for PVC ownership compatibility.
 
-## Future Work
-
-The k8s-sidecar image (`quay.io/kiwigrid/k8s-sidecar`) is still pulled from upstream. Replace with a home-built image when prioritized.
-
 ## Related
 
 - [[grafana]] — Service reference card
 - [[upgrade-grafana]] — Migration context
 - [[kustomize-grafana-deployment]] — Kustomize manifest structure
+- [[build-grafana-sidecar]] — Home-built sidecar container
 - [[build-container-image]] — Standard container build workflow

docs/how-to/grafana/build-grafana-sidecar.md

@@ -0,0 +1,39 @@
+---
+title: Build Grafana Sidecar
+modified: 2026-03-03
+last-reviewed: 2026-03-03
+tags:
+  - how-to
+  - grafana
+  - containers
+---
+
+# Build Grafana Sidecar
+
+Home-built k8s-sidecar container image published to `registry.ops.eblu.me/blumeops/grafana-sidecar`.
+
+## How It Works
+
+The Dockerfile at `containers/grafana-sidecar/Dockerfile` clones the [kiwigrid/k8s-sidecar](https://github.com/kiwigrid/k8s-sidecar) source from the forge mirror, installs Python dependencies into a venv, and copies the application into a minimal Alpine runtime image.
+
+To build and push a new version:
+
+```fish
+# Update version in Dockerfile
+# ARG CONTAINER_APP_VERSION=1.28.0
+
+mise run container-build-and-release grafana-sidecar
+```
+
+## Gotchas
+
+- **Pinned to v1.28.0:** v2.x has a 135% memory regression ([#462](https://github.com/kiwigrid/k8s-sidecar/issues/462)) and `readOnlyRootFilesystem` crashloop ([#3936](https://github.com/grafana/helm-charts/issues/3936)). Upgrade separately after upstream fixes land.
+- **UID 65534:** Matches upstream's `nobody` user convention for non-root execution.
+- **Forge mirror name:** `mirrors/kiwigrid-grafana-sidecar` (not `k8s-sidecar`).
+
+## Related
+
+- [[grafana]] — Service reference card
+- [[build-grafana-container]] — Home-built Grafana container
+- [[kustomize-grafana-deployment]] — Kustomize manifest structure
+- [[build-container-image]] — Standard container build workflow

docs/how-to/grafana/kustomize-grafana-deployment.md

@@ -16,7 +16,7 @@ Grafana is deployed via plain Kustomize manifests in `argocd/manifests/grafana/`
 | File | Purpose |
 |------|---------|
 | `kustomization.yaml` | Resource list + configMapGenerator for config files |
-| `deployment.yaml` | Grafana container + k8s-sidecar for dashboards |
+| `deployment.yaml` | Grafana container + home-built k8s-sidecar for dashboards |
 | `service.yaml` | ClusterIP on port 80 → 3000 |
 | `pvc.yaml` | 1Gi SQLite storage |
 | `grafana.ini` | Grafana server configuration (fed to configMapGenerator) |
@@ -34,4 +34,5 @@ Grafana is deployed via plain Kustomize manifests in `argocd/manifests/grafana/`
 ## Related
 
 - [[upgrade-grafana]] — Migration context
+- [[build-grafana-sidecar]] — Home-built sidecar container
 - [[grafana]] — Service reference card

docs/how-to/how-to.md

@@ -114,6 +114,7 @@ Mikado chain for upgrading Grafana to 12.x with kustomize and home-built contain
 - [[upgrade-grafana]]
 - [[kustomize-grafana-deployment]]
 - [[build-grafana-container]]
+- [[build-grafana-sidecar]]
 
 ## Forgejo Runner
 

docs/reference/services/grafana.md

@@ -19,6 +19,7 @@ Dashboards and visualization for BlumeOps observability.
 | **Namespace** | `monitoring` |
 | **Deployment** | Kustomize (`argocd/manifests/grafana/`) |
 | **Image** | `registry.ops.eblu.me/blumeops/grafana` |
+| **Sidecar Image** | `registry.ops.eblu.me/blumeops/grafana-sidecar` |
 
 ## Authentication
 
@@ -58,6 +59,7 @@ Optional annotation: `grafana_folder: "FolderName"`
 ## Related
 
 - [[build-grafana-container]] - Home-built container image
+- [[build-grafana-sidecar]] - Home-built sidecar container
 - [[kustomize-grafana-deployment]] - Kustomize manifest structure
 - [[authentik]] - OIDC identity provider for SSO
 - [[prometheus]] - Metrics datasource