From 379bcb98af29a01eedcea047b6d79596a9b0048e Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Fri, 20 Feb 2026 17:56:25 -0800 Subject: [PATCH] Create C2 Mikado cards for harden-zot-registry (#229) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Replace the old pre-Mikado plan doc (`docs/how-to/plans/harden-zot-registry.md`) with a proper C2 Mikado chain in `docs/how-to/zot/` - Root goal: `harden-zot-registry` — enable OIDC + API key auth on zot with anonymous pull preserved - Three leaf prereqs: `register-zot-oidc-client`, `wire-ci-registry-auth`, `enforce-tag-immutability` - Add Zot section to `how-to.md` index, remove plan entry from plans index - All doc checks pass (`docs-check-links`, `docs-check-index`, `docs-mikado`) ## Changes - **New:** `docs/how-to/zot/harden-zot-registry.md` — C2 Mikado root goal - **New:** `docs/how-to/zot/register-zot-oidc-client.md` — Register OIDC client in Authentik - **New:** `docs/how-to/zot/wire-ci-registry-auth.md` — Wire CI push paths with registry auth - **New:** `docs/how-to/zot/enforce-tag-immutability.md` — Prevent version tag overwrites - **Deleted:** `docs/how-to/plans/harden-zot-registry.md` — Old plan doc (content absorbed into Mikado cards) - **Updated:** `docs/how-to/how-to.md` — Add Zot section, remove plan entry - **Updated:** `docs/how-to/plans/plans.md` — Remove plan entry 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/229 --- .../changelog.d/harden-zot-mikado-cards.ai.md | 1 + docs/how-to/how-to.md | 10 +- docs/how-to/plans/harden-zot-registry.md | 211 ------------------ docs/how-to/plans/plans.md | 1 - docs/how-to/zot/enforce-tag-immutability.md | 44 ++++ docs/how-to/zot/harden-zot-registry.md | 59 +++++ docs/how-to/zot/register-zot-oidc-client.md | 51 +++++ docs/how-to/zot/wire-ci-registry-auth.md | 56 +++++ 8 files changed, 220 insertions(+), 213 deletions(-) create mode 100644 docs/changelog.d/harden-zot-mikado-cards.ai.md delete mode 100644 docs/how-to/plans/harden-zot-registry.md create mode 100644 docs/how-to/zot/enforce-tag-immutability.md create mode 100644 docs/how-to/zot/harden-zot-registry.md create mode 100644 docs/how-to/zot/register-zot-oidc-client.md create mode 100644 docs/how-to/zot/wire-ci-registry-auth.md diff --git a/docs/changelog.d/harden-zot-mikado-cards.ai.md b/docs/changelog.d/harden-zot-mikado-cards.ai.md new file mode 100644 index 0000000..13c7541 --- /dev/null +++ b/docs/changelog.d/harden-zot-mikado-cards.ai.md @@ -0,0 +1 @@ +Create C2 Mikado cards for harden-zot-registry: root goal and three prerequisite cards (register-zot-oidc-client, wire-ci-registry-auth, enforce-tag-immutability). diff --git a/docs/how-to/how-to.md b/docs/how-to/how-to.md index 4e594ef..f863881 100644 --- a/docs/how-to/how-to.md +++ b/docs/how-to/how-to.md @@ -60,11 +60,19 @@ Migration and transition plans for upcoming infrastructure changes. | [[adopt-dagger-ci]] | Adopt Dagger as CI/CD build engine | | [[upstream-fork-strategy]] | Stacked-branch forking strategy for upstream projects | | [[adopt-oidc-provider]] | Deploy OIDC identity provider for SSO across services | -| [[harden-zot-registry]] | Add authentication and tag immutability to zot registry | | [[forgejo-actions-dashboard]] | Grafana dashboard for Forgejo Actions CI metrics | | [[upgrade-grafana-helm-chart]] | Upgrade Grafana Helm chart from 8.8.2 to 11.x | | [[operationalize-reolink-camera]] | Cloud-free NVR with Frigate and ring buffer recording | +## Zot + +Mikado chain for hardening the zot registry. Track progress with `mise run docs-mikado harden-zot-registry`. + +- [[harden-zot-registry]] +- [[register-zot-oidc-client]] +- [[wire-ci-registry-auth]] +- [[enforce-tag-immutability]] + ## Authentik Mikado chain for deploying Authentik. Track progress with `mise run docs-mikado deploy-authentik`. diff --git a/docs/how-to/plans/harden-zot-registry.md b/docs/how-to/plans/harden-zot-registry.md deleted file mode 100644 index dc4a516..0000000 --- a/docs/how-to/plans/harden-zot-registry.md +++ /dev/null @@ -1,211 +0,0 @@ ---- -title: "Plan: Harden Zot Registry" -modified: 2026-02-11 -tags: - - how-to - - plans - - zot - - registry - - security ---- - -# Plan: Harden Zot Registry - -> **Status:** Planned (not yet executed) -> **Sequence:** Execute after [[adopt-dagger-ci]] and [[adopt-oidc-provider]] — the Dagger migration will change how images are built and pushed, and the OIDC provider supplies the identity layer that zot's auth and API key features depend on. - -## Background - -Zot is the BlumeOps OCI container registry, running natively on [[indri]]. It serves two roles: a pull-through cache for upstream registries (Docker Hub, GHCR, Quay) and the private image store for `blumeops/*` images. - -Currently, zot has **no authentication** — the security boundary is the Tailscale ACL. This was an acceptable starting point, but has two gaps: - -1. **Any tailnet client can push images** — there's no distinction between pull (which k8s pods need) and push (which only CI should do). A compromised service or misconfigured pod could overwrite production images. -2. **Tags are mutable** — pushing the same tag twice silently overwrites the previous image. There's no protection against accidental or malicious tag clobbering. - -### Goals - -- **Authenticated push** — only CI (Forgejo Actions / Dagger) can push images; all other clients are pull-only -- **Tag immutability** — once a version tag is pushed, it cannot be overwritten -- **No disruption to pulls** — k8s pods and pull-through caching continue to work without authentication -- **Minimal complexity** — use zot's built-in OIDC and API key features with the BlumeOps identity provider - -## Current State - -### Push Mechanism - -Images are currently pushed via the composite action at `.forgejo/actions/build-push-image/action.yaml`: - -1. `docker buildx build` creates the image -2. `docker save` exports to a tarball -3. `skopeo copy` pushes to `registry.ops.eblu.me` (no credentials needed) - -The action pushes two tags per build: a version tag (e.g., `v1.2.0`) and the git commit SHA. - -### Zot Configuration - -The config template (`ansible/roles/zot/templates/config.json.j2`) has no `accessControl` or `http.auth` section. The HTTP listener binds to `0.0.0.0:5050` with no TLS (Caddy terminates TLS at `registry.ops.eblu.me`). - -## Plan - -### 1. Add Authentication for Push (OIDC + API Keys) - -Zot supports native OIDC authentication with a built-in API key feature designed for exactly this use case. The approach: - -1. **OIDC for browser login** — zot delegates authentication to the BlumeOps OIDC provider (see [[adopt-oidc-provider]]). Human users log in via browser redirect. -2. **API keys for CI** — after logging in via OIDC, generate a scoped API key for Forgejo CI / Dagger. API keys are zot-native tokens (`zak_...`) that work with `docker login`, `skopeo`, and Dagger's `with_registry_auth()`. They can be scoped to specific repositories and given expiration dates. -3. **Access control** — `anonymousPolicy` allows unauthenticated pull; push requires authentication. - -```json -{ - "http": { - "auth": { - "openid": { - "providers": { - "oidc": { - "name": "BlumeOps", - "credentialsFile": "/Users/erichblume/.config/zot/oidc-credentials.json", - "issuer": "https://authentik.ops.eblu.me", - "scopes": ["openid", "profile", "email"] - } - } - }, - "apikey": true - }, - "accessControl": { - "repositories": { - "**": { - "anonymousPolicy": ["read"], - "defaultPolicy": ["read", "create", "update"], - "policies": [ - { - "users": ["eblume"], - "actions": ["read", "create", "update", "delete"] - } - ] - } - }, - "adminPolicy": { - "users": ["eblume"], - "actions": ["read", "create", "update", "delete"] - } - } - } -} -``` - -The OIDC credentials file (client ID and secret) is deployed by Ansible from 1Password — never committed to the repo. - -**CI push flow after setup:** -1. Log in to zot UI via browser (OIDC redirect to Authentik) -2. Generate an API key: `POST /zot/auth/apikey` with label `forgejo-ci`, scoped to `blumeops/**` -3. Store the key in 1Password (`op://blumeops/zot-ci-apikey/credential`) -4. CI uses the key: `docker login -u eblume -p zak_... registry.ops.eblu.me` - -This ensures: -- k8s pods, minikube containerd, and pull-through caching all continue to work anonymously (read-only) -- Push requires a valid API key tied to an OIDC identity -- No standalone password files (htpasswd) to manage — identity flows from the central IdP - -### 2. Enforce Tag Immutability - -Zot does not have a built-in tag immutability feature at the registry level. Options to consider during execution: - -- **Registry-side:** Check if newer zot versions (post-2.1) have added immutability policies. If so, configure in `config.json`. -- **Push-side enforcement:** The simpler approach — check whether a tag already exists before pushing. The current build-push-image action (and its eventual Dagger replacement) should query the registry API (`GET /v2//tags/list`) and **fail the build** if the version tag already exists. Commit SHA tags are inherently unique and don't need this check. - -The push-side approach is pragmatic: it prevents accidental overwrites in the normal CI flow. Combined with authenticated push, a tag can only be overwritten by someone with CI credentials who deliberately bypasses the check. - -> **See:** `.forgejo/actions/build-push-image/action.yaml` — this is where the pre-push tag check would be added in the current workflow. After [[adopt-dagger-ci]], the equivalent check goes in the Dagger `Container.publish()` wrapper. - -### 3. Update Ansible Role - -The `ansible/roles/zot/` role needs: - -- **New template:** `oidc-credentials.json.j2` (client ID and secret for the Authentik OIDC client) -- **Updated config template:** `config.json.j2` gains `http.auth` (openid + apikey) and `accessControl` sections -- **Updated config template:** `config.json.j2` gains `externalUrl` set to `https://registry.ops.eblu.me` (required for OIDC callback redirects behind Caddy) -- **New variables:** `zot_oidc_client_id` and `zot_oidc_client_secret` sourced from 1Password in the playbook's `pre_tasks` -- **Handler:** restart zot LaunchAgent after config changes (already exists) - -### 4. Update CI Push Credentials - -After [[adopt-dagger-ci]], the Dagger module will use the zot API key for registry auth: - -```python -api_key = dag.set_secret("registry-api-key", - os.environ["ZOT_CI_API_KEY"]) -container.with_registry_auth("registry.ops.eblu.me", "eblume", api_key) -container.publish("registry.ops.eblu.me/blumeops/image:tag") -``` - -### 5. Update Minikube Containerd Config - -The minikube containerd config (`ansible/roles/minikube/tasks/main.yml`) currently talks to zot without credentials. Since anonymous pull remains allowed, **no changes are needed** for containerd. - -## Execution Steps - -1. **Prerequisite: OIDC provider is running** (see [[adopt-oidc-provider]]) - - Authentik (or chosen provider) is deployed and serving `https://authentik.ops.eblu.me` - - A zot OIDC client is registered with the provider - -2. **Update Ansible role** - - Add OIDC credentials template - - Update `config.json.j2` with auth (openid + apikey) and access control - - Store OIDC client credentials in 1Password - - Test with `mise run provision-indri -- --tags zot --check --diff` - -3. **Deploy and verify pulls still work** - - `mise run provision-indri -- --tags zot` - - Verify anonymous pull: `curl -sf https://registry.ops.eblu.me/v2/_catalog` - - Verify unauthenticated push fails: `skopeo copy ... docker://registry.ops.eblu.me/blumeops/test:fail` (should get 401) - -4. **Set up OIDC login and generate CI API key** - - Log in to zot UI via browser (OIDC flow through Authentik) - - Generate an API key for CI use, store in 1Password - - Verify authenticated push works: `docker login -u eblume -p zak_... registry.ops.eblu.me` - -5. **Add tag immutability check to push workflow** - - Add pre-push tag existence check to Dagger module (or build-push-image action) - - Test by attempting to push an existing tag - -6. **Update documentation** - - Update `docs/reference/services/zot.md` security model section - - Add changelog fragment - -## Verification Checklist - -- [ ] Anonymous pull works (k8s pods, containerd, curl) -- [ ] Pull-through caching still works (pull an uncached image from docker.io) -- [ ] Unauthenticated push is rejected (401) -- [ ] OIDC browser login works (redirect to Authentik and back) -- [ ] API key generation works from zot UI -- [ ] Authenticated push with API key succeeds -- [ ] Pushing a duplicate version tag fails (immutability check) -- [ ] Pushing a new commit SHA tag succeeds -- [ ] Grafana dashboard still shows zot metrics -- [ ] `mise run services-check` passes - -## Open Questions - -- **Immutability granularity:** Should immutability apply only to semver tags (`v*`) or also to commit SHA tags? SHA tags are unique by nature, so immutability is only meaningful for version tags. -- **API key rotation:** API keys can have expiration dates. Decide on a rotation policy — e.g., annual expiry with a reminder, or no expiry with manual rotation. - -## Reference Pattern Files - -| File | Purpose | -|------|---------| -| `ansible/roles/zot/templates/config.json.j2` | Current zot config (no auth) | -| `ansible/roles/zot/tasks/main.yml` | Zot deployment tasks | -| `ansible/roles/zot/defaults/main.yml` | Zot default variables | -| `.forgejo/actions/build-push-image/action.yaml` | Current image push workflow (skopeo) | -| `ansible/roles/minikube/tasks/main.yml` | Containerd registry mirror config | -| `docs/reference/services/zot.md` | Zot reference documentation | - -## Related - -- [[adopt-oidc-provider]] — OIDC identity provider (execute first) -- [[adopt-dagger-ci]] — CI/CD engine migration (execute first) -- [[zot]] — Zot reference card -- [[forgejo]] — CI platform that pushes images -- [[cluster]] — Registry consumer diff --git a/docs/how-to/plans/plans.md b/docs/how-to/plans/plans.md index c53cfdc..81348d9 100644 --- a/docs/how-to/plans/plans.md +++ b/docs/how-to/plans/plans.md @@ -18,7 +18,6 @@ Plans differ from regular how-to guides in that they describe work that has been | [[add-unifi-pulumi-stack]] | Abandoned | Add Pulumi IaC for UniFi Express 7 (provider bugs — see doc) | | [[upstream-fork-strategy]] | Planned | Stacked-branch forking strategy for tracking upstream projects | | [[adopt-oidc-provider]] | Completed | Deploy OIDC identity provider for SSO across services | -| [[harden-zot-registry]] | Planned | Add authentication and tag immutability to zot registry | | [[forgejo-actions-dashboard]] | Planned | Grafana dashboard and custom Prometheus exporter for Forgejo Actions CI metrics | | [[upgrade-grafana-helm-chart]] | Planned | Upgrade Grafana Helm chart from 8.8.2 to 11.x (3 phases) | | [[deploy-authentik]] | Active (C2) | Deploy Authentik IdP — Mikado chain tracked in `how-to/authentik/` | diff --git a/docs/how-to/zot/enforce-tag-immutability.md b/docs/how-to/zot/enforce-tag-immutability.md new file mode 100644 index 0000000..afb8435 --- /dev/null +++ b/docs/how-to/zot/enforce-tag-immutability.md @@ -0,0 +1,44 @@ +--- +title: Enforce Tag Immutability +modified: 2026-02-20 +status: active +tags: + - how-to + - zot + - ci +--- + +# Enforce Tag Immutability + +Prevent accidental overwrite of version tags during CI push. + +## Approach + +Push-side enforcement: before pushing a version tag, query `GET /v2/blumeops//tags/list` and fail if the tag already exists. Commit SHA tags are inherently unique and skip this check. + +## Two Push Paths to Update + +1. **Dagger path:** Add tag-existence check before `ctr.publish()` in `.dagger/src/blumeops_ci/main.py` +2. **Nix/skopeo path:** Add tag-existence check before `skopeo copy` in `.forgejo/workflows/build-container-nix.yaml` + +## Key Files + +| File | Purpose | +|------|---------| +| `.dagger/src/blumeops_ci/main.py` | Add pre-publish tag check | +| `.forgejo/workflows/build-container-nix.yaml` | Add pre-copy tag check | + +## Notes + +- After auth is enabled, the tag check API call may need auth too — or it can rely on anonymous read access from `anonymousPolicy: ["read"]` +- Only version tags (e.g., `v1.2.0`) need the check; commit SHA tags are unique by nature + +## Verification + +- [ ] Pushing a new version tag succeeds +- [ ] Pushing an existing version tag fails with a clear error +- [ ] Pushing a commit SHA tag always succeeds + +## Related + +- [[harden-zot-registry]] — Parent goal diff --git a/docs/how-to/zot/harden-zot-registry.md b/docs/how-to/zot/harden-zot-registry.md new file mode 100644 index 0000000..7314bed --- /dev/null +++ b/docs/how-to/zot/harden-zot-registry.md @@ -0,0 +1,59 @@ +--- +title: Harden Zot Registry +modified: 2026-02-20 +status: active +requires: + - register-zot-oidc-client + - wire-ci-registry-auth + - enforce-tag-immutability +tags: + - how-to + - zot + - registry + - security +--- + +# Harden Zot Registry + +Enable OIDC + API key authentication on zot with anonymous pull preserved, and enforce tag immutability for version tags. This is the C2 Mikado root goal. + +## Context + +Zot currently has **no authentication** — security relies entirely on the Tailscale ACL boundary. Any tailnet client can push images, and tags are mutable. + +Both prerequisites from the original plan are now complete: +- [[adopt-oidc-provider]] — Authentik is deployed and serving OIDC +- [[adopt-dagger-ci]] — Dagger handles container builds + +## Core Change + +Update `ansible/roles/zot/templates/config.json.j2` to add: + +1. **`http.auth.openid`** — OIDC provider pointing to Authentik +2. **`http.auth.apikey: true`** — enable API key generation for CI +3. **`accessControl`** — anonymous read, authenticated write +4. **`externalUrl`** — `https://registry.ops.eblu.me` for OIDC callback redirects + +## Key Files + +| File | Purpose | +|------|---------| +| `ansible/roles/zot/templates/config.json.j2` | Zot config — add auth + access control | +| `ansible/roles/zot/defaults/main.yml` | New OIDC variables | +| `ansible/roles/zot/tasks/main.yml` | Deploy OIDC credentials file | + +## Verification + +- [ ] Anonymous pull works (`curl -sf https://registry.ops.eblu.me/v2/_catalog`) +- [ ] Unauthenticated push fails (401) +- [ ] OIDC browser login works (redirect to Authentik and back) +- [ ] API key push works (`docker login` with `zak_...` token) +- [ ] Pull-through caching still works +- [ ] `mise run services-check` passes + +## Related + +- [[register-zot-oidc-client]] — Prereq: register OIDC client in Authentik +- [[wire-ci-registry-auth]] — Prereq: update CI push paths with credentials +- [[enforce-tag-immutability]] — Prereq: prevent version tag overwrites +- [[agent-change-process]] — C2 methodology diff --git a/docs/how-to/zot/register-zot-oidc-client.md b/docs/how-to/zot/register-zot-oidc-client.md new file mode 100644 index 0000000..3e8c891 --- /dev/null +++ b/docs/how-to/zot/register-zot-oidc-client.md @@ -0,0 +1,51 @@ +--- +title: Register Zot OIDC Client +modified: 2026-02-20 +status: active +tags: + - how-to + - zot + - authentik + - oidc +--- + +# Register Zot OIDC Client + +Register a zot OAuth2 provider and application in Authentik via blueprint, following the same pattern as Grafana and Forgejo. + +## What to Do + +1. **Add `zot.yaml` blueprint section** to `argocd/manifests/authentik/configmap-blueprint.yaml`: + - OAuth2Provider: `client_id: zot`, redirect URI `https://registry.ops.eblu.me/zot/auth/callback/oidc` + - Application linked to the provider + - PolicyBinding restricting access to the admins group + +2. **Generate and store client secret** in 1Password item "Authentik (blumeops)" as field `zot-client-secret` + +3. **Add `AUTHENTIK_ZOT_CLIENT_SECRET`** to Authentik worker's ExternalSecret at `argocd/manifests/authentik/external-secret.yaml` + +4. **Blueprint references the secret** via `!Env AUTHENTIK_ZOT_CLIENT_SECRET` + +5. **Create OIDC credentials file** for zot's Ansible role: + - New template `ansible/roles/zot/templates/oidc-credentials.json.j2` containing `client_id` and `client_secret` + - Source `client_secret` from 1Password via a new pre_task in `ansible/playbooks/indri.yml` + +## Key Files + +| File | Purpose | +|------|---------| +| `argocd/manifests/authentik/configmap-blueprint.yaml` | Add zot blueprint (provider + app + policy) | +| `argocd/manifests/authentik/external-secret.yaml` | Add `AUTHENTIK_ZOT_CLIENT_SECRET` env var | +| `ansible/roles/zot/templates/oidc-credentials.json.j2` | New: OIDC credentials for zot | +| `ansible/playbooks/indri.yml` | New pre_task for zot OIDC client secret | + +## Verification + +- [ ] Authentik admin UI shows zot application +- [ ] OIDC discovery endpoint includes zot client +- [ ] Blueprint status is `successful` (check via API, not just logs) + +## Related + +- [[harden-zot-registry]] — Parent goal +- [[deploy-authentik]] — Authentik deployment (completed) diff --git a/docs/how-to/zot/wire-ci-registry-auth.md b/docs/how-to/zot/wire-ci-registry-auth.md new file mode 100644 index 0000000..ad19d00 --- /dev/null +++ b/docs/how-to/zot/wire-ci-registry-auth.md @@ -0,0 +1,56 @@ +--- +title: Wire CI Registry Auth +modified: 2026-02-20 +status: active +tags: + - how-to + - zot + - ci + - forgejo +--- + +# Wire CI Registry Auth + +Ensure both CI push paths authenticate to zot after auth is enabled. + +## Context + +There are two push paths to update: + +1. **Dagger path** (`.forgejo/workflows/build-container.yaml` → `.dagger/src/blumeops_ci/main.py`): Add `with_registry_auth()` to the Dagger `publish()` call, sourcing the API key from env var `ZOT_CI_API_KEY`. + +2. **Nix/skopeo path** (`.forgejo/workflows/build-container-nix.yaml`): Add `--dest-creds` to `skopeo copy`, sourcing the API key from the same env var. + +> **Note:** The API key must be generated manually after OIDC login is working — log in to zot UI via browser, generate an API key, and store it in 1Password. This is a manual step between [[register-zot-oidc-client]] and this card, but not modeled as a formal `requires` dependency. + +## Secret Flow + +### Indri runner (minikube) + +1Password item (new: `zot-ci-apikey`) → ExternalSecret in `forgejo-runner` namespace → env var `ZOT_CI_API_KEY` in runner pod + +### Ringtail runner (k3s) + +1Password → `/etc/forgejo-runner/zot-api-key.env` (or similar) deployed by NixOS config + +## Key Files + +| File | Purpose | +|------|---------| +| `.dagger/src/blumeops_ci/main.py` | Add `with_registry_auth()` to publish | +| `.forgejo/workflows/build-container.yaml` | Pass `ZOT_CI_API_KEY` to Dagger | +| `.forgejo/workflows/build-container-nix.yaml` | Add `--dest-creds` to skopeo | +| `argocd/manifests/forgejo-runner/deployment.yaml` | Mount secret as env var | +| `argocd/manifests/forgejo-runner/external-secret.yaml` | Pull API key from 1Password | +| `nixos/ringtail/configuration.nix` | Ringtail runner secret provisioning | + +## Verification + +- [ ] Dagger push succeeds with registry auth +- [ ] Nix/skopeo push succeeds with registry auth +- [ ] Push without credentials fails (401) + +## Related + +- [[harden-zot-registry]] — Parent goal +- [[register-zot-oidc-client]] — OIDC client registration (do first)