Create C2 Mikado cards for harden-zot-registry (#229)

## Summary
- Replace the old pre-Mikado plan doc (`docs/how-to/plans/harden-zot-registry.md`) with a proper C2 Mikado chain in `docs/how-to/zot/`
- Root goal: `harden-zot-registry` — enable OIDC + API key auth on zot with anonymous pull preserved
- Three leaf prereqs: `register-zot-oidc-client`, `wire-ci-registry-auth`, `enforce-tag-immutability`
- Add Zot section to `how-to.md` index, remove plan entry from plans index
- All doc checks pass (`docs-check-links`, `docs-check-index`, `docs-mikado`)

## Changes
- **New:** `docs/how-to/zot/harden-zot-registry.md` — C2 Mikado root goal
- **New:** `docs/how-to/zot/register-zot-oidc-client.md` — Register OIDC client in Authentik
- **New:** `docs/how-to/zot/wire-ci-registry-auth.md` — Wire CI push paths with registry auth
- **New:** `docs/how-to/zot/enforce-tag-immutability.md` — Prevent version tag overwrites
- **Deleted:** `docs/how-to/plans/harden-zot-registry.md` — Old plan doc (content absorbed into Mikado cards)
- **Updated:** `docs/how-to/how-to.md` — Add Zot section, remove plan entry
- **Updated:** `docs/how-to/plans/plans.md` — Remove plan entry

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/229

docs/how-to/how-to.md

@@ -60,11 +60,19 @@ Migration and transition plans for upcoming infrastructure changes.
 | [[adopt-dagger-ci]] | Adopt Dagger as CI/CD build engine |
 | [[upstream-fork-strategy]] | Stacked-branch forking strategy for upstream projects |
 | [[adopt-oidc-provider]] | Deploy OIDC identity provider for SSO across services |
-| [[harden-zot-registry]] | Add authentication and tag immutability to zot registry |
 | [[forgejo-actions-dashboard]] | Grafana dashboard for Forgejo Actions CI metrics |
 | [[upgrade-grafana-helm-chart]] | Upgrade Grafana Helm chart from 8.8.2 to 11.x |
 | [[operationalize-reolink-camera]] | Cloud-free NVR with Frigate and ring buffer recording |
 
+## Zot
+
+Mikado chain for hardening the zot registry. Track progress with `mise run docs-mikado harden-zot-registry`.
+
+- [[harden-zot-registry]]
+- [[register-zot-oidc-client]]
+- [[wire-ci-registry-auth]]
+- [[enforce-tag-immutability]]
+
 ## Authentik
 
 Mikado chain for deploying Authentik. Track progress with `mise run docs-mikado deploy-authentik`.

docs/how-to/plans/harden-zot-registry.md

@@ -1,211 +0,0 @@
----
-title: "Plan: Harden Zot Registry"
-modified: 2026-02-11
-tags:
-  - how-to
-  - plans
-  - zot
-  - registry
-  - security
----
-
-# Plan: Harden Zot Registry
-
-> **Status:** Planned (not yet executed)
-> **Sequence:** Execute after [[adopt-dagger-ci]] and [[adopt-oidc-provider]] — the Dagger migration will change how images are built and pushed, and the OIDC provider supplies the identity layer that zot's auth and API key features depend on.
-
-## Background
-
-Zot is the BlumeOps OCI container registry, running natively on [[indri]]. It serves two roles: a pull-through cache for upstream registries (Docker Hub, GHCR, Quay) and the private image store for `blumeops/*` images.
-
-Currently, zot has **no authentication** — the security boundary is the Tailscale ACL. This was an acceptable starting point, but has two gaps:
-
-1. **Any tailnet client can push images** — there's no distinction between pull (which k8s pods need) and push (which only CI should do). A compromised service or misconfigured pod could overwrite production images.
-2. **Tags are mutable** — pushing the same tag twice silently overwrites the previous image. There's no protection against accidental or malicious tag clobbering.
-
-### Goals
-
-- **Authenticated push** — only CI (Forgejo Actions / Dagger) can push images; all other clients are pull-only
-- **Tag immutability** — once a version tag is pushed, it cannot be overwritten
-- **No disruption to pulls** — k8s pods and pull-through caching continue to work without authentication
-- **Minimal complexity** — use zot's built-in OIDC and API key features with the BlumeOps identity provider
-
-## Current State
-
-### Push Mechanism
-
-Images are currently pushed via the composite action at `.forgejo/actions/build-push-image/action.yaml`:
-
-1. `docker buildx build` creates the image
-2. `docker save` exports to a tarball
-3. `skopeo copy` pushes to `registry.ops.eblu.me` (no credentials needed)
-
-The action pushes two tags per build: a version tag (e.g., `v1.2.0`) and the git commit SHA.
-
-### Zot Configuration
-
-The config template (`ansible/roles/zot/templates/config.json.j2`) has no `accessControl` or `http.auth` section. The HTTP listener binds to `0.0.0.0:5050` with no TLS (Caddy terminates TLS at `registry.ops.eblu.me`).
-
-## Plan
-
-### 1. Add Authentication for Push (OIDC + API Keys)
-
-Zot supports native OIDC authentication with a built-in API key feature designed for exactly this use case. The approach:
-
-1. **OIDC for browser login** — zot delegates authentication to the BlumeOps OIDC provider (see [[adopt-oidc-provider]]). Human users log in via browser redirect.
-2. **API keys for CI** — after logging in via OIDC, generate a scoped API key for Forgejo CI / Dagger. API keys are zot-native tokens (`zak_...`) that work with `docker login`, `skopeo`, and Dagger's `with_registry_auth()`. They can be scoped to specific repositories and given expiration dates.
-3. **Access control** — `anonymousPolicy` allows unauthenticated pull; push requires authentication.
-
-```json
-{
-  "http": {
-    "auth": {
-      "openid": {
-        "providers": {
-          "oidc": {
-            "name": "BlumeOps",
-            "credentialsFile": "/Users/erichblume/.config/zot/oidc-credentials.json",
-            "issuer": "https://authentik.ops.eblu.me",
-            "scopes": ["openid", "profile", "email"]
-          }
-        }
-      },
-      "apikey": true
-    },
-    "accessControl": {
-      "repositories": {
-        "**": {
-          "anonymousPolicy": ["read"],
-          "defaultPolicy": ["read", "create", "update"],
-          "policies": [
-            {
-              "users": ["eblume"],
-              "actions": ["read", "create", "update", "delete"]
-            }
-          ]
-        }
-      },
-      "adminPolicy": {
-        "users": ["eblume"],
-        "actions": ["read", "create", "update", "delete"]
-      }
-    }
-  }
-}
-```
-
-The OIDC credentials file (client ID and secret) is deployed by Ansible from 1Password — never committed to the repo.
-
-**CI push flow after setup:**
-1. Log in to zot UI via browser (OIDC redirect to Authentik)
-2. Generate an API key: `POST /zot/auth/apikey` with label `forgejo-ci`, scoped to `blumeops/**`
-3. Store the key in 1Password (`op://blumeops/zot-ci-apikey/credential`)
-4. CI uses the key: `docker login -u eblume -p zak_... registry.ops.eblu.me`
-
-This ensures:
-- k8s pods, minikube containerd, and pull-through caching all continue to work anonymously (read-only)
-- Push requires a valid API key tied to an OIDC identity
-- No standalone password files (htpasswd) to manage — identity flows from the central IdP
-
-### 2. Enforce Tag Immutability
-
-Zot does not have a built-in tag immutability feature at the registry level. Options to consider during execution:
-
-- **Registry-side:** Check if newer zot versions (post-2.1) have added immutability policies. If so, configure in `config.json`.
-- **Push-side enforcement:** The simpler approach — check whether a tag already exists before pushing. The current build-push-image action (and its eventual Dagger replacement) should query the registry API (`GET /v2/<name>/tags/list`) and **fail the build** if the version tag already exists. Commit SHA tags are inherently unique and don't need this check.
-
-The push-side approach is pragmatic: it prevents accidental overwrites in the normal CI flow. Combined with authenticated push, a tag can only be overwritten by someone with CI credentials who deliberately bypasses the check.
-
-> **See:** `.forgejo/actions/build-push-image/action.yaml` — this is where the pre-push tag check would be added in the current workflow. After [[adopt-dagger-ci]], the equivalent check goes in the Dagger `Container.publish()` wrapper.
-
-### 3. Update Ansible Role
-
-The `ansible/roles/zot/` role needs:
-
-- **New template:** `oidc-credentials.json.j2` (client ID and secret for the Authentik OIDC client)
-- **Updated config template:** `config.json.j2` gains `http.auth` (openid + apikey) and `accessControl` sections
-- **Updated config template:** `config.json.j2` gains `externalUrl` set to `https://registry.ops.eblu.me` (required for OIDC callback redirects behind Caddy)
-- **New variables:** `zot_oidc_client_id` and `zot_oidc_client_secret` sourced from 1Password in the playbook's `pre_tasks`
-- **Handler:** restart zot LaunchAgent after config changes (already exists)
-
-### 4. Update CI Push Credentials
-
-After [[adopt-dagger-ci]], the Dagger module will use the zot API key for registry auth:
-
-```python
-api_key = dag.set_secret("registry-api-key",
-    os.environ["ZOT_CI_API_KEY"])
-container.with_registry_auth("registry.ops.eblu.me", "eblume", api_key)
-container.publish("registry.ops.eblu.me/blumeops/image:tag")
-```
-
-### 5. Update Minikube Containerd Config
-
-The minikube containerd config (`ansible/roles/minikube/tasks/main.yml`) currently talks to zot without credentials. Since anonymous pull remains allowed, **no changes are needed** for containerd.
-
-## Execution Steps
-
-1. **Prerequisite: OIDC provider is running** (see [[adopt-oidc-provider]])
-   - Authentik (or chosen provider) is deployed and serving `https://authentik.ops.eblu.me`
-   - A zot OIDC client is registered with the provider
-
-2. **Update Ansible role**
-   - Add OIDC credentials template
-   - Update `config.json.j2` with auth (openid + apikey) and access control
-   - Store OIDC client credentials in 1Password
-   - Test with `mise run provision-indri -- --tags zot --check --diff`
-
-3. **Deploy and verify pulls still work**
-   - `mise run provision-indri -- --tags zot`
-   - Verify anonymous pull: `curl -sf https://registry.ops.eblu.me/v2/_catalog`
-   - Verify unauthenticated push fails: `skopeo copy ... docker://registry.ops.eblu.me/blumeops/test:fail` (should get 401)
-
-4. **Set up OIDC login and generate CI API key**
-   - Log in to zot UI via browser (OIDC flow through Authentik)
-   - Generate an API key for CI use, store in 1Password
-   - Verify authenticated push works: `docker login -u eblume -p zak_... registry.ops.eblu.me`
-
-5. **Add tag immutability check to push workflow**
-   - Add pre-push tag existence check to Dagger module (or build-push-image action)
-   - Test by attempting to push an existing tag
-
-6. **Update documentation**
-   - Update `docs/reference/services/zot.md` security model section
-   - Add changelog fragment
-
-## Verification Checklist
-
-- [ ] Anonymous pull works (k8s pods, containerd, curl)
-- [ ] Pull-through caching still works (pull an uncached image from docker.io)
-- [ ] Unauthenticated push is rejected (401)
-- [ ] OIDC browser login works (redirect to Authentik and back)
-- [ ] API key generation works from zot UI
-- [ ] Authenticated push with API key succeeds
-- [ ] Pushing a duplicate version tag fails (immutability check)
-- [ ] Pushing a new commit SHA tag succeeds
-- [ ] Grafana dashboard still shows zot metrics
-- [ ] `mise run services-check` passes
-
-## Open Questions
-
-- **Immutability granularity:** Should immutability apply only to semver tags (`v*`) or also to commit SHA tags? SHA tags are unique by nature, so immutability is only meaningful for version tags.
-- **API key rotation:** API keys can have expiration dates. Decide on a rotation policy — e.g., annual expiry with a reminder, or no expiry with manual rotation.
-
-## Reference Pattern Files
-
-| File | Purpose |
-|------|---------|
-| `ansible/roles/zot/templates/config.json.j2` | Current zot config (no auth) |
-| `ansible/roles/zot/tasks/main.yml` | Zot deployment tasks |
-| `ansible/roles/zot/defaults/main.yml` | Zot default variables |
-| `.forgejo/actions/build-push-image/action.yaml` | Current image push workflow (skopeo) |
-| `ansible/roles/minikube/tasks/main.yml` | Containerd registry mirror config |
-| `docs/reference/services/zot.md` | Zot reference documentation |
-
-## Related
-
-- [[adopt-oidc-provider]] — OIDC identity provider (execute first)
-- [[adopt-dagger-ci]] — CI/CD engine migration (execute first)
-- [[zot]] — Zot reference card
-- [[forgejo]] — CI platform that pushes images
-- [[cluster]] — Registry consumer

docs/how-to/plans/plans.md

@@ -18,7 +18,6 @@ Plans differ from regular how-to guides in that they describe work that has been
 | [[add-unifi-pulumi-stack]] | Abandoned | Add Pulumi IaC for UniFi Express 7 (provider bugs — see doc) |
 | [[upstream-fork-strategy]] | Planned | Stacked-branch forking strategy for tracking upstream projects |
 | [[adopt-oidc-provider]] | Completed | Deploy OIDC identity provider for SSO across services |
-| [[harden-zot-registry]] | Planned | Add authentication and tag immutability to zot registry |
 | [[forgejo-actions-dashboard]] | Planned | Grafana dashboard and custom Prometheus exporter for Forgejo Actions CI metrics |
 | [[upgrade-grafana-helm-chart]] | Planned | Upgrade Grafana Helm chart from 8.8.2 to 11.x (3 phases) |
 | [[deploy-authentik]] | Active (C2) | Deploy Authentik IdP — Mikado chain tracked in `how-to/authentik/` |

docs/how-to/zot/enforce-tag-immutability.md

@@ -0,0 +1,44 @@
+---
+title: Enforce Tag Immutability
+modified: 2026-02-20
+status: active
+tags:
+  - how-to
+  - zot
+  - ci
+---
+
+# Enforce Tag Immutability
+
+Prevent accidental overwrite of version tags during CI push.
+
+## Approach
+
+Push-side enforcement: before pushing a version tag, query `GET /v2/blumeops/<name>/tags/list` and fail if the tag already exists. Commit SHA tags are inherently unique and skip this check.
+
+## Two Push Paths to Update
+
+1. **Dagger path:** Add tag-existence check before `ctr.publish()` in `.dagger/src/blumeops_ci/main.py`
+2. **Nix/skopeo path:** Add tag-existence check before `skopeo copy` in `.forgejo/workflows/build-container-nix.yaml`
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `.dagger/src/blumeops_ci/main.py` | Add pre-publish tag check |
+| `.forgejo/workflows/build-container-nix.yaml` | Add pre-copy tag check |
+
+## Notes
+
+- After auth is enabled, the tag check API call may need auth too — or it can rely on anonymous read access from `anonymousPolicy: ["read"]`
+- Only version tags (e.g., `v1.2.0`) need the check; commit SHA tags are unique by nature
+
+## Verification
+
+- [ ] Pushing a new version tag succeeds
+- [ ] Pushing an existing version tag fails with a clear error
+- [ ] Pushing a commit SHA tag always succeeds
+
+## Related
+
+- [[harden-zot-registry]] — Parent goal

docs/how-to/zot/harden-zot-registry.md

@@ -0,0 +1,59 @@
+---
+title: Harden Zot Registry
+modified: 2026-02-20
+status: active
+requires:
+  - register-zot-oidc-client
+  - wire-ci-registry-auth
+  - enforce-tag-immutability
+tags:
+  - how-to
+  - zot
+  - registry
+  - security
+---
+
+# Harden Zot Registry
+
+Enable OIDC + API key authentication on zot with anonymous pull preserved, and enforce tag immutability for version tags. This is the C2 Mikado root goal.
+
+## Context
+
+Zot currently has **no authentication** — security relies entirely on the Tailscale ACL boundary. Any tailnet client can push images, and tags are mutable.
+
+Both prerequisites from the original plan are now complete:
+- [[adopt-oidc-provider]] — Authentik is deployed and serving OIDC
+- [[adopt-dagger-ci]] — Dagger handles container builds
+
+## Core Change
+
+Update `ansible/roles/zot/templates/config.json.j2` to add:
+
+1. **`http.auth.openid`** — OIDC provider pointing to Authentik
+2. **`http.auth.apikey: true`** — enable API key generation for CI
+3. **`accessControl`** — anonymous read, authenticated write
+4. **`externalUrl`** — `https://registry.ops.eblu.me` for OIDC callback redirects
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `ansible/roles/zot/templates/config.json.j2` | Zot config — add auth + access control |
+| `ansible/roles/zot/defaults/main.yml` | New OIDC variables |
+| `ansible/roles/zot/tasks/main.yml` | Deploy OIDC credentials file |
+
+## Verification
+
+- [ ] Anonymous pull works (`curl -sf https://registry.ops.eblu.me/v2/_catalog`)
+- [ ] Unauthenticated push fails (401)
+- [ ] OIDC browser login works (redirect to Authentik and back)
+- [ ] API key push works (`docker login` with `zak_...` token)
+- [ ] Pull-through caching still works
+- [ ] `mise run services-check` passes
+
+## Related
+
+- [[register-zot-oidc-client]] — Prereq: register OIDC client in Authentik
+- [[wire-ci-registry-auth]] — Prereq: update CI push paths with credentials
+- [[enforce-tag-immutability]] — Prereq: prevent version tag overwrites
+- [[agent-change-process]] — C2 methodology

docs/how-to/zot/register-zot-oidc-client.md

@@ -0,0 +1,51 @@
+---
+title: Register Zot OIDC Client
+modified: 2026-02-20
+status: active
+tags:
+  - how-to
+  - zot
+  - authentik
+  - oidc
+---
+
+# Register Zot OIDC Client
+
+Register a zot OAuth2 provider and application in Authentik via blueprint, following the same pattern as Grafana and Forgejo.
+
+## What to Do
+
+1. **Add `zot.yaml` blueprint section** to `argocd/manifests/authentik/configmap-blueprint.yaml`:
+   - OAuth2Provider: `client_id: zot`, redirect URI `https://registry.ops.eblu.me/zot/auth/callback/oidc`
+   - Application linked to the provider
+   - PolicyBinding restricting access to the admins group
+
+2. **Generate and store client secret** in 1Password item "Authentik (blumeops)" as field `zot-client-secret`
+
+3. **Add `AUTHENTIK_ZOT_CLIENT_SECRET`** to Authentik worker's ExternalSecret at `argocd/manifests/authentik/external-secret.yaml`
+
+4. **Blueprint references the secret** via `!Env AUTHENTIK_ZOT_CLIENT_SECRET`
+
+5. **Create OIDC credentials file** for zot's Ansible role:
+   - New template `ansible/roles/zot/templates/oidc-credentials.json.j2` containing `client_id` and `client_secret`
+   - Source `client_secret` from 1Password via a new pre_task in `ansible/playbooks/indri.yml`
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `argocd/manifests/authentik/configmap-blueprint.yaml` | Add zot blueprint (provider + app + policy) |
+| `argocd/manifests/authentik/external-secret.yaml` | Add `AUTHENTIK_ZOT_CLIENT_SECRET` env var |
+| `ansible/roles/zot/templates/oidc-credentials.json.j2` | New: OIDC credentials for zot |
+| `ansible/playbooks/indri.yml` | New pre_task for zot OIDC client secret |
+
+## Verification
+
+- [ ] Authentik admin UI shows zot application
+- [ ] OIDC discovery endpoint includes zot client
+- [ ] Blueprint status is `successful` (check via API, not just logs)
+
+## Related
+
+- [[harden-zot-registry]] — Parent goal
+- [[deploy-authentik]] — Authentik deployment (completed)

docs/how-to/zot/wire-ci-registry-auth.md

@@ -0,0 +1,56 @@
+---
+title: Wire CI Registry Auth
+modified: 2026-02-20
+status: active
+tags:
+  - how-to
+  - zot
+  - ci
+  - forgejo
+---
+
+# Wire CI Registry Auth
+
+Ensure both CI push paths authenticate to zot after auth is enabled.
+
+## Context
+
+There are two push paths to update:
+
+1. **Dagger path** (`.forgejo/workflows/build-container.yaml` → `.dagger/src/blumeops_ci/main.py`): Add `with_registry_auth()` to the Dagger `publish()` call, sourcing the API key from env var `ZOT_CI_API_KEY`.
+
+2. **Nix/skopeo path** (`.forgejo/workflows/build-container-nix.yaml`): Add `--dest-creds` to `skopeo copy`, sourcing the API key from the same env var.
+
+> **Note:** The API key must be generated manually after OIDC login is working — log in to zot UI via browser, generate an API key, and store it in 1Password. This is a manual step between [[register-zot-oidc-client]] and this card, but not modeled as a formal `requires` dependency.
+
+## Secret Flow
+
+### Indri runner (minikube)
+
+1Password item (new: `zot-ci-apikey`) → ExternalSecret in `forgejo-runner` namespace → env var `ZOT_CI_API_KEY` in runner pod
+
+### Ringtail runner (k3s)
+
+1Password → `/etc/forgejo-runner/zot-api-key.env` (or similar) deployed by NixOS config
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `.dagger/src/blumeops_ci/main.py` | Add `with_registry_auth()` to publish |
+| `.forgejo/workflows/build-container.yaml` | Pass `ZOT_CI_API_KEY` to Dagger |
+| `.forgejo/workflows/build-container-nix.yaml` | Add `--dest-creds` to skopeo |
+| `argocd/manifests/forgejo-runner/deployment.yaml` | Mount secret as env var |
+| `argocd/manifests/forgejo-runner/external-secret.yaml` | Pull API key from 1Password |
+| `nixos/ringtail/configuration.nix` | Ringtail runner secret provisioning |
+
+## Verification
+
+- [ ] Dagger push succeeds with registry auth
+- [ ] Nix/skopeo push succeeds with registry auth
+- [ ] Push without credentials fails (401)
+
+## Related
+
+- [[harden-zot-registry]] — Parent goal
+- [[register-zot-oidc-client]] — OIDC client registration (do first)