Enable zot registry auth + wire CI credentials (#237)

## Summary

- Enable OIDC + API key authentication on zot registry with three-tier accessControl
  - `anonymousPolicy: ["read"]` — anyone can pull
  - `artifact-workloads` group: `["read", "create"]` — CI push, no overwrite/delete
  - `admins` group: `["read", "create", "update", "delete"]` — break-glass
- Wire both CI push paths (Dagger and Nix/skopeo) with `ZOT_CI_API_KEY` credentials
- Add `artifact-workloads` PolicyBinding in Authentik blueprint for zot app access
- Add `ZOT_CI_API_KEY` to Forgejo Actions secrets via existing ansible role

Completes the `wire-ci-registry-auth` and `harden-zot-registry` Mikado cards.

## Manual Deployment Steps (after merge)

1. Deploy Authentik blueprint: `argocd app sync authentik`
2. In Authentik admin UI: set a password for the `zot-ci` service account
3. Deploy zot config: `mise run provision-indri -- --tags zot`
4. Log in to `https://registry.ops.eblu.me` as `zot-ci` via OIDC → generate API key
5. Store API key in 1Password as `zot-ci-apikey` in blumeops vault
6. Sync Forgejo secrets: `mise run provision-indri -- --tags forgejo_actions_secrets`
7. Trigger a test container build to verify CI push
8. Verify anonymous pull: `curl -sf https://registry.ops.eblu.me/v2/_catalog`

## Uncertainties

- **Zot `accessControl` group matching with OIDC:** Groups from Authentik's `profile` scope claim should map to zot policy groups, but the exact claim-to-group matching needs runtime verification
- **`http.auth.apikey: true`:** This config key is documented but needs verification against the specific zot version built from source on indri
- **API key permissions:** Need to confirm zot API keys inherit the generating user's group for accessControl evaluation

## Test Plan

- [ ] `mise run provision-indri -- --check --diff --tags zot` shows expected config changes
- [ ] Anonymous pull works after deploy
- [ ] Unauthenticated push fails (401)
- [ ] OIDC browser login redirects to Authentik and back
- [ ] API key push works after key generation
- [ ] CI push succeeds with both Dagger and skopeo paths
- [ ] `mise run services-check` passes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/237

.dagger/src/blumeops_ci/main.py

@@ -20,12 +20,16 @@ class BlumeopsCi:
         version: str,
         commit_sha: str,
         registry: str = "registry.ops.eblu.me",
+        registry_username: str = "zot-ci",
+        registry_password: dagger.Secret | None = None,
     ) -> str:
         """Build and push to registry. Returns the image ref.
 
         Tag format: {version}-{commit_sha} (e.g. v1.0.0-abc1234)
         """
         ctr = self.build(src, container_name)
+        if registry_password is not None:
+            ctr = ctr.with_registry_auth(registry, registry_username, registry_password)
         ref = f"{registry}/blumeops/{container_name}:{version}-{commit_sha}"
         return await ctr.publish(ref)
 

.forgejo/workflows/build-container-nix.yaml

@@ -129,6 +129,8 @@ jobs:
 
       - name: Push to registry
         if: steps.check.outputs.exists == 'true'
+        env:
+          ZOT_CI_API_KEY: ${{ secrets.ZOT_CI_API_KEY }}
         run: |
           CONTAINER="${{ matrix.container }}"
           VERSION="${{ steps.meta.outputs.version }}"
@@ -137,6 +139,7 @@ jobs:
 
           echo "Pushing to $IMAGE"
           skopeo copy \
+            --dest-creds="zot-ci:$ZOT_CI_API_KEY" \
             "docker-archive:result" \
             "docker://$IMAGE"
           echo "Push complete: $IMAGE"

.forgejo/workflows/build-container.yaml

@@ -97,9 +97,12 @@ jobs:
 
       - name: Publish
         if: steps.check.outputs.exists == 'true'
+        env:
+          ZOT_CI_API_KEY: ${{ secrets.ZOT_CI_API_KEY }}
         run: |
           dagger call publish \
             --src=. \
             --container-name=${{ matrix.container }} \
             --version=${{ steps.meta.outputs.version }} \
-            --commit-sha=${{ steps.meta.outputs.sha }}
+            --commit-sha=${{ steps.meta.outputs.sha }} \
+            --registry-password=env:ZOT_CI_API_KEY

ansible/playbooks/indri.yml

@@ -108,11 +108,22 @@
       check_mode: false
       tags: [forgejo_actions_secrets]
 
+    - name: Fetch Zot CI API key for Forgejo Actions
+      ansible.builtin.command:
+        cmd: op read "op://vg6xf6vvfmoh5hqjjhlhbeoaie/w3663ffnvkewbftncqxtcpeavy/zot-ci-api"
+      delegate_to: localhost
+      register: _zot_ci_api_key
+      changed_when: false
+      no_log: true
+      check_mode: false
+      tags: [forgejo_actions_secrets]
+
     - name: Set Forgejo Actions secrets facts
       ansible.builtin.set_fact:
         forgejo_api_token: "{{ _forgejo_api_token.stdout }}"
         forgejo_secret_argocd_token: "{{ _forgejo_argocd_token.stdout }}"
         forgejo_secret_fly_deploy_token: "{{ _fly_deploy_token.stdout }}"
+        forgejo_secret_zot_ci_api_key: "{{ _zot_ci_api_key.stdout }}"
       no_log: true
       tags: [forgejo_actions_secrets]
 

ansible/roles/forgejo_actions_secrets/defaults/main.yml

@@ -16,6 +16,8 @@ forgejo_actions_secrets_repos:
         value_var: forgejo_secret_argocd_token
       - name: FLY_DEPLOY_TOKEN
         value_var: forgejo_secret_fly_deploy_token
+      - name: ZOT_CI_API_KEY
+        value_var: forgejo_secret_zot_ci_api_key
   - repo: cv
     secrets:
       - name: FORGE_TOKEN

ansible/roles/zot/defaults/main.yml

@@ -5,6 +5,8 @@ zot_data_dir: /Users/erichblume/zot
 zot_config_dir: /Users/erichblume/.config/zot
 zot_port: 5050
 zot_log_dir: /Users/erichblume/Library/Logs
+zot_external_url: https://registry.ops.eblu.me
+zot_oidc_issuer: https://authentik.ops.eblu.me/application/o/zot/
 
 # Pull-through cache registries (on-demand sync)
 zot_sync_registries:

ansible/roles/zot/templates/config.json.j2

@@ -8,7 +8,41 @@
   },
   "http": {
     "address": "0.0.0.0",
-    "port": "{{ zot_port }}"
+    "port": "{{ zot_port }}",
+    "externalUrl": "{{ zot_external_url }}",
+    "auth": {
+      "openid": {
+        "providers": {
+          "oidc": {
+            "credentialsFile": "{{ zot_config_dir }}/oidc-credentials.json",
+            "issuer": "{{ zot_oidc_issuer }}",
+            "scopes": ["openid", "email", "profile"],
+            "claimMapping": {
+              "username": "preferred_username"
+            }
+          }
+        }
+      },
+      "apikey": true
+    },
+    "accessControl": {
+      "repositories": {
+        "**": {
+          "policies": [
+            {
+              "groups": ["artifact-workloads"],
+              "actions": ["read", "create"]
+            },
+            {
+              "groups": ["admins"],
+              "actions": ["read", "create", "update", "delete"]
+            }
+          ],
+          "anonymousPolicy": ["read"],
+          "defaultPolicy": ["read"]
+        }
+      }
+    }
   },
   "log": {
     "level": "info"

argocd/manifests/authentik/configmap-blueprint.yaml

@@ -218,7 +218,7 @@ data:
           meta_launch_url: https://registry.ops.eblu.me
           policy_engine_mode: any
 
-      # Policy binding — restrict Zot to admins group
+      # Policy binding — allow admins group access to Zot
       - model: authentik_policies.policybinding
         identifiers:
           order: 0
@@ -231,3 +231,17 @@ data:
           enabled: true
           negate: false
           timeout: 30
+
+      # Policy binding — allow artifact-workloads group access to Zot (CI push)
+      - model: authentik_policies.policybinding
+        identifiers:
+          order: 1
+          target: !KeyOf zot-app
+          group: !KeyOf artifact-workloads-group
+        attrs:
+          target: !KeyOf zot-app
+          group: !KeyOf artifact-workloads-group
+          order: 1
+          enabled: true
+          negate: false
+          timeout: 30

docs/changelog.d/wire-ci-registry-auth.feature.md

@@ -0,0 +1 @@
+Enable OIDC + API key authentication on zot registry with three-tier access control (anonymous read, CI create, admin full). Wire both CI push paths (Dagger and Nix/skopeo) with registry credentials via Forgejo Actions secrets.

docs/how-to/zot/harden-zot-registry.md

@@ -1,12 +1,6 @@
 ---
 title: Harden Zot Registry
 modified: 2026-02-21
-status: active
-requires:
-  - register-zot-oidc-client
-  - wire-ci-registry-auth
-  - enforce-tag-immutability
-  - adopt-commit-based-container-tags
 tags:
   - how-to
   - zot
@@ -16,44 +10,40 @@ tags:
 
 # Harden Zot Registry
 
-Enable OIDC + API key authentication on zot with anonymous pull preserved, and enforce tag immutability for version tags. This is the C2 Mikado root goal.
+OIDC + API key authentication on zot with anonymous pull preserved, and tag immutability enforced server-side via accessControl. This was a C2 Mikado goal — all prerequisites are now complete.
 
-## Context
+## What Was Done
 
-Zot currently has **no authentication** — security relies entirely on the Tailscale ACL boundary. Any tailnet client can push images, and tags are mutable.
+Updated `ansible/roles/zot/templates/config.json.j2` with:
 
-Both prerequisites from the original plan are now complete:
-- [[adopt-oidc-provider]] — Authentik is deployed and serving OIDC
-- [[adopt-dagger-ci]] — Dagger handles container builds
-
-## Core Change
-
-Update `ansible/roles/zot/templates/config.json.j2` to add:
-
-1. **`http.auth.openid`** — OIDC provider pointing to Authentik
-2. **`http.auth.apikey: true`** — enable API key generation for CI
-3. **`accessControl`** — three-tier policy enforcing tag immutability:
+1. **`http.auth.openid`** — OIDC provider pointing to Authentik (`sso.ops.eblu.me`)
+2. **`http.auth.apikey: true`** — API key generation for CI service accounts
+3. **`http.accessControl`** — three-tier policy:
    - `anonymousPolicy: ["read"]` — anyone can pull
    - `artifact-workloads` group: `["read", "create"]` — CI can push new tags but cannot overwrite or delete (immutable tags)
-   - admins group: `["read", "create", "delete"]` — break-glass for removing bad images
-4. **`externalUrl`** — `https://registry.ops.eblu.me` for OIDC callback redirects
+   - `admins` group: `["read", "create", "update", "delete"]` — break-glass
+4. **`http.externalUrl`** — `https://registry.ops.eblu.me` for OIDC callback redirects
 
-The `artifact-workloads` group must be created in Authentik (see [[register-zot-oidc-client]]) and a service account added to it for CI use.
+CI authenticates via a zot API key generated from the `zot-ci` service account's OIDC session. The key is stored in 1Password and synced to Forgejo Actions secrets.
 
 ## Key Files
 
 | File | Purpose |
 |------|---------|
-| `ansible/roles/zot/templates/config.json.j2` | Zot config — add auth + access control |
-| `ansible/roles/zot/defaults/main.yml` | New OIDC variables |
-| `ansible/roles/zot/tasks/main.yml` | Deploy OIDC credentials file |
+| `ansible/roles/zot/templates/config.json.j2` | Zot config with auth + access control |
+| `ansible/roles/zot/defaults/main.yml` | OIDC issuer and external URL variables |
+| `ansible/roles/zot/templates/oidc-credentials.json.j2` | OIDC client credentials |
+| `.dagger/src/blumeops_ci/main.py` | `publish()` with registry auth |
+| `.forgejo/workflows/build-container.yaml` | Dagger push with API key |
+| `.forgejo/workflows/build-container-nix.yaml` | Skopeo push with API key |
 
 ## Verification
 
 - [ ] Anonymous pull works (`curl -sf https://registry.ops.eblu.me/v2/_catalog`)
 - [ ] Unauthenticated push fails (401)
 - [ ] OIDC browser login works (redirect to Authentik and back)
-- [ ] API key push works (`docker login` with `zak_...` token)
+- [ ] API key push works (`docker login` with zot API key)
+- [ ] CI push succeeds (Dagger and Nix/skopeo paths)
 - [ ] Pushing an existing version tag as CI user fails (no update permission)
 - [ ] Admin can delete a tag if needed
 - [ ] Pull-through caching still works
@@ -61,8 +51,8 @@ The `artifact-workloads` group must be created in Authentik (see [[register-zot-
 
 ## Related
 
-- [[register-zot-oidc-client]] — Prereq: register OIDC client in Authentik
-- [[wire-ci-registry-auth]] — Prereq: update CI push paths with credentials
+- [[register-zot-oidc-client]] — OIDC client registration in Authentik
+- [[wire-ci-registry-auth]] — CI push path wiring
 - [[enforce-tag-immutability]] — Folded into this card (server-side via accessControl)
-- [[adopt-commit-based-container-tags]] — Prereq: commit-SHA-based image tags
+- [[adopt-commit-based-container-tags]] — Commit-SHA-based image tags
 - [[agent-change-process]] — C2 methodology

docs/how-to/zot/wire-ci-registry-auth.md

@@ -1,7 +1,6 @@
 ---
 title: Wire CI Registry Auth
 modified: 2026-02-21
-status: active
 tags:
   - how-to
   - zot
@@ -11,48 +10,39 @@ tags:
 
 # Wire CI Registry Auth
 
-Ensure both CI push paths authenticate to zot after auth is enabled.
+How CI pipelines authenticate to the zot registry after OIDC + apikey auth is enabled.
 
-## Context
+## Overview
 
-There are two push paths to update:
+The `zot-ci` service account (created in [[register-zot-oidc-client]]) belongs to the `artifact-workloads` group, granting `["read", "create"]` permissions — CI can push new tags but cannot overwrite or delete existing ones.
 
-1. **Dagger path** (`.forgejo/workflows/build-container.yaml` → `.dagger/src/blumeops_ci/main.py`): Add `with_registry_auth()` to the Dagger `publish()` call, sourcing the API key from env var `ZOT_CI_API_KEY`.
+Authentication uses a zot API key generated after the service account's first OIDC login. The key is stored in 1Password (`Forgejo Secrets` item, field `zot-ci-api`, in blumeops vault) and synced to Forgejo Actions secrets via the `forgejo_actions_secrets` ansible role. The key expires every 90 days — see [[zot#API Key Rotation]] for the rotation procedure.
 
-2. **Nix/skopeo path** (`.forgejo/workflows/build-container-nix.yaml`): Add `--dest-creds` to `skopeo copy`, sourcing the API key from the same env var.
+## Push Paths
 
-> **Note:** The API key must be generated manually after OIDC login is working — log in to zot UI via browser, generate an API key, and store it in 1Password. This is a manual step between [[register-zot-oidc-client]] and this card, but not modeled as a formal `requires` dependency.
+### Dagger path (Dockerfile containers)
 
-CI authenticates as a service account in the `artifact-workloads` group (created in [[register-zot-oidc-client]]). This group grants `["read", "create"]` — CI can push new tags but cannot overwrite or delete existing ones, enforcing tag immutability server-side.
+`.forgejo/workflows/build-container.yaml` passes `--registry-password=env:ZOT_CI_API_KEY` to the Dagger `publish()` function, which calls `with_registry_auth()` before pushing.
+
+### Nix/skopeo path (Nix containers)
+
+`.forgejo/workflows/build-container-nix.yaml` passes `--dest-creds=zot-ci:$ZOT_CI_API_KEY` to `skopeo copy`.
 
 ## Secret Flow
 
-### Indri runner (minikube)
-
-1Password item (new: `zot-ci-apikey`) → ExternalSecret in `forgejo-runner` namespace → env var `ZOT_CI_API_KEY` in runner pod
-
-### Ringtail runner (k3s)
-
-1Password → `/etc/forgejo-runner/zot-api-key.env` (or similar) deployed by NixOS config
+1Password `Forgejo Secrets` item (field `zot-ci-api`) → ansible pre_task fetches it → `forgejo_actions_secrets` role syncs to Forgejo API → both runners (k8s on indri, host on ringtail) access it as `${{ secrets.ZOT_CI_API_KEY }}`.
 
 ## Key Files
 
 | File | Purpose |
 |------|---------|
-| `.dagger/src/blumeops_ci/main.py` | Add `with_registry_auth()` to publish |
-| `.forgejo/workflows/build-container.yaml` | Pass `ZOT_CI_API_KEY` to Dagger |
-| `.forgejo/workflows/build-container-nix.yaml` | Add `--dest-creds` to skopeo |
-| `argocd/manifests/forgejo-runner/deployment.yaml` | Mount secret as env var |
-| `argocd/manifests/forgejo-runner/external-secret.yaml` | Pull API key from 1Password |
-| `nixos/ringtail/configuration.nix` | Ringtail runner secret provisioning |
-
-## Verification
-
-- [ ] Dagger push succeeds with registry auth
-- [ ] Nix/skopeo push succeeds with registry auth
-- [ ] Push without credentials fails (401)
+| `.dagger/src/blumeops_ci/main.py` | `publish()` accepts optional `registry_password` |
+| `.forgejo/workflows/build-container.yaml` | Passes API key to Dagger |
+| `.forgejo/workflows/build-container-nix.yaml` | Passes API key to skopeo |
+| `ansible/playbooks/indri.yml` | Pre_task fetches API key from 1Password |
+| `ansible/roles/forgejo_actions_secrets/defaults/main.yml` | Secret entry for `ZOT_CI_API_KEY` |
 
 ## Related
 
 - [[harden-zot-registry]] — Parent goal
-- [[register-zot-oidc-client]] — OIDC client registration (do first)
+- [[register-zot-oidc-client]] — OIDC client registration

docs/reference/services/zot.md

@@ -35,9 +35,33 @@ When [[cluster|minikube]] pulls an image, containerd checks zot first. If cached
 
 ## Security Model
 
-Network access only (no authentication). Defense is the Tailscale ACL boundary.
+OIDC authentication via [[authentik]], with API key support for CI. Three-tier access control:
+
+| Role | Permissions | Use case |
+|------|------------|----------|
+| Anonymous | read | Pull images without auth |
+| `artifact-workloads` group | read, create | CI push (new tags only, no overwrite/delete) |
+| `admins` group | read, create, update, delete | Break-glass admin access |
+
+CI authenticates with a zot API key generated from the `zot-ci` service account's OIDC session. The key is stored in the `Forgejo Secrets` 1Password item (field `zot-ci-api`) and synced to Forgejo Actions secrets via ansible.
+
+## API Key Rotation
+
+The `zot-ci` API key expires every **90 days**. To rotate:
+
+1. In Authentik admin UI, impersonate the `zot-ci` user
+2. Visit `https://registry.ops.eblu.me` — you'll land on the login page
+3. Click "SIGN IN WITH OIDC" to authenticate as zot-ci
+4. Navigate to `https://registry.ops.eblu.me/user/apikey`
+5. Generate a new API key, copy it to clipboard
+6. Update 1Password:
+   ```fish
+   pbpaste | op item edit "Forgejo Secrets" --vault blumeops "zot-ci-api[password]=-"
+   ```
+7. Sync to Forgejo: `mise run provision-indri -- --tags forgejo_actions_secrets`
 
 ## Related
 
 - [[forgejo]] - Container build CI
 - [[cluster|Cluster]] - Registry consumer
+- [[authentik]] - OIDC identity provider