From ff63679efb5e63f6eaadb1826a4f8b5d9c8285d2 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Sat, 21 Feb 2026 12:20:29 -0800 Subject: [PATCH] Enable zot registry auth + wire CI credentials (#237) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Enable OIDC + API key authentication on zot registry with three-tier accessControl - `anonymousPolicy: ["read"]` — anyone can pull - `artifact-workloads` group: `["read", "create"]` — CI push, no overwrite/delete - `admins` group: `["read", "create", "update", "delete"]` — break-glass - Wire both CI push paths (Dagger and Nix/skopeo) with `ZOT_CI_API_KEY` credentials - Add `artifact-workloads` PolicyBinding in Authentik blueprint for zot app access - Add `ZOT_CI_API_KEY` to Forgejo Actions secrets via existing ansible role Completes the `wire-ci-registry-auth` and `harden-zot-registry` Mikado cards. ## Manual Deployment Steps (after merge) 1. Deploy Authentik blueprint: `argocd app sync authentik` 2. In Authentik admin UI: set a password for the `zot-ci` service account 3. Deploy zot config: `mise run provision-indri -- --tags zot` 4. Log in to `https://registry.ops.eblu.me` as `zot-ci` via OIDC → generate API key 5. Store API key in 1Password as `zot-ci-apikey` in blumeops vault 6. Sync Forgejo secrets: `mise run provision-indri -- --tags forgejo_actions_secrets` 7. Trigger a test container build to verify CI push 8. Verify anonymous pull: `curl -sf https://registry.ops.eblu.me/v2/_catalog` ## Uncertainties - **Zot `accessControl` group matching with OIDC:** Groups from Authentik's `profile` scope claim should map to zot policy groups, but the exact claim-to-group matching needs runtime verification - **`http.auth.apikey: true`:** This config key is documented but needs verification against the specific zot version built from source on indri - **API key permissions:** Need to confirm zot API keys inherit the generating user's group for accessControl evaluation ## Test Plan - [ ] `mise run provision-indri -- --check --diff --tags zot` shows expected config changes - [ ] Anonymous pull works after deploy - [ ] Unauthenticated push fails (401) - [ ] OIDC browser login redirects to Authentik and back - [ ] API key push works after key generation - [ ] CI push succeeds with both Dagger and skopeo paths - [ ] `mise run services-check` passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/237 --- .dagger/src/blumeops_ci/main.py | 4 ++ .forgejo/workflows/build-container-nix.yaml | 3 ++ .forgejo/workflows/build-container.yaml | 5 +- ansible/playbooks/indri.yml | 11 ++++ .../forgejo_actions_secrets/defaults/main.yml | 2 + ansible/roles/zot/defaults/main.yml | 2 + ansible/roles/zot/templates/config.json.j2 | 36 ++++++++++++- .../authentik/configmap-blueprint.yaml | 16 +++++- .../wire-ci-registry-auth.feature.md | 1 + docs/how-to/zot/harden-zot-registry.md | 50 ++++++++----------- docs/how-to/zot/wire-ci-registry-auth.md | 46 +++++++---------- docs/reference/services/zot.md | 26 +++++++++- 12 files changed, 140 insertions(+), 62 deletions(-) create mode 100644 docs/changelog.d/wire-ci-registry-auth.feature.md diff --git a/.dagger/src/blumeops_ci/main.py b/.dagger/src/blumeops_ci/main.py index e8f7720..c20709f 100644 --- a/.dagger/src/blumeops_ci/main.py +++ b/.dagger/src/blumeops_ci/main.py @@ -20,12 +20,16 @@ class BlumeopsCi: version: str, commit_sha: str, registry: str = "registry.ops.eblu.me", + registry_username: str = "zot-ci", + registry_password: dagger.Secret | None = None, ) -> str: """Build and push to registry. Returns the image ref. Tag format: {version}-{commit_sha} (e.g. v1.0.0-abc1234) """ ctr = self.build(src, container_name) + if registry_password is not None: + ctr = ctr.with_registry_auth(registry, registry_username, registry_password) ref = f"{registry}/blumeops/{container_name}:{version}-{commit_sha}" return await ctr.publish(ref) diff --git a/.forgejo/workflows/build-container-nix.yaml b/.forgejo/workflows/build-container-nix.yaml index b5aab17..d68729f 100644 --- a/.forgejo/workflows/build-container-nix.yaml +++ b/.forgejo/workflows/build-container-nix.yaml @@ -129,6 +129,8 @@ jobs: - name: Push to registry if: steps.check.outputs.exists == 'true' + env: + ZOT_CI_API_KEY: ${{ secrets.ZOT_CI_API_KEY }} run: | CONTAINER="${{ matrix.container }}" VERSION="${{ steps.meta.outputs.version }}" @@ -137,6 +139,7 @@ jobs: echo "Pushing to $IMAGE" skopeo copy \ + --dest-creds="zot-ci:$ZOT_CI_API_KEY" \ "docker-archive:result" \ "docker://$IMAGE" echo "Push complete: $IMAGE" diff --git a/.forgejo/workflows/build-container.yaml b/.forgejo/workflows/build-container.yaml index 7e3775e..c589f67 100644 --- a/.forgejo/workflows/build-container.yaml +++ b/.forgejo/workflows/build-container.yaml @@ -97,9 +97,12 @@ jobs: - name: Publish if: steps.check.outputs.exists == 'true' + env: + ZOT_CI_API_KEY: ${{ secrets.ZOT_CI_API_KEY }} run: | dagger call publish \ --src=. \ --container-name=${{ matrix.container }} \ --version=${{ steps.meta.outputs.version }} \ - --commit-sha=${{ steps.meta.outputs.sha }} + --commit-sha=${{ steps.meta.outputs.sha }} \ + --registry-password=env:ZOT_CI_API_KEY diff --git a/ansible/playbooks/indri.yml b/ansible/playbooks/indri.yml index 90ea26c..c2bb67a 100644 --- a/ansible/playbooks/indri.yml +++ b/ansible/playbooks/indri.yml @@ -108,11 +108,22 @@ check_mode: false tags: [forgejo_actions_secrets] + - name: Fetch Zot CI API key for Forgejo Actions + ansible.builtin.command: + cmd: op read "op://vg6xf6vvfmoh5hqjjhlhbeoaie/w3663ffnvkewbftncqxtcpeavy/zot-ci-api" + delegate_to: localhost + register: _zot_ci_api_key + changed_when: false + no_log: true + check_mode: false + tags: [forgejo_actions_secrets] + - name: Set Forgejo Actions secrets facts ansible.builtin.set_fact: forgejo_api_token: "{{ _forgejo_api_token.stdout }}" forgejo_secret_argocd_token: "{{ _forgejo_argocd_token.stdout }}" forgejo_secret_fly_deploy_token: "{{ _fly_deploy_token.stdout }}" + forgejo_secret_zot_ci_api_key: "{{ _zot_ci_api_key.stdout }}" no_log: true tags: [forgejo_actions_secrets] diff --git a/ansible/roles/forgejo_actions_secrets/defaults/main.yml b/ansible/roles/forgejo_actions_secrets/defaults/main.yml index 3aebfc8..943b8af 100644 --- a/ansible/roles/forgejo_actions_secrets/defaults/main.yml +++ b/ansible/roles/forgejo_actions_secrets/defaults/main.yml @@ -16,6 +16,8 @@ forgejo_actions_secrets_repos: value_var: forgejo_secret_argocd_token - name: FLY_DEPLOY_TOKEN value_var: forgejo_secret_fly_deploy_token + - name: ZOT_CI_API_KEY + value_var: forgejo_secret_zot_ci_api_key - repo: cv secrets: - name: FORGE_TOKEN diff --git a/ansible/roles/zot/defaults/main.yml b/ansible/roles/zot/defaults/main.yml index 812ac51..a53acfa 100644 --- a/ansible/roles/zot/defaults/main.yml +++ b/ansible/roles/zot/defaults/main.yml @@ -5,6 +5,8 @@ zot_data_dir: /Users/erichblume/zot zot_config_dir: /Users/erichblume/.config/zot zot_port: 5050 zot_log_dir: /Users/erichblume/Library/Logs +zot_external_url: https://registry.ops.eblu.me +zot_oidc_issuer: https://authentik.ops.eblu.me/application/o/zot/ # Pull-through cache registries (on-demand sync) zot_sync_registries: diff --git a/ansible/roles/zot/templates/config.json.j2 b/ansible/roles/zot/templates/config.json.j2 index 3c5c668..1090554 100644 --- a/ansible/roles/zot/templates/config.json.j2 +++ b/ansible/roles/zot/templates/config.json.j2 @@ -8,7 +8,41 @@ }, "http": { "address": "0.0.0.0", - "port": "{{ zot_port }}" + "port": "{{ zot_port }}", + "externalUrl": "{{ zot_external_url }}", + "auth": { + "openid": { + "providers": { + "oidc": { + "credentialsFile": "{{ zot_config_dir }}/oidc-credentials.json", + "issuer": "{{ zot_oidc_issuer }}", + "scopes": ["openid", "email", "profile"], + "claimMapping": { + "username": "preferred_username" + } + } + } + }, + "apikey": true + }, + "accessControl": { + "repositories": { + "**": { + "policies": [ + { + "groups": ["artifact-workloads"], + "actions": ["read", "create"] + }, + { + "groups": ["admins"], + "actions": ["read", "create", "update", "delete"] + } + ], + "anonymousPolicy": ["read"], + "defaultPolicy": ["read"] + } + } + } }, "log": { "level": "info" diff --git a/argocd/manifests/authentik/configmap-blueprint.yaml b/argocd/manifests/authentik/configmap-blueprint.yaml index 7881614..5501811 100644 --- a/argocd/manifests/authentik/configmap-blueprint.yaml +++ b/argocd/manifests/authentik/configmap-blueprint.yaml @@ -218,7 +218,7 @@ data: meta_launch_url: https://registry.ops.eblu.me policy_engine_mode: any - # Policy binding — restrict Zot to admins group + # Policy binding — allow admins group access to Zot - model: authentik_policies.policybinding identifiers: order: 0 @@ -231,3 +231,17 @@ data: enabled: true negate: false timeout: 30 + + # Policy binding — allow artifact-workloads group access to Zot (CI push) + - model: authentik_policies.policybinding + identifiers: + order: 1 + target: !KeyOf zot-app + group: !KeyOf artifact-workloads-group + attrs: + target: !KeyOf zot-app + group: !KeyOf artifact-workloads-group + order: 1 + enabled: true + negate: false + timeout: 30 diff --git a/docs/changelog.d/wire-ci-registry-auth.feature.md b/docs/changelog.d/wire-ci-registry-auth.feature.md new file mode 100644 index 0000000..d427fbd --- /dev/null +++ b/docs/changelog.d/wire-ci-registry-auth.feature.md @@ -0,0 +1 @@ +Enable OIDC + API key authentication on zot registry with three-tier access control (anonymous read, CI create, admin full). Wire both CI push paths (Dagger and Nix/skopeo) with registry credentials via Forgejo Actions secrets. diff --git a/docs/how-to/zot/harden-zot-registry.md b/docs/how-to/zot/harden-zot-registry.md index 8c1f08a..4c8d7e0 100644 --- a/docs/how-to/zot/harden-zot-registry.md +++ b/docs/how-to/zot/harden-zot-registry.md @@ -1,12 +1,6 @@ --- title: Harden Zot Registry modified: 2026-02-21 -status: active -requires: - - register-zot-oidc-client - - wire-ci-registry-auth - - enforce-tag-immutability - - adopt-commit-based-container-tags tags: - how-to - zot @@ -16,44 +10,40 @@ tags: # Harden Zot Registry -Enable OIDC + API key authentication on zot with anonymous pull preserved, and enforce tag immutability for version tags. This is the C2 Mikado root goal. +OIDC + API key authentication on zot with anonymous pull preserved, and tag immutability enforced server-side via accessControl. This was a C2 Mikado goal — all prerequisites are now complete. -## Context +## What Was Done -Zot currently has **no authentication** — security relies entirely on the Tailscale ACL boundary. Any tailnet client can push images, and tags are mutable. +Updated `ansible/roles/zot/templates/config.json.j2` with: -Both prerequisites from the original plan are now complete: -- [[adopt-oidc-provider]] — Authentik is deployed and serving OIDC -- [[adopt-dagger-ci]] — Dagger handles container builds - -## Core Change - -Update `ansible/roles/zot/templates/config.json.j2` to add: - -1. **`http.auth.openid`** — OIDC provider pointing to Authentik -2. **`http.auth.apikey: true`** — enable API key generation for CI -3. **`accessControl`** — three-tier policy enforcing tag immutability: +1. **`http.auth.openid`** — OIDC provider pointing to Authentik (`sso.ops.eblu.me`) +2. **`http.auth.apikey: true`** — API key generation for CI service accounts +3. **`http.accessControl`** — three-tier policy: - `anonymousPolicy: ["read"]` — anyone can pull - `artifact-workloads` group: `["read", "create"]` — CI can push new tags but cannot overwrite or delete (immutable tags) - - admins group: `["read", "create", "delete"]` — break-glass for removing bad images -4. **`externalUrl`** — `https://registry.ops.eblu.me` for OIDC callback redirects + - `admins` group: `["read", "create", "update", "delete"]` — break-glass +4. **`http.externalUrl`** — `https://registry.ops.eblu.me` for OIDC callback redirects -The `artifact-workloads` group must be created in Authentik (see [[register-zot-oidc-client]]) and a service account added to it for CI use. +CI authenticates via a zot API key generated from the `zot-ci` service account's OIDC session. The key is stored in 1Password and synced to Forgejo Actions secrets. ## Key Files | File | Purpose | |------|---------| -| `ansible/roles/zot/templates/config.json.j2` | Zot config — add auth + access control | -| `ansible/roles/zot/defaults/main.yml` | New OIDC variables | -| `ansible/roles/zot/tasks/main.yml` | Deploy OIDC credentials file | +| `ansible/roles/zot/templates/config.json.j2` | Zot config with auth + access control | +| `ansible/roles/zot/defaults/main.yml` | OIDC issuer and external URL variables | +| `ansible/roles/zot/templates/oidc-credentials.json.j2` | OIDC client credentials | +| `.dagger/src/blumeops_ci/main.py` | `publish()` with registry auth | +| `.forgejo/workflows/build-container.yaml` | Dagger push with API key | +| `.forgejo/workflows/build-container-nix.yaml` | Skopeo push with API key | ## Verification - [ ] Anonymous pull works (`curl -sf https://registry.ops.eblu.me/v2/_catalog`) - [ ] Unauthenticated push fails (401) - [ ] OIDC browser login works (redirect to Authentik and back) -- [ ] API key push works (`docker login` with `zak_...` token) +- [ ] API key push works (`docker login` with zot API key) +- [ ] CI push succeeds (Dagger and Nix/skopeo paths) - [ ] Pushing an existing version tag as CI user fails (no update permission) - [ ] Admin can delete a tag if needed - [ ] Pull-through caching still works @@ -61,8 +51,8 @@ The `artifact-workloads` group must be created in Authentik (see [[register-zot- ## Related -- [[register-zot-oidc-client]] — Prereq: register OIDC client in Authentik -- [[wire-ci-registry-auth]] — Prereq: update CI push paths with credentials +- [[register-zot-oidc-client]] — OIDC client registration in Authentik +- [[wire-ci-registry-auth]] — CI push path wiring - [[enforce-tag-immutability]] — Folded into this card (server-side via accessControl) -- [[adopt-commit-based-container-tags]] — Prereq: commit-SHA-based image tags +- [[adopt-commit-based-container-tags]] — Commit-SHA-based image tags - [[agent-change-process]] — C2 methodology diff --git a/docs/how-to/zot/wire-ci-registry-auth.md b/docs/how-to/zot/wire-ci-registry-auth.md index 20d9827..cce0655 100644 --- a/docs/how-to/zot/wire-ci-registry-auth.md +++ b/docs/how-to/zot/wire-ci-registry-auth.md @@ -1,7 +1,6 @@ --- title: Wire CI Registry Auth modified: 2026-02-21 -status: active tags: - how-to - zot @@ -11,48 +10,39 @@ tags: # Wire CI Registry Auth -Ensure both CI push paths authenticate to zot after auth is enabled. +How CI pipelines authenticate to the zot registry after OIDC + apikey auth is enabled. -## Context +## Overview -There are two push paths to update: +The `zot-ci` service account (created in [[register-zot-oidc-client]]) belongs to the `artifact-workloads` group, granting `["read", "create"]` permissions — CI can push new tags but cannot overwrite or delete existing ones. -1. **Dagger path** (`.forgejo/workflows/build-container.yaml` → `.dagger/src/blumeops_ci/main.py`): Add `with_registry_auth()` to the Dagger `publish()` call, sourcing the API key from env var `ZOT_CI_API_KEY`. +Authentication uses a zot API key generated after the service account's first OIDC login. The key is stored in 1Password (`Forgejo Secrets` item, field `zot-ci-api`, in blumeops vault) and synced to Forgejo Actions secrets via the `forgejo_actions_secrets` ansible role. The key expires every 90 days — see [[zot#API Key Rotation]] for the rotation procedure. -2. **Nix/skopeo path** (`.forgejo/workflows/build-container-nix.yaml`): Add `--dest-creds` to `skopeo copy`, sourcing the API key from the same env var. +## Push Paths -> **Note:** The API key must be generated manually after OIDC login is working — log in to zot UI via browser, generate an API key, and store it in 1Password. This is a manual step between [[register-zot-oidc-client]] and this card, but not modeled as a formal `requires` dependency. +### Dagger path (Dockerfile containers) -CI authenticates as a service account in the `artifact-workloads` group (created in [[register-zot-oidc-client]]). This group grants `["read", "create"]` — CI can push new tags but cannot overwrite or delete existing ones, enforcing tag immutability server-side. +`.forgejo/workflows/build-container.yaml` passes `--registry-password=env:ZOT_CI_API_KEY` to the Dagger `publish()` function, which calls `with_registry_auth()` before pushing. + +### Nix/skopeo path (Nix containers) + +`.forgejo/workflows/build-container-nix.yaml` passes `--dest-creds=zot-ci:$ZOT_CI_API_KEY` to `skopeo copy`. ## Secret Flow -### Indri runner (minikube) - -1Password item (new: `zot-ci-apikey`) → ExternalSecret in `forgejo-runner` namespace → env var `ZOT_CI_API_KEY` in runner pod - -### Ringtail runner (k3s) - -1Password → `/etc/forgejo-runner/zot-api-key.env` (or similar) deployed by NixOS config +1Password `Forgejo Secrets` item (field `zot-ci-api`) → ansible pre_task fetches it → `forgejo_actions_secrets` role syncs to Forgejo API → both runners (k8s on indri, host on ringtail) access it as `${{ secrets.ZOT_CI_API_KEY }}`. ## Key Files | File | Purpose | |------|---------| -| `.dagger/src/blumeops_ci/main.py` | Add `with_registry_auth()` to publish | -| `.forgejo/workflows/build-container.yaml` | Pass `ZOT_CI_API_KEY` to Dagger | -| `.forgejo/workflows/build-container-nix.yaml` | Add `--dest-creds` to skopeo | -| `argocd/manifests/forgejo-runner/deployment.yaml` | Mount secret as env var | -| `argocd/manifests/forgejo-runner/external-secret.yaml` | Pull API key from 1Password | -| `nixos/ringtail/configuration.nix` | Ringtail runner secret provisioning | - -## Verification - -- [ ] Dagger push succeeds with registry auth -- [ ] Nix/skopeo push succeeds with registry auth -- [ ] Push without credentials fails (401) +| `.dagger/src/blumeops_ci/main.py` | `publish()` accepts optional `registry_password` | +| `.forgejo/workflows/build-container.yaml` | Passes API key to Dagger | +| `.forgejo/workflows/build-container-nix.yaml` | Passes API key to skopeo | +| `ansible/playbooks/indri.yml` | Pre_task fetches API key from 1Password | +| `ansible/roles/forgejo_actions_secrets/defaults/main.yml` | Secret entry for `ZOT_CI_API_KEY` | ## Related - [[harden-zot-registry]] — Parent goal -- [[register-zot-oidc-client]] — OIDC client registration (do first) +- [[register-zot-oidc-client]] — OIDC client registration diff --git a/docs/reference/services/zot.md b/docs/reference/services/zot.md index 672440d..8cfc3f5 100644 --- a/docs/reference/services/zot.md +++ b/docs/reference/services/zot.md @@ -35,9 +35,33 @@ When [[cluster|minikube]] pulls an image, containerd checks zot first. If cached ## Security Model -Network access only (no authentication). Defense is the Tailscale ACL boundary. +OIDC authentication via [[authentik]], with API key support for CI. Three-tier access control: + +| Role | Permissions | Use case | +|------|------------|----------| +| Anonymous | read | Pull images without auth | +| `artifact-workloads` group | read, create | CI push (new tags only, no overwrite/delete) | +| `admins` group | read, create, update, delete | Break-glass admin access | + +CI authenticates with a zot API key generated from the `zot-ci` service account's OIDC session. The key is stored in the `Forgejo Secrets` 1Password item (field `zot-ci-api`) and synced to Forgejo Actions secrets via ansible. + +## API Key Rotation + +The `zot-ci` API key expires every **90 days**. To rotate: + +1. In Authentik admin UI, impersonate the `zot-ci` user +2. Visit `https://registry.ops.eblu.me` — you'll land on the login page +3. Click "SIGN IN WITH OIDC" to authenticate as zot-ci +4. Navigate to `https://registry.ops.eblu.me/user/apikey` +5. Generate a new API key, copy it to clipboard +6. Update 1Password: + ```fish + pbpaste | op item edit "Forgejo Secrets" --vault blumeops "zot-ci-api[password]=-" + ``` +7. Sync to Forgejo: `mise run provision-indri -- --tags forgejo_actions_secrets` ## Related - [[forgejo]] - Container build CI - [[cluster|Cluster]] - Registry consumer +- [[authentik]] - OIDC identity provider