Enable zot registry auth + wire CI credentials (#237)

## Summary

- Enable OIDC + API key authentication on zot registry with three-tier accessControl
  - `anonymousPolicy: ["read"]` — anyone can pull
  - `artifact-workloads` group: `["read", "create"]` — CI push, no overwrite/delete
  - `admins` group: `["read", "create", "update", "delete"]` — break-glass
- Wire both CI push paths (Dagger and Nix/skopeo) with `ZOT_CI_API_KEY` credentials
- Add `artifact-workloads` PolicyBinding in Authentik blueprint for zot app access
- Add `ZOT_CI_API_KEY` to Forgejo Actions secrets via existing ansible role

Completes the `wire-ci-registry-auth` and `harden-zot-registry` Mikado cards.

## Manual Deployment Steps (after merge)

1. Deploy Authentik blueprint: `argocd app sync authentik`
2. In Authentik admin UI: set a password for the `zot-ci` service account
3. Deploy zot config: `mise run provision-indri -- --tags zot`
4. Log in to `https://registry.ops.eblu.me` as `zot-ci` via OIDC → generate API key
5. Store API key in 1Password as `zot-ci-apikey` in blumeops vault
6. Sync Forgejo secrets: `mise run provision-indri -- --tags forgejo_actions_secrets`
7. Trigger a test container build to verify CI push
8. Verify anonymous pull: `curl -sf https://registry.ops.eblu.me/v2/_catalog`

## Uncertainties

- **Zot `accessControl` group matching with OIDC:** Groups from Authentik's `profile` scope claim should map to zot policy groups, but the exact claim-to-group matching needs runtime verification
- **`http.auth.apikey: true`:** This config key is documented but needs verification against the specific zot version built from source on indri
- **API key permissions:** Need to confirm zot API keys inherit the generating user's group for accessControl evaluation

## Test Plan

- [ ] `mise run provision-indri -- --check --diff --tags zot` shows expected config changes
- [ ] Anonymous pull works after deploy
- [ ] Unauthenticated push fails (401)
- [ ] OIDC browser login redirects to Authentik and back
- [ ] API key push works after key generation
- [ ] CI push succeeds with both Dagger and skopeo paths
- [ ] `mise run services-check` passes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/237
This commit is contained in:
Erich Blume 2026-02-21 12:20:29 -08:00
commit ff63679efb
12 changed files with 139 additions and 61 deletions

View file

@ -1,12 +1,6 @@
---
title: Harden Zot Registry
modified: 2026-02-21
status: active
requires:
- register-zot-oidc-client
- wire-ci-registry-auth
- enforce-tag-immutability
- adopt-commit-based-container-tags
tags:
- how-to
- zot
@ -16,44 +10,40 @@ tags:
# Harden Zot Registry
Enable OIDC + API key authentication on zot with anonymous pull preserved, and enforce tag immutability for version tags. This is the C2 Mikado root goal.
OIDC + API key authentication on zot with anonymous pull preserved, and tag immutability enforced server-side via accessControl. This was a C2 Mikado goal — all prerequisites are now complete.
## Context
## What Was Done
Zot currently has **no authentication** — security relies entirely on the Tailscale ACL boundary. Any tailnet client can push images, and tags are mutable.
Updated `ansible/roles/zot/templates/config.json.j2` with:
Both prerequisites from the original plan are now complete:
- [[adopt-oidc-provider]] — Authentik is deployed and serving OIDC
- [[adopt-dagger-ci]] — Dagger handles container builds
## Core Change
Update `ansible/roles/zot/templates/config.json.j2` to add:
1. **`http.auth.openid`** — OIDC provider pointing to Authentik
2. **`http.auth.apikey: true`** — enable API key generation for CI
3. **`accessControl`** — three-tier policy enforcing tag immutability:
1. **`http.auth.openid`** — OIDC provider pointing to Authentik (`sso.ops.eblu.me`)
2. **`http.auth.apikey: true`** — API key generation for CI service accounts
3. **`http.accessControl`** — three-tier policy:
- `anonymousPolicy: ["read"]` — anyone can pull
- `artifact-workloads` group: `["read", "create"]` — CI can push new tags but cannot overwrite or delete (immutable tags)
- admins group: `["read", "create", "delete"]` — break-glass for removing bad images
4. **`externalUrl`** — `https://registry.ops.eblu.me` for OIDC callback redirects
- `admins` group: `["read", "create", "update", "delete"]` — break-glass
4. **`http.externalUrl`** — `https://registry.ops.eblu.me` for OIDC callback redirects
The `artifact-workloads` group must be created in Authentik (see [[register-zot-oidc-client]]) and a service account added to it for CI use.
CI authenticates via a zot API key generated from the `zot-ci` service account's OIDC session. The key is stored in 1Password and synced to Forgejo Actions secrets.
## Key Files
| File | Purpose |
|------|---------|
| `ansible/roles/zot/templates/config.json.j2` | Zot config — add auth + access control |
| `ansible/roles/zot/defaults/main.yml` | New OIDC variables |
| `ansible/roles/zot/tasks/main.yml` | Deploy OIDC credentials file |
| `ansible/roles/zot/templates/config.json.j2` | Zot config with auth + access control |
| `ansible/roles/zot/defaults/main.yml` | OIDC issuer and external URL variables |
| `ansible/roles/zot/templates/oidc-credentials.json.j2` | OIDC client credentials |
| `.dagger/src/blumeops_ci/main.py` | `publish()` with registry auth |
| `.forgejo/workflows/build-container.yaml` | Dagger push with API key |
| `.forgejo/workflows/build-container-nix.yaml` | Skopeo push with API key |
## Verification
- [ ] Anonymous pull works (`curl -sf https://registry.ops.eblu.me/v2/_catalog`)
- [ ] Unauthenticated push fails (401)
- [ ] OIDC browser login works (redirect to Authentik and back)
- [ ] API key push works (`docker login` with `zak_...` token)
- [ ] API key push works (`docker login` with zot API key)
- [ ] CI push succeeds (Dagger and Nix/skopeo paths)
- [ ] Pushing an existing version tag as CI user fails (no update permission)
- [ ] Admin can delete a tag if needed
- [ ] Pull-through caching still works
@ -61,8 +51,8 @@ The `artifact-workloads` group must be created in Authentik (see [[register-zot-
## Related
- [[register-zot-oidc-client]] — Prereq: register OIDC client in Authentik
- [[wire-ci-registry-auth]] — Prereq: update CI push paths with credentials
- [[register-zot-oidc-client]] — OIDC client registration in Authentik
- [[wire-ci-registry-auth]] — CI push path wiring
- [[enforce-tag-immutability]] — Folded into this card (server-side via accessControl)
- [[adopt-commit-based-container-tags]] — Prereq: commit-SHA-based image tags
- [[adopt-commit-based-container-tags]] — Commit-SHA-based image tags
- [[agent-change-process]] — C2 methodology

View file

@ -1,7 +1,6 @@
---
title: Wire CI Registry Auth
modified: 2026-02-21
status: active
tags:
- how-to
- zot
@ -11,48 +10,39 @@ tags:
# Wire CI Registry Auth
Ensure both CI push paths authenticate to zot after auth is enabled.
How CI pipelines authenticate to the zot registry after OIDC + apikey auth is enabled.
## Context
## Overview
There are two push paths to update:
The `zot-ci` service account (created in [[register-zot-oidc-client]]) belongs to the `artifact-workloads` group, granting `["read", "create"]` permissions — CI can push new tags but cannot overwrite or delete existing ones.
1. **Dagger path** (`.forgejo/workflows/build-container.yaml``.dagger/src/blumeops_ci/main.py`): Add `with_registry_auth()` to the Dagger `publish()` call, sourcing the API key from env var `ZOT_CI_API_KEY`.
Authentication uses a zot API key generated after the service account's first OIDC login. The key is stored in 1Password (`Forgejo Secrets` item, field `zot-ci-api`, in blumeops vault) and synced to Forgejo Actions secrets via the `forgejo_actions_secrets` ansible role. The key expires every 90 days — see [[zot#API Key Rotation]] for the rotation procedure.
2. **Nix/skopeo path** (`.forgejo/workflows/build-container-nix.yaml`): Add `--dest-creds` to `skopeo copy`, sourcing the API key from the same env var.
## Push Paths
> **Note:** The API key must be generated manually after OIDC login is working — log in to zot UI via browser, generate an API key, and store it in 1Password. This is a manual step between [[register-zot-oidc-client]] and this card, but not modeled as a formal `requires` dependency.
### Dagger path (Dockerfile containers)
CI authenticates as a service account in the `artifact-workloads` group (created in [[register-zot-oidc-client]]). This group grants `["read", "create"]` — CI can push new tags but cannot overwrite or delete existing ones, enforcing tag immutability server-side.
`.forgejo/workflows/build-container.yaml` passes `--registry-password=env:ZOT_CI_API_KEY` to the Dagger `publish()` function, which calls `with_registry_auth()` before pushing.
### Nix/skopeo path (Nix containers)
`.forgejo/workflows/build-container-nix.yaml` passes `--dest-creds=zot-ci:$ZOT_CI_API_KEY` to `skopeo copy`.
## Secret Flow
### Indri runner (minikube)
1Password item (new: `zot-ci-apikey`) → ExternalSecret in `forgejo-runner` namespace → env var `ZOT_CI_API_KEY` in runner pod
### Ringtail runner (k3s)
1Password → `/etc/forgejo-runner/zot-api-key.env` (or similar) deployed by NixOS config
1Password `Forgejo Secrets` item (field `zot-ci-api`) → ansible pre_task fetches it → `forgejo_actions_secrets` role syncs to Forgejo API → both runners (k8s on indri, host on ringtail) access it as `${{ secrets.ZOT_CI_API_KEY }}`.
## Key Files
| File | Purpose |
|------|---------|
| `.dagger/src/blumeops_ci/main.py` | Add `with_registry_auth()` to publish |
| `.forgejo/workflows/build-container.yaml` | Pass `ZOT_CI_API_KEY` to Dagger |
| `.forgejo/workflows/build-container-nix.yaml` | Add `--dest-creds` to skopeo |
| `argocd/manifests/forgejo-runner/deployment.yaml` | Mount secret as env var |
| `argocd/manifests/forgejo-runner/external-secret.yaml` | Pull API key from 1Password |
| `nixos/ringtail/configuration.nix` | Ringtail runner secret provisioning |
## Verification
- [ ] Dagger push succeeds with registry auth
- [ ] Nix/skopeo push succeeds with registry auth
- [ ] Push without credentials fails (401)
| `.dagger/src/blumeops_ci/main.py` | `publish()` accepts optional `registry_password` |
| `.forgejo/workflows/build-container.yaml` | Passes API key to Dagger |
| `.forgejo/workflows/build-container-nix.yaml` | Passes API key to skopeo |
| `ansible/playbooks/indri.yml` | Pre_task fetches API key from 1Password |
| `ansible/roles/forgejo_actions_secrets/defaults/main.yml` | Secret entry for `ZOT_CI_API_KEY` |
## Related
- [[harden-zot-registry]] — Parent goal
- [[register-zot-oidc-client]] — OIDC client registration (do first)
- [[register-zot-oidc-client]] — OIDC client registration