eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Create C2 Mikado cards for harden-zot-registry (#229)

Browse source 
## Summary
- Replace the old pre-Mikado plan doc (`docs/how-to/plans/harden-zot-registry.md`) with a proper C2 Mikado chain in `docs/how-to/zot/`
- Root goal: `harden-zot-registry` — enable OIDC + API key auth on zot with anonymous pull preserved
- Three leaf prereqs: `register-zot-oidc-client`, `wire-ci-registry-auth`, `enforce-tag-immutability`
- Add Zot section to `how-to.md` index, remove plan entry from plans index
- All doc checks pass (`docs-check-links`, `docs-check-index`, `docs-mikado`)

## Changes
- **New:** `docs/how-to/zot/harden-zot-registry.md` — C2 Mikado root goal
- **New:** `docs/how-to/zot/register-zot-oidc-client.md` — Register OIDC client in Authentik
- **New:** `docs/how-to/zot/wire-ci-registry-auth.md` — Wire CI push paths with registry auth
- **New:** `docs/how-to/zot/enforce-tag-immutability.md` — Prevent version tag overwrites
- **Deleted:** `docs/how-to/plans/harden-zot-registry.md` — Old plan doc (content absorbed into Mikado cards)
- **Updated:** `docs/how-to/how-to.md` — Add Zot section, remove plan entry
- **Updated:** `docs/how-to/plans/plans.md` — Remove plan entry

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/229
This commit is contained in:
Erich Blume 2026-02-20 17:56:25 -08:00
commit 379bcb98af
8 changed files with 220 additions and 213 deletions
Download patch file Download diff file Expand all files Collapse all files

1
docs/changelog.d/harden-zot-mikado-cards.ai.md Normal file
View file

@ -0,0 +1 @@
Create C2 Mikado cards for harden-zot-registry: root goal and three prerequisite cards (register-zot-oidc-client, wire-ci-registry-auth, enforce-tag-immutability).

10
docs/how-to/how-to.md
View file

@ -60,11 +60,19 @@ Migration and transition plans for upcoming infrastructure changes.
| [[adopt-dagger-ci]] | Adopt Dagger as CI/CD build engine |
| [[upstream-fork-strategy]] | Stacked-branch forking strategy for upstream projects |
| [[adopt-oidc-provider]] | Deploy OIDC identity provider for SSO across services |
| [[harden-zot-registry]] | Add authentication and tag immutability to zot registry |
| [[forgejo-actions-dashboard]] | Grafana dashboard for Forgejo Actions CI metrics |
| [[upgrade-grafana-helm-chart]] | Upgrade Grafana Helm chart from 8.8.2 to 11.x |
| [[operationalize-reolink-camera]] | Cloud-free NVR with Frigate and ring buffer recording |
## Zot
Mikado chain for hardening the zot registry. Track progress with `mise run docs-mikado harden-zot-registry`.
- [[harden-zot-registry]]
- [[register-zot-oidc-client]]
- [[wire-ci-registry-auth]]
- [[enforce-tag-immutability]]
## Authentik
Mikado chain for deploying Authentik. Track progress with `mise run docs-mikado deploy-authentik`.

211
docs/how-to/plans/harden-zot-registry.md
View file

@ -1,211 +0,0 @@
---
title: "Plan: Harden Zot Registry"
modified: 2026-02-11
tags:
- how-to
- plans
- zot
- registry
- security
---
# Plan: Harden Zot Registry
> **Status:** Planned (not yet executed)
> **Sequence:** Execute after [[adopt-dagger-ci]] and [[adopt-oidc-provider]] — the Dagger migration will change how images are built and pushed, and the OIDC provider supplies the identity layer that zot's auth and API key features depend on.
## Background
Zot is the BlumeOps OCI container registry, running natively on [[indri]]. It serves two roles: a pull-through cache for upstream registries (Docker Hub, GHCR, Quay) and the private image store for `blumeops/*` images.
Currently, zot has **no authentication** — the security boundary is the Tailscale ACL. This was an acceptable starting point, but has two gaps:
1. **Any tailnet client can push images** — there's no distinction between pull (which k8s pods need) and push (which only CI should do). A compromised service or misconfigured pod could overwrite production images.
2. **Tags are mutable** — pushing the same tag twice silently overwrites the previous image. There's no protection against accidental or malicious tag clobbering.
### Goals
- **Authenticated push** — only CI (Forgejo Actions / Dagger) can push images; all other clients are pull-only
- **Tag immutability** — once a version tag is pushed, it cannot be overwritten
- **No disruption to pulls** — k8s pods and pull-through caching continue to work without authentication
- **Minimal complexity** — use zot's built-in OIDC and API key features with the BlumeOps identity provider
## Current State
### Push Mechanism
Images are currently pushed via the composite action at `.forgejo/actions/build-push-image/action.yaml`:
1. `docker buildx build` creates the image
2. `docker save` exports to a tarball
3. `skopeo copy` pushes to `registry.ops.eblu.me` (no credentials needed)
The action pushes two tags per build: a version tag (e.g., `v1.2.0`) and the git commit SHA.
### Zot Configuration
The config template (`ansible/roles/zot/templates/config.json.j2`) has no `accessControl` or `http.auth` section. The HTTP listener binds to `0.0.0.0:5050` with no TLS (Caddy terminates TLS at `registry.ops.eblu.me`).
## Plan
### 1. Add Authentication for Push (OIDC + API Keys)
Zot supports native OIDC authentication with a built-in API key feature designed for exactly this use case. The approach:
1. **OIDC for browser login** — zot delegates authentication to the BlumeOps OIDC provider (see [[adopt-oidc-provider]]). Human users log in via browser redirect.
2. **API keys for CI** — after logging in via OIDC, generate a scoped API key for Forgejo CI / Dagger. API keys are zot-native tokens (`zak_...`) that work with `docker login`, `skopeo`, and Dagger's `with_registry_auth()`. They can be scoped to specific repositories and given expiration dates.
3. **Access control** — `anonymousPolicy` allows unauthenticated pull; push requires authentication.
```json
{
"http": {
"auth": {
"openid": {
"providers": {
"oidc": {
"name": "BlumeOps",
"credentialsFile": "/Users/erichblume/.config/zot/oidc-credentials.json",
"issuer": "https://authentik.ops.eblu.me",
"scopes": ["openid", "profile", "email"]
}
}
},
"apikey": true
},
"accessControl": {
"repositories": {
"**": {
"anonymousPolicy": ["read"],
"defaultPolicy": ["read", "create", "update"],
"policies": [
{
"users": ["eblume"],
"actions": ["read", "create", "update", "delete"]
}
]
}
},
"adminPolicy": {
"users": ["eblume"],
"actions": ["read", "create", "update", "delete"]
}
}
}
}
```
The OIDC credentials file (client ID and secret) is deployed by Ansible from 1Password — never committed to the repo.
**CI push flow after setup:**
1. Log in to zot UI via browser (OIDC redirect to Authentik)
2. Generate an API key: `POST /zot/auth/apikey` with label `forgejo-ci`, scoped to `blumeops/**`
3. Store the key in 1Password (`op://blumeops/zot-ci-apikey/credential`)
4. CI uses the key: `docker login -u eblume -p zak_... registry.ops.eblu.me`
This ensures:
- k8s pods, minikube containerd, and pull-through caching all continue to work anonymously (read-only)
- Push requires a valid API key tied to an OIDC identity
- No standalone password files (htpasswd) to manage — identity flows from the central IdP
### 2. Enforce Tag Immutability
Zot does not have a built-in tag immutability feature at the registry level. Options to consider during execution:
- **Registry-side:** Check if newer zot versions (post-2.1) have added immutability policies. If so, configure in `config.json`.
- **Push-side enforcement:** The simpler approach — check whether a tag already exists before pushing. The current build-push-image action (and its eventual Dagger replacement) should query the registry API (`GET /v2/<name>/tags/list`) and **fail the build** if the version tag already exists. Commit SHA tags are inherently unique and don't need this check.
The push-side approach is pragmatic: it prevents accidental overwrites in the normal CI flow. Combined with authenticated push, a tag can only be overwritten by someone with CI credentials who deliberately bypasses the check.
> **See:** `.forgejo/actions/build-push-image/action.yaml` — this is where the pre-push tag check would be added in the current workflow. After [[adopt-dagger-ci]], the equivalent check goes in the Dagger `Container.publish()` wrapper.
### 3. Update Ansible Role
The `ansible/roles/zot/` role needs:
- **New template:** `oidc-credentials.json.j2` (client ID and secret for the Authentik OIDC client)
- **Updated config template:** `config.json.j2` gains `http.auth` (openid + apikey) and `accessControl` sections
- **Updated config template:** `config.json.j2` gains `externalUrl` set to `https://registry.ops.eblu.me` (required for OIDC callback redirects behind Caddy)
- **New variables:** `zot_oidc_client_id` and `zot_oidc_client_secret` sourced from 1Password in the playbook's `pre_tasks`
- **Handler:** restart zot LaunchAgent after config changes (already exists)
### 4. Update CI Push Credentials
After [[adopt-dagger-ci]], the Dagger module will use the zot API key for registry auth:
```python
api_key = dag.set_secret("registry-api-key",
os.environ["ZOT_CI_API_KEY"])
container.with_registry_auth("registry.ops.eblu.me", "eblume", api_key)
container.publish("registry.ops.eblu.me/blumeops/image:tag")
```
### 5. Update Minikube Containerd Config
The minikube containerd config (`ansible/roles/minikube/tasks/main.yml`) currently talks to zot without credentials. Since anonymous pull remains allowed, **no changes are needed** for containerd.
## Execution Steps
1. **Prerequisite: OIDC provider is running** (see [[adopt-oidc-provider]])
- Authentik (or chosen provider) is deployed and serving `https://authentik.ops.eblu.me`
- A zot OIDC client is registered with the provider
2. **Update Ansible role**
- Add OIDC credentials template
- Update `config.json.j2` with auth (openid + apikey) and access control
- Store OIDC client credentials in 1Password
- Test with `mise run provision-indri -- --tags zot --check --diff`
3. **Deploy and verify pulls still work**
- `mise run provision-indri -- --tags zot`
- Verify anonymous pull: `curl -sf https://registry.ops.eblu.me/v2/_catalog`
- Verify unauthenticated push fails: `skopeo copy ... docker://registry.ops.eblu.me/blumeops/test:fail` (should get 401)
4. **Set up OIDC login and generate CI API key**
- Log in to zot UI via browser (OIDC flow through Authentik)
- Generate an API key for CI use, store in 1Password
- Verify authenticated push works: `docker login -u eblume -p zak_... registry.ops.eblu.me`
5. **Add tag immutability check to push workflow**
- Add pre-push tag existence check to Dagger module (or build-push-image action)
- Test by attempting to push an existing tag
6. **Update documentation**
- Update `docs/reference/services/zot.md` security model section
- Add changelog fragment
## Verification Checklist
- [ ] Anonymous pull works (k8s pods, containerd, curl)
- [ ] Pull-through caching still works (pull an uncached image from docker.io)
- [ ] Unauthenticated push is rejected (401)
- [ ] OIDC browser login works (redirect to Authentik and back)
- [ ] API key generation works from zot UI
- [ ] Authenticated push with API key succeeds
- [ ] Pushing a duplicate version tag fails (immutability check)
- [ ] Pushing a new commit SHA tag succeeds
- [ ] Grafana dashboard still shows zot metrics
- [ ] `mise run services-check` passes
## Open Questions
- **Immutability granularity:** Should immutability apply only to semver tags (`v*`) or also to commit SHA tags? SHA tags are unique by nature, so immutability is only meaningful for version tags.
- **API key rotation:** API keys can have expiration dates. Decide on a rotation policy — e.g., annual expiry with a reminder, or no expiry with manual rotation.
## Reference Pattern Files
| File | Purpose |
|------|---------|
| `ansible/roles/zot/templates/config.json.j2` | Current zot config (no auth) |
| `ansible/roles/zot/tasks/main.yml` | Zot deployment tasks |
| `ansible/roles/zot/defaults/main.yml` | Zot default variables |
| `.forgejo/actions/build-push-image/action.yaml` | Current image push workflow (skopeo) |
| `ansible/roles/minikube/tasks/main.yml` | Containerd registry mirror config |
| `docs/reference/services/zot.md` | Zot reference documentation |
## Related
- [[adopt-oidc-provider]] — OIDC identity provider (execute first)
- [[adopt-dagger-ci]] — CI/CD engine migration (execute first)
- [[zot]] — Zot reference card
- [[forgejo]] — CI platform that pushes images
- [[cluster]] — Registry consumer

1
docs/how-to/plans/plans.md
View file

@ -18,7 +18,6 @@ Plans differ from regular how-to guides in that they describe work that has been
| [[add-unifi-pulumi-stack]] | Abandoned | Add Pulumi IaC for UniFi Express 7 (provider bugs — see doc) |
| [[upstream-fork-strategy]] | Planned | Stacked-branch forking strategy for tracking upstream projects |
| [[adopt-oidc-provider]] | Completed | Deploy OIDC identity provider for SSO across services |
| [[harden-zot-registry]] | Planned | Add authentication and tag immutability to zot registry |
| [[forgejo-actions-dashboard]] | Planned | Grafana dashboard and custom Prometheus exporter for Forgejo Actions CI metrics |
| [[upgrade-grafana-helm-chart]] | Planned | Upgrade Grafana Helm chart from 8.8.2 to 11.x (3 phases) |
| [[deploy-authentik]] | Active (C2) | Deploy Authentik IdP — Mikado chain tracked in `how-to/authentik/` |

44
docs/how-to/zot/enforce-tag-immutability.md Normal file
View file

@ -0,0 +1,44 @@
---
title: Enforce Tag Immutability
modified: 2026-02-20
status: active
tags:
- how-to
- zot
- ci
---
# Enforce Tag Immutability
Prevent accidental overwrite of version tags during CI push.
## Approach
Push-side enforcement: before pushing a version tag, query `GET /v2/blumeops/<name>/tags/list` and fail if the tag already exists. Commit SHA tags are inherently unique and skip this check.
## Two Push Paths to Update
1. **Dagger path:** Add tag-existence check before `ctr.publish()` in `.dagger/src/blumeops_ci/main.py`
2. **Nix/skopeo path:** Add tag-existence check before `skopeo copy` in `.forgejo/workflows/build-container-nix.yaml`
## Key Files
| File | Purpose |
|------|---------|
| `.dagger/src/blumeops_ci/main.py` | Add pre-publish tag check |
| `.forgejo/workflows/build-container-nix.yaml` | Add pre-copy tag check |
## Notes
- After auth is enabled, the tag check API call may need auth too — or it can rely on anonymous read access from `anonymousPolicy: ["read"]`
- Only version tags (e.g., `v1.2.0`) need the check; commit SHA tags are unique by nature
## Verification
- [ ] Pushing a new version tag succeeds
- [ ] Pushing an existing version tag fails with a clear error
- [ ] Pushing a commit SHA tag always succeeds
## Related
- [[harden-zot-registry]] — Parent goal

59
docs/how-to/zot/harden-zot-registry.md Normal file
View file

@ -0,0 +1,59 @@
---
title: Harden Zot Registry
modified: 2026-02-20
status: active
requires:
- register-zot-oidc-client
- wire-ci-registry-auth
- enforce-tag-immutability
tags:
- how-to
- zot
- registry
- security
---
# Harden Zot Registry
Enable OIDC + API key authentication on zot with anonymous pull preserved, and enforce tag immutability for version tags. This is the C2 Mikado root goal.
## Context
Zot currently has **no authentication** — security relies entirely on the Tailscale ACL boundary. Any tailnet client can push images, and tags are mutable.
Both prerequisites from the original plan are now complete:
- [[adopt-oidc-provider]] — Authentik is deployed and serving OIDC
- [[adopt-dagger-ci]] — Dagger handles container builds
## Core Change
Update `ansible/roles/zot/templates/config.json.j2` to add:
1. **`http.auth.openid`** — OIDC provider pointing to Authentik
2. **`http.auth.apikey: true`** — enable API key generation for CI
3. **`accessControl`** — anonymous read, authenticated write
4. **`externalUrl`** — `https://registry.ops.eblu.me` for OIDC callback redirects
## Key Files
| File | Purpose |
|------|---------|
| `ansible/roles/zot/templates/config.json.j2` | Zot config — add auth + access control |
| `ansible/roles/zot/defaults/main.yml` | New OIDC variables |
| `ansible/roles/zot/tasks/main.yml` | Deploy OIDC credentials file |
## Verification
- [ ] Anonymous pull works (`curl -sf https://registry.ops.eblu.me/v2/_catalog`)
- [ ] Unauthenticated push fails (401)
- [ ] OIDC browser login works (redirect to Authentik and back)
- [ ] API key push works (`docker login` with `zak_...` token)
- [ ] Pull-through caching still works
- [ ] `mise run services-check` passes
## Related
- [[register-zot-oidc-client]] — Prereq: register OIDC client in Authentik
- [[wire-ci-registry-auth]] — Prereq: update CI push paths with credentials
- [[enforce-tag-immutability]] — Prereq: prevent version tag overwrites
- [[agent-change-process]] — C2 methodology

51
docs/how-to/zot/register-zot-oidc-client.md Normal file
View file

@ -0,0 +1,51 @@
---
title: Register Zot OIDC Client
modified: 2026-02-20
status: active
tags:
- how-to
- zot
- authentik
- oidc
---
# Register Zot OIDC Client
Register a zot OAuth2 provider and application in Authentik via blueprint, following the same pattern as Grafana and Forgejo.
## What to Do
1. **Add `zot.yaml` blueprint section** to `argocd/manifests/authentik/configmap-blueprint.yaml`:
- OAuth2Provider: `client_id: zot`, redirect URI `https://registry.ops.eblu.me/zot/auth/callback/oidc`
- Application linked to the provider
- PolicyBinding restricting access to the admins group
2. **Generate and store client secret** in 1Password item "Authentik (blumeops)" as field `zot-client-secret`
3. **Add `AUTHENTIK_ZOT_CLIENT_SECRET`** to Authentik worker's ExternalSecret at `argocd/manifests/authentik/external-secret.yaml`
4. **Blueprint references the secret** via `!Env AUTHENTIK_ZOT_CLIENT_SECRET`
5. **Create OIDC credentials file** for zot's Ansible role:
- New template `ansible/roles/zot/templates/oidc-credentials.json.j2` containing `client_id` and `client_secret`
- Source `client_secret` from 1Password via a new pre_task in `ansible/playbooks/indri.yml`
## Key Files
| File | Purpose |
|------|---------|
| `argocd/manifests/authentik/configmap-blueprint.yaml` | Add zot blueprint (provider + app + policy) |
| `argocd/manifests/authentik/external-secret.yaml` | Add `AUTHENTIK_ZOT_CLIENT_SECRET` env var |
| `ansible/roles/zot/templates/oidc-credentials.json.j2` | New: OIDC credentials for zot |
| `ansible/playbooks/indri.yml` | New pre_task for zot OIDC client secret |
## Verification
- [ ] Authentik admin UI shows zot application
- [ ] OIDC discovery endpoint includes zot client
- [ ] Blueprint status is `successful` (check via API, not just logs)
## Related
- [[harden-zot-registry]] — Parent goal
- [[deploy-authentik]] — Authentik deployment (completed)

56
docs/how-to/zot/wire-ci-registry-auth.md Normal file
View file

@ -0,0 +1,56 @@
---
title: Wire CI Registry Auth
modified: 2026-02-20
status: active
tags:
- how-to
- zot
- ci
- forgejo
---
# Wire CI Registry Auth
Ensure both CI push paths authenticate to zot after auth is enabled.
## Context
There are two push paths to update:
1. **Dagger path** (`.forgejo/workflows/build-container.yaml` → `.dagger/src/blumeops_ci/main.py`): Add `with_registry_auth()` to the Dagger `publish()` call, sourcing the API key from env var `ZOT_CI_API_KEY`.
2. **Nix/skopeo path** (`.forgejo/workflows/build-container-nix.yaml`): Add `--dest-creds` to `skopeo copy`, sourcing the API key from the same env var.
> **Note:** The API key must be generated manually after OIDC login is working — log in to zot UI via browser, generate an API key, and store it in 1Password. This is a manual step between [[register-zot-oidc-client]] and this card, but not modeled as a formal `requires` dependency.
## Secret Flow
### Indri runner (minikube)
1Password item (new: `zot-ci-apikey`) → ExternalSecret in `forgejo-runner` namespace → env var `ZOT_CI_API_KEY` in runner pod
### Ringtail runner (k3s)
1Password → `/etc/forgejo-runner/zot-api-key.env` (or similar) deployed by NixOS config
## Key Files
| File | Purpose |
|------|---------|
| `.dagger/src/blumeops_ci/main.py` | Add `with_registry_auth()` to publish |
| `.forgejo/workflows/build-container.yaml` | Pass `ZOT_CI_API_KEY` to Dagger |
| `.forgejo/workflows/build-container-nix.yaml` | Add `--dest-creds` to skopeo |
| `argocd/manifests/forgejo-runner/deployment.yaml` | Mount secret as env var |
| `argocd/manifests/forgejo-runner/external-secret.yaml` | Pull API key from 1Password |
| `nixos/ringtail/configuration.nix` | Ringtail runner secret provisioning |
## Verification
- [ ] Dagger push succeeds with registry auth
- [ ] Nix/skopeo push succeeds with registry auth
- [ ] Push without credentials fails (401)
## Related
- [[harden-zot-registry]] — Parent goal
- [[register-zot-oidc-client]] — OIDC client registration (do first)