diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 7b5fd1c..1797afc 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -89,6 +89,16 @@ repos: args: ['-config-file', '.github/actionlint.yaml'] files: ^\.forgejo/workflows/ + # Container version consistency + - repo: local + hooks: + - id: container-version-check + name: container-version-check + entry: mise run container-version-check + language: system + files: ^(containers/|service-versions\.yaml) + pass_filenames: false + # Documentation validation - repo: local hooks: diff --git a/docs/how-to/how-to.md b/docs/how-to/how-to.md index c8094fc..fc58ac6 100644 --- a/docs/how-to/how-to.md +++ b/docs/how-to/how-to.md @@ -76,6 +76,7 @@ Mikado chain for hardening the zot registry. Track progress with `mise run docs- - [[add-container-version-sync-check]] - [[pin-container-versions]] - [[add-dagger-nix-build]] +- [[fix-ntfy-nix-version]] ## Authentik diff --git a/docs/how-to/zot/add-container-version-sync-check.md b/docs/how-to/zot/add-container-version-sync-check.md index a657f7d..0ee478c 100644 --- a/docs/how-to/zot/add-container-version-sync-check.md +++ b/docs/how-to/zot/add-container-version-sync-check.md @@ -5,6 +5,7 @@ status: active requires: - pin-container-versions - add-dagger-nix-build + - fix-ntfy-nix-version tags: - how-to - containers @@ -20,25 +21,25 @@ Add a pre-commit check that validates version consistency across the three place Discovered during analysis of [[adopt-commit-based-container-tags]]: the new commit-SHA-based image tags need a reliable version source (`vX.Y.Z-`). Versions are currently scattered across Dockerfile ARGs (varying naming conventions), `service-versions.yaml` entries (many still `null`), and nix derivations (implicit from nixpkgs). A sync check ensures these stay consistent without adding a redundant fourth source. -## What to Do +## What Was Done -### 1. Create `mise run container-version-check` task +### 1. Created `mise run container-version-check` task -A uv-script mise task that validates version consistency across sources: +A typer-based uv-script that iterates over `containers/*/` and validates five rules per container: -1. **Dockerfile ARG** — parse `ARG CONTAINER_APP_VERSION=` (strip `v` prefix if present). Every container Dockerfile declares this as its canonical version. -2. **`service-versions.yaml`** — `current-version` field for the matching service name (strip `v` prefix). Must agree with the Dockerfile ARG. -3. **Nix derivation** — for nix-only containers (authentik), extract the version via `dagger call nix-version` (from [[add-dagger-nix-build]]). For dual containers (nettest, ntfy), the Dockerfile ARG is the primary check target; the nix version is informational. +1. Any Dockerfile must declare `ARG CONTAINER_APP_VERSION=` +2. Any `default.nix` must produce a version via `dagger call nix-version` +3. At least one build file must exist (Dockerfile or default.nix) +4. A matching `service-versions.yaml` entry must exist with non-null `current-version` +5. All resolved versions from (1), (2), and (4) must agree (v-prefix stripped for comparison) -The check also validates: -- Every `hybrid` service in `service-versions.yaml` has a non-null `current-version` -- Every container with a Dockerfile has a parseable `CONTAINER_APP_VERSION` ARG (after [[pin-container-versions]] is complete) +Scoping: by default only checks containers changed vs main. `--all-files` checks everything. If `service-versions.yaml` itself changed, all containers are checked. -Report mismatches as errors. Exit non-zero if any are found. +Blacklisted containers (utility images, not tracked services): `kubectl`, `nettest`. -### 2. Add pre-commit hook +Container-to-service name mapping: `quartz` → `docs`, `kiwix-serve` → `kiwix`. -Add a `container-version-check` entry to `.pre-commit-config.yaml` following the existing `docs-check-*` pattern: +### 2. Added pre-commit hook ```yaml - id: container-version-check @@ -49,36 +50,34 @@ Add a `container-version-check` entry to `.pre-commit-config.yaml` following the pass_filenames: false ``` -### 3. Populate `service-versions.yaml` +### 3. Populated `service-versions.yaml` -Fill in `current-version` for all hybrid services that currently have `null`. The sync check needs these values to validate against. +Filled in `current-version` for all hybrid services: navidrome (v0.60.3), miniflux (2.2.17), teslamate (v2.2.0), transmission (4.0.6-r4), kiwix (3.8.1), forgejo-runner (0.19.11). Added authentik (2025.10.1) as a new hybrid entry. -## Version extraction in CI +### ntfy nix version skew (fix forward) -The CI workflow (from [[adopt-commit-based-container-tags]]) extracts the version at build time — no VERSION file needed: - -- **Dockerfile builds**: `grep -oP 'ARG CONTAINER_APP_VERSION=\K\S+' containers/$CONTAINER/Dockerfile` -- **Nix builds**: `dagger call nix-version --src=. --package=` or `nix eval --raw nixpkgs#.version` +The check discovered that ntfy's Dockerfile pins v2.17.0 but nixpkgs has ntfy-sh 2.15.0. Rather than reverting, ntfy is excluded from `NIX_PACKAGE_MAP` and a new dependency card [[fix-ntfy-nix-version]] was created to build the nix derivation from the forge mirror at v2.17.0. ## Key Files | File | Change | |------|--------| -| `mise-tasks/container-version-check` | New: sync validation script | +| `mise-tasks/container-version-check` | New: typer CLI sync validation script | | `.pre-commit-config.yaml` | Add `container-version-check` hook | -| `service-versions.yaml` | Fill in `current-version` for hybrid services | +| `service-versions.yaml` | Populate `current-version` for all hybrid services + authentik | ## Verification -- [ ] `mise run container-version-check` passes with no errors +- [x] `mise run container-version-check --all-files` passes with no errors - [ ] Intentionally changing a Dockerfile ARG without updating `service-versions.yaml` fails the check -- [ ] Pre-commit hook catches mismatches on commit -- [ ] `service-versions.yaml` has `current-version` populated for all hybrid services -- [ ] Nix-only container versions are checked via Dagger (or gracefully skipped with a warning if Dagger unavailable) +- [x] `service-versions.yaml` has `current-version` populated for all hybrid services +- [x] Nix-only container versions (authentik) checked via Dagger +- [ ] ntfy nix version check deferred to [[fix-ntfy-nix-version]] ## Related - [[pin-container-versions]] — Prereq: containers need parseable version ARGs first - [[add-dagger-nix-build]] — Prereq: nix version extraction +- [[fix-ntfy-nix-version]] — Prereq: ntfy nix derivation version skew - [[adopt-commit-based-container-tags]] — Parent: CI uses the same version extraction at build time - [[harden-zot-registry]] — Root goal diff --git a/docs/how-to/zot/fix-ntfy-nix-version.md b/docs/how-to/zot/fix-ntfy-nix-version.md new file mode 100644 index 0000000..7f1afd1 --- /dev/null +++ b/docs/how-to/zot/fix-ntfy-nix-version.md @@ -0,0 +1,58 @@ +--- +title: Fix ntfy Nix Version +modified: 2026-02-20 +status: active +tags: + - how-to + - containers + - nix + - zot +--- + +# Fix ntfy Nix Version + +Override the nixpkgs ntfy-sh derivation to build v2.17.0 from the forge mirror, aligning the nix-built container with the Dockerfile version. + +## Context + +Discovered during [[add-container-version-sync-check]]: the ntfy container has both a Dockerfile and a `default.nix`. The Dockerfile builds v2.17.0 from `forge.ops.eblu.me/eblume/ntfy.git`, but the nix derivation uses `pkgs.ntfy-sh` from nixpkgs which is pinned at 2.15.0. The version sync check currently excludes ntfy from nix version validation as a workaround. + +## What to Do + +Override the nixpkgs `ntfy-sh` derivation in `containers/ntfy/default.nix` to build from the forge mirror at the v2.17.0 tag. The nixpkgs derivation uses `buildGoModule` with a nested `buildNpmPackage` for the web UI. + +### Hashes to Update + +| Hash | What it covers | When to update | +|------|---------------|----------------| +| `src.hash` | Source tarball integrity | Always (new source) | +| `vendorHash` | Go module dependencies | If `go.mod`/`go.sum` changed between 2.15.0 and 2.17.0 | +| `npmDepsHash` | npm dependencies | If `web/package-lock.json` changed | + +Use `lib.fakeHash` for each, attempt a build, and nix will report the expected hash. + +### Steps + +1. In `containers/ntfy/default.nix`, override `pkgs.ntfy-sh` with `fetchgit` pointing to `https://forge.ops.eblu.me/eblume/ntfy.git` at the v2.17.0 tag +2. Update all three hashes via iterative builds +3. Build and test with `dagger call build-nix --src=. --container-name=ntfy` +4. Re-enable ntfy in `NIX_PACKAGE_MAP` in `mise-tasks/container-version-check` +5. Verify `mise run container-version-check --all-files` passes + +## Key Files + +| File | Change | +|------|--------| +| `containers/ntfy/default.nix` | Override ntfy-sh derivation to build from forge | +| `mise-tasks/container-version-check` | Re-add ntfy to `NIX_PACKAGE_MAP` | + +## Verification + +- [ ] `dagger call build-nix --src=. --container-name=ntfy` produces a working image +- [ ] `dagger call nix-version --package=ntfy-sh` returns 2.17.0 (or the overridden version is extractable) +- [ ] `mise run container-version-check --all-files` passes with ntfy included + +## Related + +- [[add-container-version-sync-check]] — Parent: needs ntfy in NIX_PACKAGE_MAP +- [[harden-zot-registry]] — Root goal diff --git a/mise-tasks/container-version-check b/mise-tasks/container-version-check new file mode 100755 index 0000000..4d46469 --- /dev/null +++ b/mise-tasks/container-version-check @@ -0,0 +1,247 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.12" +# dependencies = ["pyyaml>=6.0", "rich>=13.0.0", "typer>=0.15.0"] +# /// +#MISE description="Validate container version consistency across Dockerfiles, nix derivations, and service-versions.yaml" +#USAGE flag "--all-files" help="Check all containers, not just changed ones" +"""Validate that container versions are consistent across all declaration sites. + +For each container directory under containers/, checks: +1. Any Dockerfile must declare ARG CONTAINER_APP_VERSION= +2. Any default.nix must produce a version (via dagger call nix-version) +3. At least one build file (Dockerfile or default.nix) must exist +4. A matching entry in service-versions.yaml must exist with non-null current-version +5. All resolved versions from (1), (2), and (4) must agree + +By default, only checks containers whose files differ from main. +Pass --all-files to check every container. + +Usage: + mise run container-version-check # changed containers only + mise run container-version-check --all-files # all containers +""" + +import re +import shutil +import subprocess +from pathlib import Path + +import typer +import yaml +from rich.console import Console +from rich.table import Table + +REPO_ROOT = Path(__file__).parent.parent +CONTAINERS_DIR = REPO_ROOT / "containers" +SERVICE_VERSIONS_FILE = REPO_ROOT / "service-versions.yaml" + +# Containers that are utility/test images, not tracked services +BLACKLIST = {"kubectl", "nettest"} + +# Container dir name → service-versions.yaml name (when they differ) +CONTAINER_TO_SERVICE = { + "quartz": "docs", + "kiwix-serve": "kiwix", +} + +# Container dir name → nixpkgs package name for dagger nix-version +# ntfy excluded until nix derivation is updated to build v2.17.0 from forge +# See: docs/how-to/zot/fix-ntfy-nix-version.md +NIX_PACKAGE_MAP = { + "authentik": "authentik", +} + +VERSION_ARG_PATTERN = re.compile(r"^ARG\s+CONTAINER_APP_VERSION=(\S+)", re.MULTILINE) + +app = typer.Typer() +console = Console() + + +def strip_v(version: str) -> str: + """Strip leading 'v' prefix for comparison.""" + return version.lstrip("v") + + +def changed_containers() -> set[str] | None: + """Return container names with changes vs main, or None on git failure.""" + result = subprocess.run( + ["git", "diff", "--name-only", "main...HEAD"], + capture_output=True, + text=True, + cwd=REPO_ROOT, + ) + if result.returncode != 0: + return None + + names: set[str] = set() + sv_changed = False + for line in result.stdout.splitlines(): + if line.startswith("containers/"): + parts = line.split("/") + if len(parts) >= 2: + names.add(parts[1]) + if line == "service-versions.yaml": + sv_changed = True + + # If service-versions.yaml changed, check all containers + if sv_changed: + return None + + return names + + +def get_nix_version(container_name: str) -> str | None: + """Extract nix package version via dagger. Returns None if unavailable.""" + pkg = NIX_PACKAGE_MAP.get(container_name) + if pkg is None: + return None + + if not shutil.which("dagger"): + return None + + result = subprocess.run( + ["dagger", "-m", ".dagger", "call", "nix-version", f"--package={pkg}"], + capture_output=True, + text=True, + cwd=REPO_ROOT, + ) + if result.returncode != 0: + return None + + return result.stdout.strip().splitlines()[-1].strip() + + +@app.command() +def main( + all_files: bool = typer.Option(False, "--all-files", help="Check all containers, not just changed ones"), +) -> None: + """Validate container version consistency.""" + # Determine which containers to check + if all_files: + scope = None # check all + else: + scope = changed_containers() # None means check all (fallback) + + # Load service versions + data = yaml.safe_load(SERVICE_VERSIONS_FILE.read_text()) + services = {svc["name"]: svc for svc in data.get("services", [])} + + errors: list[tuple[str, str]] = [] + results: list[dict] = [] + + for container_dir in sorted(CONTAINERS_DIR.iterdir()): + if not container_dir.is_dir(): + continue + + name = container_dir.name + if name in BLACKLIST: + continue + if scope is not None and name not in scope: + continue + + dockerfile = container_dir / "Dockerfile" + nix_file = container_dir / "default.nix" + has_dockerfile = dockerfile.exists() + has_nix = nix_file.exists() + + versions: dict[str, str] = {} + entry = { + "name": name, + "has_dockerfile": has_dockerfile, + "has_nix": has_nix, + "versions": versions, + } + results.append(entry) + + # Rule 3: at least one build file + if not has_dockerfile and not has_nix: + errors.append((name, "No Dockerfile or default.nix found")) + continue + + # Rule 1: Dockerfile must declare CONTAINER_APP_VERSION + if has_dockerfile: + match = VERSION_ARG_PATTERN.search(dockerfile.read_text()) + if match: + versions["dockerfile"] = match.group(1) + else: + errors.append((name, "Dockerfile missing ARG CONTAINER_APP_VERSION")) + + # Rule 2: nix derivation must produce a version + if has_nix: + nix_ver = get_nix_version(name) + if nix_ver is not None: + versions["nix"] = nix_ver + elif name in NIX_PACKAGE_MAP: + errors.append((name, "Failed to extract nix version via dagger")) + + # Rule 4: service-versions.yaml entry with non-null version + svc_name = CONTAINER_TO_SERVICE.get(name, name) + svc = services.get(svc_name) + if svc is None: + errors.append((name, f"No entry '{svc_name}' in service-versions.yaml")) + elif svc.get("current-version") is None: + errors.append((name, f"Null current-version for '{svc_name}' in service-versions.yaml")) + else: + versions["service-versions"] = str(svc["current-version"]) + + # Rule 5: all resolved versions must match + if len(versions) >= 2: + normalized = {src: strip_v(v) for src, v in versions.items()} + unique = set(normalized.values()) + if len(unique) > 1: + detail = ", ".join(f"{src}={v}" for src, v in sorted(versions.items())) + errors.append((name, f"Version mismatch: {detail}")) + + # Output + console.print("[bold]Container Version Sync Check[/bold]") + if scope is not None: + console.print(f"Scope: {len(scope)} container(s) changed vs main") + else: + console.print("Scope: all containers") + console.print() + + if results: + table = Table(show_header=True, header_style="bold") + table.add_column("Container") + table.add_column("Build") + table.add_column("Versions") + table.add_column("Status") + + for entry in results: + name = entry["name"] + build_parts = [] + if entry["has_dockerfile"]: + build_parts.append("dockerfile") + if entry["has_nix"]: + build_parts.append("nix") + + ver_parts = [f"{src}={v}" for src, v in sorted(entry["versions"].items())] + has_error = any(e[0] == name for e in errors) + status = "[red]FAIL[/red]" if has_error else "[green]OK[/green]" + + table.add_row( + name, + "+".join(build_parts), + ", ".join(ver_parts) or "—", + status, + ) + + console.print(table) + console.print() + + if errors: + console.print(f"[bold red]{len(errors)} error(s):[/bold red]") + for name, msg in errors: + console.print(f" {name}: {msg}") + console.print() + raise typer.Exit(code=1) + + if not results: + console.print("[dim]No containers to check.[/dim]") + else: + console.print("[bold green]All container versions are consistent![/bold green]") + + +if __name__ == "__main__": + app() diff --git a/service-versions.yaml b/service-versions.yaml index 21aed07..f74d39f 100644 --- a/service-versions.yaml +++ b/service-versions.yaml @@ -136,34 +136,40 @@ services: # --- Hybrid (custom container + ArgoCD) --- + - name: authentik + type: hybrid + last-reviewed: null + current-version: "2025.10.1" + upstream-source: https://github.com/goauthentik/authentik/releases + - name: navidrome type: hybrid last-reviewed: null - current-version: null + current-version: "v0.60.3" upstream-source: https://github.com/navidrome/navidrome/releases - name: miniflux type: hybrid last-reviewed: null - current-version: null + current-version: "2.2.17" upstream-source: https://github.com/miniflux/v2/releases - name: teslamate type: hybrid last-reviewed: null - current-version: null + current-version: "v2.2.0" upstream-source: https://github.com/teslamate-org/teslamate/releases - name: transmission type: hybrid last-reviewed: null - current-version: null + current-version: "4.0.6-r4" upstream-source: https://github.com/transmission/transmission/releases - name: kiwix type: hybrid last-reviewed: null - current-version: null + current-version: "3.8.1" upstream-source: https://github.com/kiwix/kiwix-tools/releases - name: devpi @@ -189,7 +195,7 @@ services: - name: forgejo-runner type: hybrid last-reviewed: null - current-version: null + current-version: "0.19.11" upstream-source: https://code.forgejo.org/forgejo/runner/releases # --- Ansible native ---