Pin NixOS service versions via nixpkgs-services overlay (#321)

## Summary

- Add `nixpkgs-services` flake input pinned to a specific nixpkgs commit, with an overlay that pulls `forgejo-runner`, `snowflake`, and `k3s` from it instead of the rolling `nixpkgs`
- Dagger `flake-update` pipeline now excludes `nixpkgs-services` via `--exclude`
- Fix stale nix-container-builder version in service-versions.yaml (was 12.6.4, actually running 12.7.2)
- Add k3s and minikube to service-versions.yaml tracking
- Document the pinning approach in review-services how-to and ringtail reference

## Motivation

During service review, discovered that flake updates had silently upgraded forgejo-runner from 12.6.4 → 12.7.2 without updating service-versions.yaml. This "sneak-in upgrade" bypasses the service review process. The overlay ensures these three services only change versions deliberately.

## Test plan

- [ ] Verify `nix flake update` from `nixos/ringtail/` does not change `nixpkgs-services` lock entry
- [ ] Verify `mise run provision-ringtail` builds successfully with the overlay
- [ ] Confirm running service versions unchanged after deploy

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #321

.dagger/src/blumeops_ci/main.py

@@ -258,7 +258,11 @@ class BlumeopsCi:
     async def flake_update(
         self, src: dagger.Directory, flake_path: str = "nixos/ringtail"
     ) -> dagger.File:
-        """Update all flake inputs to latest and return updated flake.lock."""
+        """Update rolling flake inputs to latest and return updated flake.lock.
+
+        Skips nixpkgs-services, which is pinned to a specific commit and should
+        only be updated deliberately during service reviews.
+        """
         return await (
             dag.container()
             .from_(NIX_IMAGE)
@@ -271,6 +275,8 @@ class BlumeopsCi:
                     "nix-command flakes",
                     "flake",
                     "update",
+                    "--exclude",
+                    "nixpkgs-services",
                     "--accept-flake-config",
                 ]
             )

ansible/roles/caddy/defaults/main.yml

@@ -82,6 +82,7 @@ caddy_services:
   - name: authentik
     host: "authentik.{{ caddy_domain }}"
     backend: "https://authentik.tail8d86e.ts.net"
+    cache_policy: spa
   - name: ntfy
     host: "ntfy.{{ caddy_domain }}"
     backend: "https://ntfy.tail8d86e.ts.net"

ansible/roles/caddy/templates/Caddyfile.j2

@@ -31,6 +31,14 @@
 {% for service in caddy_services %}
 	@{{ service.name }} host {{ service.host }}
 	handle @{{ service.name }} {
+{% if service.cache_policy | default('') == 'spa' %}
+		# SPA cache policy: hashed static assets are immutable, HTML must revalidate.
+		# Prevents stale HTML from referencing chunk hashes that no longer exist.
+		@{{ service.name }}_static path /static/dist/*
+		header @{{ service.name }}_static Cache-Control "public, max-age=31536000, immutable"
+		@{{ service.name }}_html path /if/*
+		header @{{ service.name }}_html Cache-Control "no-cache"
+{% endif %}
 {% if service.backend.startswith('https://') %}
 		reverse_proxy {{ service.backend }} {
 			# Caddy v2.11+ rewrites Host to upstream for HTTPS backends.

docs/changelog.d/pin-nixos-service-versions.infra.md

@@ -0,0 +1 @@
+Pin NixOS service versions (forgejo-runner, snowflake, k3s) via `nixpkgs-services` overlay in ringtail flake, preventing silent upgrades from `nix flake update`. Add k3s and minikube to service-versions.yaml tracking. Fix stale nix-container-builder version (was 12.6.4, actually running 12.7.2).

docs/how-to/knowledgebase/review-services.md

@@ -57,9 +57,13 @@ For all service types, start by reading the service's reference card (`docs/refe
 
 ### NixOS Services (`type: nixos`)
 
+Versioned NixOS services (forgejo-runner, snowflake, k3s) are pinned via a `nixpkgs-services` overlay in `nixos/ringtail/flake.nix`. This prevents `nix flake update` from silently upgrading them — they only change when the `nixpkgs-services` input is deliberately updated.
+
 1. Check the upstream project for new releases
-2. Review the Nix derivation or flake input for version pins
-3. If upgrading, update and deploy via `mise run provision-ringtail`
+2. Check what version nixpkgs has: `ssh ringtail 'nix eval nixpkgs#<pkg>.version'`
+3. To upgrade, update the `nixpkgs-services` rev in `flake.nix` to a nixpkgs commit that includes the desired version, then run `nix flake update nixpkgs-services` from `nixos/ringtail/`
+4. Deploy via `mise run provision-ringtail`
+5. Update `service-versions.yaml` with the new version
 
 ### Private Forge Repos (`upstream-source` under `forge.eblu.me/eblume/`)
 

docs/reference/infrastructure/ringtail.md

@@ -108,6 +108,10 @@ A native Forgejo Actions runner (`ringtail-nix-builder`) runs as a systemd servi
 
 The runner resolves `<nixpkgs>` from the flake registry at build time. Container trust policy (`/etc/containers/policy.json`) and registry search order (`/etc/containers/registries.conf`) are configured minimally in `configuration.nix` for skopeo — no full `virtualisation.containers` module needed.
 
+## Pinned Service Versions
+
+Versioned services (forgejo-runner, snowflake, k3s) are pinned via a `nixpkgs-services` overlay in `flake.nix`, separate from the rolling `nixpkgs` input. This prevents `nix flake update` from silently upgrading them. The Dagger `flake-update` pipeline excludes `nixpkgs-services` automatically. See [[review-services]] for the upgrade procedure.
+
 ## Maintenance Notes
 
 **1Password:** Desktop app must be running for `op` CLI. Use `$mod+Shift+minus` to send to scratchpad.

nixos/ringtail/flake.lock

@@ -57,11 +57,28 @@
         "type": "github"
       }
     },
+    "nixpkgs-services": {
+      "locked": {
+        "lastModified": 1774388614,
+        "narHash": "sha256-tFwzTI0DdDzovdE9+Ras6CUss0yn8P9XV4Ja6RjA+nU=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "1073dad219cb244572b74da2b20c7fe39cb3fa9e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "1073dad219cb244572b74da2b20c7fe39cb3fa9e",
+        "type": "github"
+      }
+    },
     "root": {
       "inputs": {
         "disko": "disko",
         "home-manager": "home-manager",
-        "nixpkgs": "nixpkgs"
+        "nixpkgs": "nixpkgs",
+        "nixpkgs-services": "nixpkgs-services"
       }
     }
   },

nixos/ringtail/flake.nix

@@ -3,6 +3,12 @@
 
   inputs = {
     nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.11";
+
+    # Pinned nixpkgs for versioned services (forgejo-runner, snowflake, k3s).
+    # Update this deliberately during service reviews, not via `nix flake update`.
+    # Current versions: forgejo-runner 12.7.2, snowflake 2.11.0, k3s 1.34.5+k3s1
+    nixpkgs-services.url = "github:NixOS/nixpkgs/1073dad219cb244572b74da2b20c7fe39cb3fa9e";
+
     disko = {
       url = "github:nix-community/disko";
       inputs.nixpkgs.follows = "nixpkgs";
@@ -13,7 +19,7 @@
     };
   };
 
-  outputs = { nixpkgs, disko, home-manager, ... }: {
+  outputs = { nixpkgs, nixpkgs-services, disko, home-manager, ... }: {
     nixosConfigurations.ringtail = nixpkgs.lib.nixosSystem {
       system = "x86_64-linux";
       modules = [
@@ -22,6 +28,18 @@
         ./disk-config.nix
         ./hardware-configuration.nix
         ./configuration.nix
+        # Pin versioned services to nixpkgs-services instead of the rolling nixpkgs.
+        # This prevents `nix flake update nixpkgs` from silently upgrading them.
+        # Bump nixpkgs-services explicitly during service reviews.
+        ({ ... }: {
+          nixpkgs.overlays = [
+            (final: prev: let svcPkgs = nixpkgs-services.legacyPackages.x86_64-linux; in {
+              forgejo-runner = svcPkgs.forgejo-runner;
+              snowflake = svcPkgs.snowflake;
+              k3s = svcPkgs.k3s;
+            })
+          ];
+        })
       ];
     };
   };

service-versions.yaml

@@ -252,17 +252,39 @@ services:
 
   - name: nix-container-builder
     type: nixos
-    last-reviewed: 2026-02-22
-    current-version: "12.6.4"
+    last-reviewed: 2026-04-01
+    current-version: "12.7.2"
     upstream-source: https://code.forgejo.org/forgejo/runner/releases
-    notes: Forgejo runner on ringtail via nixpkgs; version tracks flake.lock
+    notes: >-
+      Forgejo runner on ringtail; pinned via nixpkgs-services overlay in flake.nix.
+      Update nixpkgs-services rev during service reviews, not via nix flake update.
 
   - name: snowflake-proxy
     type: nixos
-    last-reviewed: 2026-03-24
+    last-reviewed: 2026-04-01
     current-version: "2.11.0"
     upstream-source: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/releases
-    notes: Tor Snowflake proxy on ringtail; anti-censorship bridge, not an exit node
+    notes: >-
+      Tor Snowflake proxy on ringtail; pinned via nixpkgs-services overlay in flake.nix.
+      Anti-censorship bridge, not an exit node.
+
+  - name: k3s
+    type: nixos
+    last-reviewed: 2026-04-01
+    current-version: "1.34.5+k3s1"
+    upstream-source: https://github.com/k3s-io/k3s/releases
+    notes: >-
+      Single-node k3s cluster on ringtail; pinned via nixpkgs-services overlay in flake.nix.
+      Update nixpkgs-services rev during service reviews.
+
+  - name: minikube
+    type: ansible
+    last-reviewed: 2026-04-01
+    current-version: "1.38.0"
+    upstream-source: https://github.com/kubernetes/minikube/releases
+    notes: >-
+      Single-node minikube on indri; installed via homebrew (not version-pinned).
+      Homebrew may silently upgrade on brew update/upgrade.
 
   - name: mealie
     type: argocd