eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Upgrade k8s forgejo-runner from v6.3.1 to v12.x (#249)

Browse source 
## Summary
- C2 Mikado chain for upgrading the k8s forgejo-runner daemon (6 major versions behind)
- Root goal card with two leaf prerequisites: workflow validation and config review
- Ringtail runner is already at ~v12.6.4 via nixpkgs, no work needed there

## Mikado Chain

```
upgrade-k8s-runner (goal)
├── validate-workflows-against-v12 (leaf)
└── review-runner-config-v12 (leaf)
```

Both leaves are actionable now. The biggest risk is workflow schema validation
(introduced in v8/v9) rejecting our existing workflows.

## Next Steps
Work the leaf nodes in a follow-up session, then attempt the goal.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/249
This commit is contained in:
Erich Blume 2026-02-22 17:12:45 -08:00
commit 00b0287bcc
8 changed files with 244 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

1
docs/changelog.d/upgrade-forgejo-runner.doc.md Normal file
View file

@ -0,0 +1 @@
Add Mikado chain for upgrading k8s forgejo-runner from v6.3.1 to v12.x

1
docs/changelog.d/upgrade-forgejo-runner.infra.md Normal file
View file

@ -0,0 +1 @@
Simplify service-versions.yaml type taxonomy to `argocd | ansible | nixos`; add nix-container-builder entry; backfill forgejo and forgejo-runner versions

60
docs/how-to/forgejo-runner/review-runner-config-v12.md Normal file
View file

@ -0,0 +1,60 @@
---
title: Review Runner Config for v12
status: active
modified: 2026-02-22
tags:
- how-to
- forgejo-runner
- ci
---
# Review Runner Config for v12
Compare the current runner ConfigMap against the v12.7.0 default config to identify new, changed, or deprecated keys.
## Background
The runner config in `argocd/manifests/forgejo-runner/configmap.yaml` was written for v6.3.1. Six major versions may have introduced new config keys, changed defaults, or deprecated options.
## Current Config
```yaml
log:
level: info
runner:
file: /data/.runner
capacity: 2
timeout: 3h
envs:
DOCKER_HOST: tcp://127.0.0.1:2375
TZ: America/Los_Angeles
container:
network: "host"
docker_host: tcp://127.0.0.1:2375
```
## Steps
1. Fetch the v12.7.0 example config:
```fish
curl -L "https://code.forgejo.org/forgejo/runner/raw/tag/v12.7.0/.forgejo-runner.example.yaml"
```
2. Diff against our current config — note new sections/keys
3. Check the release notes for each major version (v7–v12) for config-related changes:
- v7.0: `FORGEJO_*` env vars (backward compat with `GITHUB_*`)
- v8.0: Default container image change
- v12.7: `server.connections` for multi-server polling; secret URLs; ephemeral mode
4. Decide which new keys to adopt (if any) and update the ConfigMap
5. Pay attention to `container.valid_volumes` and `container.options` (added in v6.x for security) — we may want to configure these
## Key Areas to Check
- **`container.valid_volumes`** — allowlist for volume mounts in job containers (security hardening from v6.x)
- **`container.options`** — allowlist for container options
- **`runner.envs`** — are `FORGEJO_*` vars needed alongside `GITHUB_*`?
- **Ephemeral mode** (v12.7) — one-shot runners that de-register after a job. Not needed now but worth noting.
- **`server.connections`** — multi-server polling. Not needed (single Forgejo instance).
## Related
- [[upgrade-k8s-runner]] — Parent goal

55
docs/how-to/forgejo-runner/upgrade-k8s-runner.md Normal file
View file

@ -0,0 +1,55 @@
---
title: Upgrade K8s Forgejo Runner to v12
status: active
requires:
- validate-workflows-against-v12
- review-runner-config-v12
modified: 2026-02-22
tags:
- how-to
- forgejo-runner
- ci
---
# Upgrade K8s Forgejo Runner to v12
Upgrade the k8s forgejo-runner daemon from v6.3.1 to v12.7.0 (or latest v12.x at time of execution).
## Background
The k8s runner on indri (minikube) uses the upstream `code.forgejo.org/forgejo/runner` image, currently pinned to v6.3.1. The latest is v12.7.0. The runner is still in alpha and uses major version bumps for each breaking change, so v6→v12 crosses six major versions. The ringtail runner is already at ~v12.6.4 via nixpkgs and needs no work.
Blast radius is low — if the upgrade breaks CI, revert the image tag in `argocd/manifests/forgejo-runner/deployment.yaml` and sync.
## Breaking Changes Crossed
| Version | Change | Impact |
|---------|--------|--------|
| v7.0 | CLI `--gitea-instance` → `--forgejo-instance`; `FORGEJO_*` env vars | Low — our registration doesn't use the old flag |
| v8.0 | Workflow schema validation; default image → `node:22-bookworm` | Workflows must pass validation |
| v9.0 | Stricter schema + actions validation; `forgejo-runner validate` added | Same — but now we have a tool |
| v10.0 | Cache isolation; skip v10.0.0 (regression) | Low |
| v11.0 | License MIT → GPLv3 | Non-technical |
| v12.0 | Git binary required; git worktrees for remote actions | Low — OCI image includes git |
## Execution Steps
Once prerequisites are met:
1. Update `argocd/manifests/forgejo-runner/deployment.yaml`:
- Change runner image from `code.forgejo.org/forgejo/runner:6.3.1` to `code.forgejo.org/forgejo/runner:12.7.0`
2. Update `argocd/manifests/forgejo-runner/configmap.yaml` with any config changes from [[review-runner-config-v12]]
3. Push, sync ArgoCD: `argocd app sync forgejo-runner`
4. Verify runner registers and connects: check Forgejo admin → runners
5. Trigger a test workflow (manual dispatch of `build-container.yaml` or `branch-cleanup.yaml`)
6. Update `service-versions.yaml` to note the daemon version
## Rollback
Revert the image tag to `6.3.1` in `deployment.yaml`, push, and sync.
## Related
- [[forgejo]] — Forgejo service reference
- [[validate-workflows-against-v12]] — Pre-upgrade workflow validation
- [[review-runner-config-v12]] — Config format review

82
docs/how-to/forgejo-runner/validate-workflows-against-v12.md Normal file
View file

@ -0,0 +1,82 @@
---
title: Validate Workflows Against v12
status: active
modified: 2026-02-22
tags:
- how-to
- forgejo-runner
- ci
---
# Validate Workflows Against v12
Run `forgejo-runner validate` (available from v9.0+) against all workflow files to catch schema issues before upgrading the k8s runner daemon.
## Background
Forgejo-runner v8.0 introduced workflow schema validation — workflows that don't pass are rejected at runtime. v9.0 made this stricter and added a `validate` CLI command. Since we're jumping from v6.3.1, we've never run validation. All six workflows in `.forgejo/workflows/` need checking.
## Approach: Dagger Pipeline
Add a `validate_workflows` function to the [[dagger]] module (`.dagger/src/blumeops_ci/main.py`). This runs `forgejo-runner validate` inside the upstream runner container — no host-side binary management, reproducible, and version-pinned.
```python
@function
async def validate_workflows(
self,
src: dagger.Directory,
runner_version: str = "12.7.0",
) -> str:
"""Validate Forgejo Actions workflow files against runner schema."""
return await (
dag.container()
.from_(f"code.forgejo.org/forgejo/runner:{runner_version}")
.with_directory("/workspace", src)
.with_workdir("/workspace")
.with_exec([
"sh", "-c",
"for f in .forgejo/workflows/*.yaml; do "
' echo "=== $f ===" && forgejo-runner validate "$f"; '
"done"
])
.stdout()
)
```
Invoke locally with:
```fish
dagger call validate-workflows --src=.
```
### Permanent guardrail
Once the function exists, wire it into CI as a pre-commit hook or a mise task (`mise run validate-workflows`). This prevents future workflow regressions regardless of runner version changes. The `runner_version` parameter lets us pin to whatever version the k8s runner is actually running.
## Workflows to Validate
| File | Complexity | Notes |
|------|-----------|-------|
| `build-container.yaml` | High | Matrix strategy, conditional steps |
| `build-container-nix.yaml` | High | Matrix strategy, conditional steps |
| `build-blumeops.yaml` | High | Multi-step release pipeline |
| `deploy-fly.yaml` | Low | Simple deploy |
| `cv-deploy.yaml` | Medium | Version resolution + deploy |
| `branch-cleanup.yaml` | Low | Scheduled + manual dispatch |
## Fix any failures
If validation fails, fix the workflow schema issues in the same PR as the runner upgrade. Common issues in the v8/v9 changelog:
- Invalid `type:` values in `workflow_dispatch.inputs`
- Incorrect `if:` expression syntax
- Undeclared or misspelled keys
## Deliverables
1. `validate_workflows` function in `.dagger/src/blumeops_ci/main.py`
2. All 6 workflows passing validation (fix any schema issues)
3. A mise task or pre-commit hook wiring `dagger call validate-workflows` for ongoing use
## Related
- [[upgrade-k8s-runner]] — Parent goal

8
docs/how-to/how-to.md
View file

@ -93,3 +93,11 @@ Mikado chain for deploying Authentik. Track progress with `mise run docs-mikado
- [[provision-authentik-database]]
- [[create-authentik-secrets]]
- [[migrate-grafana-to-authentik]]
## Forgejo Runner
Mikado chain for upgrading the k8s forgejo-runner daemon from v6.3.1 to v12.x. Track progress with `mise run docs-mikado upgrade-k8s-runner`.
- [[upgrade-k8s-runner]]
- [[validate-workflows-against-v12]]
- [[review-runner-config-v12]]

20
mise-tasks/service-review
View file

@ -5,7 +5,7 @@
# ///
#MISE description="Review the most stale service for version freshness"
#USAGE flag "--limit <limit>" default="15" help="Number of services to show in the table"
#USAGE flag "--type <type>" help="Filter by service type (argocd, ansible, hybrid)"
#USAGE flag "--type <type>" help="Filter by service type (argocd, ansible, nixos)"
"""Review the most stale service for version freshness.
Reads ``docs/reference/services/service-versions.yaml`` and sorts services
@ -55,7 +55,7 @@ def parse_date(raw) -> date | None:
def main(
limit: Annotated[int, typer.Option(help="Number of services to show in the table")] = 15,
type: Annotated[str | None, typer.Option(help="Filter by service type (argocd, ansible, hybrid)")] = None,
type: Annotated[str | None, typer.Option(help="Filter by service type (argocd, ansible, nixos)")] = None,
) -> None:
console = Console()
today = date.today()
@ -166,14 +166,7 @@ def main(
]
svc_type = top_svc.get("type", "")
if svc_type == "hybrid":
checklist_parts += [
"\n[bold]Custom Container (hybrid):[/bold]\n",
"• Check base image for updates\n",
"• Rebuild container if needed: mise run container-build-and-release\n",
"• Update ArgoCD manifest with new image tag\n",
]
elif svc_type == "argocd":
if svc_type == "argocd":
checklist_parts += [
"\n[bold]ArgoCD Deployment:[/bold]\n",
"• Update image tag or Helm chart version in argocd/manifests/\n",
@ -185,6 +178,13 @@ def main(
f"• Check role vars for version pins: ansible/roles/{top_svc['name']}/\n",
f"• Dry run: mise run provision-indri -- --tags {top_svc['name']} --check --diff\n",
]
elif svc_type == "nixos":
checklist_parts += [
"\n[bold]NixOS Deployment:[/bold]\n",
"• Version tracks nixpkgs via flake.lock\n",
"• Update: dagger call flake-update --src=. export --path=nixos/ringtail/flake.lock\n",
"• Deploy: mise run provision-ringtail\n",
]
checklist_parts += [
"\n[bold]Health Check:[/bold]\n",

53
service-versions.yaml
View file

@ -5,15 +5,13 @@
#
# Fields:
# name - kebab-case service identifier
# type - argocd | ansible | hybrid (custom container + ArgoCD)
# type - argocd | ansible | nixos
# last-reviewed - date (YYYY-MM-DD) or null
# current-version - deployed version string or null
# upstream-source - URL to upstream releases/changelog
# notes - optional context
services:
# --- ArgoCD plain manifests ---
- name: prometheus
type: argocd
last-reviewed: 2026-02-16
@ -45,7 +43,7 @@ services:
upstream-source: https://github.com/binwiederhier/ntfy/releases
- name: homepage
type: hybrid
type: argocd
last-reviewed: 2026-02-19
current-version: "v1.10.1"
upstream-source: https://github.com/gethomepage/homepage/releases
@ -82,8 +80,6 @@ services:
current-version: "v1.94.2"
upstream-source: https://github.com/tailscale/tailscale/releases
# --- ArgoCD Helm charts ---
- name: grafana
type: argocd
last-reviewed: null
@ -119,8 +115,6 @@ services:
upstream-source: https://github.com/1Password/connect/releases
notes: Deployed via Helm chart
# --- ArgoCD infra ---
- name: argocd
type: argocd
last-reviewed: null
@ -134,77 +128,84 @@ services:
upstream-source: https://github.com/cloudnative-pg/cloudnative-pg/releases
notes: CloudNativePG Cluster resource
# --- Hybrid (custom container + ArgoCD) ---
- name: authentik
type: hybrid
type: argocd
last-reviewed: null
current-version: "2025.10.1"
upstream-source: https://github.com/goauthentik/authentik/releases
- name: navidrome
type: hybrid
type: argocd
last-reviewed: null
current-version: "v0.60.3"
upstream-source: https://github.com/navidrome/navidrome/releases
- name: miniflux
type: hybrid
type: argocd
last-reviewed: null
current-version: "2.2.17"
upstream-source: https://github.com/miniflux/v2/releases
- name: teslamate
type: hybrid
type: argocd
last-reviewed: null
current-version: "v2.2.0"
upstream-source: https://github.com/teslamate-org/teslamate/releases
- name: transmission
type: hybrid
type: argocd
last-reviewed: null
current-version: "4.0.6-r4"
upstream-source: https://github.com/transmission/transmission/releases
- name: kiwix
type: hybrid
type: argocd
last-reviewed: null
current-version: "3.8.1"
upstream-source: https://github.com/kiwix/kiwix-tools/releases
- name: devpi
type: hybrid
type: argocd
last-reviewed: null
current-version: "6.19.1"
upstream-source: https://github.com/devpi/devpi/releases
- name: cv
type: hybrid
type: argocd
last-reviewed: null
current-version: "1.0.3"
upstream-source: null
notes: Personal static site, no upstream
- name: docs
type: hybrid
type: argocd
last-reviewed: null
current-version: "1.28.2"
upstream-source: https://github.com/jackyzha0/quartz/releases
notes: Quartz static site generator; container version tracks nginx base
- name: forgejo-runner
type: hybrid
last-reviewed: null
current-version: "0.19.11"
type: argocd
last-reviewed: 2026-02-22
current-version: "6.3.1"
upstream-source: https://code.forgejo.org/forgejo/runner/releases
notes: >-
Runner daemon version. Job execution container (containers/forgejo-runner)
tracks Dagger at v0.19.11.
# --- Ansible native ---
- name: nix-container-builder
type: nixos
last-reviewed: 2026-02-22
current-version: "12.6.4"
upstream-source: https://code.forgejo.org/forgejo/runner/releases
notes: Forgejo runner on ringtail via nixpkgs; version tracks flake.lock
- name: forgejo
type: ansible
last-reviewed: null
current-version: null
last-reviewed: 2026-02-22
current-version: "14.0.2"
upstream-source: https://codeberg.org/forgejo/forgejo/releases
notes: Installed via Homebrew on indri; plan to migrate to source build
- name: alloy
type: ansible
@ -244,4 +245,4 @@ services:
last-reviewed: null
current-version: null
upstream-source: null
notes: Custom systemd service, no upstream
notes: Custom service, no upstream