diff --git a/argocd/manifests/forgejo-runner/Dockerfile b/argocd/manifests/forgejo-runner/Dockerfile new file mode 100644 index 0000000..28d545d --- /dev/null +++ b/argocd/manifests/forgejo-runner/Dockerfile @@ -0,0 +1,23 @@ +FROM code.forgejo.org/forgejo/runner:3.5.1 + +# The base image is Debian-based +# Install tools needed for GitHub Actions and builds +RUN apt-get update && apt-get install -y --no-install-recommends \ + # Required for actions/checkout and other Node-based actions + nodejs \ + npm \ + # Build essentials + git \ + curl \ + wget \ + jq \ + make \ + gcc \ + g++ \ + # For container builds + ca-certificates \ + docker.io \ + && rm -rf /var/lib/apt/lists/* + +# Verify tools are available +RUN node --version && npm --version && docker --version diff --git a/plans/ci-cd-bootstrap/00_overview.md b/plans/ci-cd-bootstrap/00_overview.md index 1943618..c0ba2af 100644 --- a/plans/ci-cd-bootstrap/00_overview.md +++ b/plans/ci-cd-bootstrap/00_overview.md @@ -38,11 +38,11 @@ This plan details the setup of Forgejo Actions as the CI/CD system for blumeops, │ KUBERNETES (minikube) │ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ │ Forgejo Runner │ │ Other Services │ │ -│ │ (act_runner) │ │ (via ArgoCD) │ │ +│ │ (host mode) │ │ (via ArgoCD) │ │ │ │ │ │ │ │ -│ │ - Polls Forgejo │ │ │ │ -│ │ - Runs workflows │ │ │ │ -│ │ - Docker-in-Docker │ │ │ │ +│ │ - Custom image │ │ │ │ +│ │ - Node.js + tools │ │ │ │ +│ │ - Docker builds │ │ │ │ │ └─────────────────────┘ └─────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ ``` @@ -51,28 +51,50 @@ This plan details the setup of Forgejo Actions as the CI/CD system for blumeops, | Phase | Name | Description | Status | |-------|------|-------------|--------| -| 1 | [Enable Actions](P1_enable_actions.md) | Configure Forgejo for Actions, deploy runner | ✅ Complete | -| 2 | [Mirror & Build](P2_mirror_and_build.md) | Mirror upstream Forgejo, create build workflow | Planning | -| 3 | [Self-Deploy](P3_self_deploy.md) | Forgejo deploys itself, transition to mcquack | Planning | -| 4 | [Container Builds](P4_container_builds.md) | Build custom container images, runner observability | Planning | +| 1 | [Enable Actions](P1_enable_actions.md) | Configure Forgejo for Actions, deploy runner in host mode | ✅ Complete | +| 2 | [Custom Runner Image](P2_mirror_and_build.md) | Build custom runner with Node.js/tools, enable standard Actions | Planning | +| 3 | [Mirror Forgejo & Build](P3_mirror_forgejo.md) | Mirror upstream Forgejo, create build workflow | Planning | +| 4 | [Self-Deploy](P4_self_deploy.md) | Forgejo deploys itself, transition to mcquack | Planning | +| 5 | [Container Builds](P5_container_builds.md) | Build custom container images (devpi, etc.) | Planning | ## The Bootstrap Problem **Chicken-and-egg**: We need Forgejo Actions to build Forgejo, but Forgejo must be running first. +**Additional complication**: The stock runner image lacks Node.js, so standard GitHub Actions don't work. + **Solution**: -1. Keep current brew-based Forgejo running during setup -2. Enable Actions, deploy runner -3. Mirror upstream Forgejo, create build workflow -4. First CI build creates the binary -5. CI deploys binary to indri as mcquack service -6. `brew services stop forgejo` and uninstall -7. Future builds: Forgejo builds and deploys itself +1. Keep current brew-based Forgejo running during setup ✅ +2. Enable Actions, deploy runner in host mode ✅ +3. **Build custom runner image** with Node.js and tools (bootstrap manually, then automate) +4. Mirror upstream Forgejo, create build workflow +5. Address cross-compilation challenge (Linux runner → macOS target) +6. First CI build creates the binary +7. CI deploys binary to indri as mcquack service +8. `brew services stop forgejo` and uninstall +9. Future builds: Forgejo builds and deploys itself + +**Cross-compilation challenge**: +The runner runs in Linux containers (k8s), but Forgejo needs to run on indri (macOS ARM64). Options: +- Cross-compile with CGO_ENABLED=1 (complex, needs OSX toolchain) +- Cross-compile with CGO_ENABLED=0 (breaks Tailscale DNS resolution) +- Build on gilbert manually, use CI only for deploy +- Run a native macOS runner on indri (outside k8s) + +This will be addressed in Phase 3. **Risk mitigation**: If self-deployment breaks Forgejo: - blumeops is mirrored to GitHub - Manual recovery: build on gilbert, scp to indri, restart service -- See Disaster Recovery section in P3 +- See Disaster Recovery section in P4 + +## Host Mode Runner + +The runner uses **host mode** (`ubuntu-latest:host`), meaning: +- Jobs run directly in the runner container (no Docker/k8s pods spawned) +- Tools must be pre-installed in the runner image +- Stock image lacks Node.js, so `actions/checkout@v4` doesn't work +- Solution: Build custom runner image with necessary tools (Phase 2) ## Ansible Role Strategy @@ -87,7 +109,7 @@ Ansible does NOT: - Deploy new versions (that's CI's job) Ansible DOES: -- Manage app.ini configuration (sans secrets) +- Manage app.ini configuration (via template with secrets from 1Password) - Manage mcquack LaunchAgent plist - Ensure service is running - Collect logs via Alloy @@ -98,25 +120,27 @@ Ansible DOES: | Path | Purpose | |------|---------| -| `argocd/apps/forgejo-runner.yaml` | ArgoCD Application for runner | -| `argocd/manifests/forgejo-runner/` | Runner k8s manifests | -| `.forgejo/workflows/build-forgejo.yml` | Build workflow in blumeops repo | -| (on forge) `eblume/forgejo/.forgejo/workflows/` | Build workflow in forgejo mirror | +| `argocd/apps/forgejo-runner.yaml` | ArgoCD Application for runner ✅ | +| `argocd/manifests/forgejo-runner/` | Runner k8s manifests ✅ | +| `argocd/manifests/forgejo-runner/Dockerfile` | Custom runner image (P2) | +| `.forgejo/workflows/build-runner.yml` | Auto-rebuild runner image (P2) | +| `.forgejo/workflows/test.yml` | Test workflow ✅ | +| (on forge) `eblume/forgejo/.forgejo/workflows/` | Build workflow in forgejo mirror (P3) | ### Modified Files | Path | Change | |------|--------| -| `ansible/roles/forgejo/` | Complete rewrite for mcquack pattern | -| `ansible/roles/alloy/defaults/main.yml` | Update forgejo log paths | +| `ansible/roles/forgejo/` | Complete rewrite for mcquack pattern (P4) | +| `ansible/roles/alloy/defaults/main.yml` | Update forgejo log paths (P4) | | zk cards | Update forgejo, argocd, blumeops cards | ### Credentials Needed | Item | Purpose | Storage | |------|---------|---------| -| Runner registration token | Runner auth to Forgejo | 1Password | -| SSH deploy key | Runner SSH to indri | 1Password + k8s secret | +| Runner registration token | Runner auth to Forgejo | 1Password ✅ | +| SSH deploy key | Runner SSH to indri (for Forgejo deploy) | 1Password + k8s secret (P3) | ## Related Plans @@ -125,6 +149,15 @@ Ansible DOES: ## Decision Log +### 2026-01-23: Custom runner image as Phase 2 + +**Decision**: Move custom runner image work from P4 to P2 + +**Rationale**: +- Stock runner lacks Node.js, can't run `actions/checkout@v4` +- Need working GitHub Actions before building Forgejo +- Bootstrap manually (podman build on gilbert), then automate + ### 2026-01-23: Forgejo Actions over Woodpecker **Decision**: Use Forgejo Actions instead of Woodpecker CI diff --git a/plans/ci-cd-bootstrap/P2_mirror_and_build.md b/plans/ci-cd-bootstrap/P2_mirror_and_build.md index 1fef474..2d146ce 100644 --- a/plans/ci-cd-bootstrap/P2_mirror_and_build.md +++ b/plans/ci-cd-bootstrap/P2_mirror_and_build.md @@ -1,138 +1,188 @@ -# Phase 2: Mirror Forgejo & Create Build Workflow +# Phase 2: Custom Runner Image -**Goal**: Mirror upstream Forgejo to forge and create a workflow that builds it from source +**Goal**: Build a custom forgejo-runner image with necessary tools, enabling standard GitHub Actions **Status**: Planning -**Prerequisites**: [Phase 1](P1_enable_actions.md) complete (Actions enabled, runner deployed) +**Prerequisites**: [Phase 1](P1_enable_actions.md) complete (Actions enabled, runner deployed in host mode) --- -## Current State +## Problem Statement -- Forgejo Actions enabled with runner in k8s -- Upstream Forgejo at https://codeberg.org/forgejo/forgejo -- No local mirror yet +The stock `code.forgejo.org/forgejo/runner:3.5.1` image lacks tools needed for standard GitHub Actions: +- **Node.js** - Required by most actions (checkout, setup-*, etc.) +- **Git** - For repository operations (present but minimal) +- **Common build tools** - make, gcc, curl, jq, etc. + +In host mode, jobs run directly in the runner container, so these tools must be pre-installed. + +### Chicken-and-Egg Problem + +We can't use `actions/checkout@v4` to build the custom runner because that action requires Node.js, which we don't have yet. Solution: Bootstrap manually, then automate. --- -## Step 1: Mirror Upstream Forgejo +## Step 1: Create Dockerfile for Custom Runner -### 1.1 User Action: Create Mirror on Forge +Create `argocd/manifests/forgejo-runner/Dockerfile`: -**Manual step** (hairpinning doesn't work from indri): +```dockerfile +FROM code.forgejo.org/forgejo/runner:3.5.1 -1. Go to https://forge.tail8d86e.ts.net -2. Click "+" → "New Migration" -3. Select "Gitea" as clone source -4. URL: `https://codeberg.org/forgejo/forgejo.git` -5. Repository name: `forgejo` -6. Check "This repository will be a mirror" -7. Click "Migrate Repository" +# The base image is Debian-based +# Install tools needed for GitHub Actions and builds +RUN apt-get update && apt-get install -y --no-install-recommends \ + # Required for actions/checkout and other Node-based actions + nodejs \ + npm \ + # Build essentials + git \ + curl \ + wget \ + jq \ + make \ + gcc \ + g++ \ + # For container builds (if we add Docker-in-Docker later) + ca-certificates \ + && rm -rf /var/lib/apt/lists/* -### 1.2 Clone Mirror Locally - -```bash -git clone ssh://forgejo@forge.tail8d86e.ts.net/eblume/forgejo.git ~/code/3rd/forgejo -cd ~/code/3rd/forgejo +# Verify Node.js is available +RUN node --version && npm --version ``` --- -## Step 2: Understand Forgejo Build Process +## Step 2: Bootstrap - Build Image Manually -### 2.1 Build Requirements +Since we can't use CI yet, build the image manually on gilbert and push to zot. -From Forgejo's `Makefile` and docs: - -- **Go**: 1.23+ (check `go.mod` for exact version) -- **Node.js**: 20+ (for frontend) -- **Make**: GNU Make -- **Git**: For version embedding - -### 2.2 Build Commands +### 2.1 Build with Podman ```bash -# Install frontend dependencies and build -make deps-frontend -make frontend +cd ~/code/personal/blumeops/argocd/manifests/forgejo-runner -# Build backend -TAGS="bindata sqlite sqlite_unlock_notify" make backend +# Build for linux/arm64 (minikube on M1 Mac) +podman build --platform linux/arm64 -t registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest . -# Or all-in-one -TAGS="bindata sqlite sqlite_unlock_notify" make build +# Push to zot (no auth required) +podman push registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest ``` -### 2.3 Output +### 2.2 Verify Image in Registry -Binary at `gitea` (yes, the binary is still named `gitea` for compatibility). +```bash +curl -s https://registry.tail8d86e.ts.net/v2/blumeops/forgejo-runner/tags/list | jq . +``` --- -## Step 3: Create Build Workflow +## Step 3: Update Runner Deployment -### 3.1 SSH Deploy Key for Runner +### 3.1 Update deployment.yaml -The runner needs SSH access to indri to deploy the binary. - -**Generate key on gilbert**: -```bash -ssh-keygen -t ed25519 -C "forgejo-runner-deploy" -f ~/.ssh/forgejo-runner-deploy -``` - -**Add public key to indri's authorized_keys**: -```bash -cat ~/.ssh/forgejo-runner-deploy.pub | ssh indri 'cat >> ~/.ssh/authorized_keys' -``` - -**Store private key in 1Password** (blumeops vault) as "Forgejo Runner Deploy Key" - -**Add to k8s as secret**: - -Create `argocd/manifests/forgejo-runner/secret-ssh.yaml.tpl`: -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: forgejo-runner-ssh - namespace: forgejo-runner -type: Opaque -stringData: - id_ed25519: | - op://blumeops//private-key - known_hosts: | - indri.tail8d86e.ts.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIxxxxxx -``` - -Get indri's host key: -```bash -ssh-keyscan indri.tail8d86e.ts.net 2>/dev/null | grep ed25519 -``` - -### 3.2 Create Workflow File - -Create `.forgejo/workflows/build.yml` in the forgejo mirror repo: +Change the image from stock to custom: ```yaml -name: Build Forgejo +# Before +image: code.forgejo.org/forgejo/runner:3.5.1 + +# After +image: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest +``` + +### 3.2 Update kustomization.yaml + +Add Dockerfile to resources (for reference, not deployed): + +```yaml +# Note: Dockerfile is for building, not k8s deployment +# It lives here for co-location with the runner manifests +``` + +### 3.3 Sync Deployment + +```bash +argocd app sync forgejo-runner + +# Verify new image is running +kubectl --context=minikube-indri -n forgejo-runner get pods -o jsonpath='{.items[*].spec.containers[*].image}' +``` + +--- + +## Step 4: Test with Real GitHub Action + +Now that we have Node.js, test with `actions/checkout@v4`. + +### 4.1 Update Test Workflow + +Update `.forgejo/workflows/test.yml`: + +```yaml +name: Test CI on: push: - tags: - - 'v*' + branches: [main] + pull_request: + workflow_dispatch: + +jobs: + test: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Verify tools + run: | + echo "Node.js: $(node --version)" + echo "npm: $(npm --version)" + echo "Git: $(git --version)" + echo "Make: $(make --version | head -1)" + + - name: Show repo info + run: | + echo "Repository: ${{ github.repository }}" + echo "Branch: ${{ github.ref_name }}" + ls -la +``` + +### 4.2 Push and Verify + +```bash +git add .forgejo/workflows/test.yml +git commit -m "Test checkout action with custom runner" +git push +``` + +Check https://forge.tail8d86e.ts.net/eblume/blumeops/actions - should see successful run with `actions/checkout@v4`. + +--- + +## Step 5: Create Auto-Build Workflow for Runner + +Now that Actions work properly, create a workflow to rebuild the runner image automatically. + +### 5.1 Create Build Workflow + +Create `.forgejo/workflows/build-runner.yml`: + +```yaml +name: Build Runner Image + +on: + push: + paths: + - 'argocd/manifests/forgejo-runner/Dockerfile' + - '.forgejo/workflows/build-runner.yml' workflow_dispatch: - inputs: - deploy: - description: 'Deploy to indri after build' - required: false - default: 'true' - type: boolean env: - GOPROXY: "https://proxy.golang.org,direct" - CGO_ENABLED: "1" - TAGS: "bindata sqlite sqlite_unlock_notify" + REGISTRY: registry.tail8d86e.ts.net + IMAGE_NAME: blumeops/forgejo-runner jobs: build: @@ -140,237 +190,158 @@ jobs: steps: - name: Checkout uses: actions/checkout@v4 - with: - fetch-depth: 0 # Need full history for version - - name: Setup Go - uses: actions/setup-go@v5 - with: - go-version-file: 'go.mod' - - - name: Setup Node - uses: actions/setup-node@v4 - with: - node-version: '20' - - - name: Get version - id: version + - name: Build image run: | - if [[ "${{ github.ref }}" == refs/tags/* ]]; then - VERSION="${{ github.ref_name }}" - else - VERSION="$(git describe --tags --always)-dev" - fi - echo "version=$VERSION" >> "$GITHUB_OUTPUT" - echo "Building version: $VERSION" + cd argocd/manifests/forgejo-runner + # Use docker build (available in runner container) + # Note: This builds for the runner's native arch + docker build -t ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} . + docker tag ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \ + ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest - - name: Build frontend + - name: Push to registry run: | - make deps-frontend - make frontend + # Zot has no auth, just push + docker push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} + docker push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest - - name: Build backend + - name: Verify push run: | - TAGS="${{ env.TAGS }}" make backend - ./gitea --version - - - name: Rename binary - run: | - mv gitea forgejo-${{ steps.version.outputs.version }}-darwin-arm64 - ls -la forgejo-* - - - name: Upload artifact - uses: actions/upload-artifact@v4 - with: - name: forgejo-${{ steps.version.outputs.version }}-darwin-arm64 - path: forgejo-${{ steps.version.outputs.version }}-darwin-arm64 - - deploy: - needs: build - runs-on: ubuntu-latest - if: github.event_name == 'push' || (github.event_name == 'workflow_dispatch' && github.event.inputs.deploy == 'true') - steps: - - name: Download artifact - uses: actions/download-artifact@v4 - with: - name: forgejo-${{ needs.build.outputs.version }}-darwin-arm64 - - - name: Setup SSH - run: | - mkdir -p ~/.ssh - echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/id_ed25519 - chmod 600 ~/.ssh/id_ed25519 - echo "${{ secrets.DEPLOY_KNOWN_HOSTS }}" > ~/.ssh/known_hosts - - - name: Deploy to indri - run: | - BINARY="forgejo-*-darwin-arm64" - chmod +x $BINARY - - # Copy binary to indri - scp $BINARY erichblume@indri.tail8d86e.ts.net:~/.local/bin/forgejo-new - - # Atomic swap and restart - ssh erichblume@indri.tail8d86e.ts.net << 'EOF' - set -e - cd ~/.local/bin - - # Verify the new binary runs - ./forgejo-new --version - - # Stop current service - launchctl unload ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist 2>/dev/null || true - - # Atomic swap - mv forgejo forgejo-old 2>/dev/null || true - mv forgejo-new forgejo - - # Start new service - launchctl load ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist - - # Verify it's running - sleep 5 - curl -sf http://localhost:3001/api/v1/version || exit 1 - - echo "Deploy successful!" - ./forgejo --version - EOF + curl -sf "https://${{ env.REGISTRY }}/v2/${{ env.IMAGE_NAME }}/tags/list" | jq . + echo "Image pushed: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}" ``` -### 3.3 Add Repository Secrets +### 5.2 Note on Docker-in-Docker -In Forgejo, go to the forgejo repo → Settings → Actions → Secrets: +The runner runs in host mode, so we need Docker CLI available. Options: -1. **DEPLOY_SSH_KEY**: Private key from 1Password -2. **DEPLOY_KNOWN_HOSTS**: Output of `ssh-keyscan indri.tail8d86e.ts.net` +1. **Add Docker CLI to the custom image** (see Dockerfile update below) +2. **Mount Docker socket from minikube** (requires deployment change) +3. **Use Podman instead** (rootless, no socket needed) + +For now, we'll add Docker CLI to the image and mount the socket. + +### 5.3 Update Dockerfile for Docker Builds + +```dockerfile +FROM code.forgejo.org/forgejo/runner:3.5.1 + +RUN apt-get update && apt-get install -y --no-install-recommends \ + nodejs \ + npm \ + git \ + curl \ + wget \ + jq \ + make \ + gcc \ + g++ \ + ca-certificates \ + # Docker CLI for building container images + docker.io \ + && rm -rf /var/lib/apt/lists/* + +RUN node --version && npm --version && docker --version +``` + +### 5.4 Update Deployment for Docker Socket + +Add Docker socket mount to `deployment.yaml`: + +```yaml +volumeMounts: + - name: runner-data + mountPath: /data + - name: runner-config + mountPath: /config + - name: docker-sock + mountPath: /var/run/docker.sock +volumes: + - name: runner-data + emptyDir: {} + - name: runner-config + configMap: + name: forgejo-runner-config + - name: docker-sock + hostPath: + path: /var/run/docker.sock + type: Socket +``` --- -## Step 4: Build Cross-Platform Consideration +## Step 6: Verification -**Important**: The runner runs Linux containers, but indri is macOS ARM64. - -**Options**: - -### Option A: Cross-compile (Simpler, may have issues) - -Add to build job: -```yaml -env: - GOOS: darwin - GOARCH: arm64 -``` - -CGO cross-compilation is tricky. May need to disable CGO or use a cross-compiler. - -### Option B: Build on macOS (More reliable) - -Run a macOS runner on indri itself (not in k8s). +### 6.1 Manual Image Build Works ```bash -# Install forgejo-runner on indri via mise -ssh indri 'mise use forgejo-runner' - -# Register as a macOS runner -ssh indri 'forgejo-runner register --labels "macos-arm64:host" ...' +# On gilbert +podman build --platform linux/arm64 -t registry.tail8d86e.ts.net/blumeops/forgejo-runner:test . +podman push registry.tail8d86e.ts.net/blumeops/forgejo-runner:test ``` -Then workflow uses: -```yaml -runs-on: macos-arm64 -``` +### 6.2 Runner Uses Custom Image -**Recommendation**: Option B is more reliable for native macOS builds. Consider deploying a runner directly on indri for macOS-specific builds. - ---- - -## Step 5: Test the Build - -### 5.1 Manual Workflow Dispatch - -1. Go to https://forge.tail8d86e.ts.net/eblume/forgejo/actions -2. Select "Build Forgejo" workflow -3. Click "Run workflow" -4. Set deploy=false for first test -5. Monitor the run - -### 5.2 Verify Artifact - -Download the artifact from the workflow run and verify it's a valid binary: ```bash -# If downloaded to gilbert -file forgejo-*-darwin-arm64 -# Should show: Mach-O 64-bit executable arm64 +kubectl --context=minikube-indri -n forgejo-runner get pods -o jsonpath='{.items[*].spec.containers[*].image}' +# Should show: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest ``` ---- +### 6.3 GitHub Actions Work -## Alternative: Build on Gilbert, Deploy via CI +- `actions/checkout@v4` succeeds +- Test workflow shows Node.js, npm, git versions -If cross-compilation proves difficult, consider a hybrid approach: +### 6.4 Auto-Build Workflow Works -1. **Build on gilbert** (has Go, Node, is macOS ARM64) -2. **CI just deploys** the built binary - -Workflow in blumeops repo: -```yaml -name: Deploy Forgejo - -on: - workflow_dispatch: - inputs: - binary_path: - description: 'Path to binary on gilbert' - required: true - -jobs: - deploy: - runs-on: ubuntu-latest - steps: - # Fetch binary from gilbert and deploy to indri - # (requires SSH access to both) -``` - -This is less elegant but more pragmatic for macOS targets. +Push a change to the Dockerfile and verify: +1. Workflow triggers +2. Image builds successfully +3. Image pushed to zot --- ## Verification Checklist -- [ ] Forgejo mirrored to forge -- [ ] SSH deploy key created and stored in 1Password -- [ ] Deploy key added to indri authorized_keys -- [ ] SSH secret added to k8s -- [ ] Workflow file created in forgejo mirror -- [ ] Repository secrets configured -- [ ] Test build completes successfully -- [ ] Binary is valid macOS ARM64 executable +- [ ] Dockerfile created for custom runner +- [ ] Image built manually on gilbert +- [ ] Image pushed to zot registry +- [ ] Runner deployment updated to use custom image +- [ ] Runner pod running with new image +- [ ] `actions/checkout@v4` works in test workflow +- [ ] Auto-build workflow created +- [ ] Docker socket mounted (for container builds) +- [ ] Auto-build workflow successfully rebuilds runner --- ## Troubleshooting -### CGO Cross-Compilation Fails +### Image Pull Fails in Minikube -If building Linux→macOS fails: -``` -# runtime/cgo -gcc: error: unrecognized command line option '-arch' +Minikube needs to be able to pull from zot. Check registry mirror config: +```bash +ssh indri 'minikube ssh -- cat /etc/containerd/certs.d/registry.tail8d86e.ts.net/hosts.toml' ``` -Either: -1. Use Option B (macOS runner on indri) -2. Build with `CGO_ENABLED=0` (loses some features) -3. Use a Docker image with macOS cross-compiler (complex) +### Docker Build Fails in Workflow -### Artifact Too Large +If Docker socket mount doesn't work: +1. Check socket exists in minikube: `minikube ssh -- ls -la /var/run/docker.sock` +2. Check permissions: runner may need to be in docker group +3. Alternative: Use `podman` (rootless) instead of Docker -Forgejo binary is ~100MB. If upload fails: -- Check Forgejo's artifact size limit in app.ini -- Consider compressing: `gzip -9 forgejo-*` +### Node.js Actions Still Fail + +Ensure the runner pod restarted after image update: +```bash +kubectl --context=minikube-indri -n forgejo-runner rollout restart deployment/forgejo-runner +kubectl --context=minikube-indri -n forgejo-runner logs -f deployment/forgejo-runner +``` --- ## Next Phase -Once build is working and produces valid binaries, proceed to [Phase 3: Self-Deploy](P3_self_deploy.md). +Once the custom runner is working with auto-build, proceed to [Phase 3: Mirror Forgejo & Build](P3_mirror_and_build.md) to set up Forgejo source builds. diff --git a/plans/ci-cd-bootstrap/P3_mirror_forgejo.md b/plans/ci-cd-bootstrap/P3_mirror_forgejo.md new file mode 100644 index 0000000..9e1e142 --- /dev/null +++ b/plans/ci-cd-bootstrap/P3_mirror_forgejo.md @@ -0,0 +1,349 @@ +# Phase 3: Mirror Forgejo & Build from Source + +**Goal**: Mirror upstream Forgejo to forge and create a workflow that builds it for macOS ARM64 + +**Status**: Planning + +**Prerequisites**: [Phase 2](P2_mirror_and_build.md) complete (custom runner image with Node.js/tools) + +--- + +## Problem Statement + +We want to build Forgejo from source to: +1. Have full control over the binary running on indri +2. Enable self-deployment via CI +3. Ensure proper macOS DNS resolution (requires CGO_ENABLED=1) + +### The Cross-Compilation Challenge + +The runner runs in a Linux container (k8s on indri), but the target is macOS ARM64 (indri itself). + +**Options**: + +| Option | Pros | Cons | +|--------|------|------| +| A. Cross-compile CGO_ENABLED=0 | Simple, no special toolchain | Breaks Tailscale MagicDNS resolution | +| B. Cross-compile CGO_ENABLED=1 | Proper DNS | Needs OSX cross-compiler (osxcross), complex | +| C. Build on gilbert manually | Works now, simple | Not automated, manual step | +| D. Native macOS runner on indri | Full native build | Runner outside k8s, different architecture | +| E. Hybrid: build on gilbert, deploy via CI | Uses existing tools | Partial automation | + +**Recommendation**: Start with Option C/E (manual build on gilbert, CI just deploys), then consider Option D if we want full automation. + +--- + +## Step 1: Mirror Upstream Forgejo + +### 1.1 User Action: Create Mirror on Forge + +**Manual step** (hairpinning doesn't work from indri): + +1. Go to https://forge.tail8d86e.ts.net +2. Click "+" → "New Migration" +3. Select "Gitea" as clone source +4. URL: `https://codeberg.org/forgejo/forgejo.git` +5. Repository name: `forgejo` +6. Check "This repository will be a mirror" +7. Click "Migrate Repository" + +### 1.2 Clone Mirror Locally + +```bash +git clone ssh://forgejo@forge.tail8d86e.ts.net/eblume/forgejo.git ~/code/3rd/forgejo +cd ~/code/3rd/forgejo +``` + +--- + +## Step 2: Understand Forgejo Build Process + +### 2.1 Build Requirements + +From Forgejo's `Makefile` and docs: + +- **Go**: 1.23+ (check `go.mod` for exact version) +- **Node.js**: 20+ (for frontend) +- **Make**: GNU Make +- **Git**: For version embedding + +### 2.2 Build Commands + +```bash +# Install frontend dependencies and build +make deps-frontend +make frontend + +# Build backend (with CGO for proper DNS on macOS) +CGO_ENABLED=1 TAGS="bindata sqlite sqlite_unlock_notify" make backend + +# Or all-in-one +CGO_ENABLED=1 TAGS="bindata sqlite sqlite_unlock_notify" make build +``` + +### 2.3 Output + +Binary at `gitea` (yes, the binary is still named `gitea` for compatibility). + +--- + +## Step 3: Build on Gilbert (Manual Bootstrap) + +For the initial bootstrap, build on gilbert (macOS ARM64 native). + +### 3.1 Setup Build Environment + +```bash +cd ~/code/3rd/forgejo +mise use go@1.23 node@20 + +# Verify tools +go version +node --version +make --version +``` + +### 3.2 Build + +```bash +# Clean build +make clean + +# Build frontend +make deps-frontend +make frontend + +# Build backend with CGO (important for macOS DNS!) +CGO_ENABLED=1 TAGS="bindata sqlite sqlite_unlock_notify" make backend + +# Verify binary +./gitea --version +file gitea # Should show: Mach-O 64-bit executable arm64 +``` + +### 3.3 Deploy to Indri + +```bash +# Copy binary +scp gitea indri:~/.local/bin/forgejo-new + +# Verify on indri +ssh indri '~/.local/bin/forgejo-new --version' +``` + +--- + +## Step 4: Create Deploy Workflow (Option E) + +Since cross-compilation is complex, use a hybrid approach: +1. Build on gilbert (manual trigger or pre-built) +2. CI workflow fetches and deploys + +### 4.1 SSH Deploy Key for Runner + +The runner needs SSH access to indri to deploy the binary. + +**Generate key on gilbert**: +```bash +ssh-keygen -t ed25519 -C "forgejo-runner-deploy" -f ~/.ssh/forgejo-runner-deploy -N "" +``` + +**Add public key to indri's authorized_keys**: +```bash +cat ~/.ssh/forgejo-runner-deploy.pub | ssh indri 'cat >> ~/.ssh/authorized_keys' +``` + +**Store private key in 1Password** (blumeops vault) as "Forgejo Runner Deploy Key" + +### 4.2 Create k8s Secret + +Create `argocd/manifests/forgejo-runner/secret-ssh.yaml.tpl`: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: forgejo-runner-ssh + namespace: forgejo-runner +type: Opaque +stringData: + id_ed25519: | + op://blumeops//private-key + known_hosts: | + # Get with: ssh-keyscan indri.tail8d86e.ts.net 2>/dev/null | grep ed25519 + indri.tail8d86e.ts.net ssh-ed25519 AAAAC3... +``` + +### 4.3 Update Deployment for SSH + +Add SSH secret mount to `deployment.yaml`: + +```yaml +volumeMounts: + - name: ssh-key + mountPath: /root/.ssh + readOnly: true +volumes: + - name: ssh-key + secret: + secretName: forgejo-runner-ssh + defaultMode: 0600 +``` + +### 4.4 Create Deploy-Only Workflow + +Create `.forgejo/workflows/deploy-forgejo.yml` in blumeops: + +```yaml +name: Deploy Forgejo + +on: + workflow_dispatch: + inputs: + version: + description: 'Version to deploy (tag or commit)' + required: true + default: 'v10.0.0' + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Deploy to indri + env: + VERSION: ${{ github.event.inputs.version }} + run: | + # SSH config + mkdir -p ~/.ssh + cp /root/.ssh/id_ed25519 ~/.ssh/ + cp /root/.ssh/known_hosts ~/.ssh/ + chmod 600 ~/.ssh/id_ed25519 + + # Deploy script + ssh erichblume@indri.tail8d86e.ts.net << 'EOF' + set -e + cd ~/.local/bin + + # Verify the new binary exists and runs + if [ ! -f forgejo-new ]; then + echo "ERROR: forgejo-new not found. Build on gilbert first:" + echo " cd ~/code/3rd/forgejo && git checkout $VERSION" + echo " CGO_ENABLED=1 TAGS='bindata sqlite sqlite_unlock_notify' make build" + echo " scp gitea indri:~/.local/bin/forgejo-new" + exit 1 + fi + + ./forgejo-new --version + + # Stop current service + launchctl unload ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist 2>/dev/null || true + + # Atomic swap + mv forgejo forgejo-old 2>/dev/null || true + mv forgejo-new forgejo + + # Start new service + launchctl load ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist + + # Verify it's running + sleep 5 + curl -sf http://localhost:3001/api/v1/version || exit 1 + + echo "Deploy successful!" + ./forgejo --version + EOF +``` + +--- + +## Future: Full CI Build (Option D) + +If we want full automation, consider running a native macOS runner on indri: + +### Native Runner on Indri + +```bash +# Install forgejo-runner on indri via mise +ssh indri 'mise use forgejo-runner' + +# Register as a macOS runner +ssh indri 'forgejo-runner register \ + --instance https://forge.tail8d86e.ts.net \ + --token "$TOKEN" \ + --name "indri-native" \ + --labels "macos-arm64:host" \ + --no-interactive' + +# Create LaunchAgent for runner +# (similar to other mcquack services) +``` + +Then workflow uses: +```yaml +runs-on: macos-arm64 +``` + +This enables full native builds in CI. Document in a future phase if needed. + +--- + +## Verification Checklist + +- [ ] Forgejo mirrored to forge +- [ ] Mirror cloned to ~/code/3rd/forgejo +- [ ] Build succeeds on gilbert +- [ ] Binary is valid macOS ARM64 executable +- [ ] Binary deployed to indri ~/.local/bin/ +- [ ] SSH deploy key created and stored in 1Password +- [ ] Deploy key added to indri authorized_keys +- [ ] (Optional) k8s SSH secret created +- [ ] (Optional) Deploy workflow created + +--- + +## Troubleshooting + +### Build Fails: Node.js Version + +``` +error: engine "node" is incompatible +``` + +Update Node.js: `mise use node@20` + +### Build Fails: Go Version + +``` +go: go.mod requires go >= 1.23 +``` + +Update Go: `mise use go@1.23` + +### Binary Crashes on indri + +Check if CGO was enabled: +```bash +# If built without CGO, DNS resolution may fail +./forgejo --version # Should work +./forgejo web # May fail to resolve Tailscale hostnames +``` + +Rebuild with `CGO_ENABLED=1`. + +### SSH Deploy Fails + +Check runner has SSH access: +```bash +# Test from inside runner pod +kubectl --context=minikube-indri -n forgejo-runner exec deployment/forgejo-runner -- \ + ssh -i /root/.ssh/id_ed25519 erichblume@indri.tail8d86e.ts.net 'echo ok' +``` + +--- + +## Next Phase + +Once Forgejo is building and deploying successfully, proceed to [Phase 4: Self-Deploy](P4_self_deploy.md) for the full mcquack transition. diff --git a/plans/ci-cd-bootstrap/P3_self_deploy.md b/plans/ci-cd-bootstrap/P4_self_deploy.md similarity index 97% rename from plans/ci-cd-bootstrap/P3_self_deploy.md rename to plans/ci-cd-bootstrap/P4_self_deploy.md index 0c2a616..8a73843 100644 --- a/plans/ci-cd-bootstrap/P3_self_deploy.md +++ b/plans/ci-cd-bootstrap/P4_self_deploy.md @@ -1,10 +1,10 @@ -# Phase 3: Self-Deploy & Transition to mcquack +# Phase 4: Self-Deploy & Transition to mcquack **Goal**: Complete the bootstrap - Forgejo deploys itself, transition from brew to mcquack LaunchAgent **Status**: Planning -**Prerequisites**: [Phase 2](P2_mirror_and_build.md) complete (build workflow produces valid binaries) +**Prerequisites**: [Phase 3](P3_mirror_forgejo.md) complete (Forgejo builds and deploys to indri) --- @@ -406,4 +406,4 @@ After recovery, switch back to Forgejo. ## Next Phase -After bootstrap is complete, proceed to [Phase 4: Container Builds](P4_container_builds.md) to set up container image building for ArgoCD. +After bootstrap is complete, proceed to [Phase 5: Container Builds](P5_container_builds.md) to set up container image building for ArgoCD. diff --git a/plans/ci-cd-bootstrap/P4_container_builds.md b/plans/ci-cd-bootstrap/P5_container_builds.md similarity index 85% rename from plans/ci-cd-bootstrap/P4_container_builds.md rename to plans/ci-cd-bootstrap/P5_container_builds.md index 60a075b..fcae2b2 100644 --- a/plans/ci-cd-bootstrap/P4_container_builds.md +++ b/plans/ci-cd-bootstrap/P5_container_builds.md @@ -1,91 +1,21 @@ -# Phase 4: Container Image Builds +# Phase 5: Container Image Builds **Goal**: Set up CI workflows to build custom container images and push to zot registry **Status**: Planning -**Prerequisites**: [Phase 3](P3_self_deploy.md) complete (Forgejo self-deploying, Actions working) +**Prerequisites**: [Phase 4](P4_self_deploy.md) complete (Forgejo self-deploying, Actions working) --- ## Overview -With Forgejo Actions operational, we can now build container images for: +With Forgejo Actions operational (including custom runner from P2), we can now build container images for: - Custom devpi with pre-installed plugins - Any other custom images needed for k8s services - Release artifacts for Python packages ---- - -## Use Case 0: Custom Runner Image - -### Problem - -The stock `forgejo/runner` image lacks tools needed for standard GitHub Actions: -- **Node.js** - Required by most actions (checkout, setup-*, etc.) -- **Docker CLI** - For building container images -- **Git** - For repository operations -- **Common build tools** - make, gcc, etc. - -In host mode, jobs run directly in the runner container, so these tools must be pre-installed. - -### Solution - -Build a custom runner image with all necessary tools: - -```dockerfile -# argocd/manifests/forgejo-runner/Dockerfile -FROM code.forgejo.org/forgejo/runner:3.5.1 - -# Install Node.js (required for most GitHub Actions) -RUN apt-get update && apt-get install -y \ - nodejs \ - npm \ - git \ - curl \ - docker.io \ - make \ - gcc \ - && rm -rf /var/lib/apt/lists/* -``` - -### Workflow - -Create `.forgejo/workflows/build-runner.yml`: - -```yaml -name: Build Runner Image - -on: - push: - paths: - - 'argocd/manifests/forgejo-runner/Dockerfile' - workflow_dispatch: - -jobs: - build: - runs-on: ubuntu-latest - steps: - - name: Checkout - run: | - git clone --depth 1 "${{ gitea.server_url }}/${{ gitea.repository }}.git" . - - - name: Build and push - run: | - cd argocd/manifests/forgejo-runner - docker build -t registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest . - docker push registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest -``` - -### Update Deployment - -Once the custom image is built, update `argocd/manifests/forgejo-runner/deployment.yaml`: - -```yaml -image: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest -``` - -This enables standard GitHub Actions like `actions/checkout@v4` to work in host mode. +**Note**: The custom runner image build is covered in [Phase 2](P2_mirror_and_build.md). This phase focuses on application container builds. ---