eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Reorganize CI/CD bootstrap phases and add custom runner Dockerfile (#50)
All checks were successful
Test CI / test (push) Successful in 2s
Details

Browse source 
## Summary
- Reorder CI/CD bootstrap phases to address chicken-and-egg problem
- P2 is now "Custom Runner Image" (stock runner lacks Node.js)
- Add P3 for "Mirror Forgejo & Build from Source"
- Rename P3 -> P4 (Self-Deploy), P4 -> P5 (Container Builds)
- Add Dockerfile for custom runner with Node.js, npm, docker, build tools
- Update overview with new phase structure, host mode notes, and cross-compilation challenge

## Key Changes

### Phase Reordering
| Old | New | Name |
|-----|-----|------|
| P1 | P1 | Enable Actions (complete) |
| P2 | P2 | **Custom Runner Image** (new focus) |
| - | P3 | **Mirror Forgejo & Build** (new) |
| P3 | P4 | Self-Deploy |
| P4 | P5 | Container Builds |

### Custom Runner Dockerfile
The stock `forgejo/runner:3.5.1` image lacks Node.js, so `actions/checkout@v4` doesn't work. The new Dockerfile adds:
- Node.js + npm (for GitHub Actions)
- Docker CLI (for container builds)
- Build tools (gcc, make, curl, jq)

### Bootstrap Strategy
1. Build custom runner image manually on gilbert (podman build)
2. Push to zot registry
3. Update deployment to use custom image
4. Then enable auto-build workflow for runner

## Deployment and Testing
- [x] Review plan changes
- [x] Build custom runner image manually and verify
- [x] Update runner deployment
- [x] Test `actions/checkout@v4` works

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/50
This commit is contained in:
Erich Blume 2026-01-23 18:50:27 -08:00
commit 5fcd122494
8 changed files with 696 additions and 373 deletions
Download patch file Download diff file Expand all files Collapse all files

41
.forgejo/workflows/test.yaml
View file

@ -10,24 +10,35 @@ jobs:
test: test:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Checkout (git clone) - name: Checkout
run: | uses: actions/checkout@v4
# For PRs use head_ref (branch name), for pushes use ref_name
BRANCH="${HEAD_REF:-$REF_NAME}"
git clone --depth 1 --branch "$BRANCH" \
"${SERVER_URL}/${REPOSITORY}.git" .
env:
GIT_SSL_NO_VERIFY: "true"
HEAD_REF: ${{ github.head_ref }}
REF_NAME: ${{ github.ref_name }}
SERVER_URL: ${{ github.server_url }}
REPOSITORY: ${{ github.repository }}
- name: Hello World - name: Verify tools
run: |
echo "=== Node.js ==="
node --version
npm --version
echo ""
echo "=== Git ==="
git --version
echo ""
echo "=== Build tools ==="
make --version | head -1
gcc --version | head -1
echo ""
echo "=== Docker ==="
docker --version
echo ""
echo "=== Other tools ==="
curl --version | head -1
jq --version
- name: Show repo info
run: | run: |
echo "Hello from Forgejo Actions!"
echo "Runner: $(hostname)"
echo "Repository: ${{ github.repository }}" echo "Repository: ${{ github.repository }}"
echo "Event: ${{ github.event_name }}" echo "Event: ${{ github.event_name }}"
echo "Ref: ${{ github.ref }}" echo "Ref: ${{ github.ref }}"
echo "Branch: ${{ github.ref_name }}"
echo ""
echo "=== Files ==="
ls -la ls -la

29
argocd/manifests/forgejo-runner/Dockerfile Normal file
View file

@ -0,0 +1,29 @@
FROM code.forgejo.org/forgejo/runner:3.5.1
# Switch to root to install packages
USER root
# The base image is Alpine Linux
# Install tools needed for GitHub Actions and builds
RUN apk add --no-cache \
# Required for actions/checkout and other Node-based actions
nodejs \
npm \
# Build essentials
git \
curl \
wget \
jq \
make \
gcc \
g++ \
musl-dev \
# For container builds
ca-certificates \
docker-cli
# Verify tools are available
RUN node --version && npm --version && docker --version
# Switch back to non-root user
USER 1000

2
argocd/manifests/forgejo-runner/deployment.yaml
View file

@ -16,7 +16,7 @@ spec:
serviceAccountName: forgejo-runner serviceAccountName: forgejo-runner
containers: containers:
- name: runner - name: runner
image: code.forgejo.org/forgejo/runner:3.5.1 image: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
env: env:
# Use internal k8s service via Tailscale operator egress # Use internal k8s service via Tailscale operator egress
- name: FORGEJO_INSTANCE_URL - name: FORGEJO_INSTANCE_URL

83
plans/ci-cd-bootstrap/00_overview.md
View file

@ -38,11 +38,11 @@ This plan details the setup of Forgejo Actions as the CI/CD system for blumeops,
│ KUBERNETES (minikube) │ │ KUBERNETES (minikube) │
│ ┌─────────────────────┐ ┌─────────────────────┐ │ │ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ Forgejo Runner │ │ Other Services │ │ │ │ Forgejo Runner │ │ Other Services │ │
│ │ (act_runner) │ │ (via ArgoCD) │ │ │ │ (host mode) │ │ (via ArgoCD) │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ - Polls Forgejo │ │ │ │ │ │ - Custom image │ │ │ │
│ │ - Runs workflows │ │ │ │ │ │ - Node.js + tools │ │ │ │
│ │ - Docker-in-Docker │ │ │ │ │ │ - Docker builds │ │ │ │
│ └─────────────────────┘ └─────────────────────┘ │ │ └─────────────────────┘ └─────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────────────┘
``` ```
@ -51,28 +51,50 @@ This plan details the setup of Forgejo Actions as the CI/CD system for blumeops,
| Phase | Name | Description | Status | | Phase | Name | Description | Status |
|-------|------|-------------|--------| |-------|------|-------------|--------|
| 1 | [Enable Actions](P1_enable_actions.md) | Configure Forgejo for Actions, deploy runner | ✅ Complete | | 1 | [Enable Actions](P1_enable_actions.md) | Configure Forgejo for Actions, deploy runner in host mode | ✅ Complete |
| 2 | [Mirror & Build](P2_mirror_and_build.md) | Mirror upstream Forgejo, create build workflow | Planning | | 2 | [Custom Runner Image](P2_mirror_and_build.md) | Build custom runner with Node.js/tools, enable standard Actions | ✅ Complete |
| 3 | [Self-Deploy](P3_self_deploy.md) | Forgejo deploys itself, transition to mcquack | Planning | | 3 | [Mirror Forgejo & Build](P3_mirror_forgejo.md) | Mirror upstream Forgejo, create build workflow | Planning |
| 4 | [Container Builds](P4_container_builds.md) | Build custom container images, runner observability | Planning | | 4 | [Self-Deploy](P4_self_deploy.md) | Forgejo deploys itself, transition to mcquack | Planning |
| 5 | [Container Builds](P5_container_builds.md) | Build custom container images (devpi, etc.) | Planning |
## The Bootstrap Problem ## The Bootstrap Problem
**Chicken-and-egg**: We need Forgejo Actions to build Forgejo, but Forgejo must be running first. **Chicken-and-egg**: We need Forgejo Actions to build Forgejo, but Forgejo must be running first.
**Additional complication**: The stock runner image lacks Node.js, so standard GitHub Actions don't work.
**Solution**: **Solution**:
1. Keep current brew-based Forgejo running during setup 1. Keep current brew-based Forgejo running during setup ✅
2. Enable Actions, deploy runner 2. Enable Actions, deploy runner in host mode ✅
3. Mirror upstream Forgejo, create build workflow 3. **Build custom runner image** with Node.js and tools (bootstrap manually, then automate)
4. First CI build creates the binary 4. Mirror upstream Forgejo, create build workflow
5. CI deploys binary to indri as mcquack service 5. Address cross-compilation challenge (Linux runner → macOS target)
6. `brew services stop forgejo` and uninstall 6. First CI build creates the binary
7. Future builds: Forgejo builds and deploys itself 7. CI deploys binary to indri as mcquack service
8. `brew services stop forgejo` and uninstall
9. Future builds: Forgejo builds and deploys itself
**Cross-compilation challenge**:
The runner runs in Linux containers (k8s), but Forgejo needs to run on indri (macOS ARM64). Options:
- Cross-compile with CGO_ENABLED=1 (complex, needs OSX toolchain)
- Cross-compile with CGO_ENABLED=0 (breaks Tailscale DNS resolution)
- Build on gilbert manually, use CI only for deploy
- Run a native macOS runner on indri (outside k8s)
This will be addressed in Phase 3.
**Risk mitigation**: If self-deployment breaks Forgejo: **Risk mitigation**: If self-deployment breaks Forgejo:
- blumeops is mirrored to GitHub - blumeops is mirrored to GitHub
- Manual recovery: build on gilbert, scp to indri, restart service - Manual recovery: build on gilbert, scp to indri, restart service
- See Disaster Recovery section in P3 - See Disaster Recovery section in P4
## Host Mode Runner
The runner uses **host mode** (`ubuntu-latest:host`), meaning:
- Jobs run directly in the runner container (no Docker/k8s pods spawned)
- Tools must be pre-installed in the runner image
- Stock image lacks Node.js, so `actions/checkout@v4` doesn't work
- Solution: Build custom runner image with necessary tools (Phase 2)
## Ansible Role Strategy ## Ansible Role Strategy
@ -87,7 +109,7 @@ Ansible does NOT:
- Deploy new versions (that's CI's job) - Deploy new versions (that's CI's job)
Ansible DOES: Ansible DOES:
- Manage app.ini configuration (sans secrets) - Manage app.ini configuration (via template with secrets from 1Password)
- Manage mcquack LaunchAgent plist - Manage mcquack LaunchAgent plist
- Ensure service is running - Ensure service is running
- Collect logs via Alloy - Collect logs via Alloy
@ -98,25 +120,27 @@ Ansible DOES:
| Path | Purpose | | Path | Purpose |
|------|---------| |------|---------|
| `argocd/apps/forgejo-runner.yaml` | ArgoCD Application for runner | | `argocd/apps/forgejo-runner.yaml` | ArgoCD Application for runner ✅ |
| `argocd/manifests/forgejo-runner/` | Runner k8s manifests | | `argocd/manifests/forgejo-runner/` | Runner k8s manifests ✅ |
| `.forgejo/workflows/build-forgejo.yml` | Build workflow in blumeops repo | | `argocd/manifests/forgejo-runner/Dockerfile` | Custom runner image (P2) |
| (on forge) `eblume/forgejo/.forgejo/workflows/` | Build workflow in forgejo mirror | | `.forgejo/workflows/build-runner.yml` | Auto-rebuild runner image (P2) |
| `.forgejo/workflows/test.yml` | Test workflow ✅ |
| (on forge) `eblume/forgejo/.forgejo/workflows/` | Build workflow in forgejo mirror (P3) |
### Modified Files ### Modified Files
| Path | Change | | Path | Change |
|------|--------| |------|--------|
| `ansible/roles/forgejo/` | Complete rewrite for mcquack pattern | | `ansible/roles/forgejo/` | Complete rewrite for mcquack pattern (P4) |
| `ansible/roles/alloy/defaults/main.yml` | Update forgejo log paths | | `ansible/roles/alloy/defaults/main.yml` | Update forgejo log paths (P4) |
| zk cards | Update forgejo, argocd, blumeops cards | | zk cards | Update forgejo, argocd, blumeops cards |
### Credentials Needed ### Credentials Needed
| Item | Purpose | Storage | | Item | Purpose | Storage |
|------|---------|---------| |------|---------|---------|
| Runner registration token | Runner auth to Forgejo | 1Password | | Runner registration token | Runner auth to Forgejo | 1Password ✅ |
| SSH deploy key | Runner SSH to indri | 1Password + k8s secret | | SSH deploy key | Runner SSH to indri (for Forgejo deploy) | 1Password + k8s secret (P3) |
## Related Plans ## Related Plans
@ -125,6 +149,15 @@ Ansible DOES:
## Decision Log ## Decision Log
### 2026-01-23: Custom runner image as Phase 2
**Decision**: Move custom runner image work from P4 to P2
**Rationale**:
- Stock runner lacks Node.js, can't run `actions/checkout@v4`
- Need working GitHub Actions before building Forgejo
- Bootstrap manually (podman build on gilbert), then automate
### 2026-01-23: Forgejo Actions over Woodpecker ### 2026-01-23: Forgejo Actions over Woodpecker
**Decision**: Use Forgejo Actions instead of Woodpecker CI **Decision**: Use Forgejo Actions instead of Woodpecker CI

539
plans/ci-cd-bootstrap/P2_mirror_and_build.md
View file

@ -1,138 +1,188 @@
# Phase 2: Mirror Forgejo & Create Build Workflow # Phase 2: Custom Runner Image
**Goal**: Mirror upstream Forgejo to forge and create a workflow that builds it from source **Goal**: Build a custom forgejo-runner image with necessary tools, enabling standard GitHub Actions
**Status**: Planning **Status**: Complete (2026-01-23)
**Prerequisites**: [Phase 1](P1_enable_actions.md) complete (Actions enabled, runner deployed) **Prerequisites**: [Phase 1](P1_enable_actions.md) complete (Actions enabled, runner deployed in host mode)
--- ---
## Current State ## Problem Statement
- Forgejo Actions enabled with runner in k8s The stock `code.forgejo.org/forgejo/runner:3.5.1` image lacks tools needed for standard GitHub Actions:
- Upstream Forgejo at https://codeberg.org/forgejo/forgejo - **Node.js** - Required by most actions (checkout, setup-*, etc.)
- No local mirror yet - **Git** - For repository operations (present but minimal)
- **Common build tools** - make, gcc, curl, jq, etc.
In host mode, jobs run directly in the runner container, so these tools must be pre-installed.
### Chicken-and-Egg Problem
We can't use `actions/checkout@v4` to build the custom runner because that action requires Node.js, which we don't have yet. Solution: Bootstrap manually, then automate.
--- ---
## Step 1: Mirror Upstream Forgejo ## Step 1: Create Dockerfile for Custom Runner
### 1.1 User Action: Create Mirror on Forge Create `argocd/manifests/forgejo-runner/Dockerfile`:
**Manual step** (hairpinning doesn't work from indri): ```dockerfile
FROM code.forgejo.org/forgejo/runner:3.5.1
1. Go to https://forge.tail8d86e.ts.net # The base image is Debian-based
2. Click "+" → "New Migration" # Install tools needed for GitHub Actions and builds
3. Select "Gitea" as clone source RUN apt-get update && apt-get install -y --no-install-recommends \
4. URL: `https://codeberg.org/forgejo/forgejo.git` # Required for actions/checkout and other Node-based actions
5. Repository name: `forgejo` nodejs \
6. Check "This repository will be a mirror" npm \
7. Click "Migrate Repository" # Build essentials
git \
curl \
wget \
jq \
make \
gcc \
g++ \
# For container builds (if we add Docker-in-Docker later)
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
### 1.2 Clone Mirror Locally # Verify Node.js is available
RUN node --version && npm --version
```bash
git clone ssh://forgejo@forge.tail8d86e.ts.net/eblume/forgejo.git ~/code/3rd/forgejo
cd ~/code/3rd/forgejo
``` ```
--- ---
## Step 2: Understand Forgejo Build Process ## Step 2: Bootstrap - Build Image Manually
### 2.1 Build Requirements Since we can't use CI yet, build the image manually on gilbert and push to zot.
From Forgejo's `Makefile` and docs: ### 2.1 Build with Podman
- **Go**: 1.23+ (check `go.mod` for exact version)
- **Node.js**: 20+ (for frontend)
- **Make**: GNU Make
- **Git**: For version embedding
### 2.2 Build Commands
```bash ```bash
# Install frontend dependencies and build cd ~/code/personal/blumeops/argocd/manifests/forgejo-runner
make deps-frontend
make frontend
# Build backend # Build for linux/arm64 (minikube on M1 Mac)
TAGS="bindata sqlite sqlite_unlock_notify" make backend podman build --platform linux/arm64 -t registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest .
# Or all-in-one # Push to zot (no auth required)
TAGS="bindata sqlite sqlite_unlock_notify" make build podman push registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
``` ```
### 2.3 Output ### 2.2 Verify Image in Registry
Binary at `gitea` (yes, the binary is still named `gitea` for compatibility). ```bash
curl -s https://registry.tail8d86e.ts.net/v2/blumeops/forgejo-runner/tags/list | jq .
```
--- ---
## Step 3: Create Build Workflow ## Step 3: Update Runner Deployment
### 3.1 SSH Deploy Key for Runner ### 3.1 Update deployment.yaml
The runner needs SSH access to indri to deploy the binary. Change the image from stock to custom:
**Generate key on gilbert**:
```bash
ssh-keygen -t ed25519 -C "forgejo-runner-deploy" -f ~/.ssh/forgejo-runner-deploy
```
**Add public key to indri's authorized_keys**:
```bash
cat ~/.ssh/forgejo-runner-deploy.pub | ssh indri 'cat >> ~/.ssh/authorized_keys'
```
**Store private key in 1Password** (blumeops vault) as "Forgejo Runner Deploy Key"
**Add to k8s as secret**:
Create `argocd/manifests/forgejo-runner/secret-ssh.yaml.tpl`:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: forgejo-runner-ssh
namespace: forgejo-runner
type: Opaque
stringData:
id_ed25519: |
op://blumeops/<deploy-key-item>/private-key
known_hosts: |
indri.tail8d86e.ts.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIxxxxxx
```
Get indri's host key:
```bash
ssh-keyscan indri.tail8d86e.ts.net 2>/dev/null | grep ed25519
```
### 3.2 Create Workflow File
Create `.forgejo/workflows/build.yml` in the forgejo mirror repo:
```yaml ```yaml
name: Build Forgejo # Before
image: code.forgejo.org/forgejo/runner:3.5.1
# After
image: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
```
### 3.2 Update kustomization.yaml
Add Dockerfile to resources (for reference, not deployed):
```yaml
# Note: Dockerfile is for building, not k8s deployment
# It lives here for co-location with the runner manifests
```
### 3.3 Sync Deployment
```bash
argocd app sync forgejo-runner
# Verify new image is running
kubectl --context=minikube-indri -n forgejo-runner get pods -o jsonpath='{.items[*].spec.containers[*].image}'
```
---
## Step 4: Test with Real GitHub Action
Now that we have Node.js, test with `actions/checkout@v4`.
### 4.1 Update Test Workflow
Update `.forgejo/workflows/test.yml`:
```yaml
name: Test CI
on: on:
push: push:
tags: branches: [main]
- 'v*' pull_request:
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Verify tools
run: |
echo "Node.js: $(node --version)"
echo "npm: $(npm --version)"
echo "Git: $(git --version)"
echo "Make: $(make --version | head -1)"
- name: Show repo info
run: |
echo "Repository: ${{ github.repository }}"
echo "Branch: ${{ github.ref_name }}"
ls -la
```
### 4.2 Push and Verify
```bash
git add .forgejo/workflows/test.yml
git commit -m "Test checkout action with custom runner"
git push
```
Check https://forge.tail8d86e.ts.net/eblume/blumeops/actions - should see successful run with `actions/checkout@v4`.
---
## Step 5: Create Auto-Build Workflow for Runner
Now that Actions work properly, create a workflow to rebuild the runner image automatically.
### 5.1 Create Build Workflow
Create `.forgejo/workflows/build-runner.yml`:
```yaml
name: Build Runner Image
on:
push:
paths:
- 'argocd/manifests/forgejo-runner/Dockerfile'
- '.forgejo/workflows/build-runner.yml'
workflow_dispatch: workflow_dispatch:
inputs:
deploy:
description: 'Deploy to indri after build'
required: false
default: 'true'
type: boolean
env: env:
GOPROXY: "https://proxy.golang.org,direct" REGISTRY: registry.tail8d86e.ts.net
CGO_ENABLED: "1" IMAGE_NAME: blumeops/forgejo-runner
TAGS: "bindata sqlite sqlite_unlock_notify"
jobs: jobs:
build: build:
@ -140,237 +190,158 @@ jobs:
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v4
with:
fetch-depth: 0 # Need full history for version
- name: Setup Go - name: Build image
uses: actions/setup-go@v5
with:
go-version-file: 'go.mod'
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Get version
id: version
run: | run: |
if [[ "${{ github.ref }}" == refs/tags/* ]]; then cd argocd/manifests/forgejo-runner
VERSION="${{ github.ref_name }}" # Use docker build (available in runner container)
else # Note: This builds for the runner's native arch
VERSION="$(git describe --tags --always)-dev" docker build -t ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} .
fi docker tag ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \
echo "version=$VERSION" >> "$GITHUB_OUTPUT" ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
echo "Building version: $VERSION"
- name: Build frontend - name: Push to registry
run: | run: |
make deps-frontend # Zot has no auth, just push
make frontend docker push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
docker push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
- name: Build backend - name: Verify push
run: | run: |
TAGS="${{ env.TAGS }}" make backend curl -sf "https://${{ env.REGISTRY }}/v2/${{ env.IMAGE_NAME }}/tags/list" | jq .
./gitea --version echo "Image pushed: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}"
- name: Rename binary
run: |
mv gitea forgejo-${{ steps.version.outputs.version }}-darwin-arm64
ls -la forgejo-*
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: forgejo-${{ steps.version.outputs.version }}-darwin-arm64
path: forgejo-${{ steps.version.outputs.version }}-darwin-arm64
deploy:
needs: build
runs-on: ubuntu-latest
if: github.event_name == 'push' || (github.event_name == 'workflow_dispatch' && github.event.inputs.deploy == 'true')
steps:
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: forgejo-${{ needs.build.outputs.version }}-darwin-arm64
- name: Setup SSH
run: |
mkdir -p ~/.ssh
echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
echo "${{ secrets.DEPLOY_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
- name: Deploy to indri
run: |
BINARY="forgejo-*-darwin-arm64"
chmod +x $BINARY
# Copy binary to indri
scp $BINARY erichblume@indri.tail8d86e.ts.net:~/.local/bin/forgejo-new
# Atomic swap and restart
ssh erichblume@indri.tail8d86e.ts.net << 'EOF'
set -e
cd ~/.local/bin
# Verify the new binary runs
./forgejo-new --version
# Stop current service
launchctl unload ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist 2>/dev/null || true
# Atomic swap
mv forgejo forgejo-old 2>/dev/null || true
mv forgejo-new forgejo
# Start new service
launchctl load ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist
# Verify it's running
sleep 5
curl -sf http://localhost:3001/api/v1/version || exit 1
echo "Deploy successful!"
./forgejo --version
EOF
``` ```
### 3.3 Add Repository Secrets ### 5.2 Note on Docker-in-Docker
In Forgejo, go to the forgejo repo → Settings → Actions → Secrets: The runner runs in host mode, so we need Docker CLI available. Options:
1. **DEPLOY_SSH_KEY**: Private key from 1Password 1. **Add Docker CLI to the custom image** (see Dockerfile update below)
2. **DEPLOY_KNOWN_HOSTS**: Output of `ssh-keyscan indri.tail8d86e.ts.net` 2. **Mount Docker socket from minikube** (requires deployment change)
3. **Use Podman instead** (rootless, no socket needed)
For now, we'll add Docker CLI to the image and mount the socket.
### 5.3 Update Dockerfile for Docker Builds
```dockerfile
FROM code.forgejo.org/forgejo/runner:3.5.1
RUN apt-get update && apt-get install -y --no-install-recommends \
nodejs \
npm \
git \
curl \
wget \
jq \
make \
gcc \
g++ \
ca-certificates \
# Docker CLI for building container images
docker.io \
&& rm -rf /var/lib/apt/lists/*
RUN node --version && npm --version && docker --version
```
### 5.4 Update Deployment for Docker Socket
Add Docker socket mount to `deployment.yaml`:
```yaml
volumeMounts:
- name: runner-data
mountPath: /data
- name: runner-config
mountPath: /config
- name: docker-sock
mountPath: /var/run/docker.sock
volumes:
- name: runner-data
emptyDir: {}
- name: runner-config
configMap:
name: forgejo-runner-config
- name: docker-sock
hostPath:
path: /var/run/docker.sock
type: Socket
```
--- ---
## Step 4: Build Cross-Platform Consideration ## Step 6: Verification
**Important**: The runner runs Linux containers, but indri is macOS ARM64. ### 6.1 Manual Image Build Works
**Options**:
### Option A: Cross-compile (Simpler, may have issues)
Add to build job:
```yaml
env:
GOOS: darwin
GOARCH: arm64
```
CGO cross-compilation is tricky. May need to disable CGO or use a cross-compiler.
### Option B: Build on macOS (More reliable)
Run a macOS runner on indri itself (not in k8s).
```bash ```bash
# Install forgejo-runner on indri via mise # On gilbert
ssh indri 'mise use forgejo-runner' podman build --platform linux/arm64 -t registry.tail8d86e.ts.net/blumeops/forgejo-runner:test .
podman push registry.tail8d86e.ts.net/blumeops/forgejo-runner:test
# Register as a macOS runner
ssh indri 'forgejo-runner register --labels "macos-arm64:host" ...'
``` ```
Then workflow uses: ### 6.2 Runner Uses Custom Image
```yaml
runs-on: macos-arm64
```
**Recommendation**: Option B is more reliable for native macOS builds. Consider deploying a runner directly on indri for macOS-specific builds.
---
## Step 5: Test the Build
### 5.1 Manual Workflow Dispatch
1. Go to https://forge.tail8d86e.ts.net/eblume/forgejo/actions
2. Select "Build Forgejo" workflow
3. Click "Run workflow"
4. Set deploy=false for first test
5. Monitor the run
### 5.2 Verify Artifact
Download the artifact from the workflow run and verify it's a valid binary:
```bash ```bash
# If downloaded to gilbert kubectl --context=minikube-indri -n forgejo-runner get pods -o jsonpath='{.items[*].spec.containers[*].image}'
file forgejo-*-darwin-arm64 # Should show: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
# Should show: Mach-O 64-bit executable arm64
``` ```
--- ### 6.3 GitHub Actions Work
## Alternative: Build on Gilbert, Deploy via CI - `actions/checkout@v4` succeeds
- Test workflow shows Node.js, npm, git versions
If cross-compilation proves difficult, consider a hybrid approach: ### 6.4 Auto-Build Workflow Works
1. **Build on gilbert** (has Go, Node, is macOS ARM64) Push a change to the Dockerfile and verify:
2. **CI just deploys** the built binary 1. Workflow triggers
2. Image builds successfully
Workflow in blumeops repo: 3. Image pushed to zot
```yaml
name: Deploy Forgejo
on:
workflow_dispatch:
inputs:
binary_path:
description: 'Path to binary on gilbert'
required: true
jobs:
deploy:
runs-on: ubuntu-latest
steps:
# Fetch binary from gilbert and deploy to indri
# (requires SSH access to both)
```
This is less elegant but more pragmatic for macOS targets.
--- ---
## Verification Checklist ## Verification Checklist
- [ ] Forgejo mirrored to forge - [x] Dockerfile created for custom runner (Alpine-based with apk)
- [ ] SSH deploy key created and stored in 1Password - [x] Image built manually on gilbert (podman build)
- [ ] Deploy key added to indri authorized_keys - [x] Image pushed to zot registry
- [ ] SSH secret added to k8s - [x] Runner deployment updated to use custom image
- [ ] Workflow file created in forgejo mirror - [x] Runner pod running with new image
- [ ] Repository secrets configured - [x] `actions/checkout@v4` works in test workflow
- [ ] Test build completes successfully - [ ] Auto-build workflow created (deferred - needs Docker socket)
- [ ] Binary is valid macOS ARM64 executable - [ ] Docker socket mounted (for container builds)
- [ ] Auto-build workflow successfully rebuilds runner
--- ---
## Troubleshooting ## Troubleshooting
### CGO Cross-Compilation Fails ### Image Pull Fails in Minikube
If building Linux→macOS fails: Minikube needs to be able to pull from zot. Check registry mirror config:
``` ```bash
# runtime/cgo ssh indri 'minikube ssh -- cat /etc/containerd/certs.d/registry.tail8d86e.ts.net/hosts.toml'
gcc: error: unrecognized command line option '-arch'
``` ```
Either: ### Docker Build Fails in Workflow
1. Use Option B (macOS runner on indri)
2. Build with `CGO_ENABLED=0` (loses some features)
3. Use a Docker image with macOS cross-compiler (complex)
### Artifact Too Large If Docker socket mount doesn't work:
1. Check socket exists in minikube: `minikube ssh -- ls -la /var/run/docker.sock`
2. Check permissions: runner may need to be in docker group
3. Alternative: Use `podman` (rootless) instead of Docker
Forgejo binary is ~100MB. If upload fails: ### Node.js Actions Still Fail
- Check Forgejo's artifact size limit in app.ini
- Consider compressing: `gzip -9 forgejo-*` Ensure the runner pod restarted after image update:
```bash
kubectl --context=minikube-indri -n forgejo-runner rollout restart deployment/forgejo-runner
kubectl --context=minikube-indri -n forgejo-runner logs -f deployment/forgejo-runner
```
--- ---
## Next Phase ## Next Phase
Once build is working and produces valid binaries, proceed to [Phase 3: Self-Deploy](P3_self_deploy.md). Once the custom runner is working with auto-build, proceed to [Phase 3: Mirror Forgejo & Build](P3_mirror_and_build.md) to set up Forgejo source builds.

349
plans/ci-cd-bootstrap/P3_mirror_forgejo.md Normal file
View file

@ -0,0 +1,349 @@
# Phase 3: Mirror Forgejo & Build from Source
**Goal**: Mirror upstream Forgejo to forge and create a workflow that builds it for macOS ARM64
**Status**: Planning
**Prerequisites**: [Phase 2](P2_mirror_and_build.md) complete (custom runner image with Node.js/tools)
---
## Problem Statement
We want to build Forgejo from source to:
1. Have full control over the binary running on indri
2. Enable self-deployment via CI
3. Ensure proper macOS DNS resolution (requires CGO_ENABLED=1)
### The Cross-Compilation Challenge
The runner runs in a Linux container (k8s on indri), but the target is macOS ARM64 (indri itself).
**Options**:
| Option | Pros | Cons |
|--------|------|------|
| A. Cross-compile CGO_ENABLED=0 | Simple, no special toolchain | Breaks Tailscale MagicDNS resolution |
| B. Cross-compile CGO_ENABLED=1 | Proper DNS | Needs OSX cross-compiler (osxcross), complex |
| C. Build on gilbert manually | Works now, simple | Not automated, manual step |
| D. Native macOS runner on indri | Full native build | Runner outside k8s, different architecture |
| E. Hybrid: build on gilbert, deploy via CI | Uses existing tools | Partial automation |
**Recommendation**: Start with Option C/E (manual build on gilbert, CI just deploys), then consider Option D if we want full automation.
---
## Step 1: Mirror Upstream Forgejo
### 1.1 User Action: Create Mirror on Forge
**Manual step** (hairpinning doesn't work from indri):
1. Go to https://forge.tail8d86e.ts.net
2. Click "+" → "New Migration"
3. Select "Gitea" as clone source
4. URL: `https://codeberg.org/forgejo/forgejo.git`
5. Repository name: `forgejo`
6. Check "This repository will be a mirror"
7. Click "Migrate Repository"
### 1.2 Clone Mirror Locally
```bash
git clone ssh://forgejo@forge.tail8d86e.ts.net/eblume/forgejo.git ~/code/3rd/forgejo
cd ~/code/3rd/forgejo
```
---
## Step 2: Understand Forgejo Build Process
### 2.1 Build Requirements
From Forgejo's `Makefile` and docs:
- **Go**: 1.23+ (check `go.mod` for exact version)
- **Node.js**: 20+ (for frontend)
- **Make**: GNU Make
- **Git**: For version embedding
### 2.2 Build Commands
```bash
# Install frontend dependencies and build
make deps-frontend
make frontend
# Build backend (with CGO for proper DNS on macOS)
CGO_ENABLED=1 TAGS="bindata sqlite sqlite_unlock_notify" make backend
# Or all-in-one
CGO_ENABLED=1 TAGS="bindata sqlite sqlite_unlock_notify" make build
```
### 2.3 Output
Binary at `gitea` (yes, the binary is still named `gitea` for compatibility).
---
## Step 3: Build on Gilbert (Manual Bootstrap)
For the initial bootstrap, build on gilbert (macOS ARM64 native).
### 3.1 Setup Build Environment
```bash
cd ~/code/3rd/forgejo
mise use go@1.23 node@20
# Verify tools
go version
node --version
make --version
```
### 3.2 Build
```bash
# Clean build
make clean
# Build frontend
make deps-frontend
make frontend
# Build backend with CGO (important for macOS DNS!)
CGO_ENABLED=1 TAGS="bindata sqlite sqlite_unlock_notify" make backend
# Verify binary
./gitea --version
file gitea # Should show: Mach-O 64-bit executable arm64
```
### 3.3 Deploy to Indri
```bash
# Copy binary
scp gitea indri:~/.local/bin/forgejo-new
# Verify on indri
ssh indri '~/.local/bin/forgejo-new --version'
```
---
## Step 4: Create Deploy Workflow (Option E)
Since cross-compilation is complex, use a hybrid approach:
1. Build on gilbert (manual trigger or pre-built)
2. CI workflow fetches and deploys
### 4.1 SSH Deploy Key for Runner
The runner needs SSH access to indri to deploy the binary.
**Generate key on gilbert**:
```bash
ssh-keygen -t ed25519 -C "forgejo-runner-deploy" -f ~/.ssh/forgejo-runner-deploy -N ""
```
**Add public key to indri's authorized_keys**:
```bash
cat ~/.ssh/forgejo-runner-deploy.pub | ssh indri 'cat >> ~/.ssh/authorized_keys'
```
**Store private key in 1Password** (blumeops vault) as "Forgejo Runner Deploy Key"
### 4.2 Create k8s Secret
Create `argocd/manifests/forgejo-runner/secret-ssh.yaml.tpl`:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: forgejo-runner-ssh
namespace: forgejo-runner
type: Opaque
stringData:
id_ed25519: |
op://blumeops/<deploy-key-item>/private-key
known_hosts: |
# Get with: ssh-keyscan indri.tail8d86e.ts.net 2>/dev/null | grep ed25519
indri.tail8d86e.ts.net ssh-ed25519 AAAAC3...
```
### 4.3 Update Deployment for SSH
Add SSH secret mount to `deployment.yaml`:
```yaml
volumeMounts:
- name: ssh-key
mountPath: /root/.ssh
readOnly: true
volumes:
- name: ssh-key
secret:
secretName: forgejo-runner-ssh
defaultMode: 0600
```
### 4.4 Create Deploy-Only Workflow
Create `.forgejo/workflows/deploy-forgejo.yml` in blumeops:
```yaml
name: Deploy Forgejo
on:
workflow_dispatch:
inputs:
version:
description: 'Version to deploy (tag or commit)'
required: true
default: 'v10.0.0'
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Deploy to indri
env:
VERSION: ${{ github.event.inputs.version }}
run: |
# SSH config
mkdir -p ~/.ssh
cp /root/.ssh/id_ed25519 ~/.ssh/
cp /root/.ssh/known_hosts ~/.ssh/
chmod 600 ~/.ssh/id_ed25519
# Deploy script
ssh erichblume@indri.tail8d86e.ts.net << 'EOF'
set -e
cd ~/.local/bin
# Verify the new binary exists and runs
if [ ! -f forgejo-new ]; then
echo "ERROR: forgejo-new not found. Build on gilbert first:"
echo " cd ~/code/3rd/forgejo && git checkout $VERSION"
echo " CGO_ENABLED=1 TAGS='bindata sqlite sqlite_unlock_notify' make build"
echo " scp gitea indri:~/.local/bin/forgejo-new"
exit 1
fi
./forgejo-new --version
# Stop current service
launchctl unload ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist 2>/dev/null || true
# Atomic swap
mv forgejo forgejo-old 2>/dev/null || true
mv forgejo-new forgejo
# Start new service
launchctl load ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist
# Verify it's running
sleep 5
curl -sf http://localhost:3001/api/v1/version || exit 1
echo "Deploy successful!"
./forgejo --version
EOF
```
---
## Future: Full CI Build (Option D)
If we want full automation, consider running a native macOS runner on indri:
### Native Runner on Indri
```bash
# Install forgejo-runner on indri via mise
ssh indri 'mise use forgejo-runner'
# Register as a macOS runner
ssh indri 'forgejo-runner register \
--instance https://forge.tail8d86e.ts.net \
--token "$TOKEN" \
--name "indri-native" \
--labels "macos-arm64:host" \
--no-interactive'
# Create LaunchAgent for runner
# (similar to other mcquack services)
```
Then workflow uses:
```yaml
runs-on: macos-arm64
```
This enables full native builds in CI. Document in a future phase if needed.
---
## Verification Checklist
- [ ] Forgejo mirrored to forge
- [ ] Mirror cloned to ~/code/3rd/forgejo
- [ ] Build succeeds on gilbert
- [ ] Binary is valid macOS ARM64 executable
- [ ] Binary deployed to indri ~/.local/bin/
- [ ] SSH deploy key created and stored in 1Password
- [ ] Deploy key added to indri authorized_keys
- [ ] (Optional) k8s SSH secret created
- [ ] (Optional) Deploy workflow created
---
## Troubleshooting
### Build Fails: Node.js Version
```
error: engine "node" is incompatible
```
Update Node.js: `mise use node@20`
### Build Fails: Go Version
```
go: go.mod requires go >= 1.23
```
Update Go: `mise use go@1.23`
### Binary Crashes on indri
Check if CGO was enabled:
```bash
# If built without CGO, DNS resolution may fail
./forgejo --version # Should work
./forgejo web # May fail to resolve Tailscale hostnames
```
Rebuild with `CGO_ENABLED=1`.
### SSH Deploy Fails
Check runner has SSH access:
```bash
# Test from inside runner pod
kubectl --context=minikube-indri -n forgejo-runner exec deployment/forgejo-runner -- \
ssh -i /root/.ssh/id_ed25519 erichblume@indri.tail8d86e.ts.net 'echo ok'
```
---
## Next Phase
Once Forgejo is building and deploying successfully, proceed to [Phase 4: Self-Deploy](P4_self_deploy.md) for the full mcquack transition.

6
plans/ci-cd-bootstrap/P3_self_deploy.md → plans/ci-cd-bootstrap/P4_self_deploy.md
View file

@ -1,10 +1,10 @@
# Phase 3: Self-Deploy & Transition to mcquack # Phase 4: Self-Deploy & Transition to mcquack
**Goal**: Complete the bootstrap - Forgejo deploys itself, transition from brew to mcquack LaunchAgent **Goal**: Complete the bootstrap - Forgejo deploys itself, transition from brew to mcquack LaunchAgent
**Status**: Planning **Status**: Planning
**Prerequisites**: [Phase 2](P2_mirror_and_build.md) complete (build workflow produces valid binaries) **Prerequisites**: [Phase 3](P3_mirror_forgejo.md) complete (Forgejo builds and deploys to indri)
--- ---
@ -406,4 +406,4 @@ After recovery, switch back to Forgejo.
## Next Phase ## Next Phase
After bootstrap is complete, proceed to [Phase 4: Container Builds](P4_container_builds.md) to set up container image building for ArgoCD. After bootstrap is complete, proceed to [Phase 5: Container Builds](P5_container_builds.md) to set up container image building for ArgoCD.

78
plans/ci-cd-bootstrap/P4_container_builds.md → plans/ci-cd-bootstrap/P5_container_builds.md
View file

@ -1,91 +1,21 @@
# Phase 4: Container Image Builds # Phase 5: Container Image Builds
**Goal**: Set up CI workflows to build custom container images and push to zot registry **Goal**: Set up CI workflows to build custom container images and push to zot registry
**Status**: Planning **Status**: Planning
**Prerequisites**: [Phase 3](P3_self_deploy.md) complete (Forgejo self-deploying, Actions working) **Prerequisites**: [Phase 4](P4_self_deploy.md) complete (Forgejo self-deploying, Actions working)
--- ---
## Overview ## Overview
With Forgejo Actions operational, we can now build container images for: With Forgejo Actions operational (including custom runner from P2), we can now build container images for:
- Custom devpi with pre-installed plugins - Custom devpi with pre-installed plugins
- Any other custom images needed for k8s services - Any other custom images needed for k8s services
- Release artifacts for Python packages - Release artifacts for Python packages
--- **Note**: The custom runner image build is covered in [Phase 2](P2_mirror_and_build.md). This phase focuses on application container builds.
## Use Case 0: Custom Runner Image
### Problem
The stock `forgejo/runner` image lacks tools needed for standard GitHub Actions:
- **Node.js** - Required by most actions (checkout, setup-*, etc.)
- **Docker CLI** - For building container images
- **Git** - For repository operations
- **Common build tools** - make, gcc, etc.
In host mode, jobs run directly in the runner container, so these tools must be pre-installed.
### Solution
Build a custom runner image with all necessary tools:
```dockerfile
# argocd/manifests/forgejo-runner/Dockerfile
FROM code.forgejo.org/forgejo/runner:3.5.1
# Install Node.js (required for most GitHub Actions)
RUN apt-get update && apt-get install -y \
nodejs \
npm \
git \
curl \
docker.io \
make \
gcc \
&& rm -rf /var/lib/apt/lists/*
```
### Workflow
Create `.forgejo/workflows/build-runner.yml`:
```yaml
name: Build Runner Image
on:
push:
paths:
- 'argocd/manifests/forgejo-runner/Dockerfile'
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
run: |
git clone --depth 1 "${{ gitea.server_url }}/${{ gitea.repository }}.git" .
- name: Build and push
run: |
cd argocd/manifests/forgejo-runner
docker build -t registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest .
docker push registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
```
### Update Deployment
Once the custom image is built, update `argocd/manifests/forgejo-runner/deployment.yaml`:
```yaml
image: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
```
This enables standard GitHub Actions like `actions/checkout@v4` to work in host mode.
--- ---