eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Reorganize CI/CD bootstrap phases and add custom runner Dockerfile #50

Merged
eblume merged 4 commits from feature/p2-mirror-and-build into main 2026-01-23 18:50:28 -08:00
Conversation 0 Commits 4 Files changed 8 +696 -373
8 changed files with 696 additions and 373 deletions
Download patch file Download diff file Expand all files Collapse all files

41
.forgejo/workflows/test.yaml
View file

@ -10,24 +10,35 @@ jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout (git clone)
run: |
# For PRs use head_ref (branch name), for pushes use ref_name
BRANCH="${HEAD_REF:-$REF_NAME}"
git clone --depth 1 --branch "$BRANCH" \
"${SERVER_URL}/${REPOSITORY}.git" .
env:
GIT_SSL_NO_VERIFY: "true"
HEAD_REF: ${{ github.head_ref }}
REF_NAME: ${{ github.ref_name }}
SERVER_URL: ${{ github.server_url }}
REPOSITORY: ${{ github.repository }}
- name: Checkout
uses: actions/checkout@v4
- name: Hello World
- name: Verify tools
run: |
echo "=== Node.js ==="
node --version
npm --version
echo ""
echo "=== Git ==="
git --version
echo ""
echo "=== Build tools ==="
make --version | head -1
gcc --version | head -1
echo ""
echo "=== Docker ==="
docker --version
echo ""
echo "=== Other tools ==="
curl --version | head -1
jq --version
- name: Show repo info
run: |
echo "Hello from Forgejo Actions!"
echo "Runner: $(hostname)"
echo "Repository: ${{ github.repository }}"
echo "Event: ${{ github.event_name }}"
echo "Ref: ${{ github.ref }}"
echo "Branch: ${{ github.ref_name }}"
echo ""
echo "=== Files ==="
ls -la

29
argocd/manifests/forgejo-runner/Dockerfile Normal file
View file

@ -0,0 +1,29 @@
FROM code.forgejo.org/forgejo/runner:3.5.1
# Switch to root to install packages
USER root
# The base image is Alpine Linux
# Install tools needed for GitHub Actions and builds
RUN apk add --no-cache \
# Required for actions/checkout and other Node-based actions
nodejs \
npm \
# Build essentials
git \
curl \
wget \
jq \
make \
gcc \
g++ \
musl-dev \
# For container builds
ca-certificates \
docker-cli
# Verify tools are available
RUN node --version && npm --version && docker --version
# Switch back to non-root user
USER 1000

2
argocd/manifests/forgejo-runner/deployment.yaml
View file

@ -16,7 +16,7 @@ spec:
serviceAccountName: forgejo-runner
containers:
- name: runner
image: code.forgejo.org/forgejo/runner:3.5.1
image: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
env:
# Use internal k8s service via Tailscale operator egress
- name: FORGEJO_INSTANCE_URL

83
plans/ci-cd-bootstrap/00_overview.md
View file

@ -38,11 +38,11 @@ This plan details the setup of Forgejo Actions as the CI/CD system for blumeops,
│ KUBERNETES (minikube) │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ Forgejo Runner │ │ Other Services │ │
│ │ (act_runner) │ │ (via ArgoCD) │ │
│ │ (host mode) │ │ (via ArgoCD) │ │
│ │ │ │ │ │
│ │ - Polls Forgejo │ │ │ │
│ │ - Runs workflows │ │ │ │
│ │ - Docker-in-Docker │ │ │ │
│ │ - Custom image │ │ │ │
│ │ - Node.js + tools │ │ │ │
│ │ - Docker builds │ │ │ │
│ └─────────────────────┘ └─────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
@ -51,28 +51,50 @@ This plan details the setup of Forgejo Actions as the CI/CD system for blumeops,
| Phase | Name | Description | Status |
|-------|------|-------------|--------|
| 1 | [Enable Actions](P1_enable_actions.md) | Configure Forgejo for Actions, deploy runner | ✅ Complete |
| 2 | [Mirror & Build](P2_mirror_and_build.md) | Mirror upstream Forgejo, create build workflow | Planning |
| 3 | [Self-Deploy](P3_self_deploy.md) | Forgejo deploys itself, transition to mcquack | Planning |
| 4 | [Container Builds](P4_container_builds.md) | Build custom container images, runner observability | Planning |
| 1 | [Enable Actions](P1_enable_actions.md) | Configure Forgejo for Actions, deploy runner in host mode | ✅ Complete |
| 2 | [Custom Runner Image](P2_mirror_and_build.md) | Build custom runner with Node.js/tools, enable standard Actions | ✅ Complete |
| 3 | [Mirror Forgejo & Build](P3_mirror_forgejo.md) | Mirror upstream Forgejo, create build workflow | Planning |
| 4 | [Self-Deploy](P4_self_deploy.md) | Forgejo deploys itself, transition to mcquack | Planning |
| 5 | [Container Builds](P5_container_builds.md) | Build custom container images (devpi, etc.) | Planning |
## The Bootstrap Problem
**Chicken-and-egg**: We need Forgejo Actions to build Forgejo, but Forgejo must be running first.
**Additional complication**: The stock runner image lacks Node.js, so standard GitHub Actions don't work.
**Solution**:
1. Keep current brew-based Forgejo running during setup
2. Enable Actions, deploy runner
3. Mirror upstream Forgejo, create build workflow
4. First CI build creates the binary
5. CI deploys binary to indri as mcquack service
6. `brew services stop forgejo` and uninstall
7. Future builds: Forgejo builds and deploys itself
1. Keep current brew-based Forgejo running during setup ✅
2. Enable Actions, deploy runner in host mode ✅
3. **Build custom runner image** with Node.js and tools (bootstrap manually, then automate)
4. Mirror upstream Forgejo, create build workflow
5. Address cross-compilation challenge (Linux runner → macOS target)
6. First CI build creates the binary
7. CI deploys binary to indri as mcquack service
8. `brew services stop forgejo` and uninstall
9. Future builds: Forgejo builds and deploys itself
**Cross-compilation challenge**:
The runner runs in Linux containers (k8s), but Forgejo needs to run on indri (macOS ARM64). Options:
- Cross-compile with CGO_ENABLED=1 (complex, needs OSX toolchain)
- Cross-compile with CGO_ENABLED=0 (breaks Tailscale DNS resolution)
- Build on gilbert manually, use CI only for deploy
- Run a native macOS runner on indri (outside k8s)
This will be addressed in Phase 3.
**Risk mitigation**: If self-deployment breaks Forgejo:
- blumeops is mirrored to GitHub
- Manual recovery: build on gilbert, scp to indri, restart service
- See Disaster Recovery section in P3
- See Disaster Recovery section in P4
## Host Mode Runner
The runner uses **host mode** (`ubuntu-latest:host`), meaning:
- Jobs run directly in the runner container (no Docker/k8s pods spawned)
- Tools must be pre-installed in the runner image
- Stock image lacks Node.js, so `actions/checkout@v4` doesn't work
- Solution: Build custom runner image with necessary tools (Phase 2)
## Ansible Role Strategy
@ -87,7 +109,7 @@ Ansible does NOT:
- Deploy new versions (that's CI's job)
Ansible DOES:
- Manage app.ini configuration (sans secrets)
- Manage app.ini configuration (via template with secrets from 1Password)
- Manage mcquack LaunchAgent plist
- Ensure service is running
- Collect logs via Alloy
@ -98,25 +120,27 @@ Ansible DOES:
| Path | Purpose |
|------|---------|
| `argocd/apps/forgejo-runner.yaml` | ArgoCD Application for runner |
| `argocd/manifests/forgejo-runner/` | Runner k8s manifests |
| `.forgejo/workflows/build-forgejo.yml` | Build workflow in blumeops repo |
| (on forge) `eblume/forgejo/.forgejo/workflows/` | Build workflow in forgejo mirror |
| `argocd/apps/forgejo-runner.yaml` | ArgoCD Application for runner ✅ |
| `argocd/manifests/forgejo-runner/` | Runner k8s manifests ✅ |
| `argocd/manifests/forgejo-runner/Dockerfile` | Custom runner image (P2) |
| `.forgejo/workflows/build-runner.yml` | Auto-rebuild runner image (P2) |
| `.forgejo/workflows/test.yml` | Test workflow ✅ |
| (on forge) `eblume/forgejo/.forgejo/workflows/` | Build workflow in forgejo mirror (P3) |
### Modified Files
| Path | Change |
|------|--------|
| `ansible/roles/forgejo/` | Complete rewrite for mcquack pattern |
| `ansible/roles/alloy/defaults/main.yml` | Update forgejo log paths |
| `ansible/roles/forgejo/` | Complete rewrite for mcquack pattern (P4) |
| `ansible/roles/alloy/defaults/main.yml` | Update forgejo log paths (P4) |
| zk cards | Update forgejo, argocd, blumeops cards |
### Credentials Needed
| Item | Purpose | Storage |
|------|---------|---------|
| Runner registration token | Runner auth to Forgejo | 1Password |
| SSH deploy key | Runner SSH to indri | 1Password + k8s secret |
| Runner registration token | Runner auth to Forgejo | 1Password ✅ |
| SSH deploy key | Runner SSH to indri (for Forgejo deploy) | 1Password + k8s secret (P3) |
## Related Plans
@ -125,6 +149,15 @@ Ansible DOES:
## Decision Log
### 2026-01-23: Custom runner image as Phase 2
**Decision**: Move custom runner image work from P4 to P2
**Rationale**:
- Stock runner lacks Node.js, can't run `actions/checkout@v4`
- Need working GitHub Actions before building Forgejo
- Bootstrap manually (podman build on gilbert), then automate
### 2026-01-23: Forgejo Actions over Woodpecker
**Decision**: Use Forgejo Actions instead of Woodpecker CI

539
plans/ci-cd-bootstrap/P2_mirror_and_build.md
View file

@ -1,138 +1,188 @@
# Phase 2: Mirror Forgejo & Create Build Workflow
# Phase 2: Custom Runner Image
**Goal**: Mirror upstream Forgejo to forge and create a workflow that builds it from source
**Goal**: Build a custom forgejo-runner image with necessary tools, enabling standard GitHub Actions
**Status**: Planning
**Status**: Complete (2026-01-23)
**Prerequisites**: [Phase 1](P1_enable_actions.md) complete (Actions enabled, runner deployed)
**Prerequisites**: [Phase 1](P1_enable_actions.md) complete (Actions enabled, runner deployed in host mode)
---
## Current State
## Problem Statement
- Forgejo Actions enabled with runner in k8s
- Upstream Forgejo at https://codeberg.org/forgejo/forgejo
- No local mirror yet
The stock `code.forgejo.org/forgejo/runner:3.5.1` image lacks tools needed for standard GitHub Actions:
- **Node.js** - Required by most actions (checkout, setup-*, etc.)
- **Git** - For repository operations (present but minimal)
- **Common build tools** - make, gcc, curl, jq, etc.
In host mode, jobs run directly in the runner container, so these tools must be pre-installed.
### Chicken-and-Egg Problem
We can't use `actions/checkout@v4` to build the custom runner because that action requires Node.js, which we don't have yet. Solution: Bootstrap manually, then automate.
---
## Step 1: Mirror Upstream Forgejo
## Step 1: Create Dockerfile for Custom Runner
### 1.1 User Action: Create Mirror on Forge
Create `argocd/manifests/forgejo-runner/Dockerfile`:
**Manual step** (hairpinning doesn't work from indri):
```dockerfile
FROM code.forgejo.org/forgejo/runner:3.5.1
1. Go to https://forge.tail8d86e.ts.net
2. Click "+" → "New Migration"
3. Select "Gitea" as clone source
4. URL: `https://codeberg.org/forgejo/forgejo.git`
5. Repository name: `forgejo`
6. Check "This repository will be a mirror"
7. Click "Migrate Repository"
# The base image is Debian-based
# Install tools needed for GitHub Actions and builds
RUN apt-get update && apt-get install -y --no-install-recommends \
# Required for actions/checkout and other Node-based actions
nodejs \
npm \
# Build essentials
git \
curl \
wget \
jq \
make \
gcc \
g++ \
# For container builds (if we add Docker-in-Docker later)
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
### 1.2 Clone Mirror Locally
```bash
git clone ssh://forgejo@forge.tail8d86e.ts.net/eblume/forgejo.git ~/code/3rd/forgejo
cd ~/code/3rd/forgejo
# Verify Node.js is available
RUN node --version && npm --version
```
---
## Step 2: Understand Forgejo Build Process
## Step 2: Bootstrap - Build Image Manually
### 2.1 Build Requirements
Since we can't use CI yet, build the image manually on gilbert and push to zot.
From Forgejo's `Makefile` and docs:
- **Go**: 1.23+ (check `go.mod` for exact version)
- **Node.js**: 20+ (for frontend)
- **Make**: GNU Make
- **Git**: For version embedding
### 2.2 Build Commands
### 2.1 Build with Podman
```bash
# Install frontend dependencies and build
make deps-frontend
make frontend
cd ~/code/personal/blumeops/argocd/manifests/forgejo-runner
# Build backend
TAGS="bindata sqlite sqlite_unlock_notify" make backend
# Build for linux/arm64 (minikube on M1 Mac)
podman build --platform linux/arm64 -t registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest .
# Or all-in-one
TAGS="bindata sqlite sqlite_unlock_notify" make build
# Push to zot (no auth required)
podman push registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
```
### 2.3 Output
### 2.2 Verify Image in Registry
Binary at `gitea` (yes, the binary is still named `gitea` for compatibility).
```bash
curl -s https://registry.tail8d86e.ts.net/v2/blumeops/forgejo-runner/tags/list | jq .
```
---
## Step 3: Create Build Workflow
## Step 3: Update Runner Deployment
### 3.1 SSH Deploy Key for Runner
### 3.1 Update deployment.yaml
The runner needs SSH access to indri to deploy the binary.
**Generate key on gilbert**:
```bash
ssh-keygen -t ed25519 -C "forgejo-runner-deploy" -f ~/.ssh/forgejo-runner-deploy
```
**Add public key to indri's authorized_keys**:
```bash
cat ~/.ssh/forgejo-runner-deploy.pub | ssh indri 'cat >> ~/.ssh/authorized_keys'
```
**Store private key in 1Password** (blumeops vault) as "Forgejo Runner Deploy Key"
**Add to k8s as secret**:
Create `argocd/manifests/forgejo-runner/secret-ssh.yaml.tpl`:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: forgejo-runner-ssh
namespace: forgejo-runner
type: Opaque
stringData:
id_ed25519: |
op://blumeops/<deploy-key-item>/private-key
known_hosts: |
indri.tail8d86e.ts.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIxxxxxx
```
Get indri's host key:
```bash
ssh-keyscan indri.tail8d86e.ts.net 2>/dev/null | grep ed25519
```
### 3.2 Create Workflow File
Create `.forgejo/workflows/build.yml` in the forgejo mirror repo:
Change the image from stock to custom:
```yaml
name: Build Forgejo
# Before
image: code.forgejo.org/forgejo/runner:3.5.1
# After
image: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
```
### 3.2 Update kustomization.yaml
Add Dockerfile to resources (for reference, not deployed):
```yaml
# Note: Dockerfile is for building, not k8s deployment
# It lives here for co-location with the runner manifests
```
### 3.3 Sync Deployment
```bash
argocd app sync forgejo-runner
# Verify new image is running
kubectl --context=minikube-indri -n forgejo-runner get pods -o jsonpath='{.items[*].spec.containers[*].image}'
```
---
## Step 4: Test with Real GitHub Action
Now that we have Node.js, test with `actions/checkout@v4`.
### 4.1 Update Test Workflow
Update `.forgejo/workflows/test.yml`:
```yaml
name: Test CI
on:
push:
tags:
- 'v*'
branches: [main]
pull_request:
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Verify tools
run: |
echo "Node.js: $(node --version)"
echo "npm: $(npm --version)"
echo "Git: $(git --version)"
echo "Make: $(make --version | head -1)"
- name: Show repo info
run: |
echo "Repository: ${{ github.repository }}"
echo "Branch: ${{ github.ref_name }}"
ls -la
```
### 4.2 Push and Verify
```bash
git add .forgejo/workflows/test.yml
git commit -m "Test checkout action with custom runner"
git push
```
Check https://forge.tail8d86e.ts.net/eblume/blumeops/actions - should see successful run with `actions/checkout@v4`.
---
## Step 5: Create Auto-Build Workflow for Runner
Now that Actions work properly, create a workflow to rebuild the runner image automatically.
### 5.1 Create Build Workflow
Create `.forgejo/workflows/build-runner.yml`:
```yaml
name: Build Runner Image
on:
push:
paths:
- 'argocd/manifests/forgejo-runner/Dockerfile'
- '.forgejo/workflows/build-runner.yml'
workflow_dispatch:
inputs:
deploy:
description: 'Deploy to indri after build'
required: false
default: 'true'
type: boolean
env:
GOPROXY: "https://proxy.golang.org,direct"
CGO_ENABLED: "1"
TAGS: "bindata sqlite sqlite_unlock_notify"
REGISTRY: registry.tail8d86e.ts.net
IMAGE_NAME: blumeops/forgejo-runner
jobs:
build:
@ -140,237 +190,158 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0 # Need full history for version
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version-file: 'go.mod'
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Get version
id: version
- name: Build image
run: |
if [[ "${{ github.ref }}" == refs/tags/* ]]; then
VERSION="${{ github.ref_name }}"
else
VERSION="$(git describe --tags --always)-dev"
fi
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Building version: $VERSION"
cd argocd/manifests/forgejo-runner
# Use docker build (available in runner container)
# Note: This builds for the runner's native arch
docker build -t ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} .
docker tag ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
- name: Build frontend
- name: Push to registry
run: |
make deps-frontend
make frontend
# Zot has no auth, just push
docker push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
docker push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
- name: Build backend
- name: Verify push
run: |
TAGS="${{ env.TAGS }}" make backend
./gitea --version
- name: Rename binary
run: |
mv gitea forgejo-${{ steps.version.outputs.version }}-darwin-arm64
ls -la forgejo-*
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: forgejo-${{ steps.version.outputs.version }}-darwin-arm64
path: forgejo-${{ steps.version.outputs.version }}-darwin-arm64
deploy:
needs: build
runs-on: ubuntu-latest
if: github.event_name == 'push' || (github.event_name == 'workflow_dispatch' && github.event.inputs.deploy == 'true')
steps:
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: forgejo-${{ needs.build.outputs.version }}-darwin-arm64
- name: Setup SSH
run: |
mkdir -p ~/.ssh
echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
echo "${{ secrets.DEPLOY_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
- name: Deploy to indri
run: |
BINARY="forgejo-*-darwin-arm64"
chmod +x $BINARY
# Copy binary to indri
scp $BINARY erichblume@indri.tail8d86e.ts.net:~/.local/bin/forgejo-new
# Atomic swap and restart
ssh erichblume@indri.tail8d86e.ts.net << 'EOF'
set -e
cd ~/.local/bin
# Verify the new binary runs
./forgejo-new --version
# Stop current service
launchctl unload ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist 2>/dev/null || true
# Atomic swap
mv forgejo forgejo-old 2>/dev/null || true
mv forgejo-new forgejo
# Start new service
launchctl load ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist
# Verify it's running
sleep 5
curl -sf http://localhost:3001/api/v1/version || exit 1
echo "Deploy successful!"
./forgejo --version
EOF
curl -sf "https://${{ env.REGISTRY }}/v2/${{ env.IMAGE_NAME }}/tags/list" | jq .
echo "Image pushed: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}"
```
### 3.3 Add Repository Secrets
### 5.2 Note on Docker-in-Docker
In Forgejo, go to the forgejo repo → Settings → Actions → Secrets:
The runner runs in host mode, so we need Docker CLI available. Options:
1. **DEPLOY_SSH_KEY**: Private key from 1Password
2. **DEPLOY_KNOWN_HOSTS**: Output of `ssh-keyscan indri.tail8d86e.ts.net`
1. **Add Docker CLI to the custom image** (see Dockerfile update below)
2. **Mount Docker socket from minikube** (requires deployment change)
3. **Use Podman instead** (rootless, no socket needed)
For now, we'll add Docker CLI to the image and mount the socket.
### 5.3 Update Dockerfile for Docker Builds
```dockerfile
FROM code.forgejo.org/forgejo/runner:3.5.1
RUN apt-get update && apt-get install -y --no-install-recommends \
nodejs \
npm \
git \
curl \
wget \
jq \
make \
gcc \
g++ \
ca-certificates \
# Docker CLI for building container images
docker.io \
&& rm -rf /var/lib/apt/lists/*
RUN node --version && npm --version && docker --version
```
### 5.4 Update Deployment for Docker Socket
Add Docker socket mount to `deployment.yaml`:
```yaml
volumeMounts:
- name: runner-data
mountPath: /data
- name: runner-config
mountPath: /config
- name: docker-sock
mountPath: /var/run/docker.sock
volumes:
- name: runner-data
emptyDir: {}
- name: runner-config
configMap:
name: forgejo-runner-config
- name: docker-sock
hostPath:
path: /var/run/docker.sock
type: Socket
```
---
## Step 4: Build Cross-Platform Consideration
## Step 6: Verification
**Important**: The runner runs Linux containers, but indri is macOS ARM64.
**Options**:
### Option A: Cross-compile (Simpler, may have issues)
Add to build job:
```yaml
env:
GOOS: darwin
GOARCH: arm64
```
CGO cross-compilation is tricky. May need to disable CGO or use a cross-compiler.
### Option B: Build on macOS (More reliable)
Run a macOS runner on indri itself (not in k8s).
### 6.1 Manual Image Build Works
```bash
# Install forgejo-runner on indri via mise
ssh indri 'mise use forgejo-runner'
# Register as a macOS runner
ssh indri 'forgejo-runner register --labels "macos-arm64:host" ...'
# On gilbert
podman build --platform linux/arm64 -t registry.tail8d86e.ts.net/blumeops/forgejo-runner:test .
podman push registry.tail8d86e.ts.net/blumeops/forgejo-runner:test
```
Then workflow uses:
```yaml
runs-on: macos-arm64
```
### 6.2 Runner Uses Custom Image
**Recommendation**: Option B is more reliable for native macOS builds. Consider deploying a runner directly on indri for macOS-specific builds.
---
## Step 5: Test the Build
### 5.1 Manual Workflow Dispatch
1. Go to https://forge.tail8d86e.ts.net/eblume/forgejo/actions
2. Select "Build Forgejo" workflow
3. Click "Run workflow"
4. Set deploy=false for first test
5. Monitor the run
### 5.2 Verify Artifact
Download the artifact from the workflow run and verify it's a valid binary:
```bash
# If downloaded to gilbert
file forgejo-*-darwin-arm64
# Should show: Mach-O 64-bit executable arm64
kubectl --context=minikube-indri -n forgejo-runner get pods -o jsonpath='{.items[*].spec.containers[*].image}'
# Should show: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
```
---
### 6.3 GitHub Actions Work
## Alternative: Build on Gilbert, Deploy via CI
- `actions/checkout@v4` succeeds
- Test workflow shows Node.js, npm, git versions
If cross-compilation proves difficult, consider a hybrid approach:
### 6.4 Auto-Build Workflow Works
1. **Build on gilbert** (has Go, Node, is macOS ARM64)
2. **CI just deploys** the built binary
Workflow in blumeops repo:
```yaml
name: Deploy Forgejo
on:
workflow_dispatch:
inputs:
binary_path:
description: 'Path to binary on gilbert'
required: true
jobs:
deploy:
runs-on: ubuntu-latest
steps:
# Fetch binary from gilbert and deploy to indri
# (requires SSH access to both)
```
This is less elegant but more pragmatic for macOS targets.
Push a change to the Dockerfile and verify:
1. Workflow triggers
2. Image builds successfully
3. Image pushed to zot
---
## Verification Checklist
- [ ] Forgejo mirrored to forge
- [ ] SSH deploy key created and stored in 1Password
- [ ] Deploy key added to indri authorized_keys
- [ ] SSH secret added to k8s
- [ ] Workflow file created in forgejo mirror
- [ ] Repository secrets configured
- [ ] Test build completes successfully
- [ ] Binary is valid macOS ARM64 executable
- [x] Dockerfile created for custom runner (Alpine-based with apk)
- [x] Image built manually on gilbert (podman build)
- [x] Image pushed to zot registry
- [x] Runner deployment updated to use custom image
- [x] Runner pod running with new image
- [x] `actions/checkout@v4` works in test workflow
- [ ] Auto-build workflow created (deferred - needs Docker socket)
- [ ] Docker socket mounted (for container builds)
- [ ] Auto-build workflow successfully rebuilds runner
---
## Troubleshooting
### CGO Cross-Compilation Fails
### Image Pull Fails in Minikube
If building Linux→macOS fails:
```
# runtime/cgo
gcc: error: unrecognized command line option '-arch'
Minikube needs to be able to pull from zot. Check registry mirror config:
```bash
ssh indri 'minikube ssh -- cat /etc/containerd/certs.d/registry.tail8d86e.ts.net/hosts.toml'
```
Either:
1. Use Option B (macOS runner on indri)
2. Build with `CGO_ENABLED=0` (loses some features)
3. Use a Docker image with macOS cross-compiler (complex)
### Docker Build Fails in Workflow
### Artifact Too Large
If Docker socket mount doesn't work:
1. Check socket exists in minikube: `minikube ssh -- ls -la /var/run/docker.sock`
2. Check permissions: runner may need to be in docker group
3. Alternative: Use `podman` (rootless) instead of Docker
Forgejo binary is ~100MB. If upload fails:
- Check Forgejo's artifact size limit in app.ini
- Consider compressing: `gzip -9 forgejo-*`
### Node.js Actions Still Fail
Ensure the runner pod restarted after image update:
```bash
kubectl --context=minikube-indri -n forgejo-runner rollout restart deployment/forgejo-runner
kubectl --context=minikube-indri -n forgejo-runner logs -f deployment/forgejo-runner
```
---
## Next Phase
Once build is working and produces valid binaries, proceed to [Phase 3: Self-Deploy](P3_self_deploy.md).
Once the custom runner is working with auto-build, proceed to [Phase 3: Mirror Forgejo & Build](P3_mirror_and_build.md) to set up Forgejo source builds.

349
plans/ci-cd-bootstrap/P3_mirror_forgejo.md Normal file
View file

@ -0,0 +1,349 @@
# Phase 3: Mirror Forgejo & Build from Source
**Goal**: Mirror upstream Forgejo to forge and create a workflow that builds it for macOS ARM64
**Status**: Planning
**Prerequisites**: [Phase 2](P2_mirror_and_build.md) complete (custom runner image with Node.js/tools)
---
## Problem Statement
We want to build Forgejo from source to:
1. Have full control over the binary running on indri
2. Enable self-deployment via CI
3. Ensure proper macOS DNS resolution (requires CGO_ENABLED=1)
### The Cross-Compilation Challenge
The runner runs in a Linux container (k8s on indri), but the target is macOS ARM64 (indri itself).
**Options**:
| Option | Pros | Cons |
|--------|------|------|
| A. Cross-compile CGO_ENABLED=0 | Simple, no special toolchain | Breaks Tailscale MagicDNS resolution |
| B. Cross-compile CGO_ENABLED=1 | Proper DNS | Needs OSX cross-compiler (osxcross), complex |
| C. Build on gilbert manually | Works now, simple | Not automated, manual step |
| D. Native macOS runner on indri | Full native build | Runner outside k8s, different architecture |
| E. Hybrid: build on gilbert, deploy via CI | Uses existing tools | Partial automation |
**Recommendation**: Start with Option C/E (manual build on gilbert, CI just deploys), then consider Option D if we want full automation.
---
## Step 1: Mirror Upstream Forgejo
### 1.1 User Action: Create Mirror on Forge
**Manual step** (hairpinning doesn't work from indri):
1. Go to https://forge.tail8d86e.ts.net
2. Click "+" → "New Migration"
3. Select "Gitea" as clone source
4. URL: `https://codeberg.org/forgejo/forgejo.git`
5. Repository name: `forgejo`
6. Check "This repository will be a mirror"
7. Click "Migrate Repository"
### 1.2 Clone Mirror Locally
```bash
git clone ssh://forgejo@forge.tail8d86e.ts.net/eblume/forgejo.git ~/code/3rd/forgejo
cd ~/code/3rd/forgejo
```
---
## Step 2: Understand Forgejo Build Process
### 2.1 Build Requirements
From Forgejo's `Makefile` and docs:
- **Go**: 1.23+ (check `go.mod` for exact version)
- **Node.js**: 20+ (for frontend)
- **Make**: GNU Make
- **Git**: For version embedding
### 2.2 Build Commands
```bash
# Install frontend dependencies and build
make deps-frontend
make frontend
# Build backend (with CGO for proper DNS on macOS)
CGO_ENABLED=1 TAGS="bindata sqlite sqlite_unlock_notify" make backend
# Or all-in-one
CGO_ENABLED=1 TAGS="bindata sqlite sqlite_unlock_notify" make build
```
### 2.3 Output
Binary at `gitea` (yes, the binary is still named `gitea` for compatibility).
---
## Step 3: Build on Gilbert (Manual Bootstrap)
For the initial bootstrap, build on gilbert (macOS ARM64 native).
### 3.1 Setup Build Environment
```bash
cd ~/code/3rd/forgejo
mise use go@1.23 node@20
# Verify tools
go version
node --version
make --version
```
### 3.2 Build
```bash
# Clean build
make clean
# Build frontend
make deps-frontend
make frontend
# Build backend with CGO (important for macOS DNS!)
CGO_ENABLED=1 TAGS="bindata sqlite sqlite_unlock_notify" make backend
# Verify binary
./gitea --version
file gitea # Should show: Mach-O 64-bit executable arm64
```
### 3.3 Deploy to Indri
```bash
# Copy binary
scp gitea indri:~/.local/bin/forgejo-new
# Verify on indri
ssh indri '~/.local/bin/forgejo-new --version'
```
---
## Step 4: Create Deploy Workflow (Option E)
Since cross-compilation is complex, use a hybrid approach:
1. Build on gilbert (manual trigger or pre-built)
2. CI workflow fetches and deploys
### 4.1 SSH Deploy Key for Runner
The runner needs SSH access to indri to deploy the binary.
**Generate key on gilbert**:
```bash
ssh-keygen -t ed25519 -C "forgejo-runner-deploy" -f ~/.ssh/forgejo-runner-deploy -N ""
```
**Add public key to indri's authorized_keys**:
```bash
cat ~/.ssh/forgejo-runner-deploy.pub | ssh indri 'cat >> ~/.ssh/authorized_keys'
```
**Store private key in 1Password** (blumeops vault) as "Forgejo Runner Deploy Key"
### 4.2 Create k8s Secret
Create `argocd/manifests/forgejo-runner/secret-ssh.yaml.tpl`:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: forgejo-runner-ssh
namespace: forgejo-runner
type: Opaque
stringData:
id_ed25519: |
op://blumeops/<deploy-key-item>/private-key
known_hosts: |
# Get with: ssh-keyscan indri.tail8d86e.ts.net 2>/dev/null | grep ed25519
indri.tail8d86e.ts.net ssh-ed25519 AAAAC3...
```
### 4.3 Update Deployment for SSH
Add SSH secret mount to `deployment.yaml`:
```yaml
volumeMounts:
- name: ssh-key
mountPath: /root/.ssh
readOnly: true
volumes:
- name: ssh-key
secret:
secretName: forgejo-runner-ssh
defaultMode: 0600
```
### 4.4 Create Deploy-Only Workflow
Create `.forgejo/workflows/deploy-forgejo.yml` in blumeops:
```yaml
name: Deploy Forgejo
on:
workflow_dispatch:
inputs:
version:
description: 'Version to deploy (tag or commit)'
required: true
default: 'v10.0.0'
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Deploy to indri
env:
VERSION: ${{ github.event.inputs.version }}
run: |
# SSH config
mkdir -p ~/.ssh
cp /root/.ssh/id_ed25519 ~/.ssh/
cp /root/.ssh/known_hosts ~/.ssh/
chmod 600 ~/.ssh/id_ed25519
# Deploy script
ssh erichblume@indri.tail8d86e.ts.net << 'EOF'
set -e
cd ~/.local/bin
# Verify the new binary exists and runs
if [ ! -f forgejo-new ]; then
echo "ERROR: forgejo-new not found. Build on gilbert first:"
echo " cd ~/code/3rd/forgejo && git checkout $VERSION"
echo " CGO_ENABLED=1 TAGS='bindata sqlite sqlite_unlock_notify' make build"
echo " scp gitea indri:~/.local/bin/forgejo-new"
exit 1
fi
./forgejo-new --version
# Stop current service
launchctl unload ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist 2>/dev/null || true
# Atomic swap
mv forgejo forgejo-old 2>/dev/null || true
mv forgejo-new forgejo
# Start new service
launchctl load ~/Library/LaunchAgents/mcquack.eblume.forgejo.plist
# Verify it's running
sleep 5
curl -sf http://localhost:3001/api/v1/version || exit 1
echo "Deploy successful!"
./forgejo --version
EOF
```
---
## Future: Full CI Build (Option D)
If we want full automation, consider running a native macOS runner on indri:
### Native Runner on Indri
```bash
# Install forgejo-runner on indri via mise
ssh indri 'mise use forgejo-runner'
# Register as a macOS runner
ssh indri 'forgejo-runner register \
--instance https://forge.tail8d86e.ts.net \
--token "$TOKEN" \
--name "indri-native" \
--labels "macos-arm64:host" \
--no-interactive'
# Create LaunchAgent for runner
# (similar to other mcquack services)
```
Then workflow uses:
```yaml
runs-on: macos-arm64
```
This enables full native builds in CI. Document in a future phase if needed.
---
## Verification Checklist
- [ ] Forgejo mirrored to forge
- [ ] Mirror cloned to ~/code/3rd/forgejo
- [ ] Build succeeds on gilbert
- [ ] Binary is valid macOS ARM64 executable
- [ ] Binary deployed to indri ~/.local/bin/
- [ ] SSH deploy key created and stored in 1Password
- [ ] Deploy key added to indri authorized_keys
- [ ] (Optional) k8s SSH secret created
- [ ] (Optional) Deploy workflow created
---
## Troubleshooting
### Build Fails: Node.js Version
```
error: engine "node" is incompatible
```
Update Node.js: `mise use node@20`
### Build Fails: Go Version
```
go: go.mod requires go >= 1.23
```
Update Go: `mise use go@1.23`
### Binary Crashes on indri
Check if CGO was enabled:
```bash
# If built without CGO, DNS resolution may fail
./forgejo --version # Should work
./forgejo web # May fail to resolve Tailscale hostnames
```
Rebuild with `CGO_ENABLED=1`.
### SSH Deploy Fails
Check runner has SSH access:
```bash
# Test from inside runner pod
kubectl --context=minikube-indri -n forgejo-runner exec deployment/forgejo-runner -- \
ssh -i /root/.ssh/id_ed25519 erichblume@indri.tail8d86e.ts.net 'echo ok'
```
---
## Next Phase
Once Forgejo is building and deploying successfully, proceed to [Phase 4: Self-Deploy](P4_self_deploy.md) for the full mcquack transition.

6
plans/ci-cd-bootstrap/P3_self_deploy.md → plans/ci-cd-bootstrap/P4_self_deploy.md
View file

@ -1,10 +1,10 @@
# Phase 3: Self-Deploy & Transition to mcquack
# Phase 4: Self-Deploy & Transition to mcquack
**Goal**: Complete the bootstrap - Forgejo deploys itself, transition from brew to mcquack LaunchAgent
**Status**: Planning
**Prerequisites**: [Phase 2](P2_mirror_and_build.md) complete (build workflow produces valid binaries)
**Prerequisites**: [Phase 3](P3_mirror_forgejo.md) complete (Forgejo builds and deploys to indri)
---
@ -406,4 +406,4 @@ After recovery, switch back to Forgejo.
## Next Phase
After bootstrap is complete, proceed to [Phase 4: Container Builds](P4_container_builds.md) to set up container image building for ArgoCD.
After bootstrap is complete, proceed to [Phase 5: Container Builds](P5_container_builds.md) to set up container image building for ArgoCD.

78
plans/ci-cd-bootstrap/P4_container_builds.md → plans/ci-cd-bootstrap/P5_container_builds.md
View file

@ -1,91 +1,21 @@
# Phase 4: Container Image Builds
# Phase 5: Container Image Builds
**Goal**: Set up CI workflows to build custom container images and push to zot registry
**Status**: Planning
**Prerequisites**: [Phase 3](P3_self_deploy.md) complete (Forgejo self-deploying, Actions working)
**Prerequisites**: [Phase 4](P4_self_deploy.md) complete (Forgejo self-deploying, Actions working)
---
## Overview
With Forgejo Actions operational, we can now build container images for:
With Forgejo Actions operational (including custom runner from P2), we can now build container images for:
- Custom devpi with pre-installed plugins
- Any other custom images needed for k8s services
- Release artifacts for Python packages
---
## Use Case 0: Custom Runner Image
### Problem
The stock `forgejo/runner` image lacks tools needed for standard GitHub Actions:
- **Node.js** - Required by most actions (checkout, setup-*, etc.)
- **Docker CLI** - For building container images
- **Git** - For repository operations
- **Common build tools** - make, gcc, etc.
In host mode, jobs run directly in the runner container, so these tools must be pre-installed.
### Solution
Build a custom runner image with all necessary tools:
```dockerfile
# argocd/manifests/forgejo-runner/Dockerfile
FROM code.forgejo.org/forgejo/runner:3.5.1
# Install Node.js (required for most GitHub Actions)
RUN apt-get update && apt-get install -y \
nodejs \
npm \
git \
curl \
docker.io \
make \
gcc \
&& rm -rf /var/lib/apt/lists/*
```
### Workflow
Create `.forgejo/workflows/build-runner.yml`:
```yaml
name: Build Runner Image
on:
push:
paths:
- 'argocd/manifests/forgejo-runner/Dockerfile'
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
run: |
git clone --depth 1 "${{ gitea.server_url }}/${{ gitea.repository }}.git" .
- name: Build and push
run: |
cd argocd/manifests/forgejo-runner
docker build -t registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest .
docker push registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
```
### Update Deployment
Once the custom image is built, update `argocd/manifests/forgejo-runner/deployment.yaml`:
```yaml
image: registry.tail8d86e.ts.net/blumeops/forgejo-runner:latest
```
This enables standard GitHub Actions like `actions/checkout@v4` to work in host mode.
**Note**: The custom runner image build is covered in [Phase 2](P2_mirror_and_build.md). This phase focuses on application container builds.
---