Add plans for Dagger CI/CD and upstream fork strategy (#150)
## Summary Two new plan documents in `docs/how-to/plans/`: - **adopt-dagger-ci** — Migrate CI/CD build logic from Forgejo Actions YAML to Dagger (Python SDK). Forgejo Actions stays as a thin trigger layer. Covers: - Container builds with local iteration (`dagger call build ... terminal`) - Docs builds with Forgejo packages migration (replacing Forgejo releases) - Runner simplification (only Docker + dagger CLI needed) - Secrets handling via Dagger's `Secret` type - Future: forked project builds, Python packages, pre-merge validation - **upstream-fork-strategy** — Stacked-branch pattern for maintaining forks of upstream projects. Covers: - Daily automated rebase with conflict detection and issue creation - Branch model: `upstream/main` → `blumeops` → `feature/*` - Quartz fork as first instance, enabling `last-reviewed` frontmatter rendering in docs - Upstream PR path for contributing changes back ## Context These plans emerged from evaluating alternatives to the GHA ecosystem (BuildKite, Concourse, Earthly) for CI/CD. Dagger was chosen for its local iteration story, Python-native pipelines, and zero-infrastructure requirements. The fork strategy is a prerequisite for customizing Quartz and other upstream tools. Neither plan is ready for execution yet — they are design documents for future work. Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/150
This commit is contained in:
parent
a106f92c38
commit
430f2c6ec5
5 changed files with 775 additions and 0 deletions
1
docs/changelog.d/feature-ci-and-fork-plans.doc.md
Normal file
1
docs/changelog.d/feature-ci-and-fork-plans.doc.md
Normal file
|
|
@ -0,0 +1 @@
|
|||
Add plan documents for Dagger CI/CD adoption and upstream fork strategy.
|
||||
|
|
@ -54,3 +54,5 @@ Migration and transition plans for upcoming infrastructure changes.
|
|||
| [[plans]] | Index of all plans |
|
||||
| [[migrate-forgejo-from-brew]] | Transition Forgejo from Homebrew to source-built binary |
|
||||
| [[add-unifi-pulumi-stack]] | Add Pulumi IaC for UniFi Express 7 |
|
||||
| [[adopt-dagger-ci]] | Adopt Dagger as CI/CD build engine |
|
||||
| [[upstream-fork-strategy]] | Stacked-branch forking strategy for upstream projects |
|
||||
|
|
|
|||
537
docs/how-to/plans/adopt-dagger-ci.md
Normal file
537
docs/how-to/plans/adopt-dagger-ci.md
Normal file
|
|
@ -0,0 +1,537 @@
|
|||
---
|
||||
title: "Plan: Adopt Dagger as CI/CD Build Engine"
|
||||
tags:
|
||||
- how-to
|
||||
- plans
|
||||
- ci-cd
|
||||
- dagger
|
||||
---
|
||||
|
||||
# Plan: Adopt Dagger as CI/CD Build Engine
|
||||
|
||||
> **Status:** Planned (not yet executed)
|
||||
|
||||
## Background
|
||||
|
||||
BlumeOps CI/CD currently runs on Forgejo Actions (GitHub Actions-compatible). While functional, the system has pain points that are inherent to the GHA ecosystem:
|
||||
|
||||
- **Hard to debug** — logs are buried in a web UI, no way to SSH into a running job, no interactive debugging
|
||||
- **No local iteration** — the only way to test a workflow change is to push and wait for CI
|
||||
- **Supply chain risk** — community actions are opaque third-party code running in your infrastructure
|
||||
- **Runner complexity** — the k8s-runner image must bundle every tool any workflow might need (Docker CLI, buildx, skopeo, Node.js, etc.)
|
||||
- **YAML as programming language** — complex workflows become unreadable
|
||||
|
||||
### Why Dagger?
|
||||
|
||||
[Dagger](https://dagger.io/) is an open-source (Apache-2.0) build engine built on BuildKit. It addresses every pain point above:
|
||||
|
||||
| Pain point | Dagger solution |
|
||||
|------------|-----------------|
|
||||
| Can't debug builds | `--interactive` drops you into a shell at the failure point; `.terminal()` adds breakpoints |
|
||||
| Can't run locally | `dagger call` runs identically on your laptop and in CI — same code path |
|
||||
| Supply chain risk | Build logic is your own Python code, not third-party actions |
|
||||
| Runner bloat | Runner only needs Docker + `dagger` CLI; all tools live inside Dagger containers |
|
||||
| YAML complexity | Pipelines are real Python (classes, decorators, async/await) — not templated YAML |
|
||||
|
||||
### What Dagger is NOT
|
||||
|
||||
Dagger is a **build engine**, not a CI scheduler. It does not handle triggers, scheduling, or webhooks. We keep Forgejo Actions as a thin trigger layer — its YAML becomes trivially simple (install dagger, run `dagger call`). All actual build logic moves to Python.
|
||||
|
||||
### Alternatives Considered
|
||||
|
||||
| System | Verdict | Reason |
|
||||
|--------|---------|--------|
|
||||
| **BuildKite** | Rejected | No fully self-hosted option (cloud control plane required); no native Forgejo integration; adds external dependency for a homelab |
|
||||
| **Concourse CI** | Rejected | Fully self-hosted and great debugging (`fly intercept`), but verbose YAML with no built-in templating; small community; 2-4GB RAM overhead for the scheduler; doesn't solve local iteration as cleanly |
|
||||
| **Earthly** | Not viable | Project discontinued April 2025, all cloud services shut down July 2025 |
|
||||
|
||||
Dagger was chosen because it delivers the best local iteration story, supports Python natively, and requires zero infrastructure beyond what we already have (Docker on the runner).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────┐ ┌──────────────────┐
|
||||
│ Forgejo Actions │ │ Your terminal │
|
||||
│ (trigger layer) │ │ (local dev) │
|
||||
│ │ │ │
|
||||
│ on: push tags │ │ mise run ... │
|
||||
│ → dagger call ... │ │ → dagger call .. │
|
||||
└──────────┬───────────┘ └────────┬──────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────┐
|
||||
│ Dagger Engine (BuildKit) │
|
||||
│ │
|
||||
│ blumeops-ci Python module │
|
||||
│ ├── build(container_name) │
|
||||
│ ├── publish(container_name, version)│
|
||||
│ ├── build_docs(version) │
|
||||
│ ├── release_docs(version, tokens) │
|
||||
│ └── validate() │
|
||||
└──────────────┬───────────────────────┘
|
||||
│
|
||||
┌────────┼────────┐
|
||||
▼ ▼ ▼
|
||||
┌─────┐ ┌──────┐ ┌───────┐
|
||||
│ Zot │ │Forgejo│ │ArgoCD │
|
||||
│ │ │Pkgs │ │ │
|
||||
└─────┘ └──────┘ └───────┘
|
||||
```
|
||||
|
||||
**Key principle:** The same `dagger call` command runs on your Mac during development and in the Forgejo runner during CI. The Forgejo Actions YAML is a thin shim that parses the trigger event and calls Dagger.
|
||||
|
||||
## Dagger Module Structure
|
||||
|
||||
```
|
||||
dagger/
|
||||
├── dagger.json # Module metadata, SDK selection
|
||||
├── pyproject.toml # Python deps (httpx, etc.)
|
||||
├── uv.lock # Locked dependencies
|
||||
└── src/blumeops_ci/
|
||||
└── __init__.py # All build functions
|
||||
```
|
||||
|
||||
## Secrets Handling
|
||||
|
||||
Dagger has a first-class `Secret` type — values are never logged, cached, or visible in traces.
|
||||
|
||||
**From CLI:**
|
||||
```bash
|
||||
dagger call release-docs \
|
||||
--src=. --version=v1.6.0 \
|
||||
--forgejo-token=env:FORGEJO_TOKEN \
|
||||
--argocd-token=env:ARGOCD_TOKEN
|
||||
```
|
||||
|
||||
The `env:VARIABLE` syntax reads from environment variables. In Forgejo Actions, secrets are injected as env vars. Locally, a mise task calls `op read` to populate them.
|
||||
|
||||
**In Python code:**
|
||||
```python
|
||||
@function
|
||||
async def release_docs(
|
||||
self,
|
||||
src: dagger.Directory,
|
||||
version: str,
|
||||
forgejo_token: dagger.Secret,
|
||||
argocd_token: dagger.Secret,
|
||||
) -> str:
|
||||
# Token is mounted securely, never exposed in logs
|
||||
token = await forgejo_token.plaintext()
|
||||
```
|
||||
|
||||
**Rule of thumb:** Simple API calls (Forgejo package upload) use Python `httpx` directly in the module runtime. CLI tools without good Python libraries (ArgoCD) run in container steps with secrets mounted as env vars via `.with_secret_variable()`.
|
||||
|
||||
## Phase 1: Container Builds
|
||||
|
||||
Migrate `build-container.yaml` to use Dagger for the build/push logic.
|
||||
|
||||
### Dagger Functions
|
||||
|
||||
```python
|
||||
@function
|
||||
def build(self, src: dagger.Directory, container_name: str) -> dagger.Container:
|
||||
"""Build a container from containers/<name>/Dockerfile."""
|
||||
context = src.directory(f"containers/{container_name}")
|
||||
return dag.container().build(context)
|
||||
|
||||
@function
|
||||
async def publish(
|
||||
self,
|
||||
src: dagger.Directory,
|
||||
container_name: str,
|
||||
version: str,
|
||||
registry: str = "registry.ops.eblu.me",
|
||||
) -> str:
|
||||
"""Build and push to zot registry."""
|
||||
ctr = self.build(src, container_name)
|
||||
ref = f"{registry}/blumeops/{container_name}:{version}"
|
||||
return await ctr.publish(ref)
|
||||
```
|
||||
|
||||
### Local Iteration Workflow
|
||||
|
||||
```bash
|
||||
# Build — validates Dockerfile, fast cached feedback
|
||||
dagger call build --src=. --container-name=devpi
|
||||
|
||||
# Build and drop into a shell to inspect the container
|
||||
dagger call build --src=. --container-name=devpi terminal
|
||||
|
||||
# Debug a failure interactively
|
||||
dagger call --interactive build --src=. --container-name=devpi
|
||||
|
||||
# Push a dev tag for testing in k8s (ArgoCD ignores it)
|
||||
dagger call publish --src=. --container-name=devpi --version=dev
|
||||
|
||||
# Publish the real version
|
||||
dagger call publish --src=. --container-name=devpi --version=v1.1.0
|
||||
```
|
||||
|
||||
### Forgejo Actions Integration
|
||||
|
||||
The existing tag-based trigger model is preserved. The workflow becomes a thin Dagger invocation:
|
||||
|
||||
```yaml
|
||||
name: Build Container
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- '*-v[0-9]*'
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: k8s
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Parse tag
|
||||
id: parse
|
||||
run: |
|
||||
TAG="${GITHUB_REF_NAME}"
|
||||
CONTAINER="${TAG%-v[0-9]*}"
|
||||
VERSION="${TAG#"${CONTAINER}"-}"
|
||||
echo "container=$CONTAINER" >> "$GITHUB_OUTPUT"
|
||||
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
|
||||
- name: Publish
|
||||
run: |
|
||||
curl -fsSL https://dl.dagger.io/dagger/install.sh | sh
|
||||
./bin/dagger call publish \
|
||||
--src=. \
|
||||
--container-name=${{ steps.parse.outputs.container }} \
|
||||
--version=${{ steps.parse.outputs.version }}
|
||||
```
|
||||
|
||||
The composite action (`.forgejo/actions/build-push-image/`), skopeo workaround, and docker save/load dance are all eliminated — that logic lives in the Dagger module.
|
||||
|
||||
### Zot Manifest Compatibility
|
||||
|
||||
The current workflow uses skopeo because Docker 27's manifest format has issues with zot. Dagger's `.publish()` uses BuildKit's push mechanism, which is different. This **must be tested** during implementation. If BuildKit's push also has zot compatibility issues, the Dagger function can shell out to skopeo inside a container step as a fallback.
|
||||
|
||||
### Release Flow (Unchanged)
|
||||
|
||||
```bash
|
||||
mise run container-tag-and-release <container> <version>
|
||||
# → creates git tag → pushes → Forgejo Action triggers → dagger call publish
|
||||
```
|
||||
|
||||
## Phase 2: Docs Build + Forgejo Packages Migration
|
||||
|
||||
Migrate `build-blumeops.yaml` to use Dagger for the build logic and switch from Forgejo releases to Forgejo generic packages for the docs tarball.
|
||||
|
||||
### Artifact Migration: Forgejo Releases → Forgejo Packages
|
||||
|
||||
**Current:** Docs tarball uploaded as a Forgejo release asset.
|
||||
```
|
||||
https://forge.ops.eblu.me/eblume/blumeops/releases/download/v1.5.2/docs-v1.5.2.tar.gz
|
||||
```
|
||||
|
||||
**New:** Docs tarball uploaded to Forgejo generic packages registry.
|
||||
```
|
||||
https://forge.ops.eblu.me/api/packages/eblume/generic/blumeops-docs/v1.6.0/docs-v1.6.0.tar.gz
|
||||
```
|
||||
|
||||
This decouples the docs artifact from git releases while keeping the versioned URL pattern. Forgejo releases can still be created for changelog/announcement purposes without carrying the tarball.
|
||||
|
||||
### Dagger Functions
|
||||
|
||||
```python
|
||||
@function
|
||||
async def build_changelog(self, src: dagger.Directory, version: str) -> dagger.Directory:
|
||||
"""Run towncrier to build changelog, return modified source tree."""
|
||||
return await (
|
||||
dag.container()
|
||||
.from_("python:3.12-slim")
|
||||
.with_exec(["pip", "install", "towncrier"])
|
||||
.with_directory("/workspace", src)
|
||||
.with_workdir("/workspace")
|
||||
.with_exec(["towncrier", "build", "--version", version, "--yes"])
|
||||
.directory("/workspace")
|
||||
)
|
||||
|
||||
@function
|
||||
async def build_docs(self, src: dagger.Directory, version: str) -> dagger.File:
|
||||
"""Build changelog, then build Quartz site, return tarball."""
|
||||
updated_src = await self.build_changelog(src, version)
|
||||
return await (
|
||||
dag.container()
|
||||
.from_("node:20-slim")
|
||||
.with_exec(["apt-get", "update", "-qq"])
|
||||
.with_exec(["apt-get", "install", "-y", "-qq", "git"])
|
||||
.with_directory("/workspace", updated_src)
|
||||
.with_workdir("/workspace")
|
||||
.with_exec(["git", "clone", "--depth=1",
|
||||
"https://github.com/jackyzha0/quartz.git", "/tmp/quartz"])
|
||||
.with_exec(["sh", "-c",
|
||||
"cp -r /tmp/quartz/quartz /tmp/quartz/package*.json "
|
||||
"/tmp/quartz/tsconfig.json ."])
|
||||
.with_exec(["npm", "ci"])
|
||||
.with_exec(["cp", "docs/quartz.config.ts", "."])
|
||||
.with_exec(["cp", "docs/quartz.layout.ts", "."])
|
||||
.with_exec(["cp", "CHANGELOG.md", "docs/"])
|
||||
.with_exec(["npx", "quartz", "build", "-d", "docs"])
|
||||
.with_exec(["sh", "-c",
|
||||
f"tar -czf /docs-{version}.tar.gz -C public ."])
|
||||
.file(f"/docs-{version}.tar.gz")
|
||||
)
|
||||
|
||||
@function
|
||||
async def upload_docs(
|
||||
self,
|
||||
tarball: dagger.File,
|
||||
version: str,
|
||||
forgejo_token: dagger.Secret,
|
||||
) -> str:
|
||||
"""Upload docs tarball to Forgejo generic packages."""
|
||||
import httpx
|
||||
|
||||
token = await forgejo_token.plaintext()
|
||||
await tarball.export(f"/tmp/docs-{version}.tar.gz")
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
with open(f"/tmp/docs-{version}.tar.gz", "rb") as f:
|
||||
resp = await client.put(
|
||||
f"https://forge.ops.eblu.me/api/packages/eblume/generic/"
|
||||
f"blumeops-docs/{version}/docs-{version}.tar.gz",
|
||||
headers={"Authorization": f"token {token}"},
|
||||
content=f.read(),
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return f"https://forge.ops.eblu.me/api/packages/eblume/generic/blumeops-docs/{version}/docs-{version}.tar.gz"
|
||||
|
||||
@function
|
||||
async def release_docs(
|
||||
self,
|
||||
src: dagger.Directory,
|
||||
version: str,
|
||||
forgejo_token: dagger.Secret,
|
||||
argocd_token: dagger.Secret,
|
||||
) -> str:
|
||||
"""Full docs release: build, upload to Forgejo packages, sync ArgoCD."""
|
||||
tarball = await self.build_docs(src, version)
|
||||
pkg_url = await self.upload_docs(tarball, version, forgejo_token)
|
||||
|
||||
# Sync ArgoCD
|
||||
await (
|
||||
dag.container()
|
||||
.from_("alpine:3.21")
|
||||
.with_exec(["apk", "add", "--no-cache", "curl"])
|
||||
.with_secret_variable("ARGOCD_AUTH_TOKEN", argocd_token)
|
||||
.with_exec(["sh", "-c",
|
||||
"curl -fSs -X POST "
|
||||
"-H \"Authorization: Bearer $ARGOCD_AUTH_TOKEN\" "
|
||||
"\"https://argocd.ops.eblu.me/api/v1/applications/docs/sync\" "
|
||||
"-d '{}'"])
|
||||
.sync()
|
||||
)
|
||||
|
||||
return pkg_url
|
||||
```
|
||||
|
||||
### Local Iteration Workflow
|
||||
|
||||
```bash
|
||||
# Test the full docs build locally — identical to CI
|
||||
dagger call build-docs --src=. --version=dev export --path=./docs-dev.tar.gz
|
||||
|
||||
# Inspect the output
|
||||
tar tf docs-dev.tar.gz | head -20
|
||||
|
||||
# Debug a Quartz build failure interactively
|
||||
dagger call --interactive build-docs --src=. --version=dev
|
||||
|
||||
# Test just the changelog build
|
||||
dagger call build-changelog --src=. --version=dev export --path=./updated-src/
|
||||
```
|
||||
|
||||
This is particularly valuable for debugging Quartz build issues and for iterating on a personal quartz fork.
|
||||
|
||||
### Forgejo Actions Integration
|
||||
|
||||
The workflow remains manually triggered (workflow_dispatch) to preserve centralized version sequencing. Dagger handles the build/upload/deploy; the workflow handles version resolution and git commit:
|
||||
|
||||
```yaml
|
||||
name: Build BlumeOps
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version_type: { type: choice, options: [BUMP_PATCH, BUMP_MINOR, BUMP_MAJOR] }
|
||||
|
||||
jobs:
|
||||
release:
|
||||
runs-on: k8s
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with: { fetch-depth: 0 }
|
||||
|
||||
- name: Resolve version
|
||||
id: version
|
||||
run: |
|
||||
# ... existing version bump logic (query Forgejo API, bump semver) ...
|
||||
|
||||
- name: Build, upload, and deploy
|
||||
env:
|
||||
FORGEJO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
ARGOCD_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
|
||||
run: |
|
||||
curl -fsSL https://dl.dagger.io/dagger/install.sh | sh
|
||||
./bin/dagger call release-docs \
|
||||
--src=. --version=${{ steps.version.outputs.version }} \
|
||||
--forgejo-token=env:FORGEJO_TOKEN \
|
||||
--argocd-token=env:ARGOCD_TOKEN
|
||||
|
||||
- name: Export changelog changes
|
||||
run: |
|
||||
./bin/dagger call build-changelog \
|
||||
--src=. --version=${{ steps.version.outputs.version }} \
|
||||
export --path=.
|
||||
|
||||
- name: Update manifest and commit
|
||||
run: |
|
||||
VERSION="${{ steps.version.outputs.version }}"
|
||||
URL="https://forge.ops.eblu.me/api/packages/eblume/generic/blumeops-docs/${VERSION}/docs-${VERSION}.tar.gz"
|
||||
sed -i "s|value: \"https://.*\"|value: \"${URL}\"|" \
|
||||
argocd/manifests/docs/deployment.yaml
|
||||
git config user.name "Forgejo Actions"
|
||||
git config user.email "actions@forge.ops.eblu.me"
|
||||
git add CHANGELOG.md docs/changelog.d/ argocd/manifests/docs/deployment.yaml
|
||||
git commit -m "Release docs $VERSION [skip ci]"
|
||||
git push origin HEAD:main
|
||||
```
|
||||
|
||||
### Manifest Update
|
||||
|
||||
The quartz container's `DOCS_RELEASE_URL` env var in `argocd/manifests/docs/deployment.yaml` must be updated to use the Forgejo packages URL format:
|
||||
|
||||
```yaml
|
||||
# Before (Forgejo releases):
|
||||
- name: DOCS_RELEASE_URL
|
||||
value: "https://forge.ops.eblu.me/eblume/blumeops/releases/download/v1.5.2/docs-v1.5.2.tar.gz"
|
||||
|
||||
# After (Forgejo generic packages):
|
||||
- name: DOCS_RELEASE_URL
|
||||
value: "https://forge.ops.eblu.me/api/packages/eblume/generic/blumeops-docs/v1.6.0/docs-v1.6.0.tar.gz"
|
||||
```
|
||||
|
||||
The quartz container's `start.sh` already downloads from `DOCS_RELEASE_URL` via curl — no container changes needed, just the URL format changes.
|
||||
|
||||
## Phase 3: Runner Simplification
|
||||
|
||||
Once container builds and docs builds both use Dagger, the k8s-runner image can be simplified.
|
||||
|
||||
### Current Runner Requirements
|
||||
|
||||
The `forgejo-runner` container (at `containers/forgejo-runner/`) bundles:
|
||||
- Docker CLI + buildx plugin (for container builds)
|
||||
- Skopeo (for zot push compatibility)
|
||||
- Node.js (for Quartz docs builds)
|
||||
- ArgoCD CLI (for deployment sync)
|
||||
- Various other tools
|
||||
|
||||
### Simplified Runner
|
||||
|
||||
With Dagger, the runner only needs:
|
||||
- Docker (for the Dagger engine — already available via DinD sidecar)
|
||||
- The `dagger` CLI binary
|
||||
- Git (for checkout)
|
||||
- Basic shell utilities
|
||||
|
||||
All other tools (Node.js, skopeo, argocd, Python, npm) live inside the Dagger containers defined by the module. Adding a new tool to a build never requires rebuilding the runner image.
|
||||
|
||||
### Implementation
|
||||
|
||||
Update `containers/forgejo-runner/Dockerfile` to remove tool-specific dependencies. Install the `dagger` CLI instead. The DinD sidecar in the Forgejo runner pod (`argocd/manifests/forgejo-runner/`) stays unchanged — Dagger's engine runs inside Docker, which the sidecar provides.
|
||||
|
||||
## Phase 4: Future Workflows
|
||||
|
||||
These are natural extensions once the Dagger module is established:
|
||||
|
||||
### Forked Project Builds
|
||||
|
||||
Once the [[upstream-fork-strategy]] is in place, forked projects (e.g., a personal quartz fork) can use the same Dagger patterns for building. The docs build function could accept a quartz source directory parameter instead of cloning upstream, enabling builds against the fork.
|
||||
|
||||
### Python Package Builds
|
||||
|
||||
If private Python packages are built for [[devpi]], Dagger is a natural fit:
|
||||
|
||||
```bash
|
||||
dagger call build-package --src=. --version=v1.0.0
|
||||
# → builds wheel/sdist → uploads to devpi
|
||||
```
|
||||
|
||||
### Pre-Merge Validation
|
||||
|
||||
A `validate` function that runs linting, doc link checks, and other pre-merge checks:
|
||||
|
||||
```bash
|
||||
dagger call validate --src=.
|
||||
# → runs docs-check-links, docs-check-index, docs-check-filenames, etc.
|
||||
```
|
||||
|
||||
Same checks run locally and in CI. Could be triggered by Forgejo Actions on PR creation.
|
||||
|
||||
## Caveats and Risks
|
||||
|
||||
### Dagger Is Pre-1.0
|
||||
|
||||
Current version is v0.19.x. API breakage between versions is possible. Mitigations:
|
||||
- Pin the Dagger CLI version in the runner image and local install
|
||||
- Test upgrades on a branch before adopting
|
||||
- The module is small enough to update quickly if APIs change
|
||||
|
||||
### Privileged Container Requirement
|
||||
|
||||
The Dagger engine requires privileged container access. The current Forgejo runner already uses DinD (privileged), so this should work. Must be verified during implementation.
|
||||
|
||||
### BuildKit Cache Persistence
|
||||
|
||||
BuildKit caches aggressively, making repeated builds fast. Since the Forgejo runner pod is persistent (not ephemeral), the cache persists between CI runs. Locally, the Dagger engine maintains its own cache. No special cache configuration should be needed.
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
### Phase 1 (Containers)
|
||||
- [ ] `dagger call build --src=. --container-name=nettest` succeeds locally
|
||||
- [ ] `dagger call build --src=. --container-name=nettest terminal` drops into container shell
|
||||
- [ ] `dagger call publish --src=. --container-name=nettest --version=test` pushes to zot
|
||||
- [ ] Zot manifest compatibility confirmed (no skopeo needed) or fallback implemented
|
||||
- [ ] Tag-triggered Forgejo Action successfully calls `dagger call publish`
|
||||
- [ ] Existing `mise run container-tag-and-release` workflow still works end-to-end
|
||||
|
||||
### Phase 2 (Docs)
|
||||
- [ ] `dagger call build-docs --src=. --version=dev` produces valid tarball locally
|
||||
- [ ] Tarball contents match current Quartz build output
|
||||
- [ ] `dagger call release-docs` uploads to Forgejo packages successfully
|
||||
- [ ] Quartz container starts and serves docs from Forgejo packages URL
|
||||
- [ ] ArgoCD sync works from within Dagger
|
||||
- [ ] Forgejo Actions workflow_dispatch completes full release cycle
|
||||
- [ ] CHANGELOG.md and fragment cleanup committed correctly
|
||||
|
||||
### Phase 3 (Runner)
|
||||
- [ ] Simplified runner image builds and runs
|
||||
- [ ] Dagger engine starts inside the runner's DinD environment
|
||||
- [ ] All existing workflows pass with the simplified runner
|
||||
|
||||
## How-To Articles to Write
|
||||
|
||||
The following how-to guides should be created alongside implementation:
|
||||
|
||||
| Article | Description |
|
||||
|---------|-------------|
|
||||
| `docs/how-to/use-dagger-containers.md` | Creating and iterating on containers with Dagger (build, terminal, publish workflow) |
|
||||
| `docs/how-to/release-docs.md` | Updated docs release process using Dagger + Forgejo packages (replaces current [[update-documentation]]) |
|
||||
|
||||
## Reference
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `.forgejo/workflows/build-container.yaml` | Current container build workflow (to be migrated) |
|
||||
| `.forgejo/workflows/build-blumeops.yaml` | Current docs build workflow (to be migrated) |
|
||||
| `.forgejo/actions/build-push-image/action.yaml` | Current composite action (to be removed) |
|
||||
| `containers/forgejo-runner/Dockerfile` | Runner image (to be simplified) |
|
||||
| `argocd/manifests/forgejo-runner/` | Runner k8s manifests |
|
||||
| `argocd/manifests/docs/deployment.yaml` | Docs deployment (DOCS_RELEASE_URL to update) |
|
||||
|
||||
## Related
|
||||
|
||||
- [[upstream-fork-strategy]] — Forking strategy plan (future Dagger integration)
|
||||
- [[forgejo]] — CI/CD infrastructure
|
||||
- [[zot]] — Container registry
|
||||
- [[apps]] — ArgoCD application registry
|
||||
|
|
@ -15,3 +15,5 @@ Plans differ from regular how-to guides in that they describe work that has been
|
|||
|------|--------|-------------|
|
||||
| [[migrate-forgejo-from-brew]] | Planned | Transition Forgejo from Homebrew to source-built binary with LaunchAgent |
|
||||
| [[add-unifi-pulumi-stack]] | Planned | Add Pulumi IaC for UniFi Express 7 home network |
|
||||
| [[adopt-dagger-ci]] | Planned | Adopt Dagger as CI/CD build engine, migrate docs artifacts to Forgejo packages |
|
||||
| [[upstream-fork-strategy]] | Planned | Stacked-branch forking strategy for tracking upstream projects |
|
||||
|
|
|
|||
233
docs/how-to/plans/upstream-fork-strategy.md
Normal file
233
docs/how-to/plans/upstream-fork-strategy.md
Normal file
|
|
@ -0,0 +1,233 @@
|
|||
---
|
||||
title: "Plan: Upstream Fork Strategy"
|
||||
tags:
|
||||
- how-to
|
||||
- plans
|
||||
- forgejo
|
||||
- git
|
||||
---
|
||||
|
||||
# Plan: Upstream Fork Strategy
|
||||
|
||||
> **Status:** Planned (design sketch — not yet executed)
|
||||
|
||||
## Background
|
||||
|
||||
Several BlumeOps projects need to track upstream repositories while maintaining local modifications. Examples include a personal Quartz fork (for docs site customization) and potentially other tools where upstream changes need to flow in continuously.
|
||||
|
||||
The current approach — Forgejo auto-tracking mirrors — works for read-only copies but breaks down when we need to:
|
||||
|
||||
1. Add BlumeOps-specific changes (delete upstream workflows, add `mise.toml`, custom config)
|
||||
2. Develop features that might eventually be upstreamed
|
||||
3. Keep all of this synchronized as upstream evolves
|
||||
|
||||
### Goals
|
||||
|
||||
- Upstream changes flow in automatically (daily rebase)
|
||||
- Rebase conflicts are detected and reported, not silently ignored
|
||||
- BlumeOps-specific patches are cleanly separated from upstream-candidate work
|
||||
- Feature branches that could become upstream PRs are maintained independently
|
||||
|
||||
## Branch Model
|
||||
|
||||
The strategy uses stacked branches — each layer builds on the one below:
|
||||
|
||||
```
|
||||
upstream/main (read-only tracking branch)
|
||||
│
|
||||
▼
|
||||
blumeops (primary branch — blumeops-specific patches)
|
||||
│ e.g., delete .github/, add mise.toml,
|
||||
│ configure for tailnet, etc.
|
||||
│
|
||||
├──▶ feature/foo (feature branch — developed on top of blumeops)
|
||||
│ │ intended for local use, may never go upstream
|
||||
│ │
|
||||
│ └──▶ feature/foo-upstream (optional — same changes rebased onto main)
|
||||
│ for submitting as an upstream PR
|
||||
│
|
||||
└──▶ feature/bar (another feature branch)
|
||||
```
|
||||
|
||||
### Branch Purposes
|
||||
|
||||
| Branch | Base | Purpose | Rebased onto |
|
||||
|--------|------|---------|--------------|
|
||||
| `upstream/main` | — | Tracks upstream's `main` (or `master`) via `git fetch` | Never rebased |
|
||||
| `blumeops` | `upstream/main` | Primary branch; BlumeOps-specific patches only | `upstream/main` (daily) |
|
||||
| `feature/*` | `blumeops` | Feature development | `blumeops` (after successful rebase) |
|
||||
| `feature/*-upstream` | `upstream/main` | Cherry-picked/rebased feature for upstream PR | `upstream/main` (on demand) |
|
||||
|
||||
### What Goes in `blumeops` vs `feature/*`
|
||||
|
||||
**`blumeops` branch** — infrastructure-level changes that are permanent and BlumeOps-specific:
|
||||
- Delete upstream CI workflows (`.github/workflows/`)
|
||||
- Add `mise.toml` for local tooling
|
||||
- Add or modify configuration for the BlumeOps environment
|
||||
- Patch version pins or dependency overrides
|
||||
|
||||
**`feature/*` branches** — functional changes to the project itself:
|
||||
- Bug fixes you want to contribute upstream
|
||||
- New features or customizations
|
||||
- Anything that could theoretically stand on its own as a PR to the upstream project
|
||||
|
||||
This separation ensures the `blumeops` branch stays small and conflict-resistant (infrastructure changes rarely conflict with upstream code changes), while feature branches carry the substantive modifications.
|
||||
|
||||
## Daily Rebase Workflow
|
||||
|
||||
A Forgejo Actions workflow runs on a schedule to keep `blumeops` rebased onto the latest upstream:
|
||||
|
||||
### Workflow Outline
|
||||
|
||||
```
|
||||
trigger: cron (daily) or manual dispatch
|
||||
|
||||
1. Fetch upstream remote
|
||||
2. Check if upstream/main has new commits since last rebase
|
||||
3. If no new commits → exit early
|
||||
4. Attempt: git rebase blumeops --onto upstream/main
|
||||
5. If rebase succeeds:
|
||||
a. Force-push blumeops
|
||||
b. For each feature/* branch:
|
||||
- Attempt rebase onto updated blumeops
|
||||
- If success → force-push
|
||||
- If conflict → skip, record failure
|
||||
6. If rebase fails (conflict):
|
||||
a. Abort rebase
|
||||
b. Create or update a Forgejo issue with conflict details
|
||||
c. Label the issue for visibility
|
||||
```
|
||||
|
||||
### Conflict Reporting
|
||||
|
||||
When a rebase fails, the workflow creates (or updates) a Forgejo issue via the API:
|
||||
|
||||
- **Title:** `Rebase conflict: blumeops onto upstream/main` (or `feature/foo onto blumeops`)
|
||||
- **Body:** Include the conflicting files, the upstream commit range, and the git output
|
||||
- **Labels:** `rebase-conflict`, `automated`
|
||||
- **Assignee:** `eblume`
|
||||
|
||||
The issue serves as a task to manually resolve the conflict. Once resolved and force-pushed, the next daily run succeeds and the issue can be closed.
|
||||
|
||||
### Safety Guards
|
||||
|
||||
- **Never force-push `upstream/main`** — this is a read-only tracking branch, only updated via `git fetch`
|
||||
- **Abort on any rebase ambiguity** — if the rebase produces unexpected state, abort and report rather than pushing garbage
|
||||
- **Dry-run mode** — the workflow should support a manual dispatch input to run in dry-run mode (rebase but don't push, just report what would happen)
|
||||
- **Lock file** — prevent concurrent rebase runs from colliding (Forgejo Actions concurrency groups)
|
||||
|
||||
## One-Time Setup Per Fork
|
||||
|
||||
### Step 1: Create the Mirror
|
||||
|
||||
Set up a Forgejo auto-tracking mirror of the upstream project:
|
||||
```
|
||||
Forgejo → New Migration → Git → URL: https://github.com/org/project.git
|
||||
```
|
||||
|
||||
### Step 2: Disable Mirroring
|
||||
|
||||
Once mirrored, disable the auto-sync in Forgejo repository settings. The repository is now a regular Forgejo repo with the upstream history.
|
||||
|
||||
### Step 3: Set Up Remotes
|
||||
|
||||
```bash
|
||||
cd ~/code/3rd/<project>
|
||||
git remote rename origin forge
|
||||
git remote add upstream https://github.com/org/project.git
|
||||
git fetch upstream
|
||||
```
|
||||
|
||||
### Step 4: Create the `blumeops` Branch
|
||||
|
||||
```bash
|
||||
git checkout upstream/main
|
||||
git checkout -b blumeops
|
||||
# Apply blumeops-specific patches
|
||||
git commit -m "BlumeOps: remove upstream workflows, add mise.toml"
|
||||
git push forge blumeops
|
||||
```
|
||||
|
||||
Set `blumeops` as the default branch in Forgejo repository settings.
|
||||
|
||||
### Step 5: Add the Rebase Workflow
|
||||
|
||||
Add `.forgejo/workflows/rebase-upstream.yaml` to the `blumeops` branch. This workflow is itself a blumeops-specific patch — upstream doesn't have it.
|
||||
|
||||
### Step 6: Protect Branches
|
||||
|
||||
Configure Forgejo branch protection:
|
||||
- `blumeops`: only the rebase workflow (and manual push) can force-push
|
||||
- `upstream/main`: read-only (only updated by the rebase workflow's `git fetch`)
|
||||
|
||||
## The Upstream PR Path
|
||||
|
||||
When a feature is ready to be proposed upstream:
|
||||
|
||||
1. Create `feature/foo-upstream` from `upstream/main`
|
||||
2. Cherry-pick or rebase `feature/foo` commits onto it (excluding any `blumeops`-specific commits)
|
||||
3. Push to the fork on the upstream platform (e.g., GitHub)
|
||||
4. Open PR from the fork to upstream
|
||||
|
||||
This branch is maintained independently — it does not participate in the daily rebase. It's a point-in-time snapshot for the PR. If the PR needs updates, rebase it manually.
|
||||
|
||||
## First Instance: Quartz Fork
|
||||
|
||||
Quartz (the documentation site generator) is the planned first fork and the primary motivation for this strategy.
|
||||
|
||||
- **Upstream:** `https://github.com/jackyzha0/quartz.git`
|
||||
- **Forge repo:** `forge.ops.eblu.me/eblume/quartz`
|
||||
- **Primary branch:** `blumeops`
|
||||
|
||||
### BlumeOps-Specific Patches (`blumeops` branch)
|
||||
|
||||
Changes that are permanently BlumeOps-specific and would never go upstream:
|
||||
|
||||
- Remove `.github/` workflows
|
||||
- Add `mise.toml` with pinned Node version
|
||||
- Configure Quartz defaults for BlumeOps site metadata
|
||||
|
||||
### Feature Work (`feature/*` branches)
|
||||
|
||||
The key feature motivating this fork is **`last-reviewed` frontmatter support**. BlumeOps documentation uses a `last-reviewed` date in frontmatter to track documentation staleness (see `mise run docs-review`). Upstream Quartz has no awareness of this field. The fork enables:
|
||||
|
||||
- **Rendering `last-reviewed` in article headers** — display when a doc was last reviewed, making staleness visible to readers without running CLI tools
|
||||
- **Staleness indicators** — visual styling (e.g., a warning banner) for docs where `last-reviewed` exceeds a threshold
|
||||
- **Sorting/filtering by review date** — Quartz explorer or listing pages that surface docs needing attention
|
||||
|
||||
This is a strong upstream PR candidate — other Quartz users maintaining knowledge bases would benefit from custom frontmatter rendering. The `feature/last-reviewed` branch would be developed on the `blumeops` branch (for local use) with a parallel `feature/last-reviewed-upstream` branch rebased onto `upstream/main` for the PR.
|
||||
|
||||
### Integration with Dagger Docs Build
|
||||
|
||||
This fork directly supports the [[adopt-dagger-ci]] plan. Once the fork exists, the Dagger `build_docs` function switches from cloning upstream Quartz to using the fork:
|
||||
|
||||
```python
|
||||
# Before (cloning upstream):
|
||||
.with_exec(["git", "clone", "--depth=1",
|
||||
"https://github.com/jackyzha0/quartz.git", "/tmp/quartz"])
|
||||
|
||||
# After (using the BlumeOps fork):
|
||||
.with_exec(["git", "clone", "--depth=1", "--branch=blumeops",
|
||||
"https://forge.ops.eblu.me/eblume/quartz.git", "/tmp/quartz"])
|
||||
```
|
||||
|
||||
This means the `build-blumeops.yaml` workflow automatically picks up fork customizations (like `last-reviewed` rendering) when building docs — no separate integration step needed. Local iteration via `dagger call build-docs` also uses the fork, so you can test Quartz customizations against actual BlumeOps content before pushing.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Rebase vs merge:** This plan uses rebase for a clean linear history. Merge commits would avoid force-pushes but create a messier history. Rebase is preferred for small forks; revisit if the commit volume grows.
|
||||
- **Notification mechanism:** Forgejo issues are proposed for conflict reporting. Alternatives: email, Slack webhook, Todoist task via API. Issues are preferred because they're visible in the forge and can carry discussion.
|
||||
- **Feature branch automation:** The daily rebase of feature branches onto `blumeops` is aggressive — it means feature branches are force-pushed daily. An alternative is to only rebase feature branches on demand (manually or via workflow dispatch). Start with manual and automate later based on experience.
|
||||
- **Multiple upstreams:** Some projects track multiple remotes (e.g., a CNCF project with a GitHub mirror and a self-hosted primary). The workflow should support configurable upstream remote URLs.
|
||||
|
||||
## Future Considerations
|
||||
|
||||
- **Renovate integration** — Renovate could watch upstream tags and open PRs to the `blumeops` branch when new releases are available, complementing the daily rebase with release-aware updates
|
||||
- **Dagger integration** — forked projects that produce build artifacts can use the BlumeOps Dagger module for builds, sharing the same local iteration and CI patterns
|
||||
- **Template repository** — once the pattern is proven with quartz, create a template repo or mise task that scaffolds the branch structure and rebase workflow for new forks
|
||||
|
||||
## Related
|
||||
|
||||
- [[adopt-dagger-ci]] — CI/CD build engine (consumes fork artifacts)
|
||||
- [[forgejo]] — Git forge hosting the forks
|
||||
- [[docs]] — Documentation site (first fork consumer)
|
||||
Loading…
Add table
Add a link
Reference in a new issue