Expose Forgejo publicly at forge.eblu.me (#278)

## Summary

Expose Forgejo publicly at `forge.eblu.me` via the Fly.io reverse proxy — the first dynamic, authenticated public-facing service.

- **Forgejo hardening:** Domain changed to forge.eblu.me, SSH stays on forge.ops.eblu.me, reverse proxy trust headers configured, local registration locked to external-only (Authentik SSO)
- **Tailscale Ingress:** ExternalName Service + Ingress in tailscale-operator creates forge.tail8d86e.ts.net endpoint
- **Fly.io proxy:** nginx server block with rate-limited auth endpoints (3r/s), fail2ban with custom nginx-deny action, security headers, /swagger blocked, WebSocket support, 512m body limit
- **Authentik:** OAuth callback updated to forge.eblu.me
- **DNS/TLS:** CNAME record in Pulumi, cert in fly-setup
- **Rename:** ~29 files updated from forge.ops.eblu.me to forge.eblu.me (HTTPS refs only; SSH, container builds, and Caddy table kept as-is)

## Deployment Order

1. `mise run provision-indri -- --tags forgejo` (config changes)
2. Verify forge.ops.eblu.me still works
3. `argocd app set tailscale-operator --revision feature/forge-public && argocd app sync tailscale-operator`
4. Verify `curl https://forge.tail8d86e.ts.net`
5. `cd fly && fly deploy`
6. Verify pre-DNS: `curl -H "Host: forge.eblu.me" https://blumeops-proxy.fly.dev/`
7. `fly certs add forge.eblu.me -a blumeops-proxy`
8. `argocd app set authentik --revision feature/forge-public && argocd app sync authentik`
9. `mise run dns-preview && mise run dns-up`
10. Full verification (see below)
11. Rehearse `mise run fly-shutoff`
12. After merge: reset ArgoCD revisions to main, re-sync

## Verification Checklist

- [ ] forge.eblu.me loads, shows public repos
- [ ] forge.ops.eblu.me still works from tailnet
- [ ] SSH clone via forge.ops.eblu.me:2222 works
- [ ] HTTPS clone via forge.eblu.me works
- [ ] UI shows forge.eblu.me for HTTPS clone, forge.ops.eblu.me for SSH
- [ ] /swagger returns 403
- [ ] Rapid login attempts trigger 429 rate limit
- [ ] fail2ban bans after 5 failed logins in 10 minutes
- [ ] ArgoCD can still sync (SSH unaffected)
- [ ] `mise run fly-shutoff` stops all public traffic
- [ ] `mise run services-check` passes

Reviewed-on: #278

docs/changelog.d/feature-forge-public.feature.md

@@ -0,0 +1 @@
+Expose Forgejo publicly at forge.eblu.me via Fly.io reverse proxy with rate limiting, fail2ban, and security hardening.

docs/how-to/authentik/build-authentik-from-source.md

@@ -36,8 +36,8 @@ The `ak` wrapper script in `default.nix` sets PATH/VIRTUAL_ENV and delegates to
 ## Source
 
 All derivations fetch from forge mirrors for supply chain control:
-- https://forge.ops.eblu.me/mirrors/authentik (upstream: `goauthentik/authentik`)
-- https://forge.ops.eblu.me/mirrors/authentik-client-go (upstream: `goauthentik/client-go`)
+- https://forge.eblu.me/mirrors/authentik (upstream: `goauthentik/authentik`)
+- https://forge.eblu.me/mirrors/authentik-client-go (upstream: `goauthentik/client-go`)
 
 Version and hashes are centralized in `containers/authentik/sources.nix`.
 

docs/how-to/configuration/expose-service-publicly.md

@@ -1,7 +1,7 @@
 ---
 title: Expose a Service Publicly
-modified: 2026-02-16
-last-reviewed: 2026-02-16
+modified: 2026-03-03
+last-reviewed: 2026-03-03
 tags:
   - how-to
   - fly-io
@@ -259,7 +259,7 @@ server {
 }
 ```
 
-**Dynamic service template** (e.g., Forgejo — hypothetical, not currently deployed):
+**Dynamic service template** (e.g., Forgejo — see `fly/nginx.conf` for the live configuration):
 
 ```nginx
 # --- forge.eblu.me (dynamic, authenticated) ---
@@ -440,32 +440,30 @@ see plan history in git).
 ### fail2ban
 
 fail2ban monitors log files for repeated failed authentication attempts
-(SSH brute force, bad login passwords, API abuse) and bans IPs via
-firewall rules.
+and bans offending IPs.
 
 **Static sites**: fail2ban does not apply. There is no login surface,
 no sessions, no credentials to brute force.
 
-**Dynamic services with authentication** (e.g., Forgejo): fail2ban is
-relevant and should be configured on **indri**, not on Fly.io. The
-nginx proxy is transparent — it forwards requests but does not see
-authentication outcomes. fail2ban watches the service's own logs on
-indri for patterns like repeated failed logins.
+**Dynamic services with authentication** (e.g., Forgejo): fail2ban
+runs in the **Fly.io container**, not on indri. Standard iptables
+banning won't work in Fly.io because `$remote_addr` is Fly's internal
+proxy IP, not the client. Instead, fail2ban uses a custom nginx-based
+ban action:
 
-Setup considerations for Forgejo specifically:
+1. fail2ban watches the nginx JSON access log for repeated 401/403
+   responses to login endpoints, keyed on the `client_ip` field
+   (populated from the `Fly-Client-IP` header)
+2. On ban, it appends the IP to `/etc/nginx/forge-deny.conf` and
+   reloads nginx
+3. nginx uses a `geo` directive keyed on `$http_fly_client_ip` to
+   check the deny list and return 403 for banned IPs
 
-- Forgejo logs failed auth attempts to its log file
-- fail2ban needs a filter matching Forgejo's log format
-- Banned IPs are blocked at indri's firewall (the Fly.io proxy IP is
-  the Tailscale address of the `flyio-proxy` node, not the end user's
-  IP)
-- **Important**: for fail2ban to see real client IPs, the nginx proxy
-  must pass `X-Real-IP` / `X-Forwarded-For` headers (included in the
-  dynamic service nginx config above), and Forgejo must be configured
-  to trust the proxy and log the forwarded IP rather than the proxy's
-  Tailscale IP
-- Disable open user registration before exposing Forgejo publicly —
-  require explicit invites
+Ban lists are **ephemeral across deploys** — nginx rate limiting
+provides the persistent baseline; fail2ban adds escalating bans for
+active attacks.
+
+See `fly/fail2ban/` for the filter, jail, and action configuration.
 
 ### Break-glass shutoff
 
@@ -504,7 +502,7 @@ dynamic, authenticated service like [[forgejo]].
 - [ ] Disable open user registration (require invites or admin approval)
 - [ ] Audit access controls and permissions
 - [ ] Configure the service to log the forwarded client IP (not the proxy IP)
-- [ ] Set up fail2ban on indri with a filter for the service's log format
+- [ ] Set up fail2ban in the Fly.io container with a filter for the service's login endpoints
 - [ ] Tag the service's Tailscale Ingress with `tag:flyio-target`
 - [ ] Test the nginx config locally or in staging before deploying
 - [ ] Rehearse the break-glass shutoff (`mise run fly-shutoff`)

docs/how-to/configuration/update-documentation.md

@@ -20,7 +20,7 @@ After merging documentation changes to main:
 2. Select version bump type (patch/minor/major) or enter a specific version
 3. The workflow builds, releases, and deploys automatically
 
-Direct link: https://forge.ops.eblu.me/eblume/blumeops/actions?workflow=build-blumeops.yaml
+Direct link: https://forge.eblu.me/eblume/blumeops/actions?workflow=build-blumeops.yaml
 
 ## What the Workflow Does
 

docs/how-to/deployment/build-container-image.md

@@ -93,7 +93,7 @@ Container image tags include the git commit SHA they were built from (e.g. `v3.9
 **The rule:** Production manifests must reference images built from a commit on main. After merging a PR that changed `containers/<name>/`:
 
 1. The merge to main automatically triggers a rebuild (the `build-container.yaml` / `build-container-nix.yaml` workflows fire on pushes to `main` that touch `containers/**`)
-2. Wait for the workflow to complete — check at `https://forge.ops.eblu.me/eblume/blumeops/actions`
+2. Wait for the workflow to complete — check at `https://forge.eblu.me/eblume/blumeops/actions`
 3. Find the new main-SHA tag:
    ```bash
    mise run container-list <name>

docs/how-to/deployment/create-release-artifact-workflow.md

@@ -48,16 +48,16 @@ The upload step uses `FORGE_TOKEN`:
       -X PUT \
       -H "Authorization: token $FORGE_TOKEN" \
       --upload-file "./$TARBALL" \
-      "https://forge.ops.eblu.me/api/packages/eblume/generic/<package>/${VERSION}/${TARBALL}"
+      "https://forge.eblu.me/api/packages/eblume/generic/<package>/${VERSION}/${TARBALL}"
 ```
 
 ## 3. Link the package to the repo
 
-After the first successful upload, the package appears under your **user-level** packages at `https://forge.ops.eblu.me/eblume/-/packages` but is not yet linked to the repo.
+After the first successful upload, the package appears under your **user-level** packages at `https://forge.eblu.me/eblume/-/packages` but is not yet linked to the repo.
 
 To link it:
 
-1. Go to `https://forge.ops.eblu.me/eblume/-/packages`
+1. Go to `https://forge.eblu.me/eblume/-/packages`
 2. Click the package name
 3. Click **Settings**
 4. Under **Link this package to a repository**, select the repo

docs/how-to/plans/completed/adopt-dagger-ci.md

@@ -222,12 +222,12 @@ Migrate `build-blumeops.yaml` to use Dagger for the build logic and switch from
 
 **Current:** Docs tarball uploaded as a Forgejo release asset.
 ```
-https://forge.ops.eblu.me/eblume/blumeops/releases/download/v1.5.2/docs-v1.5.2.tar.gz
+https://forge.eblu.me/eblume/blumeops/releases/download/v1.5.2/docs-v1.5.2.tar.gz
 ```
 
 **New:** Docs tarball uploaded to Forgejo generic packages registry.
 ```
-https://forge.ops.eblu.me/api/packages/eblume/generic/blumeops-docs/v1.6.0/docs-v1.6.0.tar.gz
+https://forge.eblu.me/api/packages/eblume/generic/blumeops-docs/v1.6.0/docs-v1.6.0.tar.gz
 ```
 
 This decouples the docs artifact from git releases while keeping the versioned URL pattern. Forgejo releases can still be created for changelog/announcement purposes without carrying the tarball.
@@ -290,13 +290,13 @@ async def upload_docs(
     async with httpx.AsyncClient() as client:
         with open(f"/tmp/docs-{version}.tar.gz", "rb") as f:
             resp = await client.put(
-                f"https://forge.ops.eblu.me/api/packages/eblume/generic/"
+                f"https://forge.eblu.me/api/packages/eblume/generic/"
                 f"blumeops-docs/{version}/docs-{version}.tar.gz",
                 headers={"Authorization": f"token {token}"},
                 content=f.read(),
             )
             resp.raise_for_status()
-    return f"https://forge.ops.eblu.me/api/packages/eblume/generic/blumeops-docs/{version}/docs-{version}.tar.gz"
+    return f"https://forge.eblu.me/api/packages/eblume/generic/blumeops-docs/{version}/docs-{version}.tar.gz"
 
 @function
 async def release_docs(
@@ -388,7 +388,7 @@ jobs:
       - name: Update manifest and commit
         run: |
           VERSION="${{ steps.version.outputs.version }}"
-          URL="https://forge.ops.eblu.me/api/packages/eblume/generic/blumeops-docs/${VERSION}/docs-${VERSION}.tar.gz"
+          URL="https://forge.eblu.me/api/packages/eblume/generic/blumeops-docs/${VERSION}/docs-${VERSION}.tar.gz"
           sed -i "s|value: \"https://.*\"|value: \"${URL}\"|" \
             argocd/manifests/docs/deployment.yaml
           git config user.name "Forgejo Actions"
@@ -405,11 +405,11 @@ The quartz container's `DOCS_RELEASE_URL` env var in `argocd/manifests/docs/depl
 ```yaml
 # Before (Forgejo releases):
 - name: DOCS_RELEASE_URL
-  value: "https://forge.ops.eblu.me/eblume/blumeops/releases/download/v1.5.2/docs-v1.5.2.tar.gz"
+  value: "https://forge.eblu.me/eblume/blumeops/releases/download/v1.5.2/docs-v1.5.2.tar.gz"
 
 # After (Forgejo generic packages):
 - name: DOCS_RELEASE_URL
-  value: "https://forge.ops.eblu.me/api/packages/eblume/generic/blumeops-docs/v1.6.0/docs-v1.6.0.tar.gz"
+  value: "https://forge.eblu.me/api/packages/eblume/generic/blumeops-docs/v1.6.0/docs-v1.6.0.tar.gz"
 ```
 
 The quartz container's `start.sh` already downloads from `DOCS_RELEASE_URL` via curl — no container changes needed, just the URL format changes.

docs/how-to/plans/migrate-forgejo-from-brew.md

@@ -32,7 +32,7 @@ https://codeberg.org/forgejo/forgejo.git
 Add the forge mirror as a secondary remote for convenience and backup:
 
 ```
-https://forge.ops.eblu.me/mirrors/forgejo.git
+https://forge.eblu.me/mirrors/forgejo.git
 ```
 
 ## One-Time Migration Steps
@@ -48,7 +48,7 @@ ssh indri 'git clone https://codeberg.org/forgejo/forgejo.git ~/code/3rd/forgejo
 ### 2. Add Forge Mirror as Secondary Remote
 
 ```fish
-ssh indri 'cd ~/code/3rd/forgejo && git remote add forge https://forge.ops.eblu.me/mirrors/forgejo.git'
+ssh indri 'cd ~/code/3rd/forgejo && git remote add forge https://forge.eblu.me/mirrors/forgejo.git'
 ```
 
 ### 3. Check Out the Desired Version Tag
@@ -155,7 +155,7 @@ Replace brew install/start with binary-check + LaunchAgent pattern (matching `an
 #    ssh indri 'git clone https://codeberg.org/forgejo/forgejo.git ~/code/3rd/forgejo'
 #
 # 2. Add forge mirror as secondary remote:
-#    ssh indri 'cd ~/code/3rd/forgejo && git remote add forge https://forge.ops.eblu.me/mirrors/forgejo.git'
+#    ssh indri 'cd ~/code/3rd/forgejo && git remote add forge https://forge.eblu.me/mirrors/forgejo.git'
 #
 # 3. Set up Go and Node via mise:
 #    ssh indri 'cd ~/code/3rd/forgejo && mise use go@1.24 node@20'
@@ -275,7 +275,7 @@ No changes needed — paths already flow through variables in `defaults/main.yml
 After running the migration and Ansible:
 
 - [ ] `ssh indri 'launchctl list mcquack.eblume.forgejo'` — shows running
-- [ ] `curl https://forge.ops.eblu.me/api/v1/version` — returns JSON with version
+- [ ] `curl https://forge.eblu.me/api/v1/version` — returns JSON with version
 - [ ] Git clone over SSH: `git clone ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git /tmp/test-clone`
 - [ ] Git push works on an existing clone
 - [ ] Ansible dry-run is clean: `mise run provision-indri -- --tags forgejo --check --diff`

docs/how-to/plans/upstream-fork-strategy.md

@@ -209,7 +209,7 @@ This fork directly supports the [[adopt-dagger-ci]] plan. Once the fork exists,
 
 # After (using the BlumeOps fork):
 .with_exec(["git", "clone", "--depth=1", "--branch=blumeops",
-             "https://forge.ops.eblu.me/mirrors/quartz.git", "/tmp/quartz"])
+             "https://forge.eblu.me/mirrors/quartz.git", "/tmp/quartz"])
 ```
 
 This means the `build-blumeops.yaml` workflow automatically picks up fork customizations (like `last-reviewed` rendering) when building docs — no separate integration step needed. Local iteration via `dagger call build-docs` also uses the fork, so you can test Quartz customizations against actual BlumeOps content before pushing.

docs/how-to/zot/register-zot-oidc-client.md

@@ -12,7 +12,7 @@ tags:
 
 Register a zot OAuth2 provider and application in Authentik via blueprint, following the same pattern as Grafana and Forgejo.
 
-Completed in PR [#236](https://forge.ops.eblu.me/eblume/blumeops/pulls/236).
+Completed in PR [#236](https://forge.eblu.me/eblume/blumeops/pulls/236).
 
 ## What Was Done
 

docs/index.md

@@ -16,7 +16,7 @@ infrastructure.
 
 BlumeOps is my personal homelab infrastructure managed entirely through code.
 Everything lives in a [single git repository](https://github.com/eblume/blumeops), from service configs to
-deployment automation. Even the [[forgejo]] instance that [hosts this repo](https://forge.ops.eblu.me/eblume/blumeops)
+deployment automation. Even the [[forgejo]] instance that [hosts this repo](https://forge.eblu.me/eblume/blumeops)
 is defined within it, making BlumeOps fully self-hosting. It's a digital life
 raft I built for myself as I went, and you can see it all from within your
 editor of choice. (I recommend vim.)

docs/quartz.layout.ts

@@ -14,7 +14,7 @@ export const sharedPageComponents: SharedLayout = {
   footer: Component.Footer({
     links: {
       "GitHub": "https://github.com/eblume/blumeops",
-      "Forge": "https://forge.ops.eblu.me/eblume/blumeops",
+      "Forge": "https://forge.eblu.me/eblume/blumeops",
     },
   }),
 }

docs/reference/infrastructure/routing.md

@@ -1,6 +1,6 @@
 ---
 title: Routing
-modified: 2026-02-09
+modified: 2026-03-03
 tags:
   - infrastructure
   - networking
@@ -49,6 +49,7 @@ DNS CNAMEs point to `blumeops-proxy.fly.dev`. TLS via Fly.io-managed Let's Encry
 | Service | URL | Description |
 |---------|-----|-------------|
 | [[docs]] | https://docs.eblu.me | Documentation site |
+| [[forgejo]] | https://forge.eblu.me | Git hosting (public) |
 
 ## Tailscale-Only Services
 

docs/reference/services/forgejo.md

@@ -1,6 +1,6 @@
 ---
 title: Forgejo
-modified: 2026-02-20
+modified: 2026-03-03
 tags:
   - service
   - git
@@ -15,7 +15,8 @@ Git forge and CI/CD platform. **Primary source of truth for blumeops** (mirrored
 
 | Property | Value |
 |----------|-------|
-| **URL** | https://forge.ops.eblu.me |
+| **URL (public)** | https://forge.eblu.me |
+| **URL (internal)** | https://forge.ops.eblu.me |
 | **SSH** | `ssh://forgejo@forge.ops.eblu.me:2222` |
 | **Local Ports** | 3001 (HTTP), 2200 (SSH) |
 | **Config** | `ansible/roles/forgejo/templates/app.ini.j2` |
@@ -71,7 +72,7 @@ mise run provision-indri -- --tags forgejo_actions_secrets
 
 The Ansible role authenticates to the Forgejo API using a Personal Access Token (PAT). This PAT must be created manually:
 
-1. Go to https://forge.ops.eblu.me/user/settings/applications
+1. Go to https://forge.eblu.me/user/settings/applications
 2. Create a new token with `write:repository` scope
 3. Store it in 1Password → "Forgejo Secrets" item → `api-token` field
 
@@ -94,23 +95,30 @@ This is a bootstrapping requirement - the PAT enables IaC for all other secrets.
 
 **Break-glass:** Local password login always works (with local MFA). Authentik SSO is additive — if Authentik is down, log in with local credentials.
 
-## Future: Public Access
+## Public Access
 
-Forgejo can be exposed publicly at `forge.eblu.me` via [[flyio-proxy]]. Since Forgejo runs natively on [[indri]] (not in k8s), the pattern is:
+Forgejo is publicly accessible at `https://forge.eblu.me` via [[flyio-proxy]]. This is the first dynamic, authenticated service exposed publicly.
 
-1. Create a k8s ExternalName Service pointing to indri's Tailscale IP
-2. Create a Tailscale Ingress with `tailscale.com/tags: "tag:k8s,tag:flyio-target"`
-3. Add the nginx server block and DNS CNAME
+| Access Method | URL | Reachable From |
+|---------------|-----|----------------|
+| **HTTPS (public)** | https://forge.eblu.me | Public internet |
+| **HTTPS (internal)** | https://forge.ops.eblu.me | Tailnet only |
+| **SSH** | `ssh://forgejo@forge.ops.eblu.me:2222` | Tailnet only |
 
-Exposing a dynamic, authenticated service like Forgejo requires a full security review before going live:
+The UI shows `forge.eblu.me` for HTTPS clone URLs and `forge.ops.eblu.me` for SSH clone URLs.
 
-- Disable all local registration — only allow login via [[authentik]] (`DISABLE_REGISTRATION = true`, `ALLOW_ONLY_EXTERNAL_REGISTRATION = true`)
-- Configure fail2ban on indri with a filter for Forgejo's log format
-- Ensure Forgejo logs the forwarded client IP (`X-Real-IP`) rather than the proxy's Tailscale IP
-- Audit repository visibility defaults and permissions
-- Rehearse the break-glass shutoff (`mise run fly-shutoff`)
+### Security Controls
 
-See [[expose-service-publicly]] for the full howto and dynamic service checklist.
+- **Registration:** Local registration disabled; only [[authentik]] SSO login allowed (`ALLOW_ONLY_EXTERNAL_REGISTRATION = true`)
+- **Reverse proxy trust:** `REVERSE_PROXY_LIMIT = 2`, `REVERSE_PROXY_TRUSTED_PROXIES = *` — Forgejo logs the real client IP from `X-Real-IP` header, not the proxy's Tailscale IP
+- **Rate limiting:** nginx rate limits login/signup/forgot-password endpoints (3r/s per client IP via `Fly-Client-IP` header)
+- **fail2ban:** Runs in the Fly.io container; bans IPs after 5 failed logins in 10 minutes via nginx deny list (ephemeral across deploys)
+- **Swagger:** Blocked at the proxy (`/swagger` returns 403); use forge.ops.eblu.me for API access
+- **OAuth dead-end:** "Sign in with Authentik" redirects to the (tailnet-only) Authentik URL — SSO only works from the tailnet
+
+### Break-glass
+
+`mise run fly-shutoff` stops all public traffic immediately. forge.ops.eblu.me continues to work from the tailnet. See [[expose-service-publicly#Break-glass shutoff]].
 
 ## Related
 

docs/tutorials/contributing.md

@@ -16,7 +16,7 @@ This tutorial walks through making your first contribution to BluemeOps - from u
 
 Before contributing, you'll need:
 - Access to the [[tailscale|Tailscale]] network (request from Erich)
-- SSH key added to [[forgejo|Forgejo]] (https://forge.ops.eblu.me)
+- SSH key added to [[forgejo|Forgejo]] (https://forge.eblu.me)
 - Development tools installed (see below)
 
 ## Tooling Setup