Add Caddy layer4 for Forgejo SSH (#56)

## Summary
- Add layer4 TCP proxy configuration to Caddyfile template for SSH services
- Configure Forgejo SSH on port 2222 → localhost:2200
- Switch HTTPS from port 8443 (testing) to 443 (production)
- Requires Caddy rebuilt with `github.com/mholt/caddy-l4` plugin

## What This Enables
Git+SSH access via `forge.ops.eblu.me:2222` is now accessible from:
- Tailnet clients (gilbert)
- Docker containers on indri
- Kubernetes pods in minikube

This solves the DNS resolution issues where containers couldn't reach Tailscale MagicDNS names.

## Testing Done
- [x] Caddy rebuilt with layer4 plugin
- [x] Validated Caddyfile syntax
- [x] Cleared `svc:forge` from tailscale serve
- [x] Verified HTTPS works: `curl https://forge.ops.eblu.me`
- [x] Verified SSH works: `ssh -p 2222 forgejo@forge.ops.eblu.me`
- [x] Verified git clone works via new endpoint
- [x] Verified minikube pods can reach both HTTPS and SSH endpoints

## Deployment
Caddy is already running with the new config on indri. This PR captures the ansible changes.

## Next Steps
- Update zk docs with new git remote format
- Migrate registry and other services to Caddy
- Retire tailscale_services ansible role

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: https://forge.tail8d86e.ts.net/eblume/blumeops/pulls/56

Brewfile

@@ -2,5 +2,5 @@
 brew "actionlint"  # GitHub/Forgejo Actions workflow linter
 brew "argocd"  # ArgoCD CLI for GitOps management
 brew "bat"  # Syntax-highlighted file concatenation
-brew "tea"  # Gitea/Forgejo CLI for forge.tail8d86e.ts.net
+brew "tea"  # Gitea/Forgejo CLI for forge.ops.eblu.me
 brew "podman"  # Container CLI (uses VM on macOS, for building/pushing images)

CLAUDE.md

@@ -143,7 +143,7 @@ mise run container-release runner v1.0.0      # Tag and trigger build workflow
 ## Third-Party Projects
 
 When a task requires cloning or using a third-party git repository (e.g., for building from source), **ask the user to mirror it on forge first**, then clone from the mirror:
-- Mirror location: `https://forge.tail8d86e.ts.net/eblume/<project>.git`
+- Mirror location: `https://forge.ops.eblu.me/eblume/<project>.git`
 - Clone to: `~/code/3rd/<project>/`
 
 This avoids external dependencies and ensures the project is available even if the upstream is unreachable.

ansible/group_vars/all.yml

@@ -1,2 +1,2 @@
 ---
-ansible_managed: "Managed by ansible - do not edit. Source: ssh://forgejo@forge.tail8d86e.ts.net/eblume/blumeops.git"
+ansible_managed: "Managed by ansible - do not edit. Source: ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git"

ansible/roles/alloy/defaults/main.yml

@@ -10,7 +10,7 @@
 # Build on dev machine (gilbert), then copy to indri:
 #
 # 1. Clone from forge mirror:
-#    git clone ssh://forgejo@forge.tail8d86e.ts.net/eblume/alloy.git ~/code/3rd/alloy
+#    git clone ssh://forgejo@forge.ops.eblu.me:2222/eblume/alloy.git ~/code/3rd/alloy
 #
 # 2. Set up build tools via mise:
 #    cd ~/code/3rd/alloy && mise use go@1.25 node yarn

ansible/roles/caddy/defaults/main.yml

@@ -15,9 +15,8 @@ caddy_gandi_token_file: /Users/erichblume/.config/caddy/gandi-token
 # Domain configuration
 caddy_domain: ops.eblu.me
 
-# Listen on Tailscale interface only (port 443)
-# Use 8443 during testing to avoid conflicts with Tailscale serve
-caddy_https_port: 8443
+# HTTPS port (443 is standard)
+caddy_https_port: 443
 
 # Services to proxy
 # Format: { name: "service", host: "hostname", backend: "url" }
@@ -35,3 +34,9 @@ caddy_services:
   # - name: grafana
   #   host: "grafana.{{ caddy_domain }}"
   #   backend: "http://minikube-ip:nodeport"
+
+# SSH services (Layer 4 TCP proxy)
+# Format: { port: external_port, backend: "host:port" }
+caddy_ssh_services:
+  - port: 2222
+    backend: "localhost:2200"  # Forgejo SSH

ansible/roles/caddy/templates/Caddyfile.j2

@@ -7,6 +7,19 @@
 {
 	# Global options
 	admin off
+
+{% if caddy_ssh_services %}
+	# Layer 4 (TCP) routing for SSH services
+	layer4 {
+{% for ssh_svc in caddy_ssh_services %}
+		:{{ ssh_svc.port }} {
+			route {
+				proxy {{ ssh_svc.backend }}
+			}
+		}
+{% endfor %}
+	}
+{% endif %}
 }
 
 # Wildcard certificate for all services

ansible/roles/forgejo/defaults/main.yml

@@ -18,7 +18,7 @@ forgejo_log_path: "{{ forgejo_work_path }}/log"
 # Server settings
 forgejo_http_addr: 0.0.0.0
 forgejo_http_port: 3001
-forgejo_domain: forge.tail8d86e.ts.net
+forgejo_domain: forge.ops.eblu.me
 forgejo_ssh_domain: "{{ forgejo_domain }}"
 forgejo_root_url: "https://{{ forgejo_domain }}/"
 forgejo_offline_mode: true
@@ -27,7 +27,7 @@ forgejo_offline_mode: true
 forgejo_disable_ssh: false
 forgejo_start_ssh_server: true
 forgejo_builtin_ssh_user: forgejo
-forgejo_ssh_port: 22
+forgejo_ssh_port: 2222
 forgejo_ssh_listen_port: 2200
 forgejo_lfs_start_server: true
 

ansible/roles/tailscale_serve/defaults/main.yml

@@ -1,16 +1,11 @@
 ---
 # Tailscale serve configuration for this host
 # Each service maps a Tailscale service name to local endpoints
+#
+# NOTE: forge has been migrated to Caddy (forge.ops.eblu.me)
+# Registry will be migrated next, then this role can be retired.
 
 tailscale_serve_services:
-  - name: svc:forge
-    https:
-      port: 443
-      upstream: http://localhost:3001
-    tcp:
-      port: 22
-      upstream: tcp://localhost:2200
-
   - name: svc:registry
     https:
       port: 443

argocd/manifests/tailscale-operator/README.md

@@ -86,5 +86,5 @@ kubectl logs -n tailscale -l app.kubernetes.io/name=operator
   annotations:
     tailscale.com/proxy-class: "default"
   ```
-- The egress proxy for forge targets `indri.tail8d86e.ts.net` directly (not `forge.tail8d86e.ts.net`)
-  because Tailscale Serve hostnames are virtual and only work via the Tailscale client.
+- The egress proxy for forge is **deprecated**. Forge is now accessible via Caddy at
+  `forge.ops.eblu.me` (HTTPS) and `forge.ops.eblu.me:2222` (SSH), which pods can reach directly.

argocd/manifests/tailscale-operator/egress-forge.yaml

@@ -1,7 +1,10 @@
-# Egress proxy to expose Forgejo (forge) to the cluster
-# Forge runs on indri:3001, exposed via Tailscale Serve as forge.tail8d86e.ts.net
-# We target indri directly since egress can't reach Tailscale Serve hostnames
+# DEPRECATED: This egress proxy is no longer needed.
+# Forge is now accessible via Caddy at forge.ops.eblu.me (HTTPS) and
+# forge.ops.eblu.me:2222 (SSH), which pods can reach directly.
 #
+# Keeping this file for reference during migration. Remove once verified.
+#
+# Original purpose: Egress proxy to expose Forgejo (forge) to the cluster
 # See: https://tailscale.com/kb/1438/kubernetes-operator-cluster-egress
 ---
 apiVersion: v1

containers/nettest/test-connectivity.sh

@@ -14,8 +14,8 @@ echo "Hostname: $(hostname)"
 echo ""
 
 # Test targets
-FORGE_HOST="forge.tail8d86e.ts.net"
-REGISTRY_HOST="registry.tail8d86e.ts.net"
+FORGE_HOST="forge.ops.eblu.me"
+REGISTRY_HOST="registry.ops.eblu.me"
 
 test_dns() {
     local host="$1"

mise-tasks/container-tag-and-release

@@ -71,4 +71,4 @@ echo "The workflow will now build and push:"
 echo "  registry.tail8d86e.ts.net/$IMAGE:$VERSION"
 echo ""
 echo "Monitor the build at:"
-echo "  https://forge.tail8d86e.ts.net/eblume/blumeops/actions"
+echo "  https://forge.ops.eblu.me/eblume/blumeops/actions"

mise-tasks/indri-runner-logs

@@ -12,7 +12,7 @@ if [[ -z "$RUN_ID" ]]; then
     echo "Only works for runs executed by the indri-host-runner."
     echo ""
     echo "Recent runs:"
-    curl -sf "https://forge.tail8d86e.ts.net/api/v1/repos/eblume/blumeops/actions/tasks" | \
+    curl -sf "https://forge.ops.eblu.me/api/v1/repos/eblume/blumeops/actions/tasks" | \
         jq -r '.workflow_runs[:10] | .[] | "  \(.id)\t\(.status)\t\(.workflow_id)\t\(.display_title | .[0:50])"'
     exit 1
 fi

mise-tasks/indri-services-check

@@ -70,7 +70,7 @@ check_http "Prometheus" "https://prometheus.tail8d86e.ts.net/-/healthy"
 check_http "Loki" "https://loki.tail8d86e.ts.net/ready"
 check_http "Grafana" "https://grafana.tail8d86e.ts.net/api/health"
 check_http "ArgoCD" "https://argocd.tail8d86e.ts.net/healthz"
-check_http "Forgejo" "https://forge.tail8d86e.ts.net/"
+check_http "Forgejo" "https://forge.ops.eblu.me/"
 check_http "Zot Registry" "https://registry.tail8d86e.ts.net/v2/_catalog"
 check_http "Kiwix" "https://kiwix.tail8d86e.ts.net/"
 check_http "Miniflux" "https://feed.tail8d86e.ts.net/healthcheck"

mise-tasks/pr-comments

@@ -20,7 +20,7 @@ import httpx
 from rich.console import Console
 from rich.text import Text
 
-FORGE_API_BASE = "https://forge.tail8d86e.ts.net/api/v1"
+FORGE_API_BASE = "https://forge.ops.eblu.me/api/v1"
 REPO_OWNER = "eblume"
 REPO_NAME = "blumeops"