blumeops/plans/k8s-migration/P5.1_qemu2_migration.md

# Phase 5.1: Migrate Minikube from Podman to QEMU2 Driver

**Goal**: Replace the podman driver with qemu2 to enable proper volume mounts (hostPath, NFS, SMB CSI)

**Status**: Complete (2026-01-21)

**Prerequisites**: [Phase 5](P5_devpi.complete.md) complete

---

## Background

During Phase 6 (Kiwix/Transmission migration), we discovered that the **podman driver has fundamental limitations** that prevent mounting external volumes:

1. **SMB CSI driver fails** with "Operation not permitted" - the rootless container lacks kernel-level mount capabilities
2. **`minikube mount` fails** - 9p mount gets "permission denied" inside the podman VM
3. **hostPath volumes** only work for paths inside the minikube container, not the macOS host

These are documented limitations of the podman driver, which is labeled "experimental" in the [minikube documentation](https://minikube.sigs.k8s.io/docs/drivers/podman/).

### Failed P6 Attempt

Branch `feature/p6-kiwix-transmission` contains the P6 implementation that was blocked by these issues. The manifests are complete and tested, but couldn't mount the torrents volume.

**What was tried:**
- NFS volume mounts - failed due to missing CAP_SYS_ADMIN in podman container
- SMB CSI driver (v1.17.0) - mount fails with EPERM (same root cause)
- `minikube mount /Volumes/torrents:/Volumes/torrents` - 9p mount permission denied
- hostPath PV pointing to `/Volumes/torrents` - path doesn't exist inside minikube container
- Installing cifs-utils in minikube VM - still fails at kernel level

All of these failures trace back to the same root cause: the podman driver runs minikube in a rootless container that lacks the kernel capabilities required for filesystem mounts.

### Why QEMU2?

Multiple sources recommend QEMU2 as the best driver for Apple Silicon Macs:

> "Qemu emulator is the best option to run a Kubernetes Cluster using minikube on MAC arm64-based systems without any issues."
> — [DevOpsCube](https://devopscube.com/minikube-mac/)

QEMU2 creates an actual VM (not a container), which has:
- Full kernel capabilities for mounts
- Proper 9p/virtio filesystem support
- Native NFS client support

---

## Prerequisites (Manual Steps)

### Create Synology User for Kubernetes Storage Access

Create a dedicated Synology user for k8s NFS/SMB access (do not use personal account):

On Synology DSM (Control Panel → User & Group):
1. Create new user: `k8s-storage`
   - Set a strong password
   - No admin privileges needed
   - Deny access to all applications (only needs file services)
2. Set permissions on the `torrents` share:
   - Give `k8s-storage` user Read/Write access
3. Store credentials in 1Password:
   - Vault: `vg6xf6vvfmoh5hqjjhlhbeoaie` (blumeops vault)
   - Item name: `synology-k8s-storage`
   - Fields: `username` (k8s-storage), `password`

---

## Plan

### 1. Export Current State

Before destroying the cluster, capture the current state:

```bash
# List all ArgoCD apps and their sync status
argocd app list

# Backup any runtime state that matters (should be minimal - everything is in git)
kubectl --context=minikube-indri get all --all-namespaces -o yaml > /tmp/k8s-backup.yaml
```

### 2. Stop and Delete Podman Minikube

```bash
# Stop the cluster
minikube stop

# Delete the cluster and all data
minikube delete

# Verify podman VM is cleaned up
podman machine list
```

### 3. Update Ansible Roles for QEMU2

The installation must be orchestrated via ansible, following the existing patterns for `podman` and `minikube` roles.

**Changes needed:**

1. **Update `ansible/roles/minikube/` role:**
   - Change driver from `podman` to `qemu2`
   - Add QEMU as a dependency (via Brewfile or role)
   - Optionally add socket_vmnet for full networking support
   - Update any driver-specific configuration

2. **Update `Brewfile`:**
   ```ruby
   brew "qemu"
   # Optional: brew "socket_vmnet"
   ```

3. **Update minikube start command in role:**
   ```bash
   minikube start \
     --driver=qemu2 \
     --cpus=4 \
     --memory=8192 \
     --disk-size=50g \
     --container-runtime=containerd \
     --kubernetes-version=stable
   ```

4. **Remove or update podman role** (may still be useful for container builds)

### 4. Run Ansible to Create QEMU2 Cluster

```bash
# Run the updated minikube role
mise run provision-indri -- --tags minikube

# Verify cluster is running
minikube status
kubectl get nodes
```

### 5. Configure Host Path Access

With QEMU2, we need to either:

**Option A: Use `minikube mount` (9p)**
```bash
# Start persistent mount (run in background or via launchd)
minikube mount /Volumes/torrents:/Volumes/torrents &
```

**Option B: Use NFS export from macOS**
```bash
# Add NFS export on macOS
echo "/Volumes/torrents -alldirs -mapall=$(id -u):$(id -g) -network 192.168.0.0 -mask 255.255.0.0" | sudo tee -a /etc/exports
sudo nfsd restart

# In k8s, use NFS volume type directly
```

### 6. Test Volume Mount with Test Pod

Create a test pod that mounts the torrents volume:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: volume-test
  namespace: default
spec:
  containers:
    - name: test
      image: busybox
      command: ["sh", "-c", "ls -la /data && sleep 3600"]
      volumeMounts:
        - name: torrents
          mountPath: /data
  volumes:
    - name: torrents
      hostPath:
        path: /Volumes/torrents
        type: Directory
```

Verify:
```bash
kubectl apply -f volume-test.yaml
kubectl logs volume-test
kubectl exec volume-test -- ls -la /data
```

### 7. Redeploy ArgoCD and Existing Apps

```bash
# Re-add ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for ArgoCD to be ready
kubectl wait --for=condition=available deployment/argocd-server -n argocd --timeout=300s

# Re-configure ArgoCD (repo credentials, etc.)
# ... follow P1 setup steps ...

# Sync all apps
argocd app sync apps
```

### 8. Verify All Services

```bash
# Run health check
mise run indri-services-check

# Verify each k8s service
argocd app list
kubectl get pods --all-namespaces
```

### 9. Clean Up Test Pod

```bash
kubectl delete pod volume-test
```

---

## Verification Checklist

- [ ] Podman minikube deleted
- [ ] QEMU2 minikube running
- [ ] `minikube mount` or NFS working
- [ ] Test pod can read `/Volumes/torrents`
- [ ] ArgoCD redeployed and synced
- [ ] All existing apps healthy (grafana, miniflux, devpi, etc.)
- [ ] PostgreSQL cluster healthy
- [ ] Test pod deleted
- [ ] `mise run indri-services-check` passes (except intentionally offline services)

---

## Rollback Plan

If QEMU2 doesn't work:

1. Delete QEMU2 cluster: `minikube delete`
2. Recreate podman cluster following P0/P1 steps
3. Redeploy apps from git

All state is in git, so cluster recreation is straightforward.

---

## Notes

- The QEMU2 VM will use more resources than podman (actual VM vs container)
- First boot may be slower due to VM initialization
- socket_vmnet provides better networking but requires sudo setup
- Consider creating a LaunchAgent for `minikube mount` if using that approach

## Implementation Notes (2026-01-21)

### What Actually Worked

**Volume mounting solution**: NFS mount on indri (host) + `minikube mount` to pass through to VM

1. Mount sifaka's torrents share on indri via NFS: `sudo mount -t nfs sifaka:/volume1/torrents /Volumes/torrents-nfs`
2. Run `minikube mount /Volumes/torrents-nfs:/mnt/torrents` from indri console (GUI session required due to macOS security)
3. Pods can access `/mnt/torrents` via hostPath

**Why NFS from inside VM didn't work**: Despite allowing 192.168.105.0/24 in Synology NFS settings, the VM got "access denied". Root cause unknown - may be Synology NFS quirk.

**Why SMB didn't work**: The minikube containerd kernel doesn't include the CIFS module.

### Zot Registry Mirror (Implemented)

The ansible role now configures containerd to redirect `registry.tail8d86e.ts.net` to `host.minikube.internal:5050`:
- Adds hosts file entry in VM
- Creates containerd registry mirror config at `/etc/containerd/certs.d/registry.tail8d86e.ts.net/hosts.toml`

### Manual Steps Still Required

These steps cannot be fully automated via ansible and must be done manually:

1. **socket_vmnet service (once per reboot)**:
   ```bash
   # On indri console:
   sudo brew services start socket_vmnet
   ```

2. **NFS mount on indri (once per reboot)**:
   ```bash
   # On indri console:
   sudo mount -t nfs sifaka:/volume1/torrents /Volumes/torrents-nfs
   ```

3. **minikube mount (must run in GUI session)**:
   ```bash
   # On indri console (not SSH - requires GUI session for macOS security):
   minikube mount /Volumes/torrents-nfs:/mnt/torrents
   # Keep this terminal open - the mount dies if process exits
   ```

### TODO: LaunchAgent for Persistent Mount

Create a LaunchAgent to run `minikube mount` at login. Challenge: must run in GUI session context for macOS security model.