P5.1: Migrate minikube from podman to QEMU2 driver #38
2 changed files with 67 additions and 23 deletions
P5.1: Add VM config to ansible role, mark phase complete
- Add hosts file entry for registry.tail8d86e.ts.net in VM - Configure containerd registry mirror to use local zot - Update P5.1 doc with implementation notes and manual steps - Mark P5.1 as complete Manual steps still required after cluster creation: 1. sudo brew services start socket_vmnet (once per reboot) 2. sudo mount -t nfs sifaka:/volume1/torrents /Volumes/torrents-nfs 3. minikube mount /Volumes/torrents-nfs:/mnt/torrents (GUI session) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
commit
26ec02e1be
|
|
@ -77,22 +77,32 @@
|
||||||
msg: "WARNING: minikube may not have started properly. Run 'minikube start' manually on indri if needed. Status: {{ minikube_final_status.stdout | default('unknown') }}"
|
msg: "WARNING: minikube may not have started properly. Run 'minikube start' manually on indri if needed. Status: {{ minikube_final_status.stdout | default('unknown') }}"
|
||||||
when: minikube_final_status.rc != 0 or 'Running' not in minikube_final_status.stdout
|
when: minikube_final_status.rc != 0 or 'Running' not in minikube_final_status.stdout
|
||||||
|
|
||||||
# Configure containerd to use zot as pull-through cache
|
# Configure VM to access zot registry on host
|
||||||
# With qemu2 driver, host is accessible via host.minikube.internal
|
# The VM can't resolve Tailscale hostnames, so we add a hosts entry
|
||||||
# Zot listens on indri:5050 (localhost:5050 from host perspective)
|
# and configure containerd to use the local zot instance
|
||||||
- name: Get host IP for registry mirror config
|
- name: Add registry hostname to VM hosts file
|
||||||
ansible.builtin.command:
|
ansible.builtin.command:
|
||||||
cmd: minikube ssh --native-ssh=false "getent hosts host.minikube.internal | awk '{print \$1}'"
|
cmd: minikube ssh --native-ssh=false "grep -q 'registry.tail8d86e.ts.net' /etc/hosts || echo '192.168.105.1 registry.tail8d86e.ts.net' | sudo tee -a /etc/hosts"
|
||||||
register: minikube_host_ip
|
register: minikube_hosts_entry
|
||||||
changed_when: false
|
changed_when: "'registry.tail8d86e.ts.net' in minikube_hosts_entry.stdout"
|
||||||
failed_when: false
|
|
||||||
when: minikube_final_status.rc == 0 and 'Running' in minikube_final_status.stdout
|
when: minikube_final_status.rc == 0 and 'Running' in minikube_final_status.stdout
|
||||||
|
|
||||||
- name: Configure insecure registry for zot
|
- name: Create containerd registry mirror directory
|
||||||
ansible.builtin.command:
|
ansible.builtin.command:
|
||||||
cmd: "minikube addons configure registry-creds"
|
cmd: minikube ssh --native-ssh=false "sudo mkdir -p /etc/containerd/certs.d/registry.tail8d86e.ts.net"
|
||||||
|
register: minikube_registry_dir
|
||||||
changed_when: false
|
changed_when: false
|
||||||
failed_when: false
|
when: minikube_final_status.rc == 0 and 'Running' in minikube_final_status.stdout
|
||||||
when: false # TODO: Configure containerd registry mirrors after basic migration works
|
|
||||||
# For now, images will be pulled directly from public registries
|
- name: Configure containerd registry mirror for zot
|
||||||
# We can add zot mirror config later via containerd config or minikube addons
|
ansible.builtin.command:
|
||||||
|
cmd: |
|
||||||
|
minikube ssh --native-ssh=false 'echo "server = \"http://host.minikube.internal:5050\"
|
||||||
|
|
||||||
|
[host.\"http://host.minikube.internal:5050\"]
|
||||||
|
capabilities = [\"pull\", \"resolve\"]
|
||||||
|
skip_verify = true" | sudo tee /etc/containerd/certs.d/registry.tail8d86e.ts.net/hosts.toml'
|
||||||
|
register: minikube_registry_config
|
||||||
|
changed_when: minikube_registry_config.rc == 0
|
||||||
|
when: minikube_final_status.rc == 0 and 'Running' in minikube_final_status.stdout
|
||||||
|
notify: Restart containerd in minikube
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
|
|
||||||
**Goal**: Replace the podman driver with qemu2 to enable proper volume mounts (hostPath, NFS, SMB CSI)
|
**Goal**: Replace the podman driver with qemu2 to enable proper volume mounts (hostPath, NFS, SMB CSI)
|
||||||
|
|
||||||
**Status**: Planning
|
**Status**: Complete (2026-01-21)
|
||||||
|
|
||||||
**Prerequisites**: [Phase 5](P5_devpi.complete.md) complete
|
**Prerequisites**: [Phase 5](P5_devpi.complete.md) complete
|
||||||
|
|
||||||
|
|
@ -254,15 +254,49 @@ All state is in git, so cluster recreation is straightforward.
|
||||||
- socket_vmnet provides better networking but requires sudo setup
|
- socket_vmnet provides better networking but requires sudo setup
|
||||||
- Consider creating a LaunchAgent for `minikube mount` if using that approach
|
- Consider creating a LaunchAgent for `minikube mount` if using that approach
|
||||||
|
|
||||||
## Post-Migration Tasks
|
## Implementation Notes (2026-01-21)
|
||||||
|
|
||||||
### Zot Registry Mirror (TODO)
|
### What Actually Worked
|
||||||
|
|
||||||
The CRI-O registry mirror config (`zot-mirror.conf`) is not compatible with containerd. After the basic QEMU2 migration is verified working, we need to:
|
**Volume mounting solution**: NFS mount on indri (host) + `minikube mount` to pass through to VM
|
||||||
|
|
||||||
1. **Reconfigure zot mirrors for containerd** - containerd uses a different config format at `/etc/containerd/config.toml`
|
1. Mount sifaka's torrents share on indri via NFS: `sudo mount -t nfs sifaka:/volume1/torrents /Volumes/torrents-nfs`
|
||||||
2. **Update host address** - `host.containers.internal` (podman-specific) won't work; use `host.minikube.internal` or the host IP instead
|
2. Run `minikube mount /Volumes/torrents-nfs:/mnt/torrents` from indri console (GUI session required due to macOS security)
|
||||||
3. **Test registry caching** - verify images are being cached through zot
|
3. Pods can access `/mnt/torrents` via hostPath
|
||||||
4. **Update ansible role** - add containerd-specific registry mirror configuration
|
|
||||||
|
|
||||||
For now, images will be pulled directly from public registries (docker.io, ghcr.io, etc.) which is fine for the migration but loses the caching benefits.
|
**Why NFS from inside VM didn't work**: Despite allowing 192.168.105.0/24 in Synology NFS settings, the VM got "access denied". Root cause unknown - may be Synology NFS quirk.
|
||||||
|
|
||||||
|
**Why SMB didn't work**: The minikube containerd kernel doesn't include the CIFS module.
|
||||||
|
|
||||||
|
### Zot Registry Mirror (Implemented)
|
||||||
|
|
||||||
|
The ansible role now configures containerd to redirect `registry.tail8d86e.ts.net` to `host.minikube.internal:5050`:
|
||||||
|
- Adds hosts file entry in VM
|
||||||
|
- Creates containerd registry mirror config at `/etc/containerd/certs.d/registry.tail8d86e.ts.net/hosts.toml`
|
||||||
|
|
||||||
|
### Manual Steps Still Required
|
||||||
|
|
||||||
|
These steps cannot be fully automated via ansible and must be done manually:
|
||||||
|
|
||||||
|
1. **socket_vmnet service (once per reboot)**:
|
||||||
|
```bash
|
||||||
|
# On indri console:
|
||||||
|
sudo brew services start socket_vmnet
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **NFS mount on indri (once per reboot)**:
|
||||||
|
```bash
|
||||||
|
# On indri console:
|
||||||
|
sudo mount -t nfs sifaka:/volume1/torrents /Volumes/torrents-nfs
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **minikube mount (must run in GUI session)**:
|
||||||
|
```bash
|
||||||
|
# On indri console (not SSH - requires GUI session for macOS security):
|
||||||
|
minikube mount /Volumes/torrents-nfs:/mnt/torrents
|
||||||
|
# Keep this terminal open - the mount dies if process exits
|
||||||
|
```
|
||||||
|
|
||||||
|
### TODO: LaunchAgent for Persistent Mount
|
||||||
|
|
||||||
|
Create a LaunchAgent to run `minikube mount` at login. Challenge: must run in GUI session context for macOS security model.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue