From 765117bb9ef81686b8c9a0225771d2102c0c761e Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 14:54:36 -0800 Subject: [PATCH] P5: Mark devpi migration complete - Add P5_devpi.complete.md with implementation notes - Document key learnings (registry mirrors, memory, env vars) - Remove incomplete P5_devpi.md Co-Authored-By: Claude Opus 4.5 --- plans/k8s-migration/P5_devpi.complete.md | 102 ++++++++++++ plans/k8s-migration/P5_devpi.md | 190 ----------------------- 2 files changed, 102 insertions(+), 190 deletions(-) create mode 100644 plans/k8s-migration/P5_devpi.complete.md delete mode 100644 plans/k8s-migration/P5_devpi.md diff --git a/plans/k8s-migration/P5_devpi.complete.md b/plans/k8s-migration/P5_devpi.complete.md new file mode 100644 index 0000000..0454a4a --- /dev/null +++ b/plans/k8s-migration/P5_devpi.complete.md @@ -0,0 +1,102 @@ +# Phase 5: devpi Migration to Kubernetes + +**Goal**: Migrate devpi PyPI caching proxy from indri to k8s + +**Status**: Complete (2026-01-20) + +**Prerequisites**: [Phase 4](P4_miniflux.complete.md) complete + +--- + +## Summary + +Successfully migrated devpi from mcquack LaunchAgent on indri to Kubernetes: +- Custom container image with devpi-server + devpi-web + auto-init startup script +- StatefulSet with 50Gi PVC for data persistence +- Tailscale Ingress at `pypi.tail8d86e.ts.net` +- Root password from 1Password secret, auto-initialized on first run +- Verified pip caching proxy and mcquack package upload + +--- + +## Key Learnings + +### Registry Mirror Configuration +- Minikube's CRI-O can't resolve Tailscale hostnames directly +- Added registry mirror config to redirect `registry.tail8d86e.ts.net` → `host.containers.internal:5050` +- Also added direct insecure registry entry for `host.containers.internal:5050` +- Config in `ansible/roles/minikube/files/zot-mirror.conf` + +### Memory Requirements +- devpi-web's Whoosh search indexer needs significant memory during PyPI index build +- Initial 512Mi limit caused OOMKills +- Solution: High limit (2Gi) with low request (256Mi) - memory reclaimed after indexing + +### Environment Variable Conflicts +- Kubernetes auto-sets `DEVPI_PORT` for service discovery +- Conflicted with our port config - renamed to `DEVPI_LISTEN_PORT` + +### Tailscale Serve Cleanup +- Use `tailscale serve status --json` to see entries (non-JSON output can be empty) +- Use `tailscale serve clear svc:` to remove entries + +### ArgoCD Workflow +- Changed `apps` to manual sync (was auto-sync with prune) +- Workflow: sync apps → set revision to feature branch → sync service → test → reset to main after merge + +--- + +## Verification Checklist + +- [x] devpi pod healthy in k8s +- [x] https://pypi.tail8d86e.ts.net accessible +- [x] Web interface shows root/pypi index +- [x] `pip install ` works through proxy +- [x] mcquack v1.0.0 uploaded to eblume/dev +- [x] `pip install --index-url https://pypi.tail8d86e.ts.net/eblume/dev/+simple/ mcquack` works +- [x] Old devpi service removed from indri +- [ ] zk documentation updated (deferred - no existing devpi card) + +--- + +## Files Changed + +### New Files +| Path | Purpose | +|------|---------| +| `argocd/apps/devpi.yaml` | ArgoCD Application definition | +| `argocd/manifests/devpi/Dockerfile` | Container image with startup script | +| `argocd/manifests/devpi/start.sh` | Auto-init startup script | +| `argocd/manifests/devpi/statefulset.yaml` | StatefulSet with PVC | +| `argocd/manifests/devpi/service.yaml` | ClusterIP Service | +| `argocd/manifests/devpi/ingress-tailscale.yaml` | Tailscale Ingress | +| `argocd/manifests/devpi/kustomization.yaml` | Kustomize configuration | +| `argocd/manifests/devpi/secret-root.yaml.tpl` | 1Password secret template | +| `argocd/manifests/devpi/README.md` | Setup documentation | + +### Modified Files +| Path | Change | +|------|--------| +| `CLAUDE.md` | Added k8s/ArgoCD workflow documentation | +| `ansible/playbooks/indri.yml` | Removed devpi and devpi_metrics roles | +| `ansible/roles/tailscale_serve/defaults/main.yml` | Removed svc:pypi | +| `ansible/roles/alloy/defaults/main.yml` | Removed devpi log collection | +| `ansible/roles/borgmatic/defaults/main.yml` | Removed devpi backup paths | +| `ansible/roles/minikube/files/zot-mirror.conf` | Added registry mirror for Tailscale hostname | +| `argocd/apps/apps.yaml` | Changed to manual sync policy | + +### Roles Kept (not deleted) +- `ansible/roles/devpi/` - Kept for reference +- `ansible/roles/devpi_metrics/` - Kept for reference + +--- + +## Post-Merge Cleanup + +After PR merge, reset ArgoCD apps to main: +```fish +argocd app set apps --revision main +argocd app sync apps +argocd app set devpi --revision main +argocd app sync devpi +``` diff --git a/plans/k8s-migration/P5_devpi.md b/plans/k8s-migration/P5_devpi.md deleted file mode 100644 index b5ea9d2..0000000 --- a/plans/k8s-migration/P5_devpi.md +++ /dev/null @@ -1,190 +0,0 @@ -# Phase 5: devpi Migration to Kubernetes - -**Goal**: Migrate devpi PyPI caching proxy from indri to k8s - -**Status**: In Progress (2026-01-20) - -**Prerequisites**: [Phase 4](P4_miniflux.complete.md) complete - ---- - -## Overview - -This phase migrates devpi from mcquack LaunchAgent on indri to Kubernetes: -1. Stop devpi on indri and clear Tailscale service -2. Build custom devpi container image (devpi-server + devpi-web) -3. Deploy as StatefulSet with PVC for data persistence -4. Expose via Tailscale Ingress at `pypi.tail8d86e.ts.net` -5. Initialize fresh (no data migration - just cache and one private package) -6. Validate with pip proxy and mcquack package upload - -**Decision: Fresh start** - The only private package is mcquack v1.0.0 which we'll re-upload. The PyPI cache is re-fetchable. - ---- - -## Steps - -### 1. Stop devpi on indri (FIRST) - -```bash -# Stop and unload the LaunchAgent -ssh indri 'launchctl unload ~/Library/LaunchAgents/mcquack.eblume.devpi.plist' - -# Also stop the metrics collector -ssh indri 'launchctl unload ~/Library/LaunchAgents/mcquack.eblume.devpi-metrics.plist' - -# Clear any tailscale serve entries (if any) -ssh indri 'tailscale serve off --service=svc:pypi' 2>/dev/null || true -``` - -**User action required**: Delete the `pypi` device/service from Tailscale admin console. - -### 2. Build devpi container image - -Create `argocd/manifests/devpi/Dockerfile`: - -```dockerfile -FROM python:3.12-slim - -# Install devpi-server and devpi-web -RUN pip install --no-cache-dir devpi-server devpi-web - -# Create data directory -RUN mkdir -p /devpi - -# Expose default port -EXPOSE 3141 - -# Use ENTRYPOINT for flexibility -ENTRYPOINT ["devpi-server"] - -# Default args (can be overridden) -CMD ["--serverdir", "/devpi", "--host", "0.0.0.0", "--port", "3141"] -``` - -Build and push to Zot: - -```bash -# From gilbert (has podman) -cd argocd/manifests/devpi -podman build -t registry.tail8d86e.ts.net/blumeops/devpi:latest . -podman push registry.tail8d86e.ts.net/blumeops/devpi:latest -``` - -### 3. Create k8s manifests - -| File | Purpose | -|------|---------| -| `argocd/apps/devpi.yaml` | ArgoCD Application definition | -| `argocd/manifests/devpi/statefulset.yaml` | StatefulSet with PVC | -| `argocd/manifests/devpi/service.yaml` | ClusterIP Service | -| `argocd/manifests/devpi/ingress-tailscale.yaml` | Tailscale Ingress for `pypi.tail8d86e.ts.net` | -| `argocd/manifests/devpi/init-job.yaml` | One-time initialization job | -| `argocd/manifests/devpi/kustomization.yaml` | Kustomize configuration | - -### 4. Deploy via ArgoCD - -```bash -# Point app at feature branch for testing -argocd app create devpi \ - --repo ssh://forgejo@indri.tail8d86e.ts.net:2200/eblume/blumeops.git \ - --path argocd/manifests/devpi \ - --dest-server https://kubernetes.default.svc \ - --dest-namespace devpi \ - --revision feature/p5-devpi - -# Or add to apps/ and let app-of-apps sync -argocd app sync apps -argocd app sync devpi -``` - -### 5. Initialize devpi - -After StatefulSet is running: - -```bash -# Exec into pod to initialize -kubectl -n devpi exec -it devpi-0 -- devpi-init --serverdir /devpi --root-passwd - -# Or use init job with secret -``` - -Create user and index: - -```bash -# From workstation -uvx devpi use https://pypi.tail8d86e.ts.net -uvx devpi login root -uvx devpi user -c eblume email=blume.erich@gmail.com -uvx devpi index -c eblume/dev bases=root/pypi -``` - -### 6. Update pip.conf on gilbert - -The existing `~/.config/pip/pip.conf` should work unchanged since we're keeping the same hostname `pypi.tail8d86e.ts.net`. - -### 7. Remove devpi ansible role - -- Remove `devpi` from `ansible/playbooks/indri.yml` roles list -- Remove `devpi_metrics` from alloy textfile collectors -- Keep devpi ansible role directory for reference (or delete) - -### 8. Validate - -```bash -# Test pip cache proxy -pip install --index-url https://pypi.tail8d86e.ts.net/root/pypi/+simple/ requests - -# Upload mcquack -cd ~/code/personal/mcquack -uv build -uv publish --publish-url https://pypi.tail8d86e.ts.net/eblume/dev/ -``` - ---- - -## New Files - -| Path | Purpose | -|------|---------| -| `argocd/apps/devpi.yaml` | ArgoCD Application definition | -| `argocd/manifests/devpi/Dockerfile` | Container image build | -| `argocd/manifests/devpi/statefulset.yaml` | StatefulSet with PVC | -| `argocd/manifests/devpi/service.yaml` | ClusterIP Service | -| `argocd/manifests/devpi/ingress-tailscale.yaml` | Tailscale Ingress | -| `argocd/manifests/devpi/kustomization.yaml` | Kustomize configuration | -| `argocd/manifests/devpi/README.md` | Setup documentation | - -## Modified Files - -| Path | Change | -|------|--------| -| `ansible/playbooks/indri.yml` | Remove devpi role | -| `ansible/roles/alloy/defaults/main.yml` | Remove devpi textfile collector | -| `ansible/roles/borgmatic/defaults/main.yml` | Remove devpi backup (no longer on indri) | - -## Deleted Files - -| Path | Reason | -|------|--------| -| `ansible/roles/devpi/` | Role no longer needed | -| `ansible/roles/devpi_metrics/` | Metrics collected differently in k8s | - ---- - -## Verification Checklist - -- [ ] devpi pod healthy in k8s -- [ ] https://pypi.tail8d86e.ts.net accessible -- [ ] Web interface shows root/pypi index -- [ ] `pip install ` works through proxy -- [ ] mcquack v1.0.0 uploaded to eblume/dev -- [ ] `pip install --index-url https://pypi.tail8d86e.ts.net/eblume/dev/+simple/ mcquack` works -- [ ] Old devpi service removed from indri -- [ ] zk documentation updated - ---- - -## Implementation Notes - -*To be filled in during implementation*