P5: Update devpi migration plan with detailed steps
- Fresh start approach (no data migration needed) - Build custom container with devpi-server + devpi-web - Use StatefulSet for persistence - Include verification steps for pip proxy and mcquack upload Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
b2307412fc
commit
b1f7dd4c3f
1 changed files with 168 additions and 15 deletions
|
|
@ -1,37 +1,190 @@
|
|||
# Phase 5: devpi Migration
|
||||
# Phase 5: devpi Migration to Kubernetes
|
||||
|
||||
**Goal**: Migrate devpi to k8s
|
||||
**Goal**: Migrate devpi PyPI caching proxy from indri to k8s
|
||||
|
||||
**Status**: Pending
|
||||
**Status**: In Progress (2026-01-20)
|
||||
|
||||
**Prerequisites**: [Phase 4](P4_miniflux.md) complete
|
||||
**Prerequisites**: [Phase 4](P4_miniflux.complete.md) complete
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This phase migrates devpi from mcquack LaunchAgent on indri to Kubernetes:
|
||||
1. Stop devpi on indri and clear Tailscale service
|
||||
2. Build custom devpi container image (devpi-server + devpi-web)
|
||||
3. Deploy as StatefulSet with PVC for data persistence
|
||||
4. Expose via Tailscale Ingress at `pypi.tail8d86e.ts.net`
|
||||
5. Initialize fresh (no data migration - just cache and one private package)
|
||||
6. Validate with pip proxy and mcquack package upload
|
||||
|
||||
**Decision: Fresh start** - The only private package is mcquack v1.0.0 which we'll re-upload. The PyPI cache is re-fetchable.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Build devpi container
|
||||
### 1. Stop devpi on indri (FIRST)
|
||||
|
||||
- Dockerfile with devpi-server + devpi-web
|
||||
- Push to local Zot registry
|
||||
```bash
|
||||
# Stop and unload the LaunchAgent
|
||||
ssh indri 'launchctl unload ~/Library/LaunchAgents/mcquack.eblume.devpi.plist'
|
||||
|
||||
# Also stop the metrics collector
|
||||
ssh indri 'launchctl unload ~/Library/LaunchAgents/mcquack.eblume.devpi-metrics.plist'
|
||||
|
||||
# Clear any tailscale serve entries (if any)
|
||||
ssh indri 'tailscale serve off --service=svc:pypi' 2>/dev/null || true
|
||||
```
|
||||
|
||||
**User action required**: Delete the `pypi` device/service from Tailscale admin console.
|
||||
|
||||
### 2. Build devpi container image
|
||||
|
||||
Create `argocd/manifests/devpi/Dockerfile`:
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.12-slim
|
||||
|
||||
# Install devpi-server and devpi-web
|
||||
RUN pip install --no-cache-dir devpi-server devpi-web
|
||||
|
||||
# Create data directory
|
||||
RUN mkdir -p /devpi
|
||||
|
||||
# Expose default port
|
||||
EXPOSE 3141
|
||||
|
||||
# Use ENTRYPOINT for flexibility
|
||||
ENTRYPOINT ["devpi-server"]
|
||||
|
||||
# Default args (can be overridden)
|
||||
CMD ["--serverdir", "/devpi", "--host", "0.0.0.0", "--port", "3141"]
|
||||
```
|
||||
|
||||
Build and push to Zot:
|
||||
|
||||
```bash
|
||||
# From gilbert (has podman)
|
||||
cd argocd/manifests/devpi
|
||||
podman build -t registry.tail8d86e.ts.net/blumeops/devpi:latest .
|
||||
podman push registry.tail8d86e.ts.net/blumeops/devpi:latest
|
||||
```
|
||||
|
||||
### 3. Create k8s manifests
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `argocd/apps/devpi.yaml` | ArgoCD Application definition |
|
||||
| `argocd/manifests/devpi/statefulset.yaml` | StatefulSet with PVC |
|
||||
| `argocd/manifests/devpi/service.yaml` | ClusterIP Service |
|
||||
| `argocd/manifests/devpi/ingress-tailscale.yaml` | Tailscale Ingress for `pypi.tail8d86e.ts.net` |
|
||||
| `argocd/manifests/devpi/init-job.yaml` | One-time initialization job |
|
||||
| `argocd/manifests/devpi/kustomization.yaml` | Kustomize configuration |
|
||||
|
||||
### 4. Deploy via ArgoCD
|
||||
|
||||
```bash
|
||||
# Point app at feature branch for testing
|
||||
argocd app create devpi \
|
||||
--repo ssh://forgejo@indri.tail8d86e.ts.net:2200/eblume/blumeops.git \
|
||||
--path argocd/manifests/devpi \
|
||||
--dest-server https://kubernetes.default.svc \
|
||||
--dest-namespace devpi \
|
||||
--revision feature/p5-devpi
|
||||
|
||||
# Or add to apps/ and let app-of-apps sync
|
||||
argocd app sync apps
|
||||
argocd app sync devpi
|
||||
```
|
||||
|
||||
### 5. Initialize devpi
|
||||
|
||||
After StatefulSet is running:
|
||||
|
||||
```bash
|
||||
# Exec into pod to initialize
|
||||
kubectl -n devpi exec -it devpi-0 -- devpi-init --serverdir /devpi --root-passwd <from-1password>
|
||||
|
||||
# Or use init job with secret
|
||||
```
|
||||
|
||||
Create user and index:
|
||||
|
||||
```bash
|
||||
# From workstation
|
||||
uvx devpi use https://pypi.tail8d86e.ts.net
|
||||
uvx devpi login root
|
||||
uvx devpi user -c eblume email=blume.erich@gmail.com
|
||||
uvx devpi index -c eblume/dev bases=root/pypi
|
||||
```
|
||||
|
||||
### 6. Update pip.conf on gilbert
|
||||
|
||||
The existing `~/.config/pip/pip.conf` should work unchanged since we're keeping the same hostname `pypi.tail8d86e.ts.net`.
|
||||
|
||||
### 7. Remove devpi ansible role
|
||||
|
||||
- Remove `devpi` from `ansible/playbooks/indri.yml` roles list
|
||||
- Remove `devpi_metrics` from alloy textfile collectors
|
||||
- Keep devpi ansible role directory for reference (or delete)
|
||||
|
||||
### 8. Validate
|
||||
|
||||
```bash
|
||||
# Test pip cache proxy
|
||||
pip install --index-url https://pypi.tail8d86e.ts.net/root/pypi/+simple/ requests
|
||||
|
||||
# Upload mcquack
|
||||
cd ~/code/personal/mcquack
|
||||
uv build
|
||||
uv publish --publish-url https://pypi.tail8d86e.ts.net/eblume/dev/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Deploy as StatefulSet
|
||||
## New Files
|
||||
|
||||
- PVC for data (50Gi)
|
||||
- Migrate existing data (excluding PyPI cache)
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `argocd/apps/devpi.yaml` | ArgoCD Application definition |
|
||||
| `argocd/manifests/devpi/Dockerfile` | Container image build |
|
||||
| `argocd/manifests/devpi/statefulset.yaml` | StatefulSet with PVC |
|
||||
| `argocd/manifests/devpi/service.yaml` | ClusterIP Service |
|
||||
| `argocd/manifests/devpi/ingress-tailscale.yaml` | Tailscale Ingress |
|
||||
| `argocd/manifests/devpi/kustomization.yaml` | Kustomize configuration |
|
||||
| `argocd/manifests/devpi/README.md` | Setup documentation |
|
||||
|
||||
## Modified Files
|
||||
|
||||
| Path | Change |
|
||||
|------|--------|
|
||||
| `ansible/playbooks/indri.yml` | Remove devpi role |
|
||||
| `ansible/roles/alloy/defaults/main.yml` | Remove devpi textfile collector |
|
||||
| `ansible/roles/borgmatic/defaults/main.yml` | Remove devpi backup (no longer on indri) |
|
||||
|
||||
## Deleted Files
|
||||
|
||||
| Path | Reason |
|
||||
|------|--------|
|
||||
| `ansible/roles/devpi/` | Role no longer needed |
|
||||
| `ansible/roles/devpi_metrics/` | Metrics collected differently in k8s |
|
||||
|
||||
---
|
||||
|
||||
### 3. Configure Tailscale LoadBalancer
|
||||
## Verification Checklist
|
||||
|
||||
Tag: `svc:pypi`
|
||||
- [ ] devpi pod healthy in k8s
|
||||
- [ ] https://pypi.tail8d86e.ts.net accessible
|
||||
- [ ] Web interface shows root/pypi index
|
||||
- [ ] `pip install <package>` works through proxy
|
||||
- [ ] mcquack v1.0.0 uploaded to eblume/dev
|
||||
- [ ] `pip install --index-url https://pypi.tail8d86e.ts.net/eblume/dev/+simple/ mcquack` works
|
||||
- [ ] Old devpi service removed from indri
|
||||
- [ ] zk documentation updated
|
||||
|
||||
---
|
||||
|
||||
### 4. Update pip.conf on gilbert
|
||||
## Implementation Notes
|
||||
|
||||
---
|
||||
|
||||
### 5. Stop mcquack devpi
|
||||
*To be filled in during implementation*
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue