From b1f7dd4c3f85b3b3581d51ca8da043fc0af0d59d Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 09:40:05 -0800 Subject: [PATCH 01/12] P5: Update devpi migration plan with detailed steps - Fresh start approach (no data migration needed) - Build custom container with devpi-server + devpi-web - Use StatefulSet for persistence - Include verification steps for pip proxy and mcquack upload Co-Authored-By: Claude Opus 4.5 --- plans/k8s-migration/P5_devpi.md | 185 +++++++++++++++++++++++++++++--- 1 file changed, 169 insertions(+), 16 deletions(-) diff --git a/plans/k8s-migration/P5_devpi.md b/plans/k8s-migration/P5_devpi.md index 7f24b7b..b5ea9d2 100644 --- a/plans/k8s-migration/P5_devpi.md +++ b/plans/k8s-migration/P5_devpi.md @@ -1,37 +1,190 @@ -# Phase 5: devpi Migration +# Phase 5: devpi Migration to Kubernetes -**Goal**: Migrate devpi to k8s +**Goal**: Migrate devpi PyPI caching proxy from indri to k8s -**Status**: Pending +**Status**: In Progress (2026-01-20) -**Prerequisites**: [Phase 4](P4_miniflux.md) complete +**Prerequisites**: [Phase 4](P4_miniflux.complete.md) complete + +--- + +## Overview + +This phase migrates devpi from mcquack LaunchAgent on indri to Kubernetes: +1. Stop devpi on indri and clear Tailscale service +2. Build custom devpi container image (devpi-server + devpi-web) +3. Deploy as StatefulSet with PVC for data persistence +4. Expose via Tailscale Ingress at `pypi.tail8d86e.ts.net` +5. Initialize fresh (no data migration - just cache and one private package) +6. Validate with pip proxy and mcquack package upload + +**Decision: Fresh start** - The only private package is mcquack v1.0.0 which we'll re-upload. The PyPI cache is re-fetchable. --- ## Steps -### 1. Build devpi container +### 1. Stop devpi on indri (FIRST) -- Dockerfile with devpi-server + devpi-web -- Push to local Zot registry +```bash +# Stop and unload the LaunchAgent +ssh indri 'launchctl unload ~/Library/LaunchAgents/mcquack.eblume.devpi.plist' + +# Also stop the metrics collector +ssh indri 'launchctl unload ~/Library/LaunchAgents/mcquack.eblume.devpi-metrics.plist' + +# Clear any tailscale serve entries (if any) +ssh indri 'tailscale serve off --service=svc:pypi' 2>/dev/null || true +``` + +**User action required**: Delete the `pypi` device/service from Tailscale admin console. + +### 2. Build devpi container image + +Create `argocd/manifests/devpi/Dockerfile`: + +```dockerfile +FROM python:3.12-slim + +# Install devpi-server and devpi-web +RUN pip install --no-cache-dir devpi-server devpi-web + +# Create data directory +RUN mkdir -p /devpi + +# Expose default port +EXPOSE 3141 + +# Use ENTRYPOINT for flexibility +ENTRYPOINT ["devpi-server"] + +# Default args (can be overridden) +CMD ["--serverdir", "/devpi", "--host", "0.0.0.0", "--port", "3141"] +``` + +Build and push to Zot: + +```bash +# From gilbert (has podman) +cd argocd/manifests/devpi +podman build -t registry.tail8d86e.ts.net/blumeops/devpi:latest . +podman push registry.tail8d86e.ts.net/blumeops/devpi:latest +``` + +### 3. Create k8s manifests + +| File | Purpose | +|------|---------| +| `argocd/apps/devpi.yaml` | ArgoCD Application definition | +| `argocd/manifests/devpi/statefulset.yaml` | StatefulSet with PVC | +| `argocd/manifests/devpi/service.yaml` | ClusterIP Service | +| `argocd/manifests/devpi/ingress-tailscale.yaml` | Tailscale Ingress for `pypi.tail8d86e.ts.net` | +| `argocd/manifests/devpi/init-job.yaml` | One-time initialization job | +| `argocd/manifests/devpi/kustomization.yaml` | Kustomize configuration | + +### 4. Deploy via ArgoCD + +```bash +# Point app at feature branch for testing +argocd app create devpi \ + --repo ssh://forgejo@indri.tail8d86e.ts.net:2200/eblume/blumeops.git \ + --path argocd/manifests/devpi \ + --dest-server https://kubernetes.default.svc \ + --dest-namespace devpi \ + --revision feature/p5-devpi + +# Or add to apps/ and let app-of-apps sync +argocd app sync apps +argocd app sync devpi +``` + +### 5. Initialize devpi + +After StatefulSet is running: + +```bash +# Exec into pod to initialize +kubectl -n devpi exec -it devpi-0 -- devpi-init --serverdir /devpi --root-passwd + +# Or use init job with secret +``` + +Create user and index: + +```bash +# From workstation +uvx devpi use https://pypi.tail8d86e.ts.net +uvx devpi login root +uvx devpi user -c eblume email=blume.erich@gmail.com +uvx devpi index -c eblume/dev bases=root/pypi +``` + +### 6. Update pip.conf on gilbert + +The existing `~/.config/pip/pip.conf` should work unchanged since we're keeping the same hostname `pypi.tail8d86e.ts.net`. + +### 7. Remove devpi ansible role + +- Remove `devpi` from `ansible/playbooks/indri.yml` roles list +- Remove `devpi_metrics` from alloy textfile collectors +- Keep devpi ansible role directory for reference (or delete) + +### 8. Validate + +```bash +# Test pip cache proxy +pip install --index-url https://pypi.tail8d86e.ts.net/root/pypi/+simple/ requests + +# Upload mcquack +cd ~/code/personal/mcquack +uv build +uv publish --publish-url https://pypi.tail8d86e.ts.net/eblume/dev/ +``` --- -### 2. Deploy as StatefulSet +## New Files -- PVC for data (50Gi) -- Migrate existing data (excluding PyPI cache) +| Path | Purpose | +|------|---------| +| `argocd/apps/devpi.yaml` | ArgoCD Application definition | +| `argocd/manifests/devpi/Dockerfile` | Container image build | +| `argocd/manifests/devpi/statefulset.yaml` | StatefulSet with PVC | +| `argocd/manifests/devpi/service.yaml` | ClusterIP Service | +| `argocd/manifests/devpi/ingress-tailscale.yaml` | Tailscale Ingress | +| `argocd/manifests/devpi/kustomization.yaml` | Kustomize configuration | +| `argocd/manifests/devpi/README.md` | Setup documentation | + +## Modified Files + +| Path | Change | +|------|--------| +| `ansible/playbooks/indri.yml` | Remove devpi role | +| `ansible/roles/alloy/defaults/main.yml` | Remove devpi textfile collector | +| `ansible/roles/borgmatic/defaults/main.yml` | Remove devpi backup (no longer on indri) | + +## Deleted Files + +| Path | Reason | +|------|--------| +| `ansible/roles/devpi/` | Role no longer needed | +| `ansible/roles/devpi_metrics/` | Metrics collected differently in k8s | --- -### 3. Configure Tailscale LoadBalancer +## Verification Checklist -Tag: `svc:pypi` +- [ ] devpi pod healthy in k8s +- [ ] https://pypi.tail8d86e.ts.net accessible +- [ ] Web interface shows root/pypi index +- [ ] `pip install ` works through proxy +- [ ] mcquack v1.0.0 uploaded to eblume/dev +- [ ] `pip install --index-url https://pypi.tail8d86e.ts.net/eblume/dev/+simple/ mcquack` works +- [ ] Old devpi service removed from indri +- [ ] zk documentation updated --- -### 4. Update pip.conf on gilbert +## Implementation Notes ---- - -### 5. Stop mcquack devpi +*To be filled in during implementation* -- 2.50.1 (Apple Git-155) From 173a9134d38298bd629359e83bee450ddf6e6355 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 09:50:15 -0800 Subject: [PATCH 02/12] P5: Add devpi k8s manifests and ArgoCD app - Dockerfile for devpi-server + devpi-web image - StatefulSet with 50Gi PVC for data persistence - Tailscale Ingress for pypi.tail8d86e.ts.net - README with setup and usage instructions Co-Authored-By: Claude Opus 4.5 --- argocd/apps/devpi.yaml | 30 ++++++++ argocd/manifests/devpi/Dockerfile | 19 +++++ argocd/manifests/devpi/README.md | 70 +++++++++++++++++++ argocd/manifests/devpi/ingress-tailscale.yaml | 17 +++++ argocd/manifests/devpi/kustomization.yaml | 9 +++ argocd/manifests/devpi/service.yaml | 13 ++++ argocd/manifests/devpi/statefulset.yaml | 63 +++++++++++++++++ 7 files changed, 221 insertions(+) create mode 100644 argocd/apps/devpi.yaml create mode 100644 argocd/manifests/devpi/Dockerfile create mode 100644 argocd/manifests/devpi/README.md create mode 100644 argocd/manifests/devpi/ingress-tailscale.yaml create mode 100644 argocd/manifests/devpi/kustomization.yaml create mode 100644 argocd/manifests/devpi/service.yaml create mode 100644 argocd/manifests/devpi/statefulset.yaml diff --git a/argocd/apps/devpi.yaml b/argocd/apps/devpi.yaml new file mode 100644 index 0000000..07e8250 --- /dev/null +++ b/argocd/apps/devpi.yaml @@ -0,0 +1,30 @@ +# devpi PyPI Caching Proxy +# Provides PyPI cache and private package hosting +# +# After first deployment, initialize devpi: +# kubectl -n devpi exec -it devpi-0 -- devpi-init --serverdir /devpi --root-passwd +# kubectl -n devpi rollout restart statefulset devpi +# +# Then create user/index: +# uvx devpi use https://pypi.tail8d86e.ts.net +# uvx devpi login root +# uvx devpi user -c eblume email=blume.erich@gmail.com +# uvx devpi index -c eblume/dev bases=root/pypi +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: devpi + namespace: argocd +spec: + project: default + source: + repoURL: ssh://forgejo@indri.tail8d86e.ts.net:2200/eblume/blumeops.git + targetRevision: main + path: argocd/manifests/devpi + destination: + server: https://kubernetes.default.svc + namespace: devpi + syncPolicy: + syncOptions: + - CreateNamespace=true + # Manual sync only - no automated sync on git push diff --git a/argocd/manifests/devpi/Dockerfile b/argocd/manifests/devpi/Dockerfile new file mode 100644 index 0000000..31e7d7c --- /dev/null +++ b/argocd/manifests/devpi/Dockerfile @@ -0,0 +1,19 @@ +FROM python:3.12-slim + +# Install devpi-server and devpi-web +RUN pip install --no-cache-dir devpi-server devpi-web + +# Create non-root user +RUN useradd -r -u 1000 devpi && mkdir -p /devpi && chown devpi:devpi /devpi + +USER devpi +WORKDIR /devpi + +# Expose default port +EXPOSE 3141 + +# Use ENTRYPOINT for flexibility +ENTRYPOINT ["devpi-server"] + +# Default args (can be overridden) +CMD ["--serverdir", "/devpi", "--host", "0.0.0.0", "--port", "3141"] diff --git a/argocd/manifests/devpi/README.md b/argocd/manifests/devpi/README.md new file mode 100644 index 0000000..b68de74 --- /dev/null +++ b/argocd/manifests/devpi/README.md @@ -0,0 +1,70 @@ +# devpi PyPI Caching Proxy + +devpi-server running in Kubernetes, providing: +- PyPI caching proxy at `root/pypi` +- Private package hosting at `eblume/dev` + +## Setup + +### 1. Deploy via ArgoCD + +```bash +argocd app sync apps +argocd app sync devpi +``` + +### 2. Initialize devpi (first time only) + +After the StatefulSet is running, initialize devpi with a root password: + +```bash +# Get the root password from 1Password +ROOT_PASSWORD=$(op --vault blumeops item get --fields password --reveal) + +# Initialize devpi +kubectl -n devpi exec -it devpi-0 -- devpi-init --serverdir /devpi --root-passwd "$ROOT_PASSWORD" + +# Restart the pod to pick up the initialized state +kubectl -n devpi rollout restart statefulset devpi +``` + +### 3. Create user and index + +```bash +# Login to devpi +uvx devpi use https://pypi.tail8d86e.ts.net +uvx devpi login root + +# Create user +uvx devpi user -c eblume email=blume.erich@gmail.com + +# Create private index inheriting from PyPI +uvx devpi index -c eblume/dev bases=root/pypi +``` + +## Usage + +### As pip index (caching proxy) + +Configure `~/.config/pip/pip.conf`: + +```ini +[global] +index-url = https://pypi.tail8d86e.ts.net/root/pypi/+simple/ +trusted-host = pypi.tail8d86e.ts.net +``` + +### Upload private packages + +```bash +# Build and publish +cd ~/code/personal/your-package +uv build +uv publish --publish-url https://pypi.tail8d86e.ts.net/eblume/dev/ +``` + +## URLs + +- Web UI: https://pypi.tail8d86e.ts.net +- PyPI cache: https://pypi.tail8d86e.ts.net/root/pypi/+simple/ +- Private index: https://pypi.tail8d86e.ts.net/eblume/dev/+simple/ diff --git a/argocd/manifests/devpi/ingress-tailscale.yaml b/argocd/manifests/devpi/ingress-tailscale.yaml new file mode 100644 index 0000000..2a1c659 --- /dev/null +++ b/argocd/manifests/devpi/ingress-tailscale.yaml @@ -0,0 +1,17 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: devpi-tailscale + namespace: devpi + annotations: + tailscale.com/proxy-class: "crio-compat" +spec: + ingressClassName: tailscale + defaultBackend: + service: + name: devpi + port: + number: 3141 + tls: + - hosts: + - pypi diff --git a/argocd/manifests/devpi/kustomization.yaml b/argocd/manifests/devpi/kustomization.yaml new file mode 100644 index 0000000..6bc7579 --- /dev/null +++ b/argocd/manifests/devpi/kustomization.yaml @@ -0,0 +1,9 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: devpi + +resources: + - statefulset.yaml + - service.yaml + - ingress-tailscale.yaml diff --git a/argocd/manifests/devpi/service.yaml b/argocd/manifests/devpi/service.yaml new file mode 100644 index 0000000..42e1543 --- /dev/null +++ b/argocd/manifests/devpi/service.yaml @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: Service +metadata: + name: devpi + namespace: devpi +spec: + selector: + app: devpi + ports: + - name: http + port: 3141 + targetPort: 3141 + protocol: TCP diff --git a/argocd/manifests/devpi/statefulset.yaml b/argocd/manifests/devpi/statefulset.yaml new file mode 100644 index 0000000..21dac52 --- /dev/null +++ b/argocd/manifests/devpi/statefulset.yaml @@ -0,0 +1,63 @@ +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: devpi + namespace: devpi +spec: + serviceName: devpi + replicas: 1 + selector: + matchLabels: + app: devpi + template: + metadata: + labels: + app: devpi + spec: + securityContext: + fsGroup: 1000 + containers: + - name: devpi + image: registry.tail8d86e.ts.net/blumeops/devpi:latest + args: + - "--serverdir" + - "/devpi" + - "--host" + - "0.0.0.0" + - "--port" + - "3141" + - "--outside-url" + - "https://pypi.tail8d86e.ts.net" + ports: + - containerPort: 3141 + name: http + volumeMounts: + - name: data + mountPath: /devpi + resources: + requests: + memory: "128Mi" + cpu: "100m" + limits: + memory: "512Mi" + cpu: "500m" + livenessProbe: + httpGet: + path: /+api + port: 3141 + initialDelaySeconds: 30 + periodSeconds: 30 + readinessProbe: + httpGet: + path: /+api + port: 3141 + initialDelaySeconds: 10 + periodSeconds: 10 + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 50Gi -- 2.50.1 (Apple Git-155) From 21f95cf0e9bde177b659c9324f80e36148b619bd Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 10:56:42 -0800 Subject: [PATCH 03/12] Update CLAUDE.md with k8s/ArgoCD workflow and project structure - Add Service Deployment section with ArgoCD PR workflow - Document app-of-apps pattern and manual sync policy - Add Tailscale hostname migration steps - Update project structure to include argocd/ directory - List services staying on indri vs moving to k8s - Use fish syntax for shell blocks Co-Authored-By: Claude Opus 4.5 --- CLAUDE.md | 96 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 82 insertions(+), 14 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 4ee5bb1..833ac74 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -33,35 +33,103 @@ The user will review your work as you go, and will merge the pr as the last step 4. Use `Brewfile` and `mise.toml` to install tools needed on the development workstation (typically hostnamed "gilbert", username "eblume"). -5. Services are typically hosted on hostname "indri" and are launched from LaunchAgents of the user `erichblume`. If a service is available from `brew services` that is typically used, otherwise there is a utility called `mcquack` (`mcquack --help`) hosted at `https://forge.tail8d86e.ts.net/eblume/mcquack` - but you can just edit the mcquack launchagents directly via ansible. +5. Services are hosted either on indri directly (via ansible) or in Kubernetes (via ArgoCD). See the "Service Deployment" section below for details. 6. Try to always test changes before applying them. Use syntax checkers, do dry runs (`--check --diff`), run commands manually via `ssh indri 'some command'`, etc. -7. **Wait for user review before deploying.** After creating a PR, do not run `mise run provision-indri` or other deployment commands until the user has had a chance to review the changes. The user will indicate when they're ready to deploy. +7. **Wait for user review before deploying.** After creating a PR, do not run deployment commands until the user has had a chance to review the changes. The user will indicate when they're ready to deploy. 8. After deploying changes, try to verify the result. Use `mise run indri-services-check` to do a general service health check. -## Project structure -Some important places you can look: +## Project Structure + ``` -./mise-tasks/ # management and utility scripts run via `mise run` -./ansible/playbooks/indri.yml # primary blumeops provisioning script -./ansible/roles/ # role dirs here give good overview of services -./pulumi/ # python (via uv) pulumi script for provisioning the tailnet and other cloud resources -~/code/personal/ # projects managed by the user -~/code/3rd/ # external projects, mirrored or downloaded -~/code/work # FORBIDDEN, never go here, avoid searching it +./mise-tasks/ # management and utility scripts run via `mise run` +./ansible/playbooks/ # ansible playbooks (indri.yml is primary) +./ansible/roles/ # ansible roles for indri-hosted services +./argocd/apps/ # ArgoCD Application definitions (app-of-apps pattern) +./argocd/manifests/ # Kubernetes manifests for each service +./pulumi/ # Pulumi IaC for tailnet ACLs and cloud resources +./plans/ # Migration and project planning documents +~/code/personal/ # projects managed by the user +~/code/3rd/ # external projects, mirrored or downloaded +~/code/work # FORBIDDEN, never go here, avoid searching it ``` +## Service Deployment + +### Kubernetes Services (via ArgoCD) + +Most services are migrating to Kubernetes. These are managed via ArgoCD using the app-of-apps pattern: + +- **Application definitions**: `argocd/apps/.yaml` +- **Manifests**: `argocd/manifests//` +- **Sync policy**: Manual sync (no auto-sync on git push) + +**PR workflow for k8s services:** + +1. Create feature branch and add/modify manifests +2. Push branch to forge +3. Sync the `apps` application to pick up new Application definitions: + ```fish + argocd app sync apps + ``` +4. Point the service app at the feature branch for testing: + ```fish + argocd app set --revision feature/branch-name + argocd app sync + ``` +5. Test the deployment +6. After PR merge, reset to main and resync: + ```fish + argocd app set --revision main + argocd app sync + ``` + +**Useful commands:** +```fish +argocd app list # List all apps +argocd app get # Get app details +argocd app diff # Preview changes before sync +argocd app sync # Sync an app +ki get pods -n # Check pods (ki = kubectl --context=minikube-indri) +k9i # k9s for the minikube-indri context +``` + +### Indri Services (via Ansible) + +Some services remain on indri outside of Kubernetes: +- **Zot Registry** - Container registry (k8s depends on it) +- **Prometheus/Loki** - Observability (must survive k8s failures) +- **Borgmatic** - Backup system +- **Grafana Alloy** - Metrics/logs collector +- **Transmission** - BitTorrent for kiwix downloads + +**Deployment:** +```fish +mise run provision-indri # Full playbook +mise run provision-indri -- --tags # Specific role +mise run provision-indri -- --check --diff # Dry run +``` + +### Tailscale Service Hostnames + +When migrating a service from indri to k8s, the Tailscale hostname must be freed: + +1. Stop the service on indri +2. Clear the tailscale serve entry: `ssh indri 'tailscale serve clear svc:'` +3. Delete the device from Tailscale admin console (user action required) +4. Deploy the k8s Ingress - it will claim the hostname + +Use `ssh indri 'tailscale serve status --json'` to check current serve entries (the non-JSON output may be empty even when entries exist). + ## Third-Party Projects When a task requires cloning or using a third-party git repository (e.g., for building from source), **ask the user to mirror it on forge first**, then clone from the mirror: - Mirror location: `https://forge.tail8d86e.ts.net/eblume/.git` - Clone to: `~/code/3rd//` -This avoids external dependencies and ensures the project is available even if the upstream is unreachable. Example mirrors: -- `https://forge.tail8d86e.ts.net/eblume/zot.git` (container registry) -- `https://forge.tail8d86e.ts.net/eblume/devpi.git` (PyPI proxy) +This avoids external dependencies and ensures the project is available even if the upstream is unreachable. ## Task Discovery -- 2.50.1 (Apple Git-155) From bfc862a9ba0b1f9f987b9782ba4304cf1f643eab Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 11:02:35 -0800 Subject: [PATCH 04/12] Change apps to manual sync policy Auto-sync was causing issues with testing feature branches - manual changes to targetRevision were being reverted. Co-Authored-By: Claude Opus 4.5 --- argocd/apps/apps.yaml | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/argocd/apps/apps.yaml b/argocd/apps/apps.yaml index ec0d5ac..3567bdc 100644 --- a/argocd/apps/apps.yaml +++ b/argocd/apps/apps.yaml @@ -15,9 +15,7 @@ spec: server: https://kubernetes.default.svc namespace: argocd syncPolicy: - automated: - prune: true - # selfHeal disabled: allows manual revision changes on child apps during development - # Sync apps app manually when adding/removing Application manifests syncOptions: - CreateNamespace=true + # Manual sync only - no automated sync on git push + # To pick up new apps: argocd app sync apps -- 2.50.1 (Apple Git-155) From 3ab67dd222f21384eebf9ed5e4f83d7efeecb17c Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 11:06:25 -0800 Subject: [PATCH 05/12] Fix devpi image path for minikube registry access - Use host.containers.internal:5050 instead of registry.tail8d86e.ts.net - Minikube's CRI-O can't resolve Tailscale DNS, but can reach Zot via podman host - Also update CLAUDE.md to spell out kubectl commands (fish abbreviations don't work in scripts) Co-Authored-By: Claude Opus 4.5 --- CLAUDE.md | 14 ++++++++------ argocd/manifests/devpi/statefulset.yaml | 3 ++- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 833ac74..bd80263 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -88,14 +88,16 @@ Most services are migrating to Kubernetes. These are managed via ArgoCD using th **Useful commands:** ```fish -argocd app list # List all apps -argocd app get # Get app details -argocd app diff # Preview changes before sync -argocd app sync # Sync an app -ki get pods -n # Check pods (ki = kubectl --context=minikube-indri) -k9i # k9s for the minikube-indri context +argocd app list # List all apps +argocd app get # Get app details +argocd app diff # Preview changes before sync +argocd app sync # Sync an app +kubectl --context=minikube-indri get pods -n # Check pods +kubectl --context=minikube-indri logs -n # View logs ``` +Note: The user has fish abbreviations `ki` for `kubectl --context=minikube-indri` and `k9i` for `k9s --context=minikube-indri`, but these only work in interactive shells. + ### Indri Services (via Ansible) Some services remain on indri outside of Kubernetes: diff --git a/argocd/manifests/devpi/statefulset.yaml b/argocd/manifests/devpi/statefulset.yaml index 21dac52..506275d 100644 --- a/argocd/manifests/devpi/statefulset.yaml +++ b/argocd/manifests/devpi/statefulset.yaml @@ -18,7 +18,8 @@ spec: fsGroup: 1000 containers: - name: devpi - image: registry.tail8d86e.ts.net/blumeops/devpi:latest + # Use host.containers.internal for minikube's CRI-O to reach Zot on indri + image: host.containers.internal:5050/blumeops/devpi:latest args: - "--serverdir" - "/devpi" -- 2.50.1 (Apple Git-155) From e2fd274ba97ba0987fad6481d2777e12f65e17c4 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 11:10:51 -0800 Subject: [PATCH 06/12] Add CRI-O registry config for private images and Tailscale mirror - Add direct access config for host.containers.internal:5050 (insecure) - Add mirror config for registry.tail8d86e.ts.net -> host.containers.internal:5050 - Revert statefulset to use registry.tail8d86e.ts.net (cleaner, if mirror works) Co-Authored-By: Claude Opus 4.5 --- ansible/roles/minikube/files/zot-mirror.conf | 16 ++++++++++++++++ argocd/manifests/devpi/statefulset.yaml | 3 +-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/ansible/roles/minikube/files/zot-mirror.conf b/ansible/roles/minikube/files/zot-mirror.conf index 434e406..df9bc11 100644 --- a/ansible/roles/minikube/files/zot-mirror.conf +++ b/ansible/roles/minikube/files/zot-mirror.conf @@ -2,6 +2,22 @@ # Uses host.containers.internal which is stable across restarts # Applied by ansible minikube role +# Direct access to Zot for private images (blumeops/*) +[[registry]] +prefix = "host.containers.internal:5050" +location = "host.containers.internal:5050" +insecure = true + +# Tailscale hostname for Zot - redirects to local access +# Allows manifests to use registry.tail8d86e.ts.net which is cleaner +[[registry]] +prefix = "registry.tail8d86e.ts.net" +location = "registry.tail8d86e.ts.net" + +[[registry.mirror]] +location = "host.containers.internal:5050" +insecure = true + [[registry]] prefix = "docker.io" location = "docker.io" diff --git a/argocd/manifests/devpi/statefulset.yaml b/argocd/manifests/devpi/statefulset.yaml index 506275d..21dac52 100644 --- a/argocd/manifests/devpi/statefulset.yaml +++ b/argocd/manifests/devpi/statefulset.yaml @@ -18,8 +18,7 @@ spec: fsGroup: 1000 containers: - name: devpi - # Use host.containers.internal for minikube's CRI-O to reach Zot on indri - image: host.containers.internal:5050/blumeops/devpi:latest + image: registry.tail8d86e.ts.net/blumeops/devpi:latest args: - "--serverdir" - "/devpi" -- 2.50.1 (Apple Git-155) From 8e9ab46335bfddce4d0d5e3430aa6bc09d717c2d Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 11:19:37 -0800 Subject: [PATCH 07/12] Add devpi auto-initialization with startup script - Startup script checks for initialization and runs devpi-init if needed - Root password passed via DEVPI_ROOT_PASSWORD env var from secret - Secret template references 1Password vault item - Updated README with setup instructions Co-Authored-By: Claude Opus 4.5 --- argocd/manifests/devpi/Dockerfile | 10 ++--- argocd/manifests/devpi/README.md | 44 +++++++++++---------- argocd/manifests/devpi/secret-root.yaml.tpl | 12 ++++++ argocd/manifests/devpi/start.sh | 30 ++++++++++++++ argocd/manifests/devpi/statefulset.yaml | 17 ++++---- 5 files changed, 78 insertions(+), 35 deletions(-) create mode 100644 argocd/manifests/devpi/secret-root.yaml.tpl create mode 100644 argocd/manifests/devpi/start.sh diff --git a/argocd/manifests/devpi/Dockerfile b/argocd/manifests/devpi/Dockerfile index 31e7d7c..6c9cdc8 100644 --- a/argocd/manifests/devpi/Dockerfile +++ b/argocd/manifests/devpi/Dockerfile @@ -6,14 +6,14 @@ RUN pip install --no-cache-dir devpi-server devpi-web # Create non-root user RUN useradd -r -u 1000 devpi && mkdir -p /devpi && chown devpi:devpi /devpi +# Add startup script +COPY --chown=devpi:devpi start.sh /usr/local/bin/start.sh +RUN chmod +x /usr/local/bin/start.sh + USER devpi WORKDIR /devpi # Expose default port EXPOSE 3141 -# Use ENTRYPOINT for flexibility -ENTRYPOINT ["devpi-server"] - -# Default args (can be overridden) -CMD ["--serverdir", "/devpi", "--host", "0.0.0.0", "--port", "3141"] +ENTRYPOINT ["/usr/local/bin/start.sh"] diff --git a/argocd/manifests/devpi/README.md b/argocd/manifests/devpi/README.md index b68de74..a349853 100644 --- a/argocd/manifests/devpi/README.md +++ b/argocd/manifests/devpi/README.md @@ -6,36 +6,33 @@ devpi-server running in Kubernetes, providing: ## Setup -### 1. Deploy via ArgoCD +### 1. Create the root password secret -```bash +```fish +kubectl create namespace devpi +op inject -i argocd/manifests/devpi/secret-root.yaml.tpl | kubectl apply -f - +``` + +### 2. Deploy via ArgoCD + +```fish argocd app sync apps argocd app sync devpi ``` -### 2. Initialize devpi (first time only) +The container will auto-initialize on first startup using the root password from the secret. -After the StatefulSet is running, initialize devpi with a root password: +### 3. Create user and index (first time only) -```bash -# Get the root password from 1Password -ROOT_PASSWORD=$(op --vault blumeops item get --fields password --reveal) +After the pod is running: -# Initialize devpi -kubectl -n devpi exec -it devpi-0 -- devpi-init --serverdir /devpi --root-passwd "$ROOT_PASSWORD" - -# Restart the pod to pick up the initialized state -kubectl -n devpi rollout restart statefulset devpi -``` - -### 3. Create user and index - -```bash -# Login to devpi +```fish +# Login to devpi as root uvx devpi use https://pypi.tail8d86e.ts.net uvx devpi login root +# Enter root password when prompted -# Create user +# Create eblume user (prompts for password - use the one from 1Password) uvx devpi user -c eblume email=blume.erich@gmail.com # Create private index inheriting from PyPI @@ -56,8 +53,7 @@ trusted-host = pypi.tail8d86e.ts.net ### Upload private packages -```bash -# Build and publish +```fish cd ~/code/personal/your-package uv build uv publish --publish-url https://pypi.tail8d86e.ts.net/eblume/dev/ @@ -68,3 +64,9 @@ uv publish --publish-url https://pypi.tail8d86e.ts.net/eblume/dev/ - Web UI: https://pypi.tail8d86e.ts.net - PyPI cache: https://pypi.tail8d86e.ts.net/root/pypi/+simple/ - Private index: https://pypi.tail8d86e.ts.net/eblume/dev/+simple/ + +## Credentials + +Stored in 1Password vault `blumeops`, item `kyhzfifryqnuk7jeyibmmjvxxm`: +- `root password` - devpi root user +- `password` - eblume user password diff --git a/argocd/manifests/devpi/secret-root.yaml.tpl b/argocd/manifests/devpi/secret-root.yaml.tpl new file mode 100644 index 0000000..d69f9a8 --- /dev/null +++ b/argocd/manifests/devpi/secret-root.yaml.tpl @@ -0,0 +1,12 @@ +# Template for devpi root password secret +# Create the secret before deploying: +# kubectl create namespace devpi +# op inject -i argocd/manifests/devpi/secret-root.yaml.tpl | kubectl apply -f - +apiVersion: v1 +kind: Secret +metadata: + name: devpi-root + namespace: devpi +type: Opaque +stringData: + password: "{{ op://vg6xf6vvfmoh5hqjjhlhbeoaie/kyhzfifryqnuk7jeyibmmjvxxm/root password }}" diff --git a/argocd/manifests/devpi/start.sh b/argocd/manifests/devpi/start.sh new file mode 100644 index 0000000..faf3139 --- /dev/null +++ b/argocd/manifests/devpi/start.sh @@ -0,0 +1,30 @@ +#!/bin/bash +set -e + +SERVERDIR="${DEVPI_SERVERDIR:-/devpi}" +HOST="${DEVPI_HOST:-0.0.0.0}" +PORT="${DEVPI_PORT:-3141}" +OUTSIDE_URL="${DEVPI_OUTSIDE_URL:-}" + +# Check if devpi is initialized +if [ ! -f "$SERVERDIR/.serverversion" ]; then + echo "Initializing devpi server..." + + if [ -z "$DEVPI_ROOT_PASSWORD" ]; then + echo "ERROR: DEVPI_ROOT_PASSWORD environment variable must be set for initialization" + exit 1 + fi + + devpi-init --serverdir "$SERVERDIR" --root-passwd "$DEVPI_ROOT_PASSWORD" + echo "Devpi initialized successfully" +fi + +# Build command +CMD="devpi-server --serverdir $SERVERDIR --host $HOST --port $PORT" + +if [ -n "$OUTSIDE_URL" ]; then + CMD="$CMD --outside-url $OUTSIDE_URL" +fi + +echo "Starting devpi-server..." +exec $CMD diff --git a/argocd/manifests/devpi/statefulset.yaml b/argocd/manifests/devpi/statefulset.yaml index 21dac52..79a7770 100644 --- a/argocd/manifests/devpi/statefulset.yaml +++ b/argocd/manifests/devpi/statefulset.yaml @@ -19,15 +19,14 @@ spec: containers: - name: devpi image: registry.tail8d86e.ts.net/blumeops/devpi:latest - args: - - "--serverdir" - - "/devpi" - - "--host" - - "0.0.0.0" - - "--port" - - "3141" - - "--outside-url" - - "https://pypi.tail8d86e.ts.net" + env: + - name: DEVPI_ROOT_PASSWORD + valueFrom: + secretKeyRef: + name: devpi-root + key: password + - name: DEVPI_OUTSIDE_URL + value: "https://pypi.tail8d86e.ts.net" ports: - containerPort: 3141 name: http -- 2.50.1 (Apple Git-155) From 5683a96057def19db7d1dd58ee2dfd202e471374 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 11:24:04 -0800 Subject: [PATCH 08/12] Fix DEVPI_PORT env var conflict with k8s service discovery Kubernetes auto-sets DEVPI_PORT for service discovery, which conflicts with our port configuration. Renamed to DEVPI_LISTEN_PORT. Co-Authored-By: Claude Opus 4.5 --- argocd/manifests/devpi/start.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/argocd/manifests/devpi/start.sh b/argocd/manifests/devpi/start.sh index faf3139..e34e60c 100644 --- a/argocd/manifests/devpi/start.sh +++ b/argocd/manifests/devpi/start.sh @@ -3,7 +3,8 @@ set -e SERVERDIR="${DEVPI_SERVERDIR:-/devpi}" HOST="${DEVPI_HOST:-0.0.0.0}" -PORT="${DEVPI_PORT:-3141}" +# Note: Can't use DEVPI_PORT - Kubernetes auto-sets it for service discovery +PORT="${DEVPI_LISTEN_PORT:-3141}" OUTSIDE_URL="${DEVPI_OUTSIDE_URL:-}" # Check if devpi is initialized -- 2.50.1 (Apple Git-155) From 0da98efb90bae08768e2a8775a1c2207b3f33fd0 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 11:25:45 -0800 Subject: [PATCH 09/12] Increase devpi memory limit to 768Mi OOM during initialization - try modest increase first. Co-Authored-By: Claude Opus 4.5 --- argocd/manifests/devpi/statefulset.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/argocd/manifests/devpi/statefulset.yaml b/argocd/manifests/devpi/statefulset.yaml index 79a7770..620a774 100644 --- a/argocd/manifests/devpi/statefulset.yaml +++ b/argocd/manifests/devpi/statefulset.yaml @@ -35,10 +35,10 @@ spec: mountPath: /devpi resources: requests: - memory: "128Mi" + memory: "256Mi" cpu: "100m" limits: - memory: "512Mi" + memory: "768Mi" cpu: "500m" livenessProbe: httpGet: -- 2.50.1 (Apple Git-155) From 82f60bcf191ac2715e4a2a955b5647e1668d0efd Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 11:35:53 -0800 Subject: [PATCH 10/12] Remove devpi from ansible - now hosted in k8s - Remove devpi and devpi_metrics roles from indri.yml - Remove svc:pypi from tailscale_serve config - Remove devpi logs from alloy collection - Remove devpi backup from borgmatic - Update README with correct uvx --from devpi-client syntax Co-Authored-By: Claude Opus 4.5 --- ansible/playbooks/indri.yml | 5 +---- ansible/roles/alloy/defaults/main.yml | 7 +------ ansible/roles/borgmatic/defaults/main.yml | 6 ++---- ansible/roles/tailscale_serve/defaults/main.yml | 7 +------ argocd/manifests/devpi/README.md | 8 ++++---- 5 files changed, 9 insertions(+), 24 deletions(-) diff --git a/ansible/playbooks/indri.yml b/ansible/playbooks/indri.yml index dac4f5f..e001da5 100644 --- a/ansible/playbooks/indri.yml +++ b/ansible/playbooks/indri.yml @@ -42,10 +42,7 @@ tags: borgmatic_metrics - role: forgejo tags: forgejo - - role: devpi - tags: devpi - - role: devpi_metrics - tags: devpi_metrics + # NOTE: devpi and devpi_metrics roles removed - now hosted in k8s (see argocd/apps/devpi.yaml) - role: zot tags: zot - role: zot_metrics diff --git a/ansible/roles/alloy/defaults/main.yml b/ansible/roles/alloy/defaults/main.yml index ccc2d8d..ec867f9 100644 --- a/ansible/roles/alloy/defaults/main.yml +++ b/ansible/roles/alloy/defaults/main.yml @@ -43,12 +43,7 @@ alloy_brew_logs: # NOTE: postgresql and miniflux removed - now hosted in k8s alloy_mcquack_logs: - - path: /Users/erichblume/Library/Logs/mcquack.devpi.out.log - service: devpi - stream: stdout - - path: /Users/erichblume/Library/Logs/mcquack.devpi.err.log - service: devpi - stream: stderr + # NOTE: devpi logs removed - now hosted in k8s - path: /Users/erichblume/Library/Logs/mcquack.kiwix-serve.out.log service: kiwix stream: stdout diff --git a/ansible/roles/borgmatic/defaults/main.yml b/ansible/roles/borgmatic/defaults/main.yml index e7807fd..49816d9 100644 --- a/ansible/roles/borgmatic/defaults/main.yml +++ b/ansible/roles/borgmatic/defaults/main.yml @@ -11,13 +11,13 @@ borgmatic_schedule_hour: 2 borgmatic_schedule_minute: 0 # Source directories to back up +# NOTE: devpi removed - now hosted in k8s (PVC handles persistence) borgmatic_source_directories: - /Users/erichblume/code/personal/zk - /opt/homebrew/var/forgejo - /Users/erichblume/.config/borgmatic - /Users/erichblume/Documents - /Users/erichblume/Pictures - - /Users/erichblume/devpi - /opt/homebrew/var/loki # Backup repository @@ -28,9 +28,7 @@ borgmatic_repositories: append_only: true # Exclude patterns -borgmatic_exclude_patterns: - # Exclude mirrored PyPI cache (only backup private packages) - - /Users/erichblume/devpi/+files/root/pypi +borgmatic_exclude_patterns: [] # Encryption passcommand (reads borg passphrase) borgmatic_encryption_passcommand: cat /Users/erichblume/.borg/config.yaml diff --git a/ansible/roles/tailscale_serve/defaults/main.yml b/ansible/roles/tailscale_serve/defaults/main.yml index b8cd4ac..a7b437f 100644 --- a/ansible/roles/tailscale_serve/defaults/main.yml +++ b/ansible/roles/tailscale_serve/defaults/main.yml @@ -3,7 +3,7 @@ # Each service maps a Tailscale service name to local endpoints tailscale_serve_services: - # NOTE: svc:grafana, svc:pg, svc:feed removed - now hosted in k8s + # NOTE: svc:grafana, svc:pg, svc:feed, svc:pypi removed - now hosted in k8s - name: svc:forge https: @@ -18,11 +18,6 @@ tailscale_serve_services: port: 443 upstream: http://localhost:5501 - - name: svc:pypi - https: - port: 443 - upstream: http://127.0.0.1:3141 - - name: svc:registry https: port: 443 diff --git a/argocd/manifests/devpi/README.md b/argocd/manifests/devpi/README.md index a349853..11fd697 100644 --- a/argocd/manifests/devpi/README.md +++ b/argocd/manifests/devpi/README.md @@ -28,15 +28,15 @@ After the pod is running: ```fish # Login to devpi as root -uvx devpi use https://pypi.tail8d86e.ts.net -uvx devpi login root +uvx --from devpi-client devpi use https://pypi.tail8d86e.ts.net +uvx --from devpi-client devpi login root # Enter root password when prompted # Create eblume user (prompts for password - use the one from 1Password) -uvx devpi user -c eblume email=blume.erich@gmail.com +uvx --from devpi-client devpi user -c eblume email=blume.erich@gmail.com # Create private index inheriting from PyPI -uvx devpi index -c eblume/dev bases=root/pypi +uvx --from devpi-client devpi index -c eblume/dev bases=root/pypi ``` ## Usage -- 2.50.1 (Apple Git-155) From 782f95c64b33c3a9e555d14ef5f188894df2fba9 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 12:51:28 -0800 Subject: [PATCH 11/12] Increase devpi memory limit to 2Gi for PyPI index build The Whoosh search indexer needs significant memory during initial PyPI index build. Using high limit with low request so memory is reclaimed after indexing completes. Co-Authored-By: Claude Opus 4.5 --- argocd/manifests/devpi/statefulset.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/argocd/manifests/devpi/statefulset.yaml b/argocd/manifests/devpi/statefulset.yaml index 620a774..5d61c13 100644 --- a/argocd/manifests/devpi/statefulset.yaml +++ b/argocd/manifests/devpi/statefulset.yaml @@ -38,7 +38,7 @@ spec: memory: "256Mi" cpu: "100m" limits: - memory: "768Mi" + memory: "2Gi" # High limit for initial PyPI index build, reclaimed after cpu: "500m" livenessProbe: httpGet: -- 2.50.1 (Apple Git-155) From 765117bb9ef81686b8c9a0225771d2102c0c761e Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 20 Jan 2026 14:54:36 -0800 Subject: [PATCH 12/12] P5: Mark devpi migration complete - Add P5_devpi.complete.md with implementation notes - Document key learnings (registry mirrors, memory, env vars) - Remove incomplete P5_devpi.md Co-Authored-By: Claude Opus 4.5 --- plans/k8s-migration/P5_devpi.complete.md | 102 ++++++++++++ plans/k8s-migration/P5_devpi.md | 190 ----------------------- 2 files changed, 102 insertions(+), 190 deletions(-) create mode 100644 plans/k8s-migration/P5_devpi.complete.md delete mode 100644 plans/k8s-migration/P5_devpi.md diff --git a/plans/k8s-migration/P5_devpi.complete.md b/plans/k8s-migration/P5_devpi.complete.md new file mode 100644 index 0000000..0454a4a --- /dev/null +++ b/plans/k8s-migration/P5_devpi.complete.md @@ -0,0 +1,102 @@ +# Phase 5: devpi Migration to Kubernetes + +**Goal**: Migrate devpi PyPI caching proxy from indri to k8s + +**Status**: Complete (2026-01-20) + +**Prerequisites**: [Phase 4](P4_miniflux.complete.md) complete + +--- + +## Summary + +Successfully migrated devpi from mcquack LaunchAgent on indri to Kubernetes: +- Custom container image with devpi-server + devpi-web + auto-init startup script +- StatefulSet with 50Gi PVC for data persistence +- Tailscale Ingress at `pypi.tail8d86e.ts.net` +- Root password from 1Password secret, auto-initialized on first run +- Verified pip caching proxy and mcquack package upload + +--- + +## Key Learnings + +### Registry Mirror Configuration +- Minikube's CRI-O can't resolve Tailscale hostnames directly +- Added registry mirror config to redirect `registry.tail8d86e.ts.net` → `host.containers.internal:5050` +- Also added direct insecure registry entry for `host.containers.internal:5050` +- Config in `ansible/roles/minikube/files/zot-mirror.conf` + +### Memory Requirements +- devpi-web's Whoosh search indexer needs significant memory during PyPI index build +- Initial 512Mi limit caused OOMKills +- Solution: High limit (2Gi) with low request (256Mi) - memory reclaimed after indexing + +### Environment Variable Conflicts +- Kubernetes auto-sets `DEVPI_PORT` for service discovery +- Conflicted with our port config - renamed to `DEVPI_LISTEN_PORT` + +### Tailscale Serve Cleanup +- Use `tailscale serve status --json` to see entries (non-JSON output can be empty) +- Use `tailscale serve clear svc:` to remove entries + +### ArgoCD Workflow +- Changed `apps` to manual sync (was auto-sync with prune) +- Workflow: sync apps → set revision to feature branch → sync service → test → reset to main after merge + +--- + +## Verification Checklist + +- [x] devpi pod healthy in k8s +- [x] https://pypi.tail8d86e.ts.net accessible +- [x] Web interface shows root/pypi index +- [x] `pip install ` works through proxy +- [x] mcquack v1.0.0 uploaded to eblume/dev +- [x] `pip install --index-url https://pypi.tail8d86e.ts.net/eblume/dev/+simple/ mcquack` works +- [x] Old devpi service removed from indri +- [ ] zk documentation updated (deferred - no existing devpi card) + +--- + +## Files Changed + +### New Files +| Path | Purpose | +|------|---------| +| `argocd/apps/devpi.yaml` | ArgoCD Application definition | +| `argocd/manifests/devpi/Dockerfile` | Container image with startup script | +| `argocd/manifests/devpi/start.sh` | Auto-init startup script | +| `argocd/manifests/devpi/statefulset.yaml` | StatefulSet with PVC | +| `argocd/manifests/devpi/service.yaml` | ClusterIP Service | +| `argocd/manifests/devpi/ingress-tailscale.yaml` | Tailscale Ingress | +| `argocd/manifests/devpi/kustomization.yaml` | Kustomize configuration | +| `argocd/manifests/devpi/secret-root.yaml.tpl` | 1Password secret template | +| `argocd/manifests/devpi/README.md` | Setup documentation | + +### Modified Files +| Path | Change | +|------|--------| +| `CLAUDE.md` | Added k8s/ArgoCD workflow documentation | +| `ansible/playbooks/indri.yml` | Removed devpi and devpi_metrics roles | +| `ansible/roles/tailscale_serve/defaults/main.yml` | Removed svc:pypi | +| `ansible/roles/alloy/defaults/main.yml` | Removed devpi log collection | +| `ansible/roles/borgmatic/defaults/main.yml` | Removed devpi backup paths | +| `ansible/roles/minikube/files/zot-mirror.conf` | Added registry mirror for Tailscale hostname | +| `argocd/apps/apps.yaml` | Changed to manual sync policy | + +### Roles Kept (not deleted) +- `ansible/roles/devpi/` - Kept for reference +- `ansible/roles/devpi_metrics/` - Kept for reference + +--- + +## Post-Merge Cleanup + +After PR merge, reset ArgoCD apps to main: +```fish +argocd app set apps --revision main +argocd app sync apps +argocd app set devpi --revision main +argocd app sync devpi +``` diff --git a/plans/k8s-migration/P5_devpi.md b/plans/k8s-migration/P5_devpi.md deleted file mode 100644 index b5ea9d2..0000000 --- a/plans/k8s-migration/P5_devpi.md +++ /dev/null @@ -1,190 +0,0 @@ -# Phase 5: devpi Migration to Kubernetes - -**Goal**: Migrate devpi PyPI caching proxy from indri to k8s - -**Status**: In Progress (2026-01-20) - -**Prerequisites**: [Phase 4](P4_miniflux.complete.md) complete - ---- - -## Overview - -This phase migrates devpi from mcquack LaunchAgent on indri to Kubernetes: -1. Stop devpi on indri and clear Tailscale service -2. Build custom devpi container image (devpi-server + devpi-web) -3. Deploy as StatefulSet with PVC for data persistence -4. Expose via Tailscale Ingress at `pypi.tail8d86e.ts.net` -5. Initialize fresh (no data migration - just cache and one private package) -6. Validate with pip proxy and mcquack package upload - -**Decision: Fresh start** - The only private package is mcquack v1.0.0 which we'll re-upload. The PyPI cache is re-fetchable. - ---- - -## Steps - -### 1. Stop devpi on indri (FIRST) - -```bash -# Stop and unload the LaunchAgent -ssh indri 'launchctl unload ~/Library/LaunchAgents/mcquack.eblume.devpi.plist' - -# Also stop the metrics collector -ssh indri 'launchctl unload ~/Library/LaunchAgents/mcquack.eblume.devpi-metrics.plist' - -# Clear any tailscale serve entries (if any) -ssh indri 'tailscale serve off --service=svc:pypi' 2>/dev/null || true -``` - -**User action required**: Delete the `pypi` device/service from Tailscale admin console. - -### 2. Build devpi container image - -Create `argocd/manifests/devpi/Dockerfile`: - -```dockerfile -FROM python:3.12-slim - -# Install devpi-server and devpi-web -RUN pip install --no-cache-dir devpi-server devpi-web - -# Create data directory -RUN mkdir -p /devpi - -# Expose default port -EXPOSE 3141 - -# Use ENTRYPOINT for flexibility -ENTRYPOINT ["devpi-server"] - -# Default args (can be overridden) -CMD ["--serverdir", "/devpi", "--host", "0.0.0.0", "--port", "3141"] -``` - -Build and push to Zot: - -```bash -# From gilbert (has podman) -cd argocd/manifests/devpi -podman build -t registry.tail8d86e.ts.net/blumeops/devpi:latest . -podman push registry.tail8d86e.ts.net/blumeops/devpi:latest -``` - -### 3. Create k8s manifests - -| File | Purpose | -|------|---------| -| `argocd/apps/devpi.yaml` | ArgoCD Application definition | -| `argocd/manifests/devpi/statefulset.yaml` | StatefulSet with PVC | -| `argocd/manifests/devpi/service.yaml` | ClusterIP Service | -| `argocd/manifests/devpi/ingress-tailscale.yaml` | Tailscale Ingress for `pypi.tail8d86e.ts.net` | -| `argocd/manifests/devpi/init-job.yaml` | One-time initialization job | -| `argocd/manifests/devpi/kustomization.yaml` | Kustomize configuration | - -### 4. Deploy via ArgoCD - -```bash -# Point app at feature branch for testing -argocd app create devpi \ - --repo ssh://forgejo@indri.tail8d86e.ts.net:2200/eblume/blumeops.git \ - --path argocd/manifests/devpi \ - --dest-server https://kubernetes.default.svc \ - --dest-namespace devpi \ - --revision feature/p5-devpi - -# Or add to apps/ and let app-of-apps sync -argocd app sync apps -argocd app sync devpi -``` - -### 5. Initialize devpi - -After StatefulSet is running: - -```bash -# Exec into pod to initialize -kubectl -n devpi exec -it devpi-0 -- devpi-init --serverdir /devpi --root-passwd - -# Or use init job with secret -``` - -Create user and index: - -```bash -# From workstation -uvx devpi use https://pypi.tail8d86e.ts.net -uvx devpi login root -uvx devpi user -c eblume email=blume.erich@gmail.com -uvx devpi index -c eblume/dev bases=root/pypi -``` - -### 6. Update pip.conf on gilbert - -The existing `~/.config/pip/pip.conf` should work unchanged since we're keeping the same hostname `pypi.tail8d86e.ts.net`. - -### 7. Remove devpi ansible role - -- Remove `devpi` from `ansible/playbooks/indri.yml` roles list -- Remove `devpi_metrics` from alloy textfile collectors -- Keep devpi ansible role directory for reference (or delete) - -### 8. Validate - -```bash -# Test pip cache proxy -pip install --index-url https://pypi.tail8d86e.ts.net/root/pypi/+simple/ requests - -# Upload mcquack -cd ~/code/personal/mcquack -uv build -uv publish --publish-url https://pypi.tail8d86e.ts.net/eblume/dev/ -``` - ---- - -## New Files - -| Path | Purpose | -|------|---------| -| `argocd/apps/devpi.yaml` | ArgoCD Application definition | -| `argocd/manifests/devpi/Dockerfile` | Container image build | -| `argocd/manifests/devpi/statefulset.yaml` | StatefulSet with PVC | -| `argocd/manifests/devpi/service.yaml` | ClusterIP Service | -| `argocd/manifests/devpi/ingress-tailscale.yaml` | Tailscale Ingress | -| `argocd/manifests/devpi/kustomization.yaml` | Kustomize configuration | -| `argocd/manifests/devpi/README.md` | Setup documentation | - -## Modified Files - -| Path | Change | -|------|--------| -| `ansible/playbooks/indri.yml` | Remove devpi role | -| `ansible/roles/alloy/defaults/main.yml` | Remove devpi textfile collector | -| `ansible/roles/borgmatic/defaults/main.yml` | Remove devpi backup (no longer on indri) | - -## Deleted Files - -| Path | Reason | -|------|--------| -| `ansible/roles/devpi/` | Role no longer needed | -| `ansible/roles/devpi_metrics/` | Metrics collected differently in k8s | - ---- - -## Verification Checklist - -- [ ] devpi pod healthy in k8s -- [ ] https://pypi.tail8d86e.ts.net accessible -- [ ] Web interface shows root/pypi index -- [ ] `pip install ` works through proxy -- [ ] mcquack v1.0.0 uploaded to eblume/dev -- [ ] `pip install --index-url https://pypi.tail8d86e.ts.net/eblume/dev/+simple/ mcquack` works -- [ ] Old devpi service removed from indri -- [ ] zk documentation updated - ---- - -## Implementation Notes - -*To be filled in during implementation* -- 2.50.1 (Apple Git-155)