From ccaef4c1a77cd7431f33d0d6e3fe516ad0ed3829 Mon Sep 17 00:00:00 2001 From: Erich Blume Date: Tue, 14 Apr 2026 07:38:06 -0700 Subject: [PATCH] Document devpi cold cache failure mode and deploy teslamate v3.0.0-08c698e MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After a DR rebuild, devpi's empty cache causes race conditions under concurrent load — metadata is served but wheel files 404. Also deploys the first container.py-built teslamate image. Co-Authored-By: Claude Opus 4.6 (1M context) --- argocd/manifests/teslamate/kustomization.yaml | 2 +- .../operations/rebuild-minikube-cluster.md | 20 +++++++++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/argocd/manifests/teslamate/kustomization.yaml b/argocd/manifests/teslamate/kustomization.yaml index ac9e1ea..a00586f 100644 --- a/argocd/manifests/teslamate/kustomization.yaml +++ b/argocd/manifests/teslamate/kustomization.yaml @@ -12,4 +12,4 @@ resources: images: - name: registry.ops.eblu.me/blumeops/teslamate - newTag: v3.0.0-613f05d + newTag: v3.0.0-08c698e diff --git a/docs/how-to/operations/rebuild-minikube-cluster.md b/docs/how-to/operations/rebuild-minikube-cluster.md index 2b97b78..26207a7 100644 --- a/docs/how-to/operations/rebuild-minikube-cluster.md +++ b/docs/how-to/operations/rebuild-minikube-cluster.md @@ -239,6 +239,26 @@ mise run services-check | Zot → Authentik OIDC | All container image pulls from Zot | Sync authentik early; Zot will crash-loop until OIDC is reachable | | ArgoCD Endpoints exclusion → forge-external | Forge Tailscale ingress has no backend | Manual `kubectl apply` for Endpoints | +## Post-Rebuild: Cold Cache Failures + +### Devpi (PyPI Cache) + +After a rebuild, devpi's package cache is empty. The first Dagger-based container build will trigger a flood of concurrent package downloads. Devpi uses lazy caching — it serves package metadata (simple index) immediately from upstream PyPI but fetches wheel files on demand. Under heavy concurrent load with a cold cache, the upstream fetch can race with the client request, causing devpi to return `no such file` (HTTP 404) for packages it knows about but hasn't finished downloading yet. + +**Symptoms:** Forgejo Actions Dagger builds fail during module initialization with errors like: +``` +Failed to download `googleapis-common-protos==1.74.0` +HTTP status client error (404 Not Found) for url (https://pypi.ops.eblu.me/root/pypi/+f/...) +``` + +**Fix:** Re-run the failed build. The first attempt warms the cache; subsequent builds succeed. Alternatively, warm the cache manually before triggering CI builds: + +```bash +# From any machine that can reach pypi.ops.eblu.me, install the Dagger SDK +# to pre-populate the most common packages: +pip install --dry-run --index-url https://pypi.ops.eblu.me/root/pypi/+simple/ dagger-io +``` + ## Related - [[restart-indri]] — Normal restart procedure (no data loss)