eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Document devpi cold cache failure mode and deploy teslamate v3.0.0-08c698e

Browse source 
After a DR rebuild, devpi's empty cache causes race conditions under
concurrent load — metadata is served but wheel files 404. Also deploys
the first container.py-built teslamate image.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Erich Blume 2026-04-14 07:38:06 -07:00
commit ccaef4c1a7
2 changed files with 21 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
argocd/manifests/teslamate/kustomization.yaml
View file

@ -12,4 +12,4 @@ resources:
images: images:
- name: registry.ops.eblu.me/blumeops/teslamate - name: registry.ops.eblu.me/blumeops/teslamate
newTag: v3.0.0-613f05d newTag: v3.0.0-08c698e

20
docs/how-to/operations/rebuild-minikube-cluster.md
View file

@ -239,6 +239,26 @@ mise run services-check
| Zot → Authentik OIDC | All container image pulls from Zot | Sync authentik early; Zot will crash-loop until OIDC is reachable | | Zot → Authentik OIDC | All container image pulls from Zot | Sync authentik early; Zot will crash-loop until OIDC is reachable |
| ArgoCD Endpoints exclusion → forge-external | Forge Tailscale ingress has no backend | Manual `kubectl apply` for Endpoints | | ArgoCD Endpoints exclusion → forge-external | Forge Tailscale ingress has no backend | Manual `kubectl apply` for Endpoints |
## Post-Rebuild: Cold Cache Failures
### Devpi (PyPI Cache)
After a rebuild, devpi's package cache is empty. The first Dagger-based container build will trigger a flood of concurrent package downloads. Devpi uses lazy caching — it serves package metadata (simple index) immediately from upstream PyPI but fetches wheel files on demand. Under heavy concurrent load with a cold cache, the upstream fetch can race with the client request, causing devpi to return `no such file` (HTTP 404) for packages it knows about but hasn't finished downloading yet.
**Symptoms:** Forgejo Actions Dagger builds fail during module initialization with errors like:
```
Failed to download `googleapis-common-protos==1.74.0`
HTTP status client error (404 Not Found) for url (https://pypi.ops.eblu.me/root/pypi/+f/...)
```
**Fix:** Re-run the failed build. The first attempt warms the cache; subsequent builds succeed. Alternatively, warm the cache manually before triggering CI builds:
```bash
# From any machine that can reach pypi.ops.eblu.me, install the Dagger SDK
# to pre-populate the most common packages:
pip install --dry-run --index-url https://pypi.ops.eblu.me/root/pypi/+simple/ dagger-io
```
## Related ## Related
- [[restart-indri]] — Normal restart procedure (no data loss) - [[restart-indri]] — Normal restart procedure (no data loss)