## Summary
- Deploy UnPoller as a k8s service on indri to export UniFi controller metrics to Prometheus
- Custom-built container from forge mirror (`containers/unpoller/Dockerfile`)
- Credentials pulled from 1Password via external-secrets
- Prometheus scrape job added, docs and service-versions updated
## Test plan
- [ ] Build container: `mise run container-release unpoller v2.34.0`
- [ ] Update kustomization tag with built image tag
- [ ] Deploy from branch: `argocd app set unpoller --revision feature/unpoller && argocd app sync unpoller`
- [ ] Verify pod connects to UX7 controller (check logs)
- [ ] Confirm `unpoller` target appears in Prometheus
- [ ] Query `unifi_` metrics in Grafana
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #298
## Summary
- Upgraded borgmatic from 2.0.13 to 2.1.3 on indri (via mise/pipx)
- Key changes: improved borg warning handling, memory/performance improvements, `source_directories_must_exist` now defaults to true (already set in our config)
- Verified: config validates, dry-run passed against both sifaka (local) and borgbase (offsite) repos
## Borg Warnings Investigation
The main concern was 2.1.0's change to treat borg warnings as errors. In 2.1.3 this was partially reverted — "file not found" warnings (exit code 107) are back to being warnings. Our config already sets `source_directories_must_exist: true`, and all four source directories were verified present on indri.
## Test plan
- [x] `borgmatic --version` confirms 2.1.3
- [x] `borgmatic config validate` passes
- [x] `borgmatic create --dry-run` succeeds against both repositories
- [x] All source directories verified present on indri
- [ ] Verify next scheduled backup (2:00 AM) completes successfully
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #297
## Summary
- Upgrade zot OCI registry from v2.1.13 to v2.1.15 on indri
- Addresses CVE-2025-30204 (golang-jwt memory) and open redirect via callback_ui
- No config template changes needed (externalUrl is auto-allowlisted)
- Requires Go 1.25.7 (bump from 1.25.6 via mise)
## Data Safety
- Data directory ~/erichblume/zot is NOT touched during build or deploy
- No schema migrations in v2.1.14 or v2.1.15
- Storage format remains OCI spec 1.1.0
## Deployment Steps
- [ ] SSH to indri: bump Go to 1.25.7 via `mise use go@1.25.7`
- [ ] Fetch and checkout v2.1.15 in ~/code/3rd/zot
- [ ] Build: `mise x -- make binary`
- [ ] Restart LaunchAgent
- [ ] Verify: `curl -s http://localhost:5050/v2/` returns 200
- [ ] Verify: `curl -s https://registry.ops.eblu.me/v2/_catalog` lists repos
- [ ] Verify: `mise run services-check`
Reviewed-on: #293
Mosquitto has been dormant since frigate-notify switched from MQTT to
webapi polling (529ba10). Tear down live infra (ArgoCD app, namespace)
and remove all manifests, service-versions entry, services-check, and
doc references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Add jobsync pod check (ringtail k3s) and HTTP endpoint to `services-check`
- Add JobSync entry to homepage dashboard under new "Apps" group
- Mark jobsync as reviewed at v1.1.4 (current with upstream)
- Changelog fragment added
## Deployment and Testing
- [ ] Sync homepage app from branch: `argocd app set homepage --revision review/jobsync && argocd app sync homepage`
- [ ] Verify JobSync appears on go.ops.eblu.me dashboard
- [ ] Run `mise run services-check` to verify new checks pass
- [ ] After merge: `argocd app set homepage --revision main && argocd app sync homepage`
Reviewed-on: #291
## Summary
C2 Mikado chain to deploy [JobSync](https://github.com/Gsync/jobsync) — a self-hosted job application tracker — to ringtail's k3s cluster.
### Mikado Graph
```
deploy-jobsync (goal)
├── build-jobsync-container
│ └── mirror-jobsync
└── integrate-jobsync-ollama
```
### What is JobSync?
Next.js app with SQLite for tracking job applications. Features resume management, application pipeline tracking, and AI-powered resume review/job matching.
### Key Decisions
- **Ringtail k3s** (not minikube-indri) — colocates with Ollama for zero-latency AI
- **Nix container** via `buildLayeredImage` — no Dockerfile, mirrors upstream source on forge
- **Ollama for AI** — uses existing deployment, no API keys needed for AI features
- **No upstream fork** — vanilla JobSync, Anthropic AI deferred to future work if needed
### Current Status
Planning phase — cards committed, ready for review before implementation begins.
Reviewed-on: #288
Phase 1 of Dagger upgrade: update the CLI in the runner container first
so CI can build the new image with the old engine version. See
[[upgrade-dagger]] for the full procedure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bare image references in manifests were ambiguous — unclear whether the
tag was intentionally omitted or managed by kustomize. Add :kustomized
sentinel to all 37 image refs overridden by kustomize images transformer.
Add sync notes for tailscale-operator proxyclass (CRD fields not processed
by kustomize). Mark devpi reviewed (6.19.1 is current).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Dagger module was upgraded to v0.20.0 in d15071a but the runner job
image still had the old CLI, causing build-blumeops to fail with a
version mismatch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Minor upstream release with doc and CI fixes. Also corrects kiwix.md
to reference the actual custom registry image and torrents.txt path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Upgrade Transmission from 4.0.6-r4 to 4.1.1-r1
- Uses Alpine edge community repo for transmission packages, keeping stable alpine:3.22 base
- Fix stale image reference in service doc (was linuxserver, now custom registry image)
- Mark transmission as reviewed in service-versions.yaml
## Context
Service review found Transmission two minor versions behind (4.0.6 → 4.1.1). Alpine 3.22 only packages 4.0.6, so transmission is installed from edge's community repo with an exact version pin.
4.1.0 added improved µTP performance, IPv6/dual-stack UDP tracker, JSON-RPC 2.0 API. 4.1.1 is a bugfix release (20+ fixes).
Dagger test build passed locally.
## Deployment and Testing
- [ ] Build container via Forgejo workflow (`mise run container-build-and-release transmission`)
- [ ] Update kustomization.yaml with new image tag
- [ ] `argocd app set torrent --revision feature/transmission-review && argocd app sync torrent`
- [ ] Verify web UI at https://torrent.ops.eblu.me
- [ ] Check Grafana Transmission dashboard still receives metrics
- [ ] After merge: `argocd app set torrent --revision main && argocd app sync torrent`
## Note
The transmission-exporter sidecar (OOMKilling every ~30min, 294 restarts) is being tracked separately as a future replacement project.
Reviewed-on: #282
Replace floating :18 tag with pinned :18.3 (upstream out-of-cycle
release fixing 18.2 regressions). Stamps service as reviewed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Widen `repo-creds-forge` URL prefix from `/eblume/` to host-wide `/` so it matches repos in all forge orgs (fixes `mirrors/` repos not getting SSH credentials)
- Update 8 ArgoCD app definitions from `eblume/<mirror>` → `mirrors/<mirror>` (immich-charts, cloudnative-pg-charts, external-secrets, connect-helm-charts)
- Fix stale alloy clone comment in Ansible defaults
- Bump immich v2.5.2 → v2.5.6 (bug-fix patches only)
- Update ArgoCD README bootstrap command and credential docs
## Context
Mirrors were migrated from `forge.ops.eblu.me/eblume/` to `forge.ops.eblu.me/mirrors/` in commit `cd57814`. Container Dockerfiles and image tags were updated, but ArgoCD app definitions and the repo credential template were missed, causing `ComparisonError` on apps that source Helm charts from mirrored repos.
## Deployment
1. Sync the ArgoCD `argocd` app first (picks up the widened credential template)
2. Sync the `apps` app (picks up new repo URLs for all 8 apps)
3. Verify immich resolves its ComparisonError: `argocd app get immich`
4. Sync immich to deploy v2.5.6: `argocd app sync immich`
5. Spot-check: `argocd app get external-secrets`, `argocd app get cloudnative-pg`, `argocd app get 1password-connect`
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/266
The forgejo-runner container is the CI job execution environment (Dagger,
ArgoCD CLI, etc.), not the runner daemon itself. Rename to runner-job-image
to fix the version-check false positive (Dagger 0.19.11 vs daemon 12.7.0)
and clarify the distinction.
RUNNER_LABELS still references the old image name — will update after
building the image under the new name.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Delete the old 3-phase Helm chart upgrade plan (predates Mikado system)
- Create C2 Mikado chain with goal card `upgrade-grafana` and two leaf prereqs:
- `kustomize-grafana-deployment` — convert Helm to kustomize manifests
- `build-grafana-container` — home-built Grafana 12.x image (no upstream containers)
- Record first-ever Grafana review: currently at v11.4.0 on Helm chart 8.8.2
- Update service-versions.yaml, how-to index, and plans index
## Service Review Findings
- Grafana is healthy and synced in ArgoCD
- Running v11.4.0, latest upstream is 12.3.3
- Breaking changes for 12.x are low-risk (React panels only, UIDs compliant)
- PVC is disposable — dashboards and datasources are all config-provisioned
## Deployment and Testing
- [ ] No deployment needed — documentation-only change
- [ ] `docs-check-links` passes
- [ ] `docs-check-index` passes
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/258
## Summary
Completes the `upgrade-k8s-runner` mikado chain. Both prerequisites (workflow validation in Dagger, config review against v12 defaults) were resolved in #250.
- Bump runner image `code.forgejo.org/forgejo/runner:6.3.1` → `12.7.0`
- Update `service-versions.yaml` to track new version
- Mark goal card complete (remove `status: active`)
## Deployment and Testing
After merge:
1. `argocd app sync forgejo-runner`
2. Verify runner registers in Forgejo admin → runners
3. Trigger a test workflow (e.g. `branch-cleanup.yaml` manual dispatch)
Rollback: revert image tag to `6.3.1`, push, sync.
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/251
## Summary
- C2 Mikado chain for upgrading the k8s forgejo-runner daemon (6 major versions behind)
- Root goal card with two leaf prerequisites: workflow validation and config review
- Ringtail runner is already at ~v12.6.4 via nixpkgs, no work needed there
## Mikado Chain
```
upgrade-k8s-runner (goal)
├── validate-workflows-against-v12 (leaf)
└── review-runner-config-v12 (leaf)
```
Both leaves are actionable now. The biggest risk is workflow schema validation
(introduced in v8/v9) rejecting our existing workflows.
## Next Steps
Work the leaf nodes in a follow-up session, then attempt the goal.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/249
## Summary
- Enable OIDC + API key authentication on zot with anonymous pull preserved
- Enforce tag immutability for version tags
- Adopt commit-SHA-based container image tagging
Details in the [[harden-zot-registry]] Mikado chain (`mise run docs-mikado harden-zot-registry`).
## Test plan
- [ ] Anonymous pull still works
- [ ] Unauthenticated push fails (401)
- [ ] CI container builds pass with new auth and tagging
- [ ] `mise run services-check` passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/231
## Summary
- Bump Frigate image from `0.16.4-standard-arm64` to `0.17.0-rc2-standard-arm64`
- Adapt `record` config to 0.17 schema: `retain.days`/`mode: all` → `continuous.days`
- Update service docs and version tracker
This is the first step toward the Apple Silicon ZMQ detector. The existing ONNX detector is kept so we can validate the upgrade independently.
## What is NOT changing
- Detector config (still `type: onnx` with YOLO-NAS-s)
- go2rtc streams, MQTT, cameras, zones, review rules
- frigate-notify, storage PVs, Grafana dashboard
## Deployment and Testing
- [ ] `argocd app set frigate --revision upgrade-frigate-0.17 && argocd app sync frigate`
- [ ] Pod starts, `/api/version` returns `0.17.0-rc2`
- [ ] No config errors in pod logs
- [ ] Frigate web UI loads at `https://nvr.ops.eblu.me`
- [ ] Live view works, detection running (`/api/stats` shows `detection_fps > 0`)
- [ ] Recordings being created (`/api/recordings/summary`)
- [ ] MQTT events flowing (check frigate-notify logs)
- [ ] After merge: `argocd app set frigate --revision main && argocd app sync frigate`
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/205
## Summary
- Upgrade ntfy from v2.11.0 to v2.17.0 (6 minor releases, no breaking changes)
- Add reference doc for ntfy service
- Add reference doc for frigate service (ntfy's sole producer via frigate-notify)
- Update reference index and service-versions.yaml tracking
## Notable upstream changes (v2.12.0–v2.17.0)
- **v2.14.0:** Declarative users/ACL config in files
- **v2.15.0:** `require-login` flag for topic-level auth
- **v2.16.0:** Dead man's switch (heartbeat) notifications, notification update/delete
- **v2.17.0:** Priority templating, crash fixes (nil pointer panics)
## Deployment and Testing
- [ ] ArgoCD sync ntfy after merge
- [ ] Verify ntfy pod healthy with new image
- [ ] Send a test notification via `curl -d "test" https://ntfy.ops.eblu.me/test`
- [ ] Verify frigate-notify still delivers alerts to ntfy
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/201
## Summary
- The `eclipse-mosquitto:2.1.2` tag doesn't exist on Docker Hub — the 2.1.x series only publishes `-alpine` variants
- Corrects the pinned tag to `2.0.22`, the latest non-alpine version (matching what the old floating `:2` tag was resolving to)
- Updates tracking file and changelog fragment accordingly
## Context
The previous PR #197 pinned mosquitto from floating `:2` to `2.1.2`, but the new pod failed with `ErrImagePull` ("manifest unknown"). The old pod is still running on `:2`.
## Test plan
- [ ] Verify `eclipse-mosquitto:2.0.22` pulls successfully
- [ ] Verify mosquitto pod restarts and passes readiness/liveness probes
- [ ] `mise run services-check` passes
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/198
## Summary
- Add `service-versions.yaml` tracking file with 33 services and upstream release URLs
- Add `mise run service-review` task (Python uv script) mirroring the docs-review UX
- Add `review-services` how-to article covering the review process by service type
- Add `[[review-services]]` link to the how-to index Knowledge Base table
## Deployment and Testing
- [x] `mise run service-review` displays 33 services, all "never reviewed"
- [x] `mise run service-review -- --type ansible` filters to 7 Ansible services
- [x] `mise run service-review -- --limit 5` shows 5 rows
- [x] `mise run docs-check-links` — no broken wiki-links
- [x] `mise run docs-check-frontmatter` — new doc passes validation
- [x] All pre-commit hooks pass
Reviewed-on: https://forge.ops.eblu.me/eblume/blumeops/pulls/196
