## Summary
- Add `containers/tempo/Dockerfile` — two-stage Go build from forge mirror, modeled on loki
- Switch kustomization from upstream `grafana/tempo` to `registry.ops.eblu.me/blumeops/tempo`
- Bump Tempo 2.10.1 → 2.10.3
## Test plan
- [ ] Kick off container build via `mise run container-build-and-release tempo`
- [ ] Update kustomization `newTag` with built image tag
- [ ] Deploy from branch: `argocd app set tempo --revision local-tempo-container && argocd app sync tempo`
- [ ] Verify Tempo health: `curl tempo.ops.eblu.me/ready`
- [ ] Verify traces flowing in Grafana Tempo datasource
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #323
Stamp single-user-cluster last-reviewed to 2026-04-01 after verifying
Tailscale ACLs and kubeconfig distribution. Add aspirational how-to card
documenting what PCI DSS evidence collection would look like (CCW,
artifacts, Drata workflow). Link from existing review process card.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Add `nixpkgs-services` flake input pinned to a specific nixpkgs commit, with an overlay that pulls `forgejo-runner`, `snowflake`, and `k3s` from it instead of the rolling `nixpkgs`
- Dagger `flake-update` pipeline now excludes `nixpkgs-services` via `--exclude`
- Fix stale nix-container-builder version in service-versions.yaml (was 12.6.4, actually running 12.7.2)
- Add k3s and minikube to service-versions.yaml tracking
- Document the pinning approach in review-services how-to and ringtail reference
## Motivation
During service review, discovered that flake updates had silently upgraded forgejo-runner from 12.6.4 → 12.7.2 without updating service-versions.yaml. This "sneak-in upgrade" bypasses the service review process. The overlay ensures these three services only change versions deliberately.
## Test plan
- [ ] Verify `nix flake update` from `nixos/ringtail/` does not change `nixpkgs-services` lock entry
- [ ] Verify `mise run provision-ringtail` builds successfully with the overlay
- [ ] Confirm running service versions unchanged after deploy
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #321
Patch upgrade picks up idempotent FetchTask API, offline registration
fix, cloudflare/circl security dep update, and custom gRPC user-agent.
No config defaults changed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Restrict backup to library/ and upload/ only (skip regenerable encoded-video/,
thumbs/, backups/). Add SSH ServerAliveInterval to prevent broken pipe on long
transfers, and checkpoint_interval so interrupted backups save progress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Kingfisher exits 200 (findings) or 205 (validated findings) on success.
Normalize these to 0 so the CronJob completes instead of restarting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mirror repos cause scan failures (likely ephemeral storage or timeout).
Scan only eblume/ repos until we investigate the root cause.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Container runs as user 65534 (nobody) but /tmp was owned by root.
Set sticky bit + world-writable (1777) like a standard /tmp.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Kingfisher needs a writable temp directory for git clones and scanning.
Nix containers don't create /tmp by default.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Nix containers don't include a shell by default. The CronJob needs
/bin/bash for the inline script that generates timestamped filenames.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Kingfisher will build via Nix on ringtail instead of Dockerfile on
indri, so the skip is no longer needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Kingfisher's Rust + Boost/vectorscan build exhausts indri's memory
(aws-sdk-ec2 alone needs 2-3GB for rustc). Build locally on Gilbert
and push manually until we have a beefier build host.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The pod-level RuntimeDefault seccomp profile (07e9c81) overrides the
DinD sidecar's privileged flag in newer Kubernetes versions, blocking
Docker daemon syscalls. Set Unconfined explicitly on the DinD container
while keeping RuntimeDefault on the runner container.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bail if upstream already has branches named 'blumeops' or 'deploy',
which would conflict with the spork branch naming strategy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Upstream can push workflows (in .github/ or .forgejo/) that execute
on our runners via any trigger mechanism including cron. Runner label
mismatch is the current defense but is fragile. No complete fix exists
short of disabling Actions entirely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
git checkout <branch> is ambiguous when both origin and mirror remotes
have the same branch name. Use -B to explicitly create from origin.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Forks from mirror repos have has_actions disabled by default.
PATCH the repo settings to enable it so the mirror-sync workflow runs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Check local path, mirror existence, and fork absence upfront.
Fail fast with clear error messages before touching forge or disk.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Trying to add remotes to an existing clone gets the origin wrong.
Better to error out and let the user handle it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Spork-create mise task sets up a floating-branch soft-fork of a
mirrored upstream project with daily mirror-sync via Forgejo Actions.
Includes explanation card, how-to guides for setup and branch
management, and the spork-create uv script.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Deploys MongoDB Kingfisher as a weekly CronJob on minikube-indri
- Scans all Forgejo repos (eblume + all orgs) for leaked secrets with live validation
- Produces timestamped HTML and JSON reports on sifaka NFS (`/volume1/reports/kingfisher/`)
- Forgejo API token sourced from 1Password via ExternalSecret
- Uses official `ghcr.io/mongodb/kingfisher:1.91.0` container image
- Runs Sunday 4am (after Prowler's 3am k8s scan)
## Resources
- CronJob, PV/PVC (sifaka NFS), ExternalSecret
- ArgoCD Application with manual sync + CreateNamespace
## Test plan
- [x] Sync ArgoCD `apps` app to pick up new kingfisher Application
- [x] Set `--revision feature/kingfisher-cronjob` on kingfisher app
- [x] Verify ExternalSecret creates the `kingfisher-forgejo-token` Secret
- [x] Trigger manual job: `kubectl create job --from=cronjob/kingfisher kingfisher-manual -n kingfisher --context=minikube-indri`
- [ ] Verify reports appear on sifaka at `/volume1/reports/kingfisher/`
- [ ] After merge: set `--revision main` and re-sync
Reviewed-on: #317
Running alongside TruffleHog to compare coverage. Kingfisher uses
staged-only mode with validation disabled for fast, offline-safe
pre-commit checks. Validation will be enabled in the planned cron job.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sifaka's Tailscale can revert to userspace networking after package
updates, causing NFS mounts to fail because the NFS daemon sees
127.0.0.1 instead of the client's Tailscale IP. Added troubleshooting
how-to doc and updated sifaka reference card with frigate export and
TUN requirement.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Resources were under wrong Helm value keys (server.resources,
machine-learning.resources) and never applied to pods. Move to correct
bjw-s chart paths (*.controllers.main.containers.main.resources).
Increase liveness/readiness probe timeouts from 1s to 5s to prevent
kubelet from killing healthy-but-busy pods during ML inference load.
Remove CPU limits (keep requests only) to avoid throttling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Adds a second borgmatic config (`photos.yaml`) that backs up `/Volumes/photos` (sifaka SMB mount, ~128 GB) to a dedicated BorgBase repo (`immich-photos`), running daily at 4 AM
- Separate launchd agent (`mcquack.eblume.borgmatic-photos`) so photo backups run independently from the main backup
- Refactors `borgmatic_metrics` script to support multiple repos with a `repo` Prometheus label
- Updates Grafana "Borg Backups" dashboard with a `repo` template variable so you can filter/compare repos
- Docs updated: `backups.md`, `borgmatic.md`
## Prerequisites (manual)
- [x] Create `immich-photos` repo on BorgBase with same SSH key
- [ ] Upgrade BorgBase plan to Small ($24/yr) if currently on free tier (128 GB exceeds 10 GB limit)
- [ ] After deploy: `borg init` the new repo (borgmatic does this automatically on first run)
## Test plan
- [ ] Dry run: `mise run provision-indri -- --check --diff --tags borgmatic,borgmatic_metrics`
- [ ] Deploy borgmatic role and verify both configs deployed
- [ ] Run `borgmatic --config ~/.config/borgmatic/photos.yaml create --verbosity 1` manually for first backup (will take hours)
- [ ] Verify metrics script collects from both repos: `~/.local/bin/borgmatic-metrics && cat /opt/homebrew/var/node_exporter/textfile/borgmatic.prom`
- [ ] Sync grafana-config in ArgoCD and verify dashboard repo selector works
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #315
## Summary
- Add `authentik` database (blumeops-pg cluster) to borgmatic pg_dump backups
- Add `immich` database (immich-pg cluster) to borgmatic pg_dump backups
- For immich-pg: new borgmatic managed role with `pg_read_all_data`, ExternalSecret, Tailscale LoadBalancer service, and Caddy L4 TCP proxy on port 5433
- Update backup docs to reflect all four CNPG databases + mealie SQLite
## Deploy plan
Deploy order matters — k8s resources must exist before ansible can route to them:
1. **ArgoCD (databases app):** sync to pick up immich-pg borgmatic role, ExternalSecret, and Tailscale service
```
argocd app set blumeops-pg --revision feature/borgmatic-all-pg-backups
argocd app sync blumeops-pg
```
2. **Wait** for `immich-pg-tailscale` service to get a Tailscale IP and `immich-pg.tail8d86e.ts.net` to resolve
3. **Ansible (caddy):** deploy Caddy L4 route for port 5433
```
mise run provision-indri -- --tags caddy
```
4. **Ansible (borgmatic):** deploy updated config and .pgpass
```
mise run provision-indri -- --tags borgmatic
```
5. **Verify:** trigger a manual borgmatic run and check all four pg_dump streams succeed
```
borgmatic --verbosity 1 2>&1 | grep -E '(Dumping|ERROR)'
```
## Test plan
- [x] `kubectl kustomize` builds cleanly
- [x] `ansible --check --diff` for borgmatic and caddy show expected changes
- [ ] ArgoCD sync succeeds for databases app
- [ ] `immich-pg.tail8d86e.ts.net` resolves
- [ ] `pg.ops.eblu.me:5433` accepts connections
- [ ] `borgmatic --verbosity 1` dumps all four databases without errors
Reviewed-on: #314
Single-file Go tool implementing the QArt technique (Russ Cox, 2012)
using only the public rsc.io/qr API. Generates QR codes whose data
modules form a recognizable image by exploiting error correction
freedom via GF(2) Gaussian elimination.
Includes a web UI with live-updating sliders for version, mask,
rotation, dx/dy offset, and scale. Keyboard shortcuts for rapid
iteration. Also works as a CLI for batch generation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update manage-lockfile doc with post-deploy steps (kernel update detection,
reboot guidance, generation pruning). Add prune-ringtail-generations mise
task that keeps the 5 most recent generations plus the most recent one
matching the booted kernel for safe rollback.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
