Commit graph

836 commits

Author SHA1 Message Date
ba5c3a6bae Add Authentik OIDC provider and application for Paperless
Blueprint with confidential client, ExternalSecret for client secret,
and worker env var injection — follows existing service pattern.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 17:49:24 -07:00
0bde34d6e1 Fix paperless port: override k8s-injected PAPERLESS_PORT env var
Kubernetes auto-injects PAPERLESS_PORT=tcp://... for a service named
'paperless', which conflicts with Granian's --port flag. Explicitly
set PAPERLESS_PORT=8000 to take precedence.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 17:39:53 -07:00
919ad04e30 Pin paperless and redis image tags from registry
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 17:34:17 -07:00
42f6299eaa Rewrite paperless Dockerfile to match upstream structure
Add syntax directive for BuildKit, use COPY --from=source instead of
inline git clone, fix s6-overlay arch mapping, use upstream jbig2enc
v0.30 trixie build, and enable RUN --mount=type=cache for Python deps.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 17:03:31 -07:00
fba339e543 Fix jbig2enc download: ADD doesn't expand shell subcommands
Use curl in a RUN instead of ADD so $(dpkg --print-architecture)
is evaluated by the shell.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 16:44:11 -07:00
d036782b43 Fix paperless admin username and email
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 16:38:57 -07:00
041c47acfb Deploy Paperless-ngx document management system
Add paperless-ngx (v2.20.13) as a new ArgoCD-managed service on indri
with Authentik OIDC SSO, PostgreSQL on blumeops-pg, Redis sidecar, and
NFS document storage on sifaka. Includes Dockerfile built from forge
mirror, full k8s manifests, Caddy route, 1Password secrets, and
reference documentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 16:33:34 -07:00
e04455c911 Add changelog fragment for adding-a-service tutorial review
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 11:29:54 -07:00
d3235c5ca9 Review adding-a-service tutorial: fix ingress, repoURL, add kustomize and reference card steps
- Fix Tailscale Ingress: move hostname to tls.hosts, remove from rules (ProxyGroup compat)
- Update ArgoCD repoURL to forge.ops.eblu.me:2222
- Add kustomization.yaml section with :kustomized sentinel tag pattern
- Add Step 5: Create a Reference Card (keep under 30s reading time)
- Set last-reviewed: 2026-04-08

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 11:28:46 -07:00
22b77ac141 Fix Frigate preview config and services-check NoData detection
preview.quality was at the top level (invalid); moved under record
with a valid preset (very_low). Also fix services-check to catch
Grafana "Alerting (NoData)" state which was silently passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 11:12:42 -07:00
ec63d560f3 Deploy authentik 2026.2.2 container to ringtail
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:56:50 -07:00
2eb28301e4 Upgrade authentik 2026.2.0 → 2026.2.2 (patch release)
All checks were successful
Build Container / detect (push) Successful in 2s
Build Container / build-nix (authentik) (push) Successful in 1m6s
Bug-fix release with web UI fixes, LDAP page size, and SAML SLO
redirect. Also bumps client-go to v3.2026.2.1.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:53:03 -07:00
0366a0346b Set Frigate preview quality to CRF 8 for faster timeline loading
Previews are ~4MB/hour at default quality (CRF 1), served over NFS from
sifaka. Reducing to CRF 8 shrinks preview files to improve review page
load times when scrubbing older footage.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 08:43:43 -07:00
936d29bbe1 Fix UnPoller dashboard UIDs exceeding Grafana 12's 40-char limit
Strip redundant "unifi-poller-" prefix from generated slugs, bringing
UIDs from 45-48 chars down to 32-35 chars.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 07:03:39 -07:00
3c894e659d Pin kube-state-metrics to main-SHA container tags
C0 follow-up to #327: update from branch-SHA tags to main-SHA tags
after squash-merge rebuild.

indri: v2.18.0-f59f885
ringtail: v2.18.0-f59f885-nix

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 16:10:14 -07:00
f59f8859dc Localize kube-state-metrics container (Dockerfile + nix) (#327)
All checks were successful
Build Container / detect (push) Successful in 2s
Build Container / build-dockerfile (kube-state-metrics) (push) Successful in 5s
Build Container / build-nix (kube-state-metrics) (push) Successful in 7s
## Summary

- Build kube-state-metrics v2.18.0 locally from forge mirror, replacing upstream `registry.k8s.io` image
- Dockerfile (two-stage Go build) for indri/minikube
- default.nix (buildGoModule + buildLayeredImage) for ringtail/k3s
- Both kustomization files updated with `newName` pointing to local registry

## Verification

- [x] Nix build succeeded on ringtail (`nix-build` → 10-layer image)
- [x] Dockerfile build succeeded locally (`dagger call build` → ~2min)
- [x] `container-version-check --all-files` passes (2.18.0 consistent across Dockerfile, nix, service-versions.yaml)
- [ ] CI builds container images from this branch
- [ ] Update kustomization `newTag` with SHA-tagged version from CI
- [ ] ArgoCD sync on both clusters

## Test plan

- Trigger CI build: `mise run container-build-and-release kube-state-metrics`
- Verify tags: `mise run container-list kube-state-metrics`
- Update newTag in kustomization files with CI-produced tag
- Sync ArgoCD on indri: `argocd app sync kube-state-metrics`
- Sync ArgoCD on ringtail: `argocd app sync kube-state-metrics --context=k3s-ringtail` (note: argocd uses its own auth, not kubectl context)
- Verify metrics still flowing to Prometheus

Reviewed-on: #327
2026-04-07 16:09:25 -07:00
84eda0301f Bump authentik worker memory limit 1Gi → 2Gi (OOMKilled after ringtail restart)
Worker forks 4 Dramatiq processes each loading the full Django app
(~250MB each), hitting the 1Gi limit on startup. Ringtail has ample
RAM headroom.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 15:39:29 -07:00
efae404d1e Remove superuser from teslamate PG role, transfer extension ownership
teslamate had superuser on the shared blumeops-pg cluster (which also
hosts miniflux and authentik). Downgraded to plain database owner with
extension ownership (cube, earthdistance) transferred manually so it
can still ALTER EXTENSION UPDATE. earthdistance is untrusted in PG so
DROP+CREATE would need temporary superuser escalation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 15:36:39 -07:00
fc34a7da5b Review postgresql.md: add authentik user/db, immich-pg borgmatic secret
Doc review found the authentik database, user, and external secret were
missing, along with the immich-pg borgmatic secret. Added Cluster column
to Users table for clarity. Set last-reviewed: 2026-04-07.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 15:21:48 -07:00
1fd8aae8f6 Upgrade ArgoCD v3.3.2 → v3.3.6, SHA-pin install manifest
Patch upgrade with bug fixes (diff normalization, installation ID cache).
Pin the upstream manifest URL to commit SHA for supply chain integrity.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 08:21:11 -07:00
e85c71e73f Add changelog fragments for seccomp hardening and bracket fix
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 10:38:41 -07:00
d3d67272a7 Fix blumeops-tasks swallowing bracket content in descriptions
Rich markup parser interprets [text] as style tags, stripping
wiki-links like [[review-compensating-controls]] to empty [].
Escape description lines with rich.markup.escape().

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 10:37:40 -07:00
59f3422d3e Review compensating control: tailscale-network-isolation
Verified: tailscale serve status shows only svc:k8s, ACLs restrict
tag:flyio-target to port 443 with admin/operator ownership only,
indri has no flyio-target tag. All 10 muted findings remain valid.

Noted gap: no automated alerting on new flyio-target devices.
Tracked in Todoist as MC4 (Manual Compliance Control Check CronJob).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 10:35:13 -07:00
18fe172a54 Add seccomp RuntimeDefault profiles to alloy-k8s and immich pods
Resolves 4 unmuted Prowler core_seccomp_profile_docker_default
findings on alloy, immich-server, immich-machine-learning, and
immich-valkey.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 10:21:23 -07:00
a059d81314 Add review-compliance-reports task and reorganize report storage
New mise task fetches Prowler reports from sifaka, parses with proper
muted/unmuted distinction, shows week-over-week delta, and includes
a scaffold for Kingfisher once JSON/CSV output is available upstream.

Moved all legacy top-level reports on sifaka into date subdirectories
to match the current CronJob output structure. Updated
read-compliance-reports doc with task reference and links.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 10:16:46 -07:00
54213ab810 Fix flake-update pipeline and update ringtail flake inputs
The `--exclude` flag added in #321 never existed in nix — it was
introduced broken and never tested. Replace with dynamic input
discovery: query `nix flake metadata --json` for all input names,
filter out skip_inputs (default: nixpkgs-services), pass the rest
as positional args. Also bump NIX_IMAGE 2.33.3 → 2.34.4.

Updated inputs: nixpkgs, home-manager, disko.
nixpkgs-services stays pinned.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:27:31 -07:00
Forgejo Actions
370a3574b2 Update docs release to v1.15.4
- Built changelog from towncrier fragments

[skip ci]
2026-04-06 07:53:54 -07:00
0eaf8680fd Rewrite observability stack tutorial to match actual practices v1.15.4
Replace generic Helm install instructions with kustomize/ArgoCD patterns
that reflect how BlumeOps actually deploys Prometheus, Loki, Grafana, and
Alloy. Fix "BluemeOps" typos, document Alloy as a core (not optional)
component, remove hardcoded admin password, add proper prerequisites and
cross-references.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 07:52:35 -07:00
f42fa2d558 Remove stale Helm chart mirror references from forgejo docs
All Helm chart mirrors (grafana-helm-charts, connect-helm-charts,
cloudnative-pg-charts) have been deleted from forge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 07:37:21 -07:00
c7e5af6d51 Migrate 1Password Connect from Helm to kustomize (1.8.1 → 1.8.2) (#326)
## Summary

- Renders manifests from `connect-helm-charts v2.4.1` as plain kustomize (deployment + service)
- Bumps 1Password Connect from 1.8.1 → 1.8.2
- Completes the no-helm-policy migration — all services now use kustomize
- Retains all production hardening from the Helm chart (securityContext, runAsNonRoot, drop ALL, seccomp, resource limits)

## Changes

- **New:** `deployment.yaml`, `service.yaml`, `kustomization.yaml` in `argocd/manifests/1password-connect/`
- **Rewritten:** Both ArgoCD app definitions (indri + ringtail) — single source kustomize instead of multi-source Helm
- **Deleted:** `values.yaml` (Helm values no longer needed)
- **Updated:** `no-helm-policy.md`, `service-versions.yaml`, `README.md`

## Deployment plan

1. Sync `apps` app to pick up the new app definitions
2. `argocd app set 1password-connect --revision 1password-connect-kustomize`
3. `argocd app sync 1password-connect` — verify on indri
4. Repeat for ringtail
5. After merge: reset revision to main, re-sync both

## Test plan

- [ ] `kubectl kustomize` renders cleanly (verified locally)
- [ ] ArgoCD diff shows expected changes (Helm labels removed, images bumped)
- [ ] Pods come up healthy on indri
- [ ] External Secrets still resolves 1Password items
- [ ] Repeat on ringtail

Reviewed-on: #326
2026-04-06 07:31:40 -07:00
Forgejo Actions
facb803010 Update docs release to v1.15.3
- Built changelog from towncrier fragments

[skip ci]
2026-04-05 21:24:25 -07:00
f9397b7fa0 Review core-services tutorial: add SSH rationale, runner example, TODO notice v1.15.3
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 21:22:00 -07:00
5597e02467 Fix Homepage pod-selector for Immich (Helm labels → kustomize labels)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 12:12:48 -07:00
0f1143a5bd Merge pull request 'Migrate Immich from Helm to kustomize (v2.5.6 → v2.6.3)' (#324) from immich-kustomize-v2.6.3 into main 2026-04-04 12:09:40 -07:00
6cab5091ea Add storage-provisioner health check to minikube Ansible role
The storage-provisioner is a bare Pod with no controller. If the node
restarts via Docker Desktop (rather than `minikube start`), kubelet
restores static pods but bare pods are lost. Detect this and re-run
`minikube start` to restore addons.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 12:04:25 -07:00
3c819cf16e Fix Homepage migration date: 2025-12 → 2026-02 (per git history)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 09:53:48 -07:00
6e06efa6d0 Add AI-drafted disclaimer to no-helm-policy explanation doc
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 09:50:53 -07:00
64200a55c5 Migrate Immich from Helm chart to kustomize manifests (v2.5.6 → v2.6.3)
Replace the Helm chart deployment with plain kustomize manifests following
the Authentik pattern (separate deployments per component). Consolidate
the immich-storage ArgoCD app into the main immich app. Add no-helm-policy
doc establishing kustomize as the standard deployment mechanism.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 09:42:25 -07:00
464e3222d2 Document upstream fix for Prowler --registry bug (pending release)
PR #10470 merged 2026-03-30; initContainer workaround stays until a
Prowler release includes the fix (latest is 5.22.0).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 20:21:19 -07:00
afb184fefc Add Sway fullscreen rule for RDR2 (gamescope broken on NVIDIA 580.x)
Gamescope 3.16.17 segfaults on NVIDIA 580.x in nested Wayland/Sway due
to explicit sync issues (ValveSoftware/gamescope#1662). Use a Sway
window rule to force RDR2 fullscreen instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 16:04:29 -07:00
5de2ed9f96 Add gaming.nix for ringtail: gamescope + consolidate Steam config
Move Steam config from configuration.nix to a dedicated gaming.nix module
and add gamescope for fullscreen/resolution management with Proton games.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 15:48:36 -07:00
306f580bdb Point Tempo at main-built container v2.10.3-75f9ba4
C0 follow-up: update tag from branch-built image to main-SHA image.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 13:45:57 -07:00
75f9ba4943 Build Tempo container from source (2.10.3) (#323)
All checks were successful
Build Container / detect (push) Successful in 2s
Build Container / build-dockerfile (tempo) (push) Successful in 6s
## Summary
- Add `containers/tempo/Dockerfile` — two-stage Go build from forge mirror, modeled on loki
- Switch kustomization from upstream `grafana/tempo` to `registry.ops.eblu.me/blumeops/tempo`
- Bump Tempo 2.10.1 → 2.10.3

## Test plan
- [ ] Kick off container build via `mise run container-build-and-release tempo`
- [ ] Update kustomization `newTag` with built image tag
- [ ] Deploy from branch: `argocd app set tempo --revision local-tempo-container && argocd app sync tempo`
- [ ] Verify Tempo health: `curl tempo.ops.eblu.me/ready`
- [ ] Verify traces flowing in Grafana Tempo datasource

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #323
2026-04-02 13:45:02 -07:00
b1e2811077 Upgrade Grafana 12.3.3 → 12.4.2 (#322)
All checks were successful
Build Container / detect (push) Successful in 2s
Build Container / build-dockerfile (grafana) (push) Successful in 7s
## Summary

- Bumps Grafana from 12.3.3 to 12.4.2
- Patches 7 CVEs, notably CVE-2026-27880 (unauthenticated OOM DoS, CVSS 7.5) and CVE-2026-27879 (authenticated OOM via resample queries)
- No config changes required — reviewed alerting, datasources, OIDC, and feature toggles against 12.4.x breaking changes

## Breaking changes reviewed

| Change | Impact |
|--------|--------|
| Alerting: pending period applies to NoData/Error | Net positive — reduces noise from transient blips |
| Default notification uses empty receiver | No impact — we explicitly set `ntfy-infra` |
| Removed feature toggles (4) | No impact — none configured |
| OAuth ID token signature validation | Low risk — verify OIDC login post-deploy |
| OpsGenie deprecated | No impact — using webhook |

## Test plan

- [ ] Container build completes at forge
- [ ] Update kustomization.yaml with new image tag
- [ ] `argocd app set grafana --revision upgrade/grafana-12.4.2 && argocd app sync grafana`
- [ ] Verify Grafana UI loads at grafana.ops.eblu.me
- [ ] Verify OIDC login via Authentik
- [ ] Verify dashboards and datasources load
- [ ] Check alerting rules are intact

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #322
2026-04-02 11:33:19 -07:00
08d57ef4d4 Review pulumi reference doc: fix Tailscale auth, add how-to links
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 10:55:57 -07:00
1e67975acb Add changelog fragment for compensating control review
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 22:04:38 -07:00
baee7ae54b Review single-user-cluster control and add evidence collection card
Stamp single-user-cluster last-reviewed to 2026-04-01 after verifying
Tailscale ACLs and kubeconfig distribution. Add aspirational how-to card
documenting what PCI DSS evidence collection would look like (CCW,
artifacts, Drata workflow). Link from existing review process card.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 22:01:57 -07:00
a18a424866 Pin NixOS service versions via nixpkgs-services overlay (#321)
## Summary

- Add `nixpkgs-services` flake input pinned to a specific nixpkgs commit, with an overlay that pulls `forgejo-runner`, `snowflake`, and `k3s` from it instead of the rolling `nixpkgs`
- Dagger `flake-update` pipeline now excludes `nixpkgs-services` via `--exclude`
- Fix stale nix-container-builder version in service-versions.yaml (was 12.6.4, actually running 12.7.2)
- Add k3s and minikube to service-versions.yaml tracking
- Document the pinning approach in review-services how-to and ringtail reference

## Motivation

During service review, discovered that flake updates had silently upgraded forgejo-runner from 12.6.4 → 12.7.2 without updating service-versions.yaml. This "sneak-in upgrade" bypasses the service review process. The overlay ensures these three services only change versions deliberately.

## Test plan

- [ ] Verify `nix flake update` from `nixos/ringtail/` does not change `nixpkgs-services` lock entry
- [ ] Verify `mise run provision-ringtail` builds successfully with the overlay
- [ ] Confirm running service versions unchanged after deploy

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #321
2026-04-01 21:37:57 -07:00
cfbf4cadbd Review argocd-cli reference doc: stamp last-reviewed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 20:35:52 -07:00
Forgejo Actions
2b7b21dc9b Update docs release to v1.15.2
- Built changelog from towncrier fragments

[skip ci]
2026-03-30 17:48:40 -07:00