eblume/blumeops
1
1
Fork 0
Code Issues Pull requests Projects Releases 83 Packages Wiki Activity Actions

Retire Prowler image + IaC scans (keep K8s CIS only) #372

Merged
eblume merged 2 commits from retire-prowler-image-iac-scans into main 2026-06-08 09:30:11 -07:00
Conversation 0 Commits 2 Files changed 10 +67 -298
10 changed files with 67 additions and 298 deletions
Download patch file Download diff file Expand all files Collapse all files

54
argocd/manifests/prowler/cronjob-iac-scan.yaml
View file

@ -1,54 +0,0 @@
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: prowler-iac-scan
namespace: prowler
spec:
schedule: "0 2 * * 6" # Saturday 2am
concurrencyPolicy: Forbid
jobTemplate:
spec:
ttlSecondsAfterFinished: 604800 # Auto-delete after 7 days
template:
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: prowler
image: registry.ops.eblu.me/blumeops/prowler:kustomized
command: ["/bin/sh", "-c"]
# Prowler's --mutelist-file is a no-op for the IaC provider
# (it delegates to Trivy). The Prowler image's trivy shim
# injects --ignorefile $TRIVY_IGNOREFILE when set; see
# containers/prowler/Dockerfile.
env:
- name: TRIVY_IGNOREFILE
value: /mutelist/trivyignore.yaml
args:
- |
DATEDIR=/reports/prowler-iac/$(date +%Y-%m-%d)
mkdir -p "$DATEDIR"
prowler iac \
--scan-repository-url https://forge.ops.eblu.me/eblume/blumeops.git \
-z \
--output-formats html csv json-ocsf \
--output-directory "$DATEDIR"
volumeMounts:
- name: reports
mountPath: /reports
- name: mutelist
mountPath: /mutelist
readOnly: true
restartPolicy: OnFailure
volumes:
- name: reports
persistentVolumeClaim:
claimName: prowler-reports
- name: mutelist
configMap:
name: prowler-mutelist
items:
- key: trivyignore.yaml
path: trivyignore.yaml

39
argocd/manifests/prowler/cronjob-image-scan.yaml
View file

@ -1,39 +0,0 @@
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: prowler-image-scan
namespace: prowler
spec:
schedule: "0 3 * * 6" # Saturday 3am
concurrencyPolicy: Forbid
jobTemplate:
spec:
ttlSecondsAfterFinished: 604800 # Auto-delete after 7 days
template:
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: prowler
image: registry.ops.eblu.me/blumeops/prowler:kustomized
command: ["/bin/sh", "-c"]
args:
- |
DATEDIR=/reports/prowler-images/$(date +%Y-%m-%d)
mkdir -p "$DATEDIR"
prowler image \
--registry https://registry.ops.eblu.me \
--image-filter "^blumeops/" \
-z \
--output-formats html csv json-ocsf \
--output-directory "$DATEDIR"
volumeMounts:
- name: reports
mountPath: /reports
restartPolicy: OnFailure
volumes:
- name: reports
persistentVolumeClaim:
claimName: prowler-reports

3
argocd/manifests/prowler/kustomization.yaml
View file

@ -10,8 +10,6 @@ resources:
- pv-nfs.yaml
- pvc.yaml
- cronjob.yaml
- cronjob-image-scan.yaml
- cronjob-iac-scan.yaml
configMapGenerator:
- name: prowler-mutelist
@ -23,7 +21,6 @@ configMapGenerator:
- mutelist/core-pod-security.yaml
- mutelist/manual-node-checks.yaml
- mutelist/rbac.yaml
- mutelist/trivyignore.yaml
images:
- name: registry.ops.eblu.me/blumeops/prowler

37
argocd/manifests/prowler/mutelist/trivyignore.yaml
View file

@ -1,37 +0,0 @@
# Trivy ignorefile for Prowler IaC scan.
#
# Prowler's `--mutelist-file` flag is a no-op for the IaC provider
# (iac_provider.py sets self._mutelist = None and delegates to Trivy).
# Trivy in turn does not auto-discover this YAML form from cwd, so the
# Prowler image ships a shim wrapper around `trivy` that injects
# --ignorefile $TRIVY_IGNOREFILE when the env var is set. The cronjob
# mounts this file and sets TRIVY_IGNOREFILE accordingly.
#
# Schema: https://trivy.dev/latest/docs/configuration/filtering/
# IDs use the hyphenated form Trivy displays (KSV-0041, not KSV0041).
misconfigurations:
- id: KSV-0041
paths:
- "argocd/manifests/external-secrets/rbac.yaml"
statement: >-
external-secrets-operator's entire function is to read and
synthesize Secret objects; ClusterRole over secrets is its
purpose. Both the controller and cert-controller are
upstream-defined.
- id: KSV-0041
paths:
- "argocd/manifests/kube-state-metrics/rbac.yaml"
- "argocd/manifests/kube-state-metrics-ringtail/rbac.yaml"
statement: >-
KSM exposes only Secret metadata (name, namespace, type, labels),
never the data field. list/watch on secrets is required for
kube_secret_info / kube_secret_labels metrics.
- id: KSV-0114
paths:
- "argocd/manifests/external-secrets/rbac.yaml"
statement: >-
cert-controller manages the external-secrets validating webhook
configurations to inject its own rotating CA bundle. RBAC is
scoped to two named webhooks (secretstore-validate,
externalsecret-validate) via resourceNames; KSV-0114 doesn't see
the resourceNames restriction so reports the full ClusterRole.

1
docs/changelog.d/retire-prowler-image-iac-scans.infra.md Normal file
View file

@ -0,0 +1 @@
Retired the Prowler container-image CVE scan and IaC scan, keeping only the K8s CIS benchmark scan. The two retired scans generated tens of thousands of un-actioned, un-muted findings every week (~20,000 image findings and growing, mostly unpatchable upstream-image CVEs; ~650 systemic Trivy KSV pod-security warnings) — the weekly `mise run review-compliance-reports` re-surfaced them all as "action needed" though none were ever triaged. The K8s CIS scan is fully mutelisted and runs clean, so it stays. Removed the two CronJobs, the now-unused `trivyignore.yaml` mutelist, and the grouped-findings rendering in the review tool that existed solely for the high-volume scans.

49
docs/how-to/operations/deploy-prowler.md
View file

@ -1,6 +1,6 @@
---
title: Deploy Prowler CIS Scanner
modified: 2026-03-24
modified: 2026-06-08
last-reviewed: 2026-03-24
tags:
- how-to
@ -11,7 +11,20 @@ tags:
# Deploy Prowler CIS Scanner
Prowler runs weekly CIS Kubernetes Benchmark scans against minikube-indri and writes HTML/CSV/JSON reports to the NFS share on sifaka.
Prowler runs a weekly CIS Kubernetes Benchmark scan against minikube-indri and writes HTML/CSV/JSON reports to the NFS share on sifaka.
## Why only the K8s CIS scan
Prowler originally ran three CronJobs: K8s CIS, container-image CVE scanning, and IaC scanning. The image and IaC scans were **retired in 2026-06**.
Both were pure toil with no realized value:
- **Image scan** produced ~20,000 unmuted findings per run and growing, none ever triaged or muted. They were overwhelmingly CVEs in *upstream* base images we don't control and can't patch, and the job re-scanned every historical tag still in the registry, multiplying the count.
- **IaC scan** produced ~650 Trivy KSV findings (`runAsNonRoot`, `readOnlyRootFilesystem`, drop-capabilities, …) against our own manifests — real but systemic, homelab-acceptable, and likewise never muted, so the weekly review re-surfaced all of them indefinitely.
The K8s CIS scan, by contrast, is fully mutelisted and runs clean (0 unmuted findings week over week), so it stays. The guiding principle matches [[ai-scraper-mitigation]]: don't keep generating a firehose of output that has no audience. If image-CVE signal is wanted later, the right shape is critical-severity-only, currently-deployed-tags-only, alert-on-new — a rebuild, not a revival (tracked as the "Trivy for image/IaC scanning" task).
Note that the K8s CIS scan itself is tied to minikube-indri, which is slated for retirement; on k3s only ~22 of 70 checks produce results (no static pods). Re-pointing a lean posture check at ringtail is tracked separately ("prowler scan against ringtail").
## What it checks
@ -33,38 +46,6 @@ Prowler's Kubernetes provider runs ~70 checks from the CIS Kubernetes Benchmark
**k3s note:** k3s embeds the control plane in a single binary — no static pods exist. Only core + RBAC checks (~22 of 70) produce results. Consider `kube-bench` for k3s control plane checks.
### Image vulnerability scanning (Saturday 3am)
Prowler's image provider scans all `blumeops/*` container images in `registry.ops.eblu.me` for:
- **CVEs** — known vulnerabilities from NVD, Alpine SecDB, Debian Security Tracker, and other sources
- **Embedded secrets** — credentials or API keys baked into image layers
- **Misconfigurations** — Dockerfile best practices (running as root, missing HEALTHCHECK, etc.)
Uses Trivy under the hood. Reports are written to `sifaka:/volume1/reports/prowler-images/`.
To run an ad-hoc image scan:
```fish
kubectl create job --from=cronjob/prowler-image-scan prowler-image-manual -n prowler --context=minikube-indri
```
### IaC scanning (Saturday 2am)
Prowler's IaC provider scans the blumeops repository (cloned at scan time) for misconfigurations in:
- **Dockerfiles** — running as root, using `latest` tags, missing `HEALTHCHECK`
- **Kubernetes manifests** — missing resource limits, privileged containers, insecure settings
- **Other IaC files** — Terraform, CloudFormation, etc. if present
Uses Trivy under the hood. Reports are written to `sifaka:/volume1/reports/prowler-iac/`.
To run an ad-hoc IaC scan:
```fish
kubectl create job --from=cronjob/prowler-iac-scan prowler-iac-manual -n prowler --context=minikube-indri
```
## Reports
Reports are written to `sifaka:/volume1/reports/prowler/` with timestamped filenames. See [[read-compliance-reports]] for how to access and interpret them.

11
docs/how-to/operations/read-compliance-reports.md
View file

@ -1,6 +1,6 @@
---
title: Read Compliance Reports
modified: 2026-04-06
modified: 2026-06-08
last-reviewed: 2026-04-06
tags:
- how-to
@ -27,8 +27,13 @@ Reports are stored on sifaka at `/volume1/reports/`. Each scanner writes to its
| Scanner | Path | Schedule |
|---------|------|----------|
| [[prowler]] K8s CIS | `sifaka:/volume1/reports/prowler/` | Weekly (Sunday 3am) |
| [[prowler]] Image | `sifaka:/volume1/reports/prowler-images/` | Weekly (Saturday 3am) |
| [[prowler]] IaC | `sifaka:/volume1/reports/prowler-iac/` | Weekly (Saturday 2am) |
> **Retired (2026-06):** the Prowler **image** (`prowler-images/`) and **IaC**
> (`prowler-iac/`) scans were retired. They produced tens of thousands of
> un-actioned, un-muted findings every week — mostly unpatchable upstream-image
> CVEs and systemic pod-security KSV warnings — and nobody triaged them. See
> [[deploy-prowler#Why only the K8s CIS scan]] for the rationale. Their stale
> report directories may linger on sifaka until manually removed.
Copy reports to your local machine (remember `scp -O` for sifaka):

8
docs/reference/operations/security.md
View file

@ -1,6 +1,6 @@
---
title: Security & Compliance
modified: 2026-03-24
modified: 2026-06-08
last-reviewed: 2026-03-24
tags:
- operations
@ -21,7 +21,7 @@ Security posture and compliance scanning for BlumeOps infrastructure.
## Scanning tools
- [[prowler]] — CIS Kubernetes Benchmark scanner (weekly CronJob)
- [[prowler]] — CIS Kubernetes Benchmark scanner (weekly CronJob). The container-image CVE scan and IaC scan were retired in 2026-06 (un-actioned noise — see [[deploy-prowler#Why only the K8s CIS scan]]); only the K8s CIS scan remains.
- [[deploy-prowler]] — deployment and ad-hoc scan how-to
- [[read-compliance-reports]] — accessing and interpreting reports
- [[kingfisher]] — Secret detection and live validation for Forgejo repos (weekly CronJob + prek hook)
@ -52,5 +52,5 @@ Suppressed findings are kept in Prowler mutelist YAML under `argocd/manifests/pr
- No SOC 2 compliance mapping for Kubernetes (Prowler only maps SOC 2 for AWS/Azure/GCP)
- k3s control plane checks produce no results (embedded binary, no static pods) — consider kube-bench
- Container image scanning covers `blumeops/*` images only — upstream images (ollama, immich, etc.) are not scanned
- IaC scanning covers the blumeops repo only — no scanning of third-party Helm charts or vendored manifests
- No container-image CVE scanning (the Prowler image scan was retired 2026-06 as un-actioned noise). If reintroduced, scope it to critical-severity, currently-deployed tags, alert-on-new
- No automated IaC misconfiguration scanning (the Prowler IaC scan was retired 2026-06). Manifest pod-security hardening is now an accept-and-document decision rather than a weekly report

12
docs/reference/services/prowler.md
View file

@ -1,6 +1,6 @@
---
title: Prowler
modified: 2026-03-24
modified: 2026-06-08
last-reviewed: 2026-03-24
tags:
- service
@ -17,20 +17,20 @@ CIS Kubernetes Benchmark scanner for compliance posture reporting.
|----------|-------|
| **Namespace** | `prowler` |
| **Image** | `registry.ops.eblu.me/blumeops/prowler` (see `argocd/manifests/prowler/kustomization.yaml` for current tag) |
| **Schedule** | K8s CIS: Sunday 3am / Image: Saturday 3am / IaC: Saturday 2am |
| **Reports** | `sifaka:/volume1/reports/prowler/`, `prowler-images/`, `prowler-iac/` (NFS) |
| **Schedule** | K8s CIS: Sunday 3am |
| **Reports** | `sifaka:/volume1/reports/prowler/` (NFS) |
| **Manifests** | `argocd/manifests/prowler/` |
## What it does
Runs Prowler 5 as two CronJobs:
Runs Prowler 5 as a single CronJob:
- **K8s CIS scan** (Sunday) — CIS Kubernetes Benchmark v1.11 checks across pod security, RBAC, apiserver, etcd, kubelet, controller-manager, and scheduler
- **Image scan** (Saturday) — CVE, secret, and misconfiguration scanning of all `blumeops/*` container images in the registry via Trivy
- **IaC scan** (Saturday) — static analysis of Dockerfiles, K8s manifests, and other IaC files in the repo via Trivy
Reports are written in HTML, CSV, and JSON-OCSF to the NFS share on sifaka.
The **image** and **IaC** scans (formerly Saturday CronJobs) were retired in 2026-06 — they generated tens of thousands of un-actioned findings weekly. See [[deploy-prowler#Why only the K8s CIS scan]].
## See also
- [[security]] — security & compliance posture overview

147
mise-tasks/review-compliance-reports
View file

@ -10,19 +10,19 @@
Covers:
- Prowler K8s CIS (in-cluster): per-finding detail
- Prowler container image scans: grouped by check + resource
- Prowler IaC manifest scans: grouped by check + resource
- Kingfisher secret scanning: TODO — pending upstream JSON/CSV output
support (currently HTML-only; contribute from spork)
For each Prowler scan, copies the two most recent CSV reports, parses
The Prowler container-image CVE scan and IaC scan were retired in 2026-06
(see docs/how-to/operations/deploy-prowler.md) — they produced tens of
thousands of un-actioned findings weekly. Only the K8s CIS scan remains.
For the Prowler scan, copies the two most recent CSV reports, parses
them, and displays:
1. Overall status (pass/fail/manual/muted counts)
2. Unmuted failures by severity
3. Delta from the previous report (new vs resolved)
4. Actionable unmuted failures (per-finding for in-cluster; grouped
by check ID and resource for image/IaC because they have far too
many findings to list individually)
4. Actionable unmuted failures (per-finding detail)
This is the primary tool for the weekly compliance report review.
"""
@ -39,11 +39,9 @@ from rich.console import Console
from rich.panel import Panel
from rich.table import Table
PROWLER_SCANS: list[tuple[str, str, bool]] = [
# (label, sifaka base path, group_findings)
("K8s CIS (In-Cluster)", "/volume1/reports/prowler", False),
("Container Images", "/volume1/reports/prowler-images", True),
("IaC (manifests)", "/volume1/reports/prowler-iac", True),
PROWLER_SCANS: list[tuple[str, str]] = [
# (label, sifaka base path)
("K8s CIS (In-Cluster)", "/volume1/reports/prowler"),
]
console = Console()
@ -334,14 +332,8 @@ def summarize_report(
tmpdir: str,
*,
show_muted: bool = False,
group_findings: bool = False,
) -> None:
"""Fetch and summarize the latest Prowler report under `base`.
When `group_findings` is True, top-N CHECK_ID and RESOURCE_NAME tables
are shown instead of a per-finding detail table — appropriate for
image and IaC scans that produce thousands of findings.
"""
"""Fetch and summarize the latest Prowler report under `base`."""
console.rule(f"[bold]{label}[/bold]")
csvs = list_reports(base)
if not csvs:
@ -458,36 +450,29 @@ def summarize_report(
)
console.print()
# For grouped scans the new/resolved listings are too noisy
# (potentially thousands of lines). Skip the listings; the count
# is in the panel above and detail is in the grouped tables.
if not group_findings:
if new_keys:
console.print("[bold red]New Unmuted Failures:[/bold red]")
for k in sorted(new_keys):
r = curr_keys[k]
console.print(
f" [{r['SEVERITY']}] {r['CHECK_ID']}: "
f"{r['STATUS_EXTENDED'][:120]}"
)
console.print()
if new_keys:
console.print("[bold red]New Unmuted Failures:[/bold red]")
for k in sorted(new_keys):
r = curr_keys[k]
console.print(
f" [{r['SEVERITY']}] {r['CHECK_ID']}: "
f"{r['STATUS_EXTENDED'][:120]}"
)
console.print()
if resolved_keys:
console.print("[bold green]Resolved:[/bold green]")
for k in sorted(resolved_keys):
r = prev_keys[k]
console.print(
f" [dim][{r['SEVERITY']}] {r['CHECK_ID']}: "
f"{r['STATUS_EXTENDED'][:120]}[/dim]"
)
console.print()
if resolved_keys:
console.print("[bold green]Resolved:[/bold green]")
for k in sorted(resolved_keys):
r = prev_keys[k]
console.print(
f" [dim][{r['SEVERITY']}] {r['CHECK_ID']}: "
f"{r['STATUS_EXTENDED'][:120]}[/dim]"
)
console.print()
# --- Unmuted failure details (grouped or per-finding) ---
# --- Unmuted failure details ---
if latest["unmuted"]:
if group_findings:
_print_grouped_findings(latest["unmuted"])
else:
_print_findings_detail(latest["unmuted"])
_print_findings_detail(latest["unmuted"])
# --- Muted findings summary ---
if show_muted and latest["muted"]:
@ -566,75 +551,6 @@ def _print_findings_detail(unmuted: list[dict]) -> None:
console.print()
def _worst_severity(rows: list[dict]) -> str:
"""Return the most severe severity label across `rows`."""
if not rows:
return ""
return min(
(r["SEVERITY"] for r in rows),
key=lambda s: severity_sort({"SEVERITY": s}),
)
def _print_grouped_findings(unmuted: list[dict], top_n: int = 15) -> None:
"""Top-N tables grouped by CHECK_ID and RESOURCE_NAME.
Used for image and IaC scans where per-finding tables would be too
large to be useful. Shows count and worst severity for each group.
"""
by_check: dict[str, list[dict]] = {}
by_resource: dict[str, list[dict]] = {}
for r in unmuted:
by_check.setdefault(r["CHECK_ID"], []).append(r)
by_resource.setdefault(r.get("RESOURCE_NAME", "") or "(no resource)", []).append(r)
check_table = Table(
show_header=True,
header_style="bold",
title=f"Top {top_n} Checks by Unmuted Finding Count",
)
check_table.add_column("Worst Sev")
check_table.add_column("Check ID")
check_table.add_column("Count", justify="right")
for check, rows in sorted(
by_check.items(), key=lambda kv: -len(kv[1])
)[:top_n]:
worst = _worst_severity(rows)
style = _sev_style(worst)
check_table.add_row(
f"[{style}]{worst}[/{style}]" if style else worst,
check,
str(len(rows)),
)
console.print(check_table)
console.print()
res_table = Table(
show_header=True,
header_style="bold",
title=f"Top {top_n} Resources by Unmuted Finding Count",
)
res_table.add_column("Worst Sev")
res_table.add_column("Resource")
res_table.add_column("Count", justify="right")
for resource, rows in sorted(
by_resource.items(), key=lambda kv: -len(kv[1])
)[:top_n]:
worst = _worst_severity(rows)
style = _sev_style(worst)
res_table.add_row(
f"[{style}]{worst}[/{style}]" if style else worst,
resource[:80],
str(len(rows)),
)
console.print(res_table)
console.print()
def main(
full: Annotated[
bool, typer.Option(help="(reserved) currently a no-op; all unmuted failures already shown")
@ -646,13 +562,12 @@ def main(
del full # historical flag, kept for backwards compatibility
with tempfile.TemporaryDirectory() as tmpdir:
for label, base, group in PROWLER_SCANS:
for label, base in PROWLER_SCANS:
summarize_report(
label,
base,
tmpdir,
show_muted=show_muted,
group_findings=group,
)
# --- Node-level MANUAL check verification ---