forked from mirrors/kingfisher
- Fixed local filesystem scans to keep open_path_as_is enabled when opening Git repositories and only disable it for diff-based scans.
- Created Linux and Windows specific installer script - Updated diff-focused scanning so --branch-root-commit can be provided alongside --branch, letting you diff from a chosen commit while targeting a specific branch tip (still defaulting back to the --branch ref when the commit is omitted).
This commit is contained in:
parent
43d5a06b15
commit
7d9d3be132
23 changed files with 1608 additions and 21 deletions
|
|
@ -2,6 +2,11 @@
|
|||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
## [v1.61.0]
|
||||
- Fixed local filesystem scans to keep `open_path_as_is` enabled when opening Git repositories and only disable it for diff-based scans.
|
||||
- Created Linux and Windows specific installer script
|
||||
- Updated diff-focused scanning so `--branch-root-commit` can be provided alongside `--branch`, letting you diff from a chosen commit while targeting a specific branch tip (still defaulting back to the `--branch` ref when the commit is omitted).
|
||||
|
||||
## [v1.60.0]
|
||||
- Removed the `--bitbucket-username`, `--bitbucket-token`, and `--bitbucket-oauth-token` flags in favour of `KF_BITBUCKET_*` environment variables when authenticating to Bitbucket.
|
||||
- Added provider-specific `kingfisher scan` subcommands (for example `kingfisher scan github …`) that translate into the legacy flags under the hood. The new layout keeps backwards compatibility while removing the wall of provider options from `kingfisher scan --help`.
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ publish = false
|
|||
|
||||
[package]
|
||||
name = "kingfisher"
|
||||
version = "1.60.0"
|
||||
version = "1.61.0"
|
||||
description = "MongoDB's blazingly fast and accurate secret scanning and validation tool"
|
||||
edition.workspace = true
|
||||
rust-version.workspace = true
|
||||
|
|
|
|||
47
README.md
47
README.md
|
|
@ -166,17 +166,23 @@ brew install kingfisher
|
|||
|
||||
<details>
|
||||
|
||||
You can easily install using [ubi](https://github.com/houseabsolute/ubi), which downloads the correct binary for your platform.
|
||||
Use the bundled installer script to fetch the latest release and place it in
|
||||
`~/.local/bin` (or a directory of your choice):
|
||||
|
||||
```bash
|
||||
# Linux, macOS
|
||||
curl --silent --location \
|
||||
https://raw.githubusercontent.com/houseabsolute/ubi/master/bootstrap/bootstrap-ubi.sh | \
|
||||
sh && \
|
||||
ubi --project mongodb/kingfisher --in "$HOME/.local/bin"
|
||||
https://raw.githubusercontent.com/mongodb/kingfisher/main/scripts/install-kingfisher.sh | \
|
||||
bash
|
||||
```
|
||||
|
||||
This installs and runs `ubi` and then places the `kingfisher` executable in `~/.local/bin` on Unix-like systems.
|
||||
To install into a custom location, pass the desired directory as an argument:
|
||||
|
||||
```bash
|
||||
curl --silent --location \
|
||||
https://raw.githubusercontent.com/mongodb/kingfisher/main/scripts/install-kingfisher.sh | \
|
||||
bash -s -- /opt/kingfisher
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
|
|
@ -184,14 +190,21 @@ This installs and runs `ubi` and then places the `kingfisher` executable in `~/.
|
|||
|
||||
<details>
|
||||
|
||||
You can easily install using [ubi](https://github.com/houseabsolute/ubi), which downloads the correct binary for your platform.
|
||||
Download and run the PowerShell installer to place the binary in
|
||||
`$env:USERPROFILE\bin` (or another directory you specify):
|
||||
|
||||
```powershell
|
||||
# Windows
|
||||
powershell -exec bypass -c "Invoke-WebRequest -URI 'https://raw.githubusercontent.com/houseabsolute/ubi/master/bootstrap/bootstrap-ubi.ps1' -UseBasicParsing | Invoke-Expression" && ubi --project mongodb/kingfisher --in .
|
||||
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force
|
||||
Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/mongodb/kingfisher/main/scripts/install-kingfisher.ps1' -OutFile install-kingfisher.ps1
|
||||
./install-kingfisher.ps1
|
||||
```
|
||||
|
||||
This installs and runs `ubi` and then places the `kingfisher` executable in the current directory on Windows.
|
||||
You can provide a custom destination using the `-InstallDir` parameter:
|
||||
|
||||
```powershell
|
||||
./install-kingfisher.ps1 -InstallDir 'C:\Tools\Kingfisher'
|
||||
```
|
||||
</details>
|
||||
|
||||
|
||||
|
|
@ -415,6 +428,11 @@ kingfisher scan ./my-project \
|
|||
|
||||
Limit scanning to the delta between your default branch and a pull request branch by combining `--since-commit` with `--branch` (defaults to `HEAD`). This only scans files that differ between the two references, which keeps CI runs fast while still blocking new secrets.
|
||||
|
||||
Use `--branch-root-commit` alongside `--branch` when you need to include a specific commit (and everything after it) in a diff-focused scan without re-examining earlier history. Provide the branch tip (or other comparison ref) via `--branch`, and pass the commit or merge-base you want to include with `--branch-root-commit`. If you omit `--branch-root-commit`, you can still enable `--branch-root` to fall back to treating the `--branch` ref itself as the inclusive root for backwards compatibility. This is especially useful in long-lived branches where you want to resume scanning from a previous review point or from the commit where a hotfix forked.
|
||||
|
||||
> **How is this different from `--since-commit`?**
|
||||
> `--since-commit` computes a diff between the branch tip and another ref, so it only inspects files that changed between those two points in history. `--branch-root-commit` rewinds to the parent of the commit you provide and then scans everything introduced from that commit forward, even if the files are unchanged relative to another baseline. Reach for `--since-commit` to keep CI scans fast by checking only the latest delta, and use `--branch-root-commit` when you want to re-audit the full contents of a branch starting at a specific commit.
|
||||
|
||||
```bash
|
||||
kingfisher scan . \
|
||||
--since-commit origin/main \
|
||||
|
|
@ -436,6 +454,19 @@ kingfisher scan /tmp/SecretsTest --branch feature-1 \
|
|||
# scan only a specific commit
|
||||
kingfisher scan /tmp/dev/SecretsTest \
|
||||
--branch baba6ccb453963d3f6136d1ace843e48d7007c3f
|
||||
#
|
||||
# scan feature-1 starting at a specific commit (inclusive)
|
||||
kingfisher scan /tmp/SecretsTest --branch feature-1 \
|
||||
--branch-root-commit baba6ccb453963d3f6136d1ace843e48d7007c3f
|
||||
#
|
||||
# scan feature-1 starting from the commit where the branch diverged from main
|
||||
kingfisher scan /tmp/SecretsTest --branch feature-1 \
|
||||
--branch-root-commit $(git -C /tmp/SecretsTest merge-base main feature-1)
|
||||
#
|
||||
# scan from a hotfix commit that should be re-checked before merging
|
||||
HOTFIX_COMMIT=$(git -C /tmp/SecretsTest rev-parse hotfix~1)
|
||||
kingfisher scan /tmp/SecretsTest --branch hotfix \
|
||||
--branch-root-commit "$HOTFIX_COMMIT"
|
||||
```
|
||||
|
||||
When the branch under test is already checked out, `--branch HEAD` or omitting `--branch` entirely is sufficient. Kingfisher exits with `200` when any findings are discovered and `205` when validated secrets are present, allowing CI jobs to fail automatically if new credentials slip in.
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ rules:
|
|||
(?:.|[\n\r]){0,32}?
|
||||
\b
|
||||
(
|
||||
[a-zA-Z0-9]{24}
|
||||
[A-Z0-9]{24}
|
||||
)
|
||||
\b
|
||||
confidence: medium
|
||||
|
|
|
|||
80
scripts/install-kingfisher.ps1
Normal file
80
scripts/install-kingfisher.ps1
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
<#
|
||||
.SYNOPSIS
|
||||
Download and install the latest Kingfisher release for Windows.
|
||||
|
||||
.DESCRIPTION
|
||||
Fetches the most recent GitHub release for mongodb/kingfisher, downloads the
|
||||
Windows x64 archive, and extracts kingfisher.exe to the destination folder.
|
||||
By default the script installs into "$env:USERPROFILE\bin".
|
||||
|
||||
.PARAMETER InstallDir
|
||||
Optional destination directory for the kingfisher.exe binary.
|
||||
|
||||
.EXAMPLE
|
||||
./install-kingfisher.ps1
|
||||
|
||||
.EXAMPLE
|
||||
./install-kingfisher.ps1 -InstallDir "C:\\Tools"
|
||||
#>
|
||||
param(
|
||||
[Parameter(Position = 0)]
|
||||
[string]$InstallDir = (Join-Path $env:USERPROFILE 'bin')
|
||||
)
|
||||
|
||||
$repo = 'mongodb/kingfisher'
|
||||
$apiUrl = "https://api.github.com/repos/$repo/releases/latest"
|
||||
$assetName = 'kingfisher-windows-x64.zip'
|
||||
|
||||
if (-not (Get-Command Invoke-WebRequest -ErrorAction SilentlyContinue)) {
|
||||
throw 'Invoke-WebRequest is required to download releases.'
|
||||
}
|
||||
|
||||
if (-not (Get-Command Expand-Archive -ErrorAction SilentlyContinue)) {
|
||||
throw 'Expand-Archive is required to extract the release archive. Install the PowerShell archive module.'
|
||||
}
|
||||
|
||||
Write-Host "Fetching latest release metadata for $repo…"
|
||||
try {
|
||||
$response = Invoke-WebRequest -Uri $apiUrl -UseBasicParsing
|
||||
$release = $response.Content | ConvertFrom-Json
|
||||
} catch {
|
||||
throw "Failed to retrieve release information from GitHub: $_"
|
||||
}
|
||||
|
||||
$releaseTag = $release.tag_name
|
||||
$asset = $release.assets | Where-Object { $_.name -eq $assetName }
|
||||
if (-not $asset) {
|
||||
throw "Could not find asset '$assetName' in the latest release."
|
||||
}
|
||||
|
||||
$tempDir = New-Item -ItemType Directory -Path ([System.IO.Path]::GetTempPath()) -Name ([System.Guid]::NewGuid().ToString())
|
||||
$archivePath = Join-Path $tempDir.FullName $assetName
|
||||
|
||||
try {
|
||||
if ($releaseTag) {
|
||||
Write-Host "Latest release: $releaseTag"
|
||||
}
|
||||
|
||||
Write-Host "Downloading $assetName…"
|
||||
Invoke-WebRequest -Uri $asset.browser_download_url -OutFile $archivePath -UseBasicParsing
|
||||
|
||||
Write-Host 'Extracting archive…'
|
||||
Expand-Archive -Path $archivePath -DestinationPath $tempDir.FullName -Force
|
||||
|
||||
$binaryPath = Join-Path $tempDir.FullName 'kingfisher.exe'
|
||||
if (-not (Test-Path $binaryPath)) {
|
||||
throw 'Extracted archive did not contain kingfisher.exe.'
|
||||
}
|
||||
|
||||
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
|
||||
$destination = Join-Path $InstallDir 'kingfisher.exe'
|
||||
Copy-Item -Path $binaryPath -Destination $destination -Force
|
||||
|
||||
Write-Host "Kingfisher installed to: $destination"
|
||||
Write-Host "Ensure '$InstallDir' is in your PATH environment variable."
|
||||
}
|
||||
finally {
|
||||
if ($tempDir -and (Test-Path $tempDir.FullName)) {
|
||||
Remove-Item -Path $tempDir.FullName -Recurse -Force
|
||||
}
|
||||
}
|
||||
151
scripts/install-kingfisher.sh
Executable file
151
scripts/install-kingfisher.sh
Executable file
|
|
@ -0,0 +1,151 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
REPO="mongodb/kingfisher"
|
||||
API_URL="https://api.github.com/repos/${REPO}/releases/latest"
|
||||
DEFAULT_INSTALL_DIR="$HOME/.local/bin"
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage: install-kingfisher.sh [INSTALL_DIR]
|
||||
|
||||
Downloads the latest Kingfisher release for Linux or macOS and installs the
|
||||
binary into INSTALL_DIR (default: ~/.local/bin).
|
||||
|
||||
The script requires curl, tar, and python3.
|
||||
USAGE
|
||||
}
|
||||
|
||||
if [[ "${1-}" == "-h" || "${1-}" == "--help" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
INSTALL_DIR="${1:-$DEFAULT_INSTALL_DIR}"
|
||||
|
||||
if ! command -v curl >/dev/null 2>&1; then
|
||||
echo "Error: curl is required to download releases." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v tar >/dev/null 2>&1; then
|
||||
echo "Error: tar is required to extract the release archive." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "Error: python3 is required to process the GitHub API response." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
OS=$(uname -s)
|
||||
ARCH=$(uname -m)
|
||||
|
||||
case "$OS" in
|
||||
Linux)
|
||||
platform="linux"
|
||||
;;
|
||||
Darwin)
|
||||
platform="darwin"
|
||||
;;
|
||||
*)
|
||||
echo "Error: Unsupported operating system '$OS'." >&2
|
||||
echo "This installer currently supports Linux and macOS." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
case "$ARCH" in
|
||||
x86_64|amd64)
|
||||
arch_suffix="x64"
|
||||
;;
|
||||
arm64|aarch64)
|
||||
arch_suffix="arm64"
|
||||
;;
|
||||
*)
|
||||
echo "Error: Unsupported architecture '$ARCH'." >&2
|
||||
echo "This installer currently supports x86_64/amd64 and arm64/aarch64." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
asset_name="kingfisher-${platform}-${arch_suffix}.tgz"
|
||||
|
||||
echo "Fetching latest release metadata for ${REPO}…"
|
||||
release_json=$(curl -fsSL "$API_URL")
|
||||
|
||||
if [[ -z "$release_json" ]]; then
|
||||
echo "Error: Failed to retrieve release information from GitHub." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
download_url=$(RELEASE_JSON="$release_json" python3 - "$asset_name" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
import os
|
||||
|
||||
asset_name = sys.argv[1]
|
||||
try:
|
||||
release = json.loads(os.environ["RELEASE_JSON"])
|
||||
except (json.JSONDecodeError, KeyError) as exc:
|
||||
sys.stderr.write(f"Error: Failed to parse GitHub response: {exc}\n")
|
||||
sys.exit(1)
|
||||
|
||||
for asset in release.get("assets", []):
|
||||
if asset.get("name") == asset_name:
|
||||
print(asset.get("browser_download_url", ""))
|
||||
sys.exit(0)
|
||||
|
||||
sys.stderr.write(f"Error: Could not find asset '{asset_name}' in the latest release.\n")
|
||||
sys.exit(1)
|
||||
PY
|
||||
)
|
||||
|
||||
if [[ -z "$download_url" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
release_tag=$(RELEASE_JSON="$release_json" python3 - <<'PY'
|
||||
import json
|
||||
import sys
|
||||
import os
|
||||
|
||||
try:
|
||||
release = json.loads(os.environ["RELEASE_JSON"])
|
||||
except (json.JSONDecodeError, KeyError) as exc:
|
||||
sys.stderr.write(f"Error: Failed to parse GitHub response: {exc}\n")
|
||||
sys.exit(1)
|
||||
|
||||
print(release.get("tag_name", ""))
|
||||
PY
|
||||
)
|
||||
|
||||
tmpdir=$(mktemp -d)
|
||||
cleanup() {
|
||||
rm -rf "$tmpdir"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
archive_path="$tmpdir/$asset_name"
|
||||
|
||||
if [[ -n "$release_tag" ]]; then
|
||||
echo "Latest release: $release_tag"
|
||||
fi
|
||||
|
||||
echo "Downloading $asset_name…"
|
||||
curl -fsSL "$download_url" -o "$archive_path"
|
||||
|
||||
echo "Extracting archive…"
|
||||
tar -C "$tmpdir" -xzf "$archive_path"
|
||||
|
||||
if [[ ! -f "$tmpdir/kingfisher" ]]; then
|
||||
echo "Error: Extracted archive did not contain the kingfisher binary." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$INSTALL_DIR"
|
||||
install -m 755 "$tmpdir/kingfisher" "$INSTALL_DIR/kingfisher"
|
||||
|
||||
printf 'Kingfisher installed to: %s/kingfisher\n\n' "$INSTALL_DIR"
|
||||
printf 'Add the following to your shell configuration if the directory is not already in your PATH:\n export PATH="%s:$PATH"\n' "$INSTALL_DIR"
|
||||
|
||||
|
|
@ -332,6 +332,32 @@ pub struct InputSpecifierArgs {
|
|||
visible_alias = "ref"
|
||||
)]
|
||||
pub branch: Option<String>,
|
||||
|
||||
/// Treat the `--branch` commit or ref as the inclusive root for the scan.
|
||||
///
|
||||
/// When enabled, Kingfisher diffs from the parent of the selected commit
|
||||
/// through the current HEAD of the repository, ensuring the chosen commit
|
||||
/// and every descendant is scanned exactly once. Providing
|
||||
/// `--branch-root-commit` will also enable this behaviour automatically.
|
||||
#[arg(
|
||||
long = "branch-root",
|
||||
help_heading = "Git Options",
|
||||
requires = "branch",
|
||||
conflicts_with = "since_commit",
|
||||
action = clap::ArgAction::SetTrue
|
||||
)]
|
||||
pub branch_root: bool,
|
||||
|
||||
/// Explicit commit or ref to use as the inclusive branch root. Supplying
|
||||
/// this flag implicitly enables branch-root scanning even if `--branch-root`
|
||||
/// is omitted.
|
||||
#[arg(
|
||||
long = "branch-root-commit",
|
||||
value_name = "GIT-REF",
|
||||
help_heading = "Git Options",
|
||||
conflicts_with = "since_commit"
|
||||
)]
|
||||
pub branch_root_commit: Option<String>,
|
||||
}
|
||||
|
||||
impl InputSpecifierArgs {
|
||||
|
|
|
|||
12
src/lib.rs
12
src/lib.rs
|
|
@ -62,6 +62,7 @@ use tracing::debug;
|
|||
pub struct GitDiffConfig {
|
||||
pub since_ref: Option<String>,
|
||||
pub branch_ref: String,
|
||||
pub branch_root: Option<String>,
|
||||
}
|
||||
|
||||
struct EnumeratorConfig {
|
||||
|
|
@ -332,7 +333,16 @@ impl FilesystemEnumerator {
|
|||
|
||||
/// Opens the given Git repository if it exists, returning None if not.
|
||||
pub fn open_git_repo(path: &Path) -> Result<Option<Repository>> {
|
||||
let opts = Options::isolated().open_path_as_is(false);
|
||||
open_git_repo_with_options(path, true)
|
||||
}
|
||||
|
||||
/// Opens the given Git repository with explicit control over the
|
||||
/// `open_path_as_is` option, returning None if not.
|
||||
pub fn open_git_repo_with_options(
|
||||
path: &Path,
|
||||
open_path_as_is: bool,
|
||||
) -> Result<Option<Repository>> {
|
||||
let opts = Options::isolated().open_path_as_is(open_path_as_is);
|
||||
match open_opts(path, opts) {
|
||||
Err(gix::open::Error::NotARepository { .. }) => Ok(None),
|
||||
Err(err) => Err(err.into()),
|
||||
|
|
|
|||
|
|
@ -418,6 +418,8 @@ fn create_default_scan_args() -> cli::commands::scan::ScanArgs {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
extra_ignore_comments: Vec::new(),
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
|
|
|
|||
|
|
@ -779,6 +779,8 @@ mod tests {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
extra_ignore_comments: Vec::new(),
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
|
|
|
|||
|
|
@ -153,6 +153,8 @@ mod tests {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
extra_ignore_comments: Vec::new(),
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ use crate::{
|
|||
git_commit_metadata::CommitMetadata,
|
||||
git_repo_enumerator::GitBlobMetadata,
|
||||
matcher::{Matcher, MatcherStats},
|
||||
open_git_repo,
|
||||
open_git_repo_with_options,
|
||||
origin::{Origin, OriginSet},
|
||||
rule_profiling::ConcurrentRuleProfiler,
|
||||
rules_database::RulesDatabase,
|
||||
|
|
@ -60,16 +60,29 @@ pub fn enumerate_filesystem_inputs(
|
|||
) -> Result<()> {
|
||||
let repo_scan_timeout = Duration::from_secs(args.git_repo_timeout);
|
||||
|
||||
let branch_root_enabled = args.input_specifier_args.branch_root
|
||||
|| args.input_specifier_args.branch_root_commit.is_some();
|
||||
|
||||
let diff_config = if args.input_specifier_args.since_commit.is_some()
|
||||
|| args.input_specifier_args.branch.is_some()
|
||||
|| branch_root_enabled
|
||||
{
|
||||
let branch_arg = args.input_specifier_args.branch.clone();
|
||||
let branch_root_commit = args.input_specifier_args.branch_root_commit.clone();
|
||||
let (branch_ref, branch_root) = if branch_root_enabled {
|
||||
if let Some(explicit_root) = branch_root_commit {
|
||||
(branch_arg.clone().unwrap_or_else(|| "HEAD".to_string()), Some(explicit_root))
|
||||
} else {
|
||||
("HEAD".to_string(), branch_arg.clone())
|
||||
}
|
||||
} else {
|
||||
(branch_arg.clone().unwrap_or_else(|| "HEAD".to_string()), None)
|
||||
};
|
||||
|
||||
Some(GitDiffConfig {
|
||||
since_ref: args.input_specifier_args.since_commit.clone(),
|
||||
branch_ref: args
|
||||
.input_specifier_args
|
||||
.branch
|
||||
.clone()
|
||||
.unwrap_or_else(|| "HEAD".to_string()),
|
||||
branch_ref,
|
||||
branch_root,
|
||||
})
|
||||
} else {
|
||||
None
|
||||
|
|
@ -609,13 +622,14 @@ impl<'cfg> ParallelBlobIterator for (&'cfg EnumeratorConfig, FoundInput) {
|
|||
// ───────────── directory (possible Git repo) ─────────────
|
||||
FoundInput::Directory(i) => {
|
||||
let path = &i.path;
|
||||
let open_path_as_is = cfg.git_diff.is_none();
|
||||
|
||||
if cfg.git_diff.is_none() && !cfg.enumerate_git_history {
|
||||
if open_path_as_is && !cfg.enumerate_git_history {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
// Try to open a Git repository at that path
|
||||
let repository = match open_git_repo(path)? {
|
||||
let repository = match open_git_repo_with_options(path, open_path_as_is)? {
|
||||
Some(r) => r,
|
||||
None => return Ok(None),
|
||||
};
|
||||
|
|
@ -719,7 +733,7 @@ fn enumerate_git_diff_repo(
|
|||
exclude_globset: Option<std::sync::Arc<globset::GlobSet>>,
|
||||
collect_commit_metadata: bool,
|
||||
) -> Result<GitRepoResult> {
|
||||
let GitDiffConfig { since_ref, branch_ref } = diff_cfg;
|
||||
let GitDiffConfig { since_ref, branch_ref, branch_root } = diff_cfg;
|
||||
|
||||
let blobs = {
|
||||
let head_id = resolve_diff_ref(&repository, path, &branch_ref).with_context(|| {
|
||||
|
|
@ -760,6 +774,40 @@ fn enumerate_git_diff_repo(
|
|||
.with_context(|| format!("Failed to read tree for commit {}", base_id.to_hex()))?;
|
||||
|
||||
base_tree = Some(tree);
|
||||
} else if let Some(ref branch_root_value) = branch_root {
|
||||
let root_id =
|
||||
resolve_diff_ref(&repository, path, branch_root_value).with_context(|| {
|
||||
format!(
|
||||
"Failed to resolve --branch-root '{}' in repository {}",
|
||||
branch_root_value,
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
|
||||
let root_commit = root_id
|
||||
.object()
|
||||
.with_context(|| format!("Failed to load commit {} for diffing", root_id.to_hex()))?
|
||||
.try_into_commit()
|
||||
.with_context(|| {
|
||||
format!("Referenced object {} is not a commit", root_id.to_hex())
|
||||
})?;
|
||||
|
||||
let mut parent_ids = root_commit.parent_ids();
|
||||
if let Some(parent_id) = parent_ids.next() {
|
||||
let parent_commit = parent_id
|
||||
.object()
|
||||
.with_context(|| {
|
||||
format!("Failed to load parent commit {} for diffing", parent_id.to_hex())
|
||||
})?
|
||||
.try_into_commit()
|
||||
.with_context(|| {
|
||||
format!("Referenced object {} is not a commit", parent_id.to_hex())
|
||||
})?;
|
||||
let parent_tree = parent_commit.tree().with_context(|| {
|
||||
format!("Failed to read tree for commit {}", parent_id.to_hex())
|
||||
})?;
|
||||
base_tree = Some(parent_tree);
|
||||
}
|
||||
}
|
||||
|
||||
let changes = repository
|
||||
|
|
@ -1008,7 +1056,11 @@ mod tests {
|
|||
let result = enumerate_git_diff_repo(
|
||||
&repo_path,
|
||||
gix_repo,
|
||||
GitDiffConfig { since_ref: None, branch_ref: "featurefake".to_string() },
|
||||
GitDiffConfig {
|
||||
since_ref: None,
|
||||
branch_ref: "featurefake".to_string(),
|
||||
branch_root: None,
|
||||
},
|
||||
None,
|
||||
false,
|
||||
)?;
|
||||
|
|
|
|||
1070
src/scanner/enumerate.rs.orig
Normal file
1070
src/scanner/enumerate.rs.orig
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -120,6 +120,8 @@ fn run_skiplist(skip_regex: Vec<String>, skip_skipword: Vec<String>) -> Result<u
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
extra_ignore_comments: Vec::new(),
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
|
|
|
|||
|
|
@ -120,6 +120,8 @@ fn test_bitbucket_remote_scan() -> Result<()> {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
max_file_size_mb: 25.0,
|
||||
|
|
|
|||
|
|
@ -140,6 +140,8 @@ rules:
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
max_file_size_mb: 5.0,
|
||||
|
|
|
|||
|
|
@ -127,6 +127,8 @@ fn test_github_remote_scan() -> Result<()> {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
max_file_size_mb: 25.0,
|
||||
|
|
|
|||
|
|
@ -125,6 +125,8 @@ fn test_gitlab_remote_scan() -> Result<()> {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
extra_ignore_comments: Vec::new(),
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
|
|
@ -271,6 +273,8 @@ fn test_gitlab_remote_scan_no_history() -> Result<()> {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
gcs_bucket: None,
|
||||
gcs_prefix: None,
|
||||
gcs_service_account: None,
|
||||
|
|
|
|||
|
|
@ -103,6 +103,8 @@ async fn test_redact_hashes_finding_values() -> Result<()> {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
max_file_size_mb: 25.0,
|
||||
|
|
|
|||
|
|
@ -111,6 +111,8 @@ impl TestContext {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
extra_ignore_comments: Vec::new(),
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
|
|
@ -248,6 +250,8 @@ async fn test_scan_slack_messages() -> Result<()> {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
max_file_size_mb: 25.0,
|
||||
|
|
|
|||
|
|
@ -183,6 +183,8 @@ async fn test_validation_cache_and_depvars() -> Result<()> {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
max_file_size_mb: 25.0,
|
||||
|
|
|
|||
|
|
@ -126,6 +126,8 @@ impl TestContext {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
},
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
max_file_size_mb: 25.0,
|
||||
|
|
@ -247,6 +249,8 @@ impl TestContext {
|
|||
scan_nested_repos: true,
|
||||
since_commit: None,
|
||||
branch: None,
|
||||
branch_root: false,
|
||||
branch_root_commit: None,
|
||||
|
||||
gcs_bucket: None,
|
||||
gcs_prefix: None,
|
||||
|
|
|
|||
|
|
@ -117,3 +117,135 @@ aws_secret_access_key = efnegoUp/WXc3XwlL77dXu1aKIICzvz+n+7Sz88i
|
|||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scan_branch_root_inclusive_history() -> anyhow::Result<()> {
|
||||
let dir = tempdir()?;
|
||||
let repo_dir = dir.path().join("repo");
|
||||
let repo = Repository::init(&repo_dir)?;
|
||||
let signature = Signature::now("tester", "tester@example.com")?;
|
||||
|
||||
let secrets_path = repo_dir.join("secrets.txt");
|
||||
|
||||
let aws_value = "UpUbsQANRHLf2uuQ7QOlNXPbbtV5fmseW/GgTs5D/";
|
||||
let gcp_value = "c4c474d61701fd6fd4191883b8fea9a8411bf771";
|
||||
let slack_value = "xoxb-123465789012-0987654321123-AbDcEfGhIjKlMnOpQrStUvWx";
|
||||
let github_value = "ghp_aBcDeFgHiJkLmNoqpRsTuVwXyZ1243567890";
|
||||
let stripe_value =
|
||||
"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";
|
||||
|
||||
let aws_line = "AWS_SECRET_ACCESS_KEY = 'UpUbsQANRHLf2uuQ7QOlNXPbbtV5fmseW/GgTs5D/'";
|
||||
let gcp_line = "GCP_PRIVATE_KEY_ID = 'c4c474d61701fd6fd4191883b8fea9a8411bf771'";
|
||||
let slack_line = "SLACK_BOT_TOKEN = 'xoxb-123465789012-0987654321123-AbDcEfGhIjKlMnOpQrStUvWx'";
|
||||
let github_line = "GITHUB_TOKEN = 'ghp_aBcDeFgHiJkLmNoqpRsTuVwXyZ1243567890'";
|
||||
let stripe_line = concat!(
|
||||
"STRIPE_SECRET_KEY = '",
|
||||
"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",
|
||||
"'",
|
||||
);
|
||||
|
||||
fs::write(&secrets_path, aws_line)?;
|
||||
|
||||
let mut index = repo.index()?;
|
||||
index.add_path(Path::new("secrets.txt"))?;
|
||||
let tree_id = index.write_tree()?;
|
||||
let tree = repo.find_tree(tree_id)?;
|
||||
let initial_commit_id =
|
||||
repo.commit(Some("HEAD"), &signature, &signature, "Add AWS secret", &tree, &[])?;
|
||||
let initial_commit = repo.find_commit(initial_commit_id)?;
|
||||
let initial_commit_hex = initial_commit_id.to_string();
|
||||
|
||||
let additions = [
|
||||
("Add GCP private key id", gcp_line),
|
||||
("Add Slack bot token", slack_line),
|
||||
("Add GitHub PAT", github_line),
|
||||
("Add Stripe API key", stripe_line),
|
||||
];
|
||||
|
||||
let mut parent_commit = initial_commit;
|
||||
let mut contents = String::from(aws_line);
|
||||
|
||||
for (message, line) in additions {
|
||||
contents.push('\n');
|
||||
contents.push_str(line);
|
||||
fs::write(&secrets_path, &contents)?;
|
||||
|
||||
let mut index = repo.index()?;
|
||||
index.add_path(Path::new("secrets.txt"))?;
|
||||
let tree_id = index.write_tree()?;
|
||||
let tree = repo.find_tree(tree_id)?;
|
||||
let new_commit_id =
|
||||
repo.commit(Some("HEAD"), &signature, &signature, message, &tree, &[&parent_commit])?;
|
||||
parent_commit = repo.find_commit(new_commit_id)?;
|
||||
}
|
||||
|
||||
let latest_commit_hex = parent_commit.id().to_string();
|
||||
repo.branch("long-lived", &parent_commit, true)?;
|
||||
|
||||
// Scanning the initial commit without --branch-root should report only the
|
||||
// secret present at that commit.
|
||||
Command::cargo_bin("kingfisher")?
|
||||
.args([
|
||||
"scan",
|
||||
repo_dir.to_str().unwrap(),
|
||||
"--branch",
|
||||
initial_commit_hex.as_str(),
|
||||
"--no-validate",
|
||||
"--no-update-check",
|
||||
])
|
||||
.assert()
|
||||
.code(200)
|
||||
.stdout(
|
||||
contains(aws_value)
|
||||
.and(contains(gcp_value).not())
|
||||
.and(contains(slack_value).not())
|
||||
.and(contains(github_value).not())
|
||||
.and(contains(stripe_value).not()),
|
||||
);
|
||||
|
||||
// Using --branch-root should include the selected commit and the remaining
|
||||
// branch history up to HEAD, surfacing the later secrets too.
|
||||
Command::cargo_bin("kingfisher")?
|
||||
.args([
|
||||
"scan",
|
||||
repo_dir.to_str().unwrap(),
|
||||
"--branch",
|
||||
initial_commit_hex.as_str(),
|
||||
"--branch-root",
|
||||
"--no-validate",
|
||||
"--no-update-check",
|
||||
])
|
||||
.assert()
|
||||
.code(200)
|
||||
.stdout(
|
||||
contains(aws_value)
|
||||
.and(contains(gcp_value))
|
||||
.and(contains(slack_value))
|
||||
.and(contains(github_value))
|
||||
.and(contains(stripe_value)),
|
||||
);
|
||||
|
||||
Command::cargo_bin("kingfisher")?
|
||||
.args([
|
||||
"scan",
|
||||
repo_dir.to_str().unwrap(),
|
||||
"--branch",
|
||||
"long-lived",
|
||||
"--branch-root-commit",
|
||||
initial_commit_hex.as_str(),
|
||||
"--no-validate",
|
||||
"--no-update-check",
|
||||
])
|
||||
.assert()
|
||||
.code(200)
|
||||
.stdout(
|
||||
contains(aws_value)
|
||||
.and(contains(gcp_value))
|
||||
.and(contains(slack_value))
|
||||
.and(contains(github_value))
|
||||
.and(contains(stripe_value))
|
||||
.and(contains(latest_commit_hex.as_str())),
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue