forked from mirrors/kingfisher
commit
fa72b106cf
20 changed files with 312 additions and 86 deletions
|
|
@ -2,6 +2,11 @@
|
|||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
## [1.44.0]
|
||||
- Fixed issue with self-update on Linux
|
||||
- Reverted the change to json and jsonl outputs by rule
|
||||
- Added `--skip-regex` and `--skip-word` flags to ignore secrets matching custom patterns or skipwords
|
||||
|
||||
## [1.43.0]
|
||||
- Added rules for clearbit, kickbox, azure container registry, improved Azure Storage key
|
||||
- Grouped JSON and JSONL outputs by rule, restoring `matches` arrays in reports
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ publish = false
|
|||
|
||||
[package]
|
||||
name = "kingfisher"
|
||||
version = "1.43.0"
|
||||
version = "1.44.0"
|
||||
description = "MongoDB's blazingly fast secret scanning and validation tool"
|
||||
edition.workspace = true
|
||||
rust-version.workspace = true
|
||||
|
|
|
|||
58
README.md
58
README.md
|
|
@ -15,6 +15,7 @@ Kingfisher originated as a fork of Praetorian's Nosey Parker, and is built atop
|
|||
- **Extra targets**: GitLab repos, S3 buckets, Docker images, Jira issues, Confluence pages, and Slack messages
|
||||
- **Compressed Files**: Supports extracting and scanning compressed files for secrets
|
||||
- **Baseline mode**: ignore known secrets, flag only new ones
|
||||
- **Allowlist support**: suppress false positives with custom regexes or words
|
||||
- **Language-aware detection** (source-code parsing) for ~20 languages
|
||||
- **Native Windows** binary
|
||||
|
||||
|
|
@ -158,8 +159,8 @@ Kingfisher ships with hundreds of rules that cover everything from classic cloud
|
|||
|----------|---------------|
|
||||
| **AI / LLM APIs** | OpenAI, Anthropic, Google Gemini, Cohere, Mistral, Stability AI, Replicate, xAI (Grok), and more
|
||||
| **Cloud Providers** | AWS, Azure, GCP, Alibaba Cloud, DigitalOcean, IBM Cloud, Cloudflare, and more
|
||||
| **Dev & CI/CD** | GitHub/GitLab tokens, CircleCI, TravisCI, TeamCity, Docker Hub, npm & PyPI publish token, and more
|
||||
| **Messaging & Comms** | Slack, Discord, Microsoft Teams, Twilio, Mailgun/SendGrid/Mailchimp, and more
|
||||
| **Dev & CI/CD** | GitHub/GitLab tokens, CircleCI, TravisCI, TeamCity, Docker Hub, npm, PyPI, and more
|
||||
| **Messaging & Comms** | Slack, Discord, Microsoft Teams, Twilio, Mailgun, SendGrid, Mailchimp, and more
|
||||
| **Databases & Data Ops** | MongoDB Atlas, PlanetScale, Postgres DSNs, Grafana Cloud, Datadog, Dynatrace, and more
|
||||
| **Payments & Billing** | Stripe, PayPal, Square, GoCardless, and more
|
||||
| **Security & DevSecOps** | Snyk, Dependency-Track, CodeClimate, Codacy, OpsGenie, PagerDuty, and more
|
||||
|
|
@ -179,7 +180,7 @@ Once you've done that, you can provide your custom rules (defined in a YAML file
|
|||
|
||||
## Basic Examples
|
||||
|
||||
> **Note** `kingfisher scan` detects whether the input is a Git repository or a plain directory—no extra flags required.
|
||||
> **Note** `kingfisher scan` detects whether the input is a Git repository or a plain directory, no extra flags required.
|
||||
|
||||
### Scan with secret validation
|
||||
|
||||
|
|
@ -597,7 +598,31 @@ kingfisher github repos list --organization my-org
|
|||
- `--exclude <PATTERN>`: Skip any file or directory whose path matches this glob pattern (repeatable, uses gitignore-style syntax, case sensitive)
|
||||
- `--baseline-file <FILE>`: Ignore matches listed in a baseline YAML file
|
||||
- `--manage-baseline`: Create or update the baseline file with current findings
|
||||
- `--skip-regex <PATTERN>`: Ignore findings whose text matches this regex (repeatable)
|
||||
- `--skip-word <WORD>`: Ignore findings containing this case-insensitive word (repeatable)
|
||||
|
||||
### Ignore known false positives
|
||||
|
||||
Use `--skip-regex` and `--skip-word` to suppress findings you know are benign. Both flags may be provided multiple times and are tested against the secret value **and** the full match context.
|
||||
|
||||
With `--skip-regex`, these should be Rust compatible regular expressions, which you can test out at [regex101](https://regex101.com)
|
||||
|
||||
```bash
|
||||
# Skip any finding where the finding mentions TEST_KEY
|
||||
kingfisher scan --skip-regex '(?i)TEST_KEY' path/
|
||||
|
||||
# Skip findings that contain the word "dummy" anywhere in the match
|
||||
kingfisher scan --skip-word dummy path/
|
||||
|
||||
# Combine multiple patterns
|
||||
kingfisher scan \
|
||||
--skip-regex 'AKIA[0-9A-Z]{16}' \
|
||||
--skip-word placeholder \
|
||||
--skip-word dummy \
|
||||
path/
|
||||
```
|
||||
|
||||
If a `--skip-regex` regular expression fails to compile, the scan aborts with an error so that typos are caught early.
|
||||
|
||||
## Finding Fingerprint
|
||||
|
||||
|
|
@ -614,36 +639,11 @@ Use `--rule-stats` to collect timing information for every rule. After scanning,
|
|||
kingfisher scan --help
|
||||
```
|
||||
|
||||
## Business Value
|
||||
|
||||
By integrating Kingfisher into your development lifecycle, you can:
|
||||
|
||||
- **Prevent Costly Breaches**
|
||||
Early detection of embedded credentials avoids expensive incident response, legal fees, and reputation damage
|
||||
- **Automate Compliance**
|
||||
Enforce secret‑scanning policies across GitOps, CI/CD, and pull requests to help satisfy SOC 2, PCI‑DSS, GDPR, and other standards
|
||||
- **Reduce Noise, Focus on Real Threats**
|
||||
Validation logic filters out false positives and highlights only active, valid secrets (`--only-valid`)
|
||||
- **Accelerate Dev Workflows**
|
||||
Run in parallel across dozens of languages, integrate with GitHub Actions or any pipeline, and shift security left to minimize delays
|
||||
|
||||
## The Risk of Leaked Secrets
|
||||
|
||||
Real breaches show how one exposed key can snowball into a full-scale incident:
|
||||
|
||||
- **Uber (2016):** GitHub-hosted AWS key let attackers access data on 57 M riders and 600 k drivers. [[BBC](https://www.bbc.com/news/technology-42075306)] [[Ars](https://arstechnica.com/tech-policy/2017/11/report-uber-paid-hackers-100000-to-keep-2016-data-breach-quiet/)]
|
||||
- **AWS engineer (2020):** Pushed log files with root credentials to GitHub. [[Register](https://www.theregister.com/2020/01/23/aws_engineer_credentials_github/)] [[UpGuard](https://www.upguard.com/breaches/identity-and-access-misstep-how-an-amazon-engineer-exposed-credentials-and-more)]
|
||||
- **Infosys (2023):** Full-admin AWS key left in a public PyPI package for a year. [[Stack](https://www.thestack.technology/infosys-leak-aws-key-exposed-on-pypi/)] [[Blog](https://tomforb.es/blog/infosys-leak/)]
|
||||
- **Microsoft (2023):** Azure SAS token in an AI repo exposed 38 TB of internal data. [[Wiz](https://www.wiz.io/blog/38-terabytes-of-private-data-accidentally-exposed-by-microsoft-ai-researchers)] [[TechCrunch](https://techcrunch.com/2023/09/18/microsoft-ai-researchers-accidentally-exposed-terabytes-of-internal-sensitive-data/)]
|
||||
- **GitHub (2023):** RSA SSH host key briefly went public; company rotated it. [[GitHub](https://github.blog/news-insights/company-news/we-updated-our-rsa-ssh-host-key/)]
|
||||
|
||||
Leaked secrets fuel unauthorized access, lateral movement, regulatory fines, and brand-damaging incident-response costs.
|
||||
|
||||
# Roadmap
|
||||
|
||||
- More rules
|
||||
- More targets
|
||||
- Please file a [feature request](https://github.com/mongodb/kingfisher/issues) if you have specific features you'd like added
|
||||
- Please file a [feature request](https://github.com/mongodb/kingfisher/issues), or open a PR, if you have features you'd like added
|
||||
|
||||
# License
|
||||
|
||||
|
|
|
|||
|
|
@ -106,6 +106,14 @@ pub struct ScanArgs {
|
|||
/// Create or update the baseline file with current findings
|
||||
#[arg(long, default_value_t = false)]
|
||||
pub manage_baseline: bool,
|
||||
|
||||
/// Regex patterns to allow-list secret matches (repeatable)
|
||||
#[arg(long = "skip-regex", value_name = "PATTERN")]
|
||||
pub skip_regex: Vec<String>,
|
||||
|
||||
/// Skipwords to allow-list secret matches (case-insensitive, repeatable)
|
||||
#[arg(long = "skip-word", value_name = "WORD")]
|
||||
pub skip_word: Vec<String>,
|
||||
}
|
||||
|
||||
/// Confidence levels for findings
|
||||
|
|
|
|||
|
|
@ -325,6 +325,8 @@ fn create_default_scan_args() -> cli::commands::scan::ScanArgs {
|
|||
no_dedup: false,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
output_args: OutputArgs { output: None, format: ReportOutputFormat::Pretty },
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ use crate::{
|
|||
rule_profiling::{ConcurrentRuleProfiler, RuleStats, RuleTimer},
|
||||
rules::rule::Rule,
|
||||
rules_database::RulesDatabase,
|
||||
safe_list::is_safe_match,
|
||||
safe_list::{is_safe_match, is_user_match},
|
||||
scanner_pool::ScannerPool,
|
||||
snippet::Base64BString,
|
||||
util::{intern, redact_value},
|
||||
|
|
@ -472,16 +472,16 @@ fn filter_match<'b>(
|
|||
None => Cow::Borrowed(&blob.bytes()[start..end]),
|
||||
};
|
||||
for captures in re.captures_iter(byte_slice.as_ref()) {
|
||||
let matching_input = captures.get(1).or_else(|| captures.get(0)).unwrap();
|
||||
// let str_input = std::str::from_utf8(matching_input.as_bytes()).unwrap_or("");
|
||||
// let calculated_entropy = calculate_shannon_entropy(str_input);
|
||||
// if calculated_entropy <= rule.min_entropy() || is_safe_match(str_input) {
|
||||
// continue;
|
||||
// }
|
||||
let full_capture = captures.get(0).unwrap();
|
||||
let matching_input = captures.get(1).unwrap_or(full_capture);
|
||||
let min_entropy = rule.min_entropy();
|
||||
let mi_bytes = matching_input.as_bytes();
|
||||
let full_bytes = full_capture.as_bytes();
|
||||
let calculated_entropy = calculate_shannon_entropy(mi_bytes);
|
||||
if calculated_entropy <= min_entropy || is_safe_match(mi_bytes) {
|
||||
if calculated_entropy <= min_entropy
|
||||
|| is_safe_match(mi_bytes)
|
||||
|| is_user_match(mi_bytes, full_bytes)
|
||||
{
|
||||
debug!(
|
||||
"Skipping match with entropy {} <= {} or safe match",
|
||||
calculated_entropy, min_entropy
|
||||
|
|
|
|||
|
|
@ -517,12 +517,6 @@ pub struct FindingRecordData {
|
|||
pub git_metadata: Option<serde_json::Value>,
|
||||
}
|
||||
|
||||
#[derive(Serialize, JsonSchema, Clone, Debug)]
|
||||
pub struct RuleMatches {
|
||||
pub id: String,
|
||||
pub matches: Vec<FindingReporterRecord>,
|
||||
}
|
||||
|
||||
impl From<finding_data::FindingDataEntry> for ReportMatch {
|
||||
fn from(e: finding_data::FindingDataEntry) -> Self {
|
||||
ReportMatch {
|
||||
|
|
|
|||
|
|
@ -1,5 +1,4 @@
|
|||
use super::*;
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
impl DetailsReporter {
|
||||
pub fn json_format<W: std::io::Write>(
|
||||
|
|
@ -9,13 +8,7 @@ impl DetailsReporter {
|
|||
) -> Result<()> {
|
||||
let records = self.build_finding_records(args)?;
|
||||
if !records.is_empty() {
|
||||
let mut grouped: BTreeMap<String, Vec<FindingReporterRecord>> = BTreeMap::new();
|
||||
for record in records {
|
||||
grouped.entry(record.rule.id.clone()).or_default().push(record);
|
||||
}
|
||||
let groups: Vec<RuleMatches> =
|
||||
grouped.into_iter().map(|(id, matches)| RuleMatches { id, matches }).collect();
|
||||
serde_json::to_writer_pretty(&mut writer, &groups)?;
|
||||
serde_json::to_writer_pretty(&mut writer, &records)?;
|
||||
writeln!(writer)?;
|
||||
}
|
||||
Ok(())
|
||||
|
|
@ -27,16 +20,9 @@ impl DetailsReporter {
|
|||
args: &cli::commands::scan::ScanArgs,
|
||||
) -> Result<()> {
|
||||
let records = self.build_finding_records(args)?;
|
||||
if !records.is_empty() {
|
||||
let mut grouped: BTreeMap<String, Vec<FindingReporterRecord>> = BTreeMap::new();
|
||||
for record in records {
|
||||
grouped.entry(record.rule.id.clone()).or_default().push(record);
|
||||
}
|
||||
for (id, matches) in grouped {
|
||||
let group = RuleMatches { id, matches };
|
||||
serde_json::to_writer(&mut writer, &group)?;
|
||||
writeln!(writer)?;
|
||||
}
|
||||
for record in records {
|
||||
serde_json::to_writer(&mut writer, &record)?;
|
||||
writeln!(writer)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -139,6 +125,8 @@ mod tests {
|
|||
output_args: OutputArgs { output: None, format: ReportOutputFormat::Pretty },
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -237,10 +225,7 @@ mod tests {
|
|||
reporter.json_format(&mut output, &create_default_args())?;
|
||||
let json_output: Vec<serde_json::Value> = serde_json::from_slice(&output.into_inner())?;
|
||||
assert!(!json_output.is_empty(), "JSON output should not be empty");
|
||||
let first_group = &json_output[0];
|
||||
assert_eq!(first_group["id"], "mock_rule_1");
|
||||
let matches = first_group["matches"].as_array().unwrap();
|
||||
let first = &matches[0];
|
||||
let first = &json_output[0];
|
||||
assert_eq!(first["rule"]["name"], "MockRule");
|
||||
assert_eq!(first["finding"]["language"], "Rust");
|
||||
Ok(())
|
||||
|
|
@ -281,10 +266,8 @@ mod tests {
|
|||
reporter.json_format(&mut output, &create_default_args())?;
|
||||
let json_output: Vec<serde_json::Value> = serde_json::from_slice(&output.into_inner())?;
|
||||
assert!(!json_output.is_empty(), "JSON output should not be empty");
|
||||
let first_group = &json_output[0];
|
||||
let first_match = &first_group["matches"][0];
|
||||
let validation_status =
|
||||
first_match["finding"]["validation"]["status"].as_str().unwrap();
|
||||
let first = &json_output[0];
|
||||
let validation_status = first["finding"]["validation"]["status"].as_str().unwrap();
|
||||
assert_eq!(validation_status, expected_status);
|
||||
}
|
||||
Ok(())
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@
|
|||
|
||||
use once_cell::sync::Lazy;
|
||||
use regex::bytes::Regex;
|
||||
use std::sync::Mutex;
|
||||
use tracing::debug;
|
||||
|
||||
/// A rule that describes *why* a match is considered safe/benign.
|
||||
|
|
@ -131,6 +132,63 @@ static SAFE_LIST_FILTER_RULES: Lazy<Vec<SafeRule>> = Lazy::new(|| {
|
|||
]
|
||||
});
|
||||
|
||||
// User-supplied allow-list patterns (regexes) and skipwords. These are empty by
|
||||
// default and populated via CLI flags at runtime.
|
||||
static USER_SAFE_REGEXES: Lazy<Mutex<Vec<Regex>>> = Lazy::new(|| Mutex::new(Vec::new()));
|
||||
static USER_SAFE_SKIPWORDS: Lazy<Mutex<Vec<String>>> = Lazy::new(|| Mutex::new(Vec::new()));
|
||||
|
||||
/// Register an additional allow-list regex provided by the user.
|
||||
/// If the pattern fails to compile, the error is returned so the caller can
|
||||
/// surface it.
|
||||
pub fn add_user_regex(pattern: &str) -> std::result::Result<(), regex::Error> {
|
||||
let re = Regex::new(pattern)?;
|
||||
USER_SAFE_REGEXES.lock().unwrap().push(re);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Register an allow-list skipword provided by the user. Comparisons are
|
||||
/// case-insensitive.
|
||||
pub fn add_user_skipword(word: &str) {
|
||||
USER_SAFE_SKIPWORDS.lock().unwrap().push(word.to_lowercase());
|
||||
}
|
||||
|
||||
/// Returns `true` if the given input matches any user-supplied allow-list
|
||||
/// patterns (regexes or skipwords).
|
||||
///
|
||||
/// `secret` is the primary capture group (typically just the secret value)
|
||||
/// while `full_match` includes the entire match, allowing regexes to target
|
||||
/// surrounding context such as variable names.
|
||||
pub fn is_user_match(secret: &[u8], full_match: &[u8]) -> bool {
|
||||
{
|
||||
let regexes = USER_SAFE_REGEXES.lock().unwrap();
|
||||
if regexes.iter().any(|re| re.is_match(secret) || re.is_match(full_match)) {
|
||||
debug!("Safe match: user skip-regex");
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
let skipwords = USER_SAFE_SKIPWORDS.lock().unwrap();
|
||||
if skipwords.is_empty() {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Check skipwords against both the secret and full match (case-insensitive)
|
||||
let contains_skipword = |bytes: &[u8]| -> bool {
|
||||
if let Ok(s) = std::str::from_utf8(bytes) {
|
||||
let lower = s.to_lowercase();
|
||||
return skipwords.iter().any(|w| lower.contains(w));
|
||||
}
|
||||
false
|
||||
};
|
||||
|
||||
if contains_skipword(secret) || contains_skipword(full_match) {
|
||||
debug!("Safe match: user skip-word");
|
||||
return true;
|
||||
}
|
||||
|
||||
false
|
||||
}
|
||||
|
||||
/// Returns `Some(&'static str)` with the rule description if the input likely
|
||||
/// contains *benign* placeholder/test strings; otherwise `None`.
|
||||
pub fn is_safe_match_reason(input: &[u8]) -> Option<&'static str> {
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ use crate::{
|
|||
rule_loader::RuleLoader,
|
||||
rule_profiling::ConcurrentRuleProfiler,
|
||||
rules_database::RulesDatabase,
|
||||
safe_list,
|
||||
scanner::{
|
||||
clone_or_update_git_repos, enumerate_filesystem_inputs, enumerate_github_repos,
|
||||
repos::{
|
||||
|
|
@ -52,6 +53,15 @@ pub async fn run_async_scan(
|
|||
}
|
||||
}
|
||||
|
||||
// Register user-provided allow-list patterns
|
||||
for pattern in &args.skip_regex {
|
||||
safe_list::add_user_regex(pattern)
|
||||
.map_err(|e| anyhow::anyhow!("Invalid skip-regex '{pattern}': {e}"))?;
|
||||
}
|
||||
for word in &args.skip_word {
|
||||
safe_list::add_user_skipword(word);
|
||||
}
|
||||
|
||||
let start_time = Instant::now();
|
||||
|
||||
trace!("Args:\n{global_args:#?}\n{args:#?}");
|
||||
|
|
|
|||
|
|
@ -95,6 +95,17 @@ pub fn check_for_update(global_args: &GlobalArgs, base_url: Option<&str>) -> Opt
|
|||
#[cfg(all(target_os = "windows", target_arch = "x86_64"))]
|
||||
builder.target("windows-x64");
|
||||
|
||||
// ──────────────────────────────────────────────────────
|
||||
// Disambiguate archive format to avoid picking .deb packages.
|
||||
// Linux and macOS releases use `.tgz`; Windows uses `.zip`.
|
||||
// ──────────────────────────────────────────────────────
|
||||
#[cfg(target_os = "windows")]
|
||||
builder.identifier("zip");
|
||||
|
||||
// Linux releases also ship as .deb and .rpm packages; select the .tgz asset for self‑updates
|
||||
#[cfg(not(target_os = "windows"))]
|
||||
builder.identifier("tgz");
|
||||
|
||||
// Build the updater.
|
||||
let Ok(updater) = builder.build() else {
|
||||
warn!("Failed to configure update checker");
|
||||
|
|
|
|||
143
tests/int_allowlist.rs
Normal file
143
tests/int_allowlist.rs
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
use std::{
|
||||
fs,
|
||||
sync::{Arc, Mutex},
|
||||
};
|
||||
|
||||
use anyhow::Result;
|
||||
use kingfisher::{
|
||||
cli::{
|
||||
commands::{
|
||||
github::{GitCloneMode, GitHistoryMode, GitHubRepoType},
|
||||
gitlab::GitLabRepoType,
|
||||
inputs::{ContentFilteringArgs, InputSpecifierArgs},
|
||||
output::{OutputArgs, ReportOutputFormat},
|
||||
rules::RuleSpecifierArgs,
|
||||
scan::{ConfidenceLevel, ScanArgs},
|
||||
},
|
||||
global::{AdvancedArgs, Mode},
|
||||
GlobalArgs,
|
||||
},
|
||||
findings_store::FindingsStore,
|
||||
rule_loader::RuleLoader,
|
||||
rules_database::RulesDatabase,
|
||||
scanner::run_async_scan,
|
||||
};
|
||||
use tempfile::TempDir;
|
||||
use tokio::runtime::Runtime;
|
||||
use url::Url;
|
||||
|
||||
fn run_skiplist(skip_regex: Vec<String>, skip_skipword: Vec<String>) -> Result<usize> {
|
||||
let rt = Runtime::new().unwrap();
|
||||
let work = TempDir::new()?;
|
||||
let rules_dir = work.path().join("rules");
|
||||
fs::create_dir_all(&rules_dir)?;
|
||||
let inputs_dir = work.path().join("in");
|
||||
fs::create_dir_all(&inputs_dir)?;
|
||||
|
||||
fs::write(
|
||||
rules_dir.join("demo.yml"),
|
||||
r#"rules:
|
||||
- id: demo.token
|
||||
name: Demo token
|
||||
pattern: 'token_(\w+)'
|
||||
confidence: low
|
||||
"#,
|
||||
)?;
|
||||
|
||||
fs::write(inputs_dir.join("a.txt"), "token_realvalue\ntoken_testvalue\n")?;
|
||||
|
||||
let scan_args = ScanArgs {
|
||||
num_jobs: 2,
|
||||
rules: RuleSpecifierArgs {
|
||||
rules_path: vec![rules_dir.clone()],
|
||||
rule: vec!["all".into()],
|
||||
load_builtins: false,
|
||||
},
|
||||
input_specifier_args: InputSpecifierArgs {
|
||||
path_inputs: vec![inputs_dir.join("a.txt")],
|
||||
git_url: Vec::new(),
|
||||
github_user: Vec::new(),
|
||||
github_organization: Vec::new(),
|
||||
all_github_organizations: false,
|
||||
github_api_url: Url::parse("https://api.github.com/").unwrap(),
|
||||
github_repo_type: GitHubRepoType::Source,
|
||||
gitlab_user: Vec::new(),
|
||||
gitlab_group: Vec::new(),
|
||||
all_gitlab_groups: false,
|
||||
gitlab_api_url: Url::parse("https://gitlab.com/").unwrap(),
|
||||
gitlab_repo_type: GitLabRepoType::Owner,
|
||||
gitlab_include_subgroups: false,
|
||||
jira_url: None,
|
||||
jql: None,
|
||||
confluence_url: None,
|
||||
cql: None,
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
max_results: 100,
|
||||
s3_bucket: None,
|
||||
s3_prefix: None,
|
||||
role_arn: None,
|
||||
aws_local_profile: None,
|
||||
docker_image: Vec::new(),
|
||||
git_clone: GitCloneMode::Bare,
|
||||
git_history: GitHistoryMode::Full,
|
||||
scan_nested_repos: true,
|
||||
commit_metadata: true,
|
||||
},
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
max_file_size_mb: 5.0,
|
||||
exclude: Vec::new(),
|
||||
no_extract_archives: false,
|
||||
extraction_depth: 1,
|
||||
no_binary: true,
|
||||
},
|
||||
confidence: ConfidenceLevel::Low,
|
||||
no_validate: true,
|
||||
rule_stats: false,
|
||||
only_valid: false,
|
||||
min_entropy: Some(0.0),
|
||||
redact: false,
|
||||
git_repo_timeout: 1800,
|
||||
output_args: OutputArgs { output: None, format: ReportOutputFormat::Pretty },
|
||||
no_dedup: false,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: skip_regex,
|
||||
skip_word: skip_skipword,
|
||||
};
|
||||
|
||||
let global_args = GlobalArgs {
|
||||
verbose: 0,
|
||||
quiet: true,
|
||||
color: Mode::Never,
|
||||
progress: Mode::Never,
|
||||
no_update_check: true,
|
||||
self_update: false,
|
||||
ignore_certs: false,
|
||||
advanced: AdvancedArgs { rlimit_nofile: 8192 },
|
||||
};
|
||||
|
||||
let loaded = RuleLoader::from_rule_specifiers(&scan_args.rules).load(&scan_args)?;
|
||||
let resolved = loaded.resolve_enabled_rules()?;
|
||||
let rules_db = Arc::new(RulesDatabase::from_rules(resolved.into_iter().cloned().collect())?);
|
||||
|
||||
let datastore = Arc::new(Mutex::new(FindingsStore::new(work.path().join("store"))));
|
||||
|
||||
rt.block_on(run_async_scan(&global_args, &scan_args, Arc::clone(&datastore), &rules_db))?;
|
||||
|
||||
let x = Ok(datastore.lock().unwrap().get_matches().len()); x
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn skip_regex_filters_match() -> Result<()> {
|
||||
let count = run_skiplist(vec!["token_realvalue".into()], Vec::new())?;
|
||||
assert_eq!(count, 1);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn skip_skipword_filters_match() -> Result<()> {
|
||||
let count = run_skiplist(Vec::new(), vec!["test".into()])?;
|
||||
assert_eq!(count, 1);
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -118,6 +118,8 @@ rules:
|
|||
no_dedup,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
};
|
||||
|
||||
let global_args = GlobalArgs {
|
||||
|
|
|
|||
|
|
@ -105,6 +105,8 @@ fn test_github_remote_scan() -> Result<()> {
|
|||
no_dedup: true,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
};
|
||||
// Create global arguments
|
||||
let global_args = GlobalArgs {
|
||||
|
|
|
|||
|
|
@ -103,6 +103,8 @@ fn test_gitlab_remote_scan() -> Result<()> {
|
|||
no_dedup: true,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
};
|
||||
|
||||
let global_args = GlobalArgs {
|
||||
|
|
@ -207,6 +209,8 @@ fn test_gitlab_remote_scan_no_history() -> Result<()> {
|
|||
no_dedup: true,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
};
|
||||
|
||||
let global_args = GlobalArgs {
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ async fn test_redact_hashes_finding_values() -> Result<()> {
|
|||
no_dedup: true,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
};
|
||||
|
||||
let global_args = GlobalArgs {
|
||||
|
|
|
|||
|
|
@ -41,23 +41,15 @@ fn scan_rules_has_no_validated_findings() -> Result<()> {
|
|||
return Ok(());
|
||||
}
|
||||
|
||||
let groups: Vec<Value> = serde_json::from_str(json_array_str)?;
|
||||
let findings: Vec<Value> = serde_json::from_str(json_array_str)?;
|
||||
for finding in findings {
|
||||
let rule_id = finding["rule"]["id"].as_str().unwrap_or("unknown");
|
||||
|
||||
for group in groups {
|
||||
let rule_id = group["id"].as_str().unwrap_or("unknown");
|
||||
if let Some(matches) = group["matches"].as_array() {
|
||||
for finding in matches {
|
||||
let status = finding["finding"]["validation"]["status"]
|
||||
.as_str()
|
||||
.unwrap_or("")
|
||||
.to_ascii_lowercase();
|
||||
// Fail only on genuinely validated secrets
|
||||
assert_ne!(
|
||||
&status, "active credential",
|
||||
"Validated finding detected in rule {rule_id}"
|
||||
);
|
||||
}
|
||||
}
|
||||
let status =
|
||||
finding["finding"]["validation"]["status"].as_str().unwrap_or("").to_ascii_lowercase();
|
||||
|
||||
// Fail only on genuinely validated secrets
|
||||
assert_ne!(&status, "active credential", "Validated finding detected in rule {rule_id}");
|
||||
}
|
||||
|
||||
Ok(())
|
||||
|
|
|
|||
|
|
@ -91,6 +91,8 @@ impl TestContext {
|
|||
no_dedup: true,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
};
|
||||
|
||||
let loaded = RuleLoader::from_rule_specifiers(&scan_args.rules).load(&scan_args)?;
|
||||
|
|
@ -185,6 +187,8 @@ async fn test_scan_slack_messages() -> Result<()> {
|
|||
no_dedup: true,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
};
|
||||
|
||||
let global_args = GlobalArgs {
|
||||
|
|
|
|||
|
|
@ -161,6 +161,8 @@ async fn test_validation_cache_and_depvars() -> Result<()> {
|
|||
no_dedup: true, // keep duplicates so the cache is stressed
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
};
|
||||
|
||||
/* --------------------------------------------------------- *
|
||||
|
|
|
|||
|
|
@ -104,6 +104,8 @@ impl TestContext {
|
|||
no_dedup: true,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
};
|
||||
|
||||
let loaded = RuleLoader::from_rule_specifiers(&scan_args.rules)
|
||||
|
|
@ -183,6 +185,8 @@ impl TestContext {
|
|||
no_dedup: true,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
skip_regex: Vec::new(),
|
||||
skip_word: Vec::new(),
|
||||
};
|
||||
|
||||
let global_args = GlobalArgs {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue