forked from mirrors/kingfisher
commit
c3e4ec1df3
22 changed files with 462 additions and 92 deletions
1
.github/workflows/release.yml
vendored
1
.github/workflows/release.yml
vendored
|
|
@ -189,7 +189,6 @@ jobs:
|
|||
- name: Cache vcpkg artifacts
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
# Adjust these paths if your vcpkg root is somewhere else
|
||||
path: |
|
||||
C:\vcpkg\buildtrees
|
||||
C:\vcpkg\packages
|
||||
|
|
|
|||
|
|
@ -2,6 +2,9 @@
|
|||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
## [1.28.0]
|
||||
- Added support for scanning Slack
|
||||
|
||||
## [1.27.0]
|
||||
- Added Buildkite rule
|
||||
- Added support for scanning Docker images via `--docker-image`
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ publish = false
|
|||
|
||||
[package]
|
||||
name = "kingfisher"
|
||||
version = "1.27.0"
|
||||
version = "1.28.0"
|
||||
description = "MongoDB's blazingly fast secret scanning and validation tool"
|
||||
edition.workspace = true
|
||||
rust-version.workspace = true
|
||||
|
|
|
|||
78
README.md
78
README.md
|
|
@ -6,31 +6,37 @@
|
|||
[](https://opensource.org/licenses/Apache-2.0)
|
||||
|
||||
Kingfisher is a blazingly fast secret‑scanning and validation tool built in Rust. It combines Intel’s hardware‑accelerated Hyperscan regex engine with language‑aware parsing via Tree‑Sitter, and **ships with hundreds of built‑in rules** to detect, validate, and triage secrets before they ever reach production
|
||||
|
||||
</p>
|
||||
|
||||
Kingfisher originated as a fork of [Nosey Parker](https://github.com/praetorian-inc/noseyparker) by Praetorian Security, Inc, and is built atop their incredible work and the work contributed by the Nosey Parker community.
|
||||
Kingfisher originated as a fork of Praetorian's [Nosey Parker](https://github.com/praetorian-inc/noseyparker), and is built atop their incredible work and the work contributed by the Nosey Parker community.
|
||||
|
||||
Kingfisher extends Nosey Parker by:
|
||||
1. **Validating secrets** in real time via cloud-provider APIs
|
||||
2. Enhancing regex-based detection with **source-code parsing** for improved accuracy
|
||||
3. Adding **GitLab** repository scanning support
|
||||
4. Adding support for scanning **Docker** images via `--docker-image`
|
||||
5. Providing **Jira** scanning capabilities
|
||||
6. Introducing a baseline feature that suppresses known secrets and reports only newly introduced ones
|
||||
7. Offering native **Windows** support
|
||||
## What Kingfisher Adds
|
||||
- **Live validation** via cloud-provider APIs
|
||||
- **Language-aware detection** (source-code parsing) for ~20 languages
|
||||
- **Extra targets**: GitLab repos, Docker images, Jira issues, and Slack messages
|
||||
- **Baseline mode**: ignore known secrets, flag only new ones
|
||||
- **Native Windows** binary
|
||||
|
||||
**MongoDB Blog**: [Introducing Kingfisher: Real-Time Secret Detection and Validation](https://www.mongodb.com/blog/post/product-release-announcements/introducing-kingfisher-real-time-secret-detection-validation)
|
||||
|
||||
## Key Features
|
||||
- **Performance**: multithreaded, Hyperscan‑powered scanning built for huge codebases
|
||||
- **Extensible rules**: hundreds of built-in detectors plus YAML-defined custom rules ([docs/RULES.md](/docs/RULES.md))
|
||||
- **Multiple targets**:
|
||||
- **Git history**: local repos or GitHub/GitLab orgs/users
|
||||
- **Docker images**: public or private via `--docker-image`
|
||||
- **Jira issues**: JQL‑driven scans with `--jira-url` and `--jql`
|
||||
- **Slack messages**: query‑based scans with `--slack-query`
|
||||
- **Baseline management**: generate and track baselines to suppress known secrets ([docs/BASELINE.md](/docs/BASELINE.md))
|
||||
|
||||
- **Performance**: Multi‑threaded, Hyperscan‑powered scanning for massive codebases
|
||||
- **Language‑Aware Accuracy**: AST parsing in 20+ languages via Tree‑Sitter reduces contextless regex matches. see [docs/PARSING.md](/docs/PARSING.md)
|
||||
- **Built-In Validation**: Hundreds of built-in detection rules, many with live-credential validators that call the relevant service APIs (AWS, Azure, GCP, Stripe, etc.) to confirm a secret is active. You can extend or override the library by adding YAML-defined rules on the command line—see [docs/RULES.md](/docs/RULES.md) for details
|
||||
- **Git History Scanning**: Scan local repos, remote GitHub/GitLab orgs/users, or arbitrary GitHub/GitLab repos
|
||||
- **Jira Scanning**: Scan issues returned from a JQL search using `--jira-url` and `--jql`
|
||||
- **Docker Image Scanning**: Scan public or private docker images via `--docker-image`
|
||||
- **Baseline Support:** Generate and manage baseline files to ignore known secrets and report only newly introduced ones. See ([docs/BASELINE.md](docs/BASELINE.md)) for details.
|
||||
**Learn more:** [Introducing Kingfisher: Real‑Time Secret Detection and Validation](https://www.mongodb.com/blog/post/product-release-announcements/introducing-kingfisher-real-time-secret-detection-validation)
|
||||
|
||||
# Benchmark Results
|
||||
|
||||
See ([docs/COMPARISON.md](docs/COMPARISON.md))
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/runtime-comparison.png" alt="Kingfisher Runtime Comparison" style="vertical-align: center;" />
|
||||
</p>
|
||||
|
||||
# Getting Started
|
||||
## Installation
|
||||
|
|
@ -353,7 +359,20 @@ KF_JIRA_TOKEN="token" kingfisher scan \
|
|||
--max-results 1000
|
||||
```
|
||||
---
|
||||
## Scanning Slack
|
||||
|
||||
### Scan Slack messages matching a search query
|
||||
|
||||
```bash
|
||||
KF_SLACK_TOKEN="xoxp-1234..." kingfisher scan \
|
||||
--slack-query "from:username has:link" \
|
||||
--max-results 1000
|
||||
|
||||
KF_SLACK_TOKEN="xoxp-1234..." kingfisher scan \
|
||||
--slack-query "akia" \
|
||||
--max-results 1000
|
||||
```
|
||||
*The Slack token must be a user token with the `search:read` scope. Bot tokens (those beginning with `xoxb-`) cannot call the Slack search API.*
|
||||
|
||||
## Environment Variables for Tokens
|
||||
|
||||
|
|
@ -362,8 +381,8 @@ KF_JIRA_TOKEN="token" kingfisher scan \
|
|||
| `KF_GITHUB_TOKEN` | GitHub Personal Access Token |
|
||||
| `KF_GITLAB_TOKEN` | GitLab Personal Access Token |
|
||||
| `KF_JIRA_TOKEN` | Jira API token |
|
||||
| `KF_SLACK_TOKEN` | Slack API token |
|
||||
| `KF_DOCKER_TOKEN` | Docker registry token (`user:pass` or bearer token). If unset, credentials from the Docker keychain are used |
|
||||
|
||||
Set them temporarily per command:
|
||||
|
||||
```bash
|
||||
|
|
@ -413,15 +432,6 @@ This creates `.git/hooks/pre-commit` that scans the files staged for commit with
|
|||
|
||||
Installs a global pre-commit hook at `$HOME/.git/hooks/pre-commit`; for every Git repository you use, it runs `kingfisher scan --no-update-check` on the staged files and cancels the commit if any secrets are detected.
|
||||
|
||||
To check incoming pushes on a server-side repository, install the pre-receive hook:
|
||||
|
||||
```bash
|
||||
./install-prereceive-hook.sh
|
||||
```
|
||||
|
||||
The resulting `.git/hooks/pre-receive` script scans the files in each pushed commit and rejects the push if any secrets are detected.
|
||||
|
||||
|
||||
## Update Checks
|
||||
|
||||
Kingfisher automatically queries GitHub for a newer release when it starts and tells you whether an update is available.
|
||||
|
|
@ -547,20 +557,10 @@ Real breaches show how one exposed key can snowball into a full-scale incident:
|
|||
|
||||
Leaked secrets fuel unauthorized access, lateral movement, regulatory fines, and brand-damaging incident-response costs.
|
||||
|
||||
# Benchmark Results
|
||||
|
||||
See ([docs/COMPARISON.md](docs/COMPARISON.md))
|
||||
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/runtime-comparison.png" alt="Kingfisher Runtime Comparison" style="vertical-align: center;" />
|
||||
</p>
|
||||
|
||||
|
||||
# Roadmap
|
||||
|
||||
- More rules
|
||||
- Packages for Linux (deb, rpm)
|
||||
- More targets
|
||||
- Please file a [feature request](https://github.com/mongodb/kingfisher/issues) if you have specific features you'd like added
|
||||
|
||||
# License
|
||||
|
|
|
|||
|
|
@ -1,39 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
HOOK_DIR="$(git rev-parse --git-dir)/hooks"
|
||||
HOOK_PATH="$HOOK_DIR/pre-receive"
|
||||
|
||||
if [ -e "$HOOK_PATH" ]; then
|
||||
echo "Error: $HOOK_PATH already exists. Move or remove the existing hook to continue." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cat > "$HOOK_PATH" <<'HOOK'
|
||||
#!/usr/bin/env bash
|
||||
# Pre-receive hook to scan pushed commits with Kingfisher
|
||||
set -euo pipefail
|
||||
|
||||
if ! command -v kingfisher >/dev/null 2>&1; then
|
||||
echo "kingfisher not found in PATH" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
while read -r oldrev newrev refname; do
|
||||
if [ "$oldrev" = "0000000000000000000000000000000000000000" ]; then
|
||||
git diff-tree --name-only -r "$newrev" -z |
|
||||
xargs -0 --no-run-if-empty kingfisher scan --no-update-check
|
||||
else
|
||||
git diff-tree --no-commit-id --name-only -r "$oldrev" "$newrev" -z |
|
||||
xargs -0 --no-run-if-empty kingfisher scan --no-update-check
|
||||
fi
|
||||
status=$?
|
||||
if [ "$status" -ne 0 ]; then
|
||||
echo "Kingfisher detected secrets in push. Push rejected." >&2
|
||||
exit "$status"
|
||||
fi
|
||||
done
|
||||
HOOK
|
||||
|
||||
chmod +x "$HOOK_PATH"
|
||||
echo "Pre-receive hook installed to $HOOK_PATH"
|
||||
|
|
@ -27,7 +27,8 @@ pub struct InputSpecifierArgs {
|
|||
"all_github_organizations",
|
||||
"all_gitlab_groups",
|
||||
"jira_url",
|
||||
"docker_image"
|
||||
"docker_image",
|
||||
"slack_query"
|
||||
]),
|
||||
value_hint = ValueHint::AnyPath
|
||||
)]
|
||||
|
|
@ -94,7 +95,15 @@ pub struct InputSpecifierArgs {
|
|||
#[arg(long, requires = "jira_url")]
|
||||
pub jql: Option<String>,
|
||||
|
||||
/// Maximum number of Jira results to fetch
|
||||
/// Slack search query
|
||||
#[arg(long)]
|
||||
pub slack_query: Option<String>,
|
||||
|
||||
/// Use the specified URL for Slack API access
|
||||
#[arg(long, default_value = "https://slack.com/api/", value_hint = ValueHint::Url)]
|
||||
pub slack_api_url: Url,
|
||||
|
||||
/// Maximum number of Slack or Jira results to fetch
|
||||
#[arg(long, default_value_t = 100)]
|
||||
pub max_results: usize,
|
||||
|
||||
|
|
@ -102,7 +111,6 @@ pub struct InputSpecifierArgs {
|
|||
#[arg(long = "docker-image")]
|
||||
pub docker_image: Vec<String>,
|
||||
|
||||
|
||||
/// Select how to clone Git repositories
|
||||
#[arg(long, default_value_t=GitCloneMode::Bare, alias="git-clone-mode")]
|
||||
pub git_clone: GitCloneMode,
|
||||
|
|
|
|||
|
|
@ -53,6 +53,7 @@ pub struct FindingsStore {
|
|||
blob_meta: FxHashMap<BlobId, Arc<BlobMetadata>>,
|
||||
origin_meta: FxHashMap<u64, Arc<OriginSet>>,
|
||||
docker_images: FxHashMap<PathBuf, String>,
|
||||
slack_links: FxHashMap<PathBuf, String>,
|
||||
}
|
||||
impl FindingsStore {
|
||||
pub fn new(clone_dir: PathBuf) -> Self {
|
||||
|
|
@ -71,6 +72,7 @@ impl FindingsStore {
|
|||
seen_bloom,
|
||||
bloom_items: 0,
|
||||
docker_images: FxHashMap::default(),
|
||||
slack_links: FxHashMap::default(),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -296,6 +298,14 @@ impl FindingsStore {
|
|||
&self.docker_images
|
||||
}
|
||||
|
||||
pub fn register_slack_message(&mut self, path: PathBuf, permalink: String) {
|
||||
self.slack_links.insert(path, permalink);
|
||||
}
|
||||
|
||||
pub fn slack_links(&self) -> &FxHashMap<PathBuf, String> {
|
||||
&self.slack_links
|
||||
}
|
||||
|
||||
pub fn get_finding_data_iter(
|
||||
&self,
|
||||
) -> impl Iterator<Item = finding_data::FindingMetadata> + '_ {
|
||||
|
|
|
|||
|
|
@ -33,6 +33,7 @@ pub mod safe_list;
|
|||
pub mod scanner;
|
||||
pub mod scanner_pool;
|
||||
pub mod serde_utils;
|
||||
pub mod slack;
|
||||
pub mod snippet;
|
||||
pub mod update;
|
||||
pub mod util;
|
||||
|
|
|
|||
|
|
@ -282,6 +282,10 @@ fn create_default_scan_args() -> cli::commands::scan::ScanArgs {
|
|||
jira_url: None,
|
||||
jql: None,
|
||||
max_results: 100,
|
||||
// Slack query
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
|
||||
// Docker image scanning
|
||||
docker_image: Vec::new(),
|
||||
|
||||
|
|
|
|||
|
|
@ -122,13 +122,8 @@ impl DetailsReporter {
|
|||
args: &cli::commands::scan::ScanArgs,
|
||||
) -> Option<String> {
|
||||
// drop any trailing slash so we don’t end up with “//browse/…”
|
||||
let jira_url = args
|
||||
.input_specifier_args
|
||||
.jira_url
|
||||
.as_ref()?
|
||||
.as_str()
|
||||
.trim_end_matches('/');
|
||||
|
||||
let jira_url = args.input_specifier_args.jira_url.as_ref()?.as_str().trim_end_matches('/');
|
||||
|
||||
let ds = self.datastore.lock().ok()?;
|
||||
let root = ds.clone_root();
|
||||
let jira_dir = root.join("jira_issues");
|
||||
|
|
@ -140,6 +135,13 @@ impl DetailsReporter {
|
|||
}
|
||||
}
|
||||
|
||||
/// If the given file path corresponds to a Slack message downloaded to disk,
|
||||
/// return the permalink for that message.
|
||||
fn slack_message_url(&self, path: &std::path::Path) -> Option<String> {
|
||||
let ds = self.datastore.lock().ok()?;
|
||||
ds.slack_links().get(path).cloned()
|
||||
}
|
||||
|
||||
fn docker_display_path(&self, path: &std::path::Path) -> Option<String> {
|
||||
let ds = self.datastore.lock().ok()?;
|
||||
for (dir, image) in ds.docker_images().iter() {
|
||||
|
|
|
|||
|
|
@ -103,6 +103,8 @@ impl DetailsReporter {
|
|||
if let Origin::File(e) = origin {
|
||||
if let Some(url) = self.jira_issue_url(&e.path, args) {
|
||||
Some(url)
|
||||
} else if let Some(url) = self.slack_message_url(&e.path) {
|
||||
Some(url)
|
||||
} else if let Some(mapped) = self.docker_display_path(&e.path) {
|
||||
Some(mapped)
|
||||
} else {
|
||||
|
|
@ -254,6 +256,8 @@ impl DetailsReporter {
|
|||
if let Origin::File(e) = origin {
|
||||
if let Some(url) = self.jira_issue_url(&e.path, args) {
|
||||
Some(url)
|
||||
} else if let Some(url) = self.slack_message_url(&e.path) {
|
||||
Some(url)
|
||||
} else if let Some(mapped) = self.docker_display_path(&e.path) {
|
||||
Some(mapped)
|
||||
} else {
|
||||
|
|
@ -434,6 +438,10 @@ mod tests {
|
|||
jql: None,
|
||||
max_results: 100,
|
||||
// Docker image scanning
|
||||
// Slack options
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
|
||||
docker_image: Vec::new(),
|
||||
// clone / history options
|
||||
git_clone: GitCloneMode::Bare,
|
||||
|
|
|
|||
|
|
@ -216,6 +216,8 @@ impl<'a> Display for PrettyFinding<'a> {
|
|||
Origin::File(e) => {
|
||||
let display_path = if let Some(url) = reporter.jira_issue_url(&e.path, args) {
|
||||
url
|
||||
} else if let Some(url) = reporter.slack_message_url(&e.path) {
|
||||
url
|
||||
} else if let Some(mapped) = reporter.docker_display_path(&e.path) {
|
||||
mapped
|
||||
} else {
|
||||
|
|
@ -347,6 +349,10 @@ fn test_pretty_format_with_nan_entropy_panics() {
|
|||
jira_url: None,
|
||||
jql: None,
|
||||
max_results: 100,
|
||||
|
||||
// Slack options
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
// Docker image scanning
|
||||
docker_image: Vec::new(),
|
||||
// git clone / history options
|
||||
|
|
|
|||
|
|
@ -73,6 +73,8 @@ impl DetailsReporter {
|
|||
Origin::File(e) => {
|
||||
let uri = if let Some(url) = self.jira_issue_url(&e.path, args) {
|
||||
url
|
||||
} else if let Some(url) = self.slack_message_url(&e.path) {
|
||||
url
|
||||
} else {
|
||||
e.path.display().to_string()
|
||||
};
|
||||
|
|
@ -209,6 +211,8 @@ impl DetailsReporter {
|
|||
|
||||
let uri = if let Some(url) = self.jira_issue_url(&e.path, args) {
|
||||
url
|
||||
} else if let Some(url) = self.slack_message_url(&e.path) {
|
||||
url
|
||||
} else {
|
||||
e.path.display().to_string()
|
||||
};
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ use crate::{
|
|||
github, gitlab, jira,
|
||||
matcher::Match,
|
||||
origin::OriginSet,
|
||||
PathBuf,
|
||||
slack, PathBuf,
|
||||
};
|
||||
pub type DatastoreMessage = (OriginSet, BlobMetadata, Vec<(Option<f64>, Match)>);
|
||||
|
||||
|
|
@ -252,4 +252,36 @@ pub async fn fetch_jira_issues(
|
|||
)
|
||||
.await?;
|
||||
Ok(vec![output_dir])
|
||||
}
|
||||
|
||||
pub async fn fetch_slack_messages(
|
||||
args: &scan::ScanArgs,
|
||||
global_args: &global::GlobalArgs,
|
||||
datastore: &Arc<Mutex<findings_store::FindingsStore>>,
|
||||
) -> Result<Vec<PathBuf>> {
|
||||
let Some(query) = args.input_specifier_args.slack_query.as_deref() else {
|
||||
return Ok(Vec::new());
|
||||
};
|
||||
let api_url = args.input_specifier_args.slack_api_url.clone();
|
||||
let max_results = args.input_specifier_args.max_results;
|
||||
let output_root = {
|
||||
let ds = datastore.lock().unwrap();
|
||||
ds.clone_root()
|
||||
};
|
||||
let output_dir = output_root.join("slack_messages");
|
||||
let paths = slack::download_messages_to_dir(
|
||||
api_url,
|
||||
query,
|
||||
max_results,
|
||||
global_args.ignore_certs,
|
||||
&output_dir,
|
||||
)
|
||||
.await?;
|
||||
{
|
||||
let mut ds = datastore.lock().unwrap();
|
||||
for (path, link) in &paths {
|
||||
ds.register_slack_message(path.clone(), link.clone());
|
||||
}
|
||||
}
|
||||
Ok(vec![output_dir])
|
||||
}
|
||||
|
|
@ -18,7 +18,7 @@ use crate::{
|
|||
rules_database::RulesDatabase,
|
||||
scanner::{
|
||||
clone_or_update_git_repos, enumerate_filesystem_inputs, enumerate_github_repos,
|
||||
repos::{enumerate_gitlab_repos, fetch_jira_issues},
|
||||
repos::{enumerate_gitlab_repos, fetch_jira_issues, fetch_slack_messages},
|
||||
run_secret_validation, save_docker_images,
|
||||
summary::print_scan_summary,
|
||||
},
|
||||
|
|
@ -68,6 +68,10 @@ pub async fn run_async_scan(
|
|||
let jira_dirs = fetch_jira_issues(args, global_args, &datastore).await?;
|
||||
input_roots.extend(jira_dirs);
|
||||
|
||||
// Fetch Slack messages if requested
|
||||
let slack_dirs = fetch_slack_messages(args, global_args, &datastore).await?;
|
||||
input_roots.extend(slack_dirs);
|
||||
|
||||
// Save Docker images if specified
|
||||
if !args.input_specifier_args.docker_image.is_empty() {
|
||||
let clone_root = {
|
||||
|
|
|
|||
118
src/slack.rs
Normal file
118
src/slack.rs
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
use anyhow::{Context, Result};
|
||||
use reqwest::Client;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::path::PathBuf;
|
||||
use url::Url;
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize)]
|
||||
pub struct SlackMessage {
|
||||
pub permalink: String,
|
||||
pub text: Option<String>,
|
||||
pub ts: String,
|
||||
pub channel: SlackChannel,
|
||||
}
|
||||
#[derive(Debug, Deserialize, Serialize)]
|
||||
pub struct SlackChannel {
|
||||
pub id: String,
|
||||
pub name: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct SlackPagination {
|
||||
page: Option<u32>,
|
||||
page_count: Option<u32>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct SlackMessages {
|
||||
matches: Vec<SlackMessage>,
|
||||
pagination: Option<SlackPagination>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct SlackSearchResponse {
|
||||
ok: bool,
|
||||
error: Option<String>,
|
||||
messages: Option<SlackMessages>,
|
||||
}
|
||||
|
||||
pub async fn search_messages(
|
||||
api_url: Url,
|
||||
query: &str,
|
||||
max_results: usize,
|
||||
ignore_certs: bool,
|
||||
) -> Result<Vec<SlackMessage>> {
|
||||
let token = std::env::var("KF_SLACK_TOKEN")
|
||||
.context("KF_SLACK_TOKEN environment variable must be set")?;
|
||||
|
||||
let client = Client::builder()
|
||||
.danger_accept_invalid_certs(ignore_certs)
|
||||
.build()
|
||||
.context("Failed to build HTTP client")?;
|
||||
|
||||
let mut page = 1u32;
|
||||
let mut messages = Vec::new();
|
||||
|
||||
loop {
|
||||
let url = api_url.join("search.messages").context("Failed to build Slack API URL")?;
|
||||
|
||||
let resp = client
|
||||
.get(url)
|
||||
.bearer_auth(&token)
|
||||
.query(&[("query", query), ("count", "100"), ("page", &page.to_string())])
|
||||
.send()
|
||||
.await
|
||||
.context("Failed to send Slack request")?;
|
||||
|
||||
let body: SlackSearchResponse =
|
||||
resp.json().await.context("Failed to parse Slack response")?;
|
||||
|
||||
if !body.ok {
|
||||
let err = body.error.unwrap_or_else(|| "unknown".to_string());
|
||||
if err == "not_allowed_token_type" {
|
||||
return Err(anyhow::anyhow!(
|
||||
"Slack API error: not_allowed_token_type - use a user token with the `search:read` scope"
|
||||
));
|
||||
}
|
||||
return Err(anyhow::anyhow!("Slack API error: {}", err));
|
||||
}
|
||||
|
||||
let Some(msgs) = body.messages else {
|
||||
break;
|
||||
};
|
||||
for m in msgs.matches {
|
||||
messages.push(m);
|
||||
if messages.len() >= max_results {
|
||||
return Ok(messages);
|
||||
}
|
||||
}
|
||||
let next_page =
|
||||
msgs.pagination.as_ref().and_then(|p| p.page).map(|p| p + 1).unwrap_or(page + 1);
|
||||
let page_count = msgs.pagination.as_ref().and_then(|p| p.page_count).unwrap_or(next_page);
|
||||
if next_page > page_count {
|
||||
break;
|
||||
}
|
||||
page = next_page;
|
||||
}
|
||||
|
||||
Ok(messages)
|
||||
}
|
||||
|
||||
pub async fn download_messages_to_dir(
|
||||
api_url: Url,
|
||||
query: &str,
|
||||
max_results: usize,
|
||||
ignore_certs: bool,
|
||||
output_dir: &PathBuf,
|
||||
) -> Result<Vec<(PathBuf, String)>> {
|
||||
std::fs::create_dir_all(output_dir)?;
|
||||
let messages = search_messages(api_url, query, max_results, ignore_certs).await?;
|
||||
let mut paths = Vec::new();
|
||||
for msg in messages {
|
||||
let ts = msg.ts.replace('.', "_");
|
||||
let file = output_dir.join(format!("{}_{}.json", msg.channel.id, ts));
|
||||
std::fs::write(&file, serde_json::to_vec(&msg)?)?;
|
||||
paths.push((file, msg.permalink));
|
||||
}
|
||||
Ok(paths)
|
||||
}
|
||||
|
|
@ -82,6 +82,8 @@ rules:
|
|||
jira_url: None,
|
||||
jql: None,
|
||||
max_results: 100,
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
// Docker image scanning
|
||||
docker_image: Vec::new(),
|
||||
// git clone / history options
|
||||
|
|
|
|||
|
|
@ -69,6 +69,8 @@ fn test_github_remote_scan() -> Result<()> {
|
|||
jira_url: None,
|
||||
jql: None,
|
||||
max_results: 100,
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
// Docker image scanning
|
||||
docker_image: Vec::new(),
|
||||
// git clone / history options
|
||||
|
|
|
|||
|
|
@ -68,6 +68,8 @@ fn test_gitlab_remote_scan() -> Result<()> {
|
|||
jira_url: None,
|
||||
jql: None,
|
||||
max_results: 100,
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
// Docker image scanning
|
||||
docker_image: Vec::new(),
|
||||
git_clone: GitCloneMode::Bare,
|
||||
|
|
|
|||
198
tests/int_slack.rs
Normal file
198
tests/int_slack.rs
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
use std::{
|
||||
env,
|
||||
sync::{Arc, Mutex},
|
||||
};
|
||||
|
||||
use anyhow::Result;
|
||||
use kingfisher::{
|
||||
cli::{
|
||||
commands::{
|
||||
github::{GitCloneMode, GitHistoryMode, GitHubRepoType},
|
||||
gitlab::GitLabRepoType,
|
||||
inputs::{ContentFilteringArgs, InputSpecifierArgs},
|
||||
output::{OutputArgs, ReportOutputFormat},
|
||||
rules::RuleSpecifierArgs,
|
||||
scan::{ConfidenceLevel, ScanArgs},
|
||||
},
|
||||
global::{AdvancedArgs, Mode},
|
||||
GlobalArgs,
|
||||
},
|
||||
findings_store::FindingsStore,
|
||||
rule_loader::RuleLoader,
|
||||
rules_database::RulesDatabase,
|
||||
scanner::run_async_scan,
|
||||
};
|
||||
use tempfile::TempDir;
|
||||
use url::Url;
|
||||
use wiremock::{
|
||||
matchers::{method, path},
|
||||
Mock, MockServer, ResponseTemplate,
|
||||
};
|
||||
|
||||
struct TestContext {
|
||||
rules_db: Arc<RulesDatabase>,
|
||||
}
|
||||
|
||||
impl TestContext {
|
||||
fn new() -> Result<Self> {
|
||||
let scan_args = ScanArgs {
|
||||
num_jobs: 2,
|
||||
rules: RuleSpecifierArgs {
|
||||
rules_path: Vec::new(),
|
||||
rule: vec!["all".into()],
|
||||
load_builtins: true,
|
||||
},
|
||||
input_specifier_args: InputSpecifierArgs {
|
||||
path_inputs: Vec::new(),
|
||||
git_url: Vec::new(),
|
||||
github_user: Vec::new(),
|
||||
github_organization: Vec::new(),
|
||||
all_github_organizations: false,
|
||||
github_api_url: Url::parse("https://api.github.com/").unwrap(),
|
||||
github_repo_type: GitHubRepoType::Source,
|
||||
gitlab_user: Vec::new(),
|
||||
gitlab_group: Vec::new(),
|
||||
all_gitlab_groups: false,
|
||||
gitlab_api_url: Url::parse("https://gitlab.com/").unwrap(),
|
||||
gitlab_repo_type: GitLabRepoType::Owner,
|
||||
jira_url: None,
|
||||
jql: None,
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
max_results: 10,
|
||||
docker_image: Vec::new(),
|
||||
git_clone: GitCloneMode::Bare,
|
||||
git_history: GitHistoryMode::Full,
|
||||
scan_nested_repos: true,
|
||||
commit_metadata: true,
|
||||
},
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
max_file_size_mb: 25.0,
|
||||
extraction_depth: 2,
|
||||
no_binary: true,
|
||||
no_extract_archives: false,
|
||||
exclude: Vec::new(),
|
||||
},
|
||||
confidence: ConfidenceLevel::Low,
|
||||
no_validate: true,
|
||||
rule_stats: false,
|
||||
only_valid: false,
|
||||
min_entropy: Some(0.0),
|
||||
redact: false,
|
||||
git_repo_timeout: 1800,
|
||||
output_args: OutputArgs { output: None, format: ReportOutputFormat::Pretty },
|
||||
no_dedup: true,
|
||||
snippet_length: 128,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
};
|
||||
|
||||
let loaded = RuleLoader::from_rule_specifiers(&scan_args.rules).load(&scan_args)?;
|
||||
let resolved = loaded.resolve_enabled_rules()?;
|
||||
let rules_db = RulesDatabase::from_rules(resolved.into_iter().cloned().collect())?;
|
||||
Ok(Self { rules_db: Arc::new(rules_db) })
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_scan_slack_messages() -> Result<()> {
|
||||
let ctx = TestContext::new()?;
|
||||
|
||||
let server = MockServer::start().await;
|
||||
let response = serde_json::json!({
|
||||
"ok": true,
|
||||
"messages": {
|
||||
"matches": [{
|
||||
"permalink": "https://example.slack.com/archives/C123/p1234",
|
||||
"text": "This contains a github token ghp_1wuHFikBKQtCcH3EB2FBUkyn8krXhP2qLqPa",
|
||||
"ts": "1234.56",
|
||||
"channel": {"id": "C123", "name": "general"}
|
||||
}],
|
||||
"pagination": {"page": 1, "page_count": 1}
|
||||
}
|
||||
});
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/search.messages"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(response))
|
||||
.mount(&server)
|
||||
.await;
|
||||
|
||||
env::set_var("KF_SLACK_TOKEN", "xoxp-test");
|
||||
|
||||
let temp_dir = TempDir::new()?;
|
||||
let clone_dir = temp_dir.path().to_path_buf();
|
||||
|
||||
let scan_args = ScanArgs {
|
||||
num_jobs: 2,
|
||||
rules: RuleSpecifierArgs {
|
||||
rules_path: Vec::new(),
|
||||
rule: vec!["all".into()],
|
||||
load_builtins: true,
|
||||
},
|
||||
input_specifier_args: InputSpecifierArgs {
|
||||
path_inputs: Vec::new(),
|
||||
git_url: Vec::new(),
|
||||
github_user: Vec::new(),
|
||||
github_organization: Vec::new(),
|
||||
all_github_organizations: false,
|
||||
github_api_url: Url::parse("https://api.github.com/").unwrap(),
|
||||
github_repo_type: GitHubRepoType::Source,
|
||||
gitlab_user: Vec::new(),
|
||||
gitlab_group: Vec::new(),
|
||||
all_gitlab_groups: false,
|
||||
gitlab_api_url: Url::parse("https://gitlab.com/").unwrap(),
|
||||
gitlab_repo_type: GitLabRepoType::Owner,
|
||||
jira_url: None,
|
||||
jql: None,
|
||||
slack_query: Some("test".into()),
|
||||
slack_api_url: Url::parse(&format!("{}/", server.uri()))?,
|
||||
max_results: 10,
|
||||
docker_image: Vec::new(),
|
||||
git_clone: GitCloneMode::Bare,
|
||||
git_history: GitHistoryMode::Full,
|
||||
scan_nested_repos: true,
|
||||
commit_metadata: true,
|
||||
},
|
||||
content_filtering_args: ContentFilteringArgs {
|
||||
max_file_size_mb: 25.0,
|
||||
extraction_depth: 2,
|
||||
no_binary: true,
|
||||
no_extract_archives: false,
|
||||
exclude: Vec::new(),
|
||||
},
|
||||
confidence: ConfidenceLevel::Low,
|
||||
no_validate: true,
|
||||
rule_stats: false,
|
||||
only_valid: false,
|
||||
min_entropy: Some(0.0),
|
||||
redact: false,
|
||||
git_repo_timeout: 1800,
|
||||
output_args: OutputArgs { output: None, format: ReportOutputFormat::Pretty },
|
||||
no_dedup: true,
|
||||
snippet_length: 128,
|
||||
baseline_file: None,
|
||||
manage_baseline: false,
|
||||
};
|
||||
|
||||
let global_args = GlobalArgs {
|
||||
verbose: 0,
|
||||
quiet: true,
|
||||
color: Mode::Auto,
|
||||
no_update_check: false,
|
||||
self_update: false,
|
||||
progress: Mode::Never,
|
||||
ignore_certs: false,
|
||||
advanced: AdvancedArgs { rlimit_nofile: 16384 },
|
||||
};
|
||||
|
||||
let datastore = Arc::new(Mutex::new(FindingsStore::new(clone_dir)));
|
||||
|
||||
run_async_scan(&global_args, &scan_args, Arc::clone(&datastore), &ctx.rules_db).await?;
|
||||
|
||||
let findings = {
|
||||
let ds = datastore.lock().unwrap();
|
||||
ds.get_matches().len()
|
||||
};
|
||||
assert!(findings > 0);
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -125,6 +125,8 @@ async fn test_validation_cache_and_depvars() -> Result<()> {
|
|||
jira_url: None,
|
||||
jql: None,
|
||||
max_results: 100,
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
// Docker image scanning
|
||||
docker_image: Vec::new(),
|
||||
// git clone / history options
|
||||
|
|
|
|||
|
|
@ -68,6 +68,8 @@ impl TestContext {
|
|||
jira_url: None,
|
||||
jql: None,
|
||||
max_results: 100,
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
// Docker image scanning
|
||||
docker_image: Vec::new(),
|
||||
// git clone / history options
|
||||
|
|
@ -138,6 +140,8 @@ impl TestContext {
|
|||
jira_url: None,
|
||||
jql: None,
|
||||
max_results: 100,
|
||||
slack_query: None,
|
||||
slack_api_url: Url::parse("https://slack.com/api/").unwrap(),
|
||||
// Docker image scanning
|
||||
docker_image: Vec::new(),
|
||||
// git clone / history options
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue