Commit graph

89 commits

Author SHA1 Message Date
Mick Grove
fa640e2c38 Python bytecode (.pyc) scanning: extracts string constants from compiled Python 2026-02-23 20:06:43 -07:00
Mick Grove
1f4ccb8144 Automatically extracts and scans SQLite database contents for secrets stored in table rows 2026-02-22 23:35:18 -07:00
Mick Grove
7845cfa727 being discovered, overlapping I/O with pattern matching.
- Performance: skip blobs smaller than 20 bytes during enumeration (too small to contain any secret).
- Performance: preserve pack-ascending blob order in the metadata path for better I/O locality when Rayon splits work.
2026-02-22 22:59:42 -07:00
Mick Grove
02538a6ac4 added more access-maps 2026-02-19 20:51:12 -08:00
Mick Grove
05002fe4d6 added more access-maps 2026-02-19 20:39:07 -08:00
Mick Grove
a9c5d8524f added more access-maps 2026-02-19 18:19:20 -08:00
Mick Grove
3b1085baa6 added buildkit and harness to access-map 2026-02-17 22:58:29 -08:00
Mick Grove
39a4e217e3 Kingfisher can now generate an auditor-friendly HTML report 2026-02-15 14:29:42 -08:00
Mick Grove
d3e659491d refactored code 2026-02-14 13:12:26 -08:00
Mick Grove
816d5c40ba wip 1.83 2026-02-13 16:41:28 -08:00
Mick Grove
e72f40b169 Fixed CI runner failure when executing tests 2026-02-12 16:51:55 -08:00
Mick Grove
5882468177 Added optional validation rate limiting via --validation-rps (global) and repeatable --validation-rps-rule <RULE_SELECTOR=RPS> (per-rule override) for both scan and validate. Throttling now applies across built-in validator types (HTTP/gRPC plus AWS, GCP, Coinbase, MongoDB, Postgres, MySQL, JDBC, JWT, and Azure Storage). Rule selectors support the short form (for example, github=2 matches kingfisher.github.*) with longest-prefix precedence when multiple selectors apply. 2026-02-12 12:33:59 -08:00
Mick Grove
65251b7213 more changes for v1.78.0 2026-02-03 09:32:06 -08:00
Mick Grove
5253204c2a preparing for v1.78.0 2026-02-02 23:22:08 -08:00
Mick Grove
63f1d515ae preparing for v1.78.0 2026-02-02 18:39:24 -08:00
Mick Grove
76be1df60c Refactored into multiple crates. Added the 'validate' subcommand 2026-01-28 10:27:24 -08:00
Mick Grove
b4feb86f47 - Fixed validation deduplication for rules with nested unnamed captures (e.g. (?<REGEX>...(ABC|DEF)...)) to use the primary capture for grouping, ensuring each unique match triggers a separate validation request.
- Added trace-level (-vv) logging for internal validation dedup keys and grouping to aid debugging.
2026-01-21 13:13:43 -08:00
Mick Grove
caaa31562c Skipped per-repository report writes when an output file is specified and emit a single aggregated report after multi-repository scans to preserve full output content in files. 2026-01-16 10:03:59 -08:00
Mick Grove
26f41fcf7a - Enhanced Access Map View: added fingerprint display, enabled searching by fingerprint, and implemented bidirectional navigation between Findings and Access Map nodes.
- Added Slack Access Map support with granular permissions in the tree view.
2026-01-14 17:19:02 -08:00
Mick Grove
7237a931d5 v1.73.0 2026-01-01 22:24:57 -08:00
Mick Grove
14d41d560f updated README 2025-12-16 21:13:00 -08:00
Mick Grove
195f086afc added dark mode for finding + access map viewer 2025-12-12 17:21:17 -08:00
Mick Grove
f603c474f6 Updated precommit behavior and docs 2025-12-09 14:56:32 -08:00
Mick Grove
f1a77a736c Updated precommit behavior and docs 2025-12-09 12:56:55 -08:00
Mick Grove
078fa16e6a - Reduced per-match memory usage by compacting stored source locations and interning repeated capture names.
- Stored optional validation response bodies as boxed strings to avoid allocating empty payloads and to streamline validator caches.
- Parallelized git cloning based on the configured job count and begin scanning repositories as soon as each clone finishes to reduce end-to-end scan times.
- Combined per-repository results into a single aggregate summary after scans complete.
- Added initial access-map support and report viewer html file. Currently beta features.
2025-12-04 22:02:30 -08:00
Mick Grove
eeafe2fe6b updated tests 2025-11-24 11:08:31 -08:00
Mick Grove
0acaaa0680 Updated Summary to include scan date, kingfisher version ran, and latest kingfisher version available 2025-11-24 10:36:58 -08:00
Mick Grove
d6c1dfc9d0 updated allocator 2025-11-11 13:24:06 -08:00
Mick Grove
f606f59f93 Added an optional exclude_words list to PatternRequirements so matches containing case-insensitive placeholder words are filtered out, with accompanying tests to cover the new behavior. 2025-11-05 17:19:11 -08:00
Mick Grove
4ee15b9363 - Fixed local filesystem scans to keep open_path_as_is enabled when opening Git repositories and only disable it for diff-based scans.
- Created Linux and Windows specific installer script
- Updated diff-focused scanning so --branch-root-commit can be provided alongside --branch, letting you diff from a chosen commit while targeting a specific branch tip (still defaulting back to the --branch ref when the commit is omitted).
2025-10-25 17:13:16 -07:00
Mick Grove
7d9d3be132 - Fixed local filesystem scans to keep open_path_as_is enabled when opening Git repositories and only disable it for diff-based scans.
- Created Linux and Windows specific installer script
- Updated diff-focused scanning so --branch-root-commit can be provided alongside --branch, letting you diff from a chosen commit while targeting a specific branch tip (still defaulting back to the --branch ref when the commit is omitted).
2025-10-25 17:12:51 -07:00
Mick Grove
9f91cbdab6 added tests for --branch and --since-commit feature 2025-10-23 17:02:31 -07:00
Mick Grove
1b181a368a - Added provider-specific kingfisher scan subcommands (for example kingfisher scan github …) that translate into the legacy flags under the hood. The new layout keeps backwards compatibility while removing the wall of provider options from kingfisher scan --help.
- Updated the README so every provider example (GitHub, GitLab, Bitbucket, Azure Repos, Gitea, Hugging Face, Slack, Jira, Confluence, S3, GCS, Docker) uses the new subcommand style.
- Restored the direct kingfisher scan /path/to/dir flow for local filesystem scans while adding a --list-only switch to each provider subcommand so repository enumeration no longer requires the standalone github repos, gitlab repos, etc. commands.
- Removed the legacy top-level provider commands (kingfisher github, kingfisher gitlab, kingfisher gitea, kingfisher bitbucket, kingfisher azure, kingfisher huggingface) now that enumeration lives under kingfisher scan <provider> --list-only.
- Fixed kingfisher scan github … (and other provider-specific subcommands) so they no longer demand placeholder path arguments before the CLI accepts the request.
- Removed the --bitbucket-username, --bitbucket-token, and --bitbucket-oauth-token flags in favour of KF_BITBUCKET_* environment variables when authenticating to Bitbucket.
2025-10-22 16:24:09 -07:00
amone
47027d2ce5 update: just sort and dedup once 2025-10-21 16:36:05 +08:00
Mick Grove
122885199d - Fixed kingfisher scan so that providing --branch without --since-commit now diffs the branch against the empty tree and scans every commit reachable from that branch.
- Added rules for meraki, duffel, finnhub, frameio, freshbooks, gitter, infracost, launchdarkly, lob, maxmind, messagebird, nytimes, prefect, salingo, sendinblue, sentry, shippo, twitch, typeform
2025-10-20 18:23:12 -07:00
Mick Grove
03d7364888 - Added first-class Hugging Face scanning support, including CLI enumeration, token authentication, and integration with remote scans.
- Condensed GitError formatting to report the exit status and the first informative lines from stdout/stderr, producing concise git clone failure logs.
- Added support for scanning Google Cloud Storage buckets via --gcs-bucket, including optional prefixes and service-account authentication.
- Added --skip-aws-account (now accepting comma-separated values) and --skip-aws-account-file to bypass live AWS validation for known canary/honey-token account IDs without triggering alerts. Kingfisher now ships with several canary AWS account IDs pre-seeded in the skip list and now reports matching findings as "Not Attempted" with the "Response" containing "(skip list entry)" so its clear that validation was intentionally skipped and why.
2025-10-15 22:47:40 -07:00
Mick Grove
3647d759a3 - Added a --no-ignore CLI flag to disable inline directives when you need every potential secret reported
- Added: repeatable --ignore-comment <TOKEN> flag to reuse inline directives from other scanners (for example NOSONAR,
  kics-scan ignore, gitleaks:allow, etc)
2025-10-10 16:23:41 -07:00
Mick Grove
a003b732fa - Added kingfisher:ignore (or kingfisher:allow) to silence a finding inline within a file
- Added: to reuse existing inline directives from other scanners, pass --compat-ignore-comments to also accept NOSONAR, kics-scan ignore,  gitleaks:allow and trufflehog:ignore
2025-10-09 20:11:31 -07:00
Mick Grove
ec1d640b74 Added first-class Azure Repos support, including CLI commands, enumeration, and documentation updates 2025-10-04 23:12:28 -07:00
Mick Grove
d6d854c168 - Improved performance of tree-sitter parsing
- Updated Windows build script to ensure static binary is produced
2025-10-03 17:22:28 -07:00
Mick Grove
6a974907ee Added support for Gitea 2025-09-23 13:07:45 -07:00
Mick Grove
5c70fdc8e5 Added support for BitBucket 2025-09-22 18:21:03 -07:00
Mick Grove
866bf63202 Added diff-only Git scanning via --since-commit and --branch, including remote-aware ref resolution so CI jobs can pair --git-url clones with pull request branches 2025-09-16 14:20:43 -07:00
Mick Grove
563fa66d46 Added --github-exclude and --gitlab-exclude options to skip specific repositories when scanning or listing GitHub and GitLab sources, including support for gitignore-style glob patterns 2025-09-15 21:26:51 -07:00
Mick Grove
ba12a5b2be preparing for v1.48.0 2025-09-05 09:31:52 -07:00
Mick Grove
80aef7e6d7 preparing for v1.48.0 2025-09-05 09:31:43 -07:00
Mick Grove
8d15c8eabf - Improved error message when self-update cannot find the current binary
- Optimized memory usage via string interning and extensive data sharing
- Replaced quadratic match filtering with a per-rule span map, fixing missed secrets in extremely large files and improving scan performance
- Support scanning extremely large files by chunking input into 1 GiB segments with small overlaps, avoiding vectorscan buffer limits while preserving match offsets
- Always use chunked vectorscan, eliminating the slow regex fallback for blobs over 4 GiB
- Skip Base64 scanning for blobs over 64 MB to avoid a second pass over massive files
- Increased max-file-size default to 64 MB (up from 25 MB)
2025-09-04 21:51:24 -07:00
Mick Grove
52b2c02ee9 Optimized memory usage via string interning and extensive data sharing 2025-09-03 09:52:49 -07:00
Mick Grove
c3513ea206 Optimized memory usage via string interning and extensive data sharing 2025-09-02 19:54:44 -07:00
Mick Grove
5638a6cb45 Decode Base64 blobs and scan their contents for secrets while skipping short strings for performance. This has a small performance impact and can be disabled with --no-base64 2025-08-30 19:40:11 -07:00