eblume/kingfisher
1
0
Fork 0
forked from mirrors/kingfisher
Code Pull requests Activity Actions

Merge pull request #328 from mongodb/development

Browse source
This commit is contained in:
Mick Grove 2026-04-08 22:55:28 -07:00 • committed by GitHub
commit ef3e72c0e2
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
138 changed files with 8784 additions and 2450 deletions
Download patch file Download diff file Expand all files Collapse all files

6
.github/workflows/docs.yml vendored
View file

@ -27,7 +27,7 @@ jobs:
- uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.12'
@ -46,7 +46,7 @@ jobs:
CI: true
- name: Upload artifact
uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3.0.1
uses: actions/upload-pages-artifact@7b1f4a764d45c48632c6b24a0339c27f5614fb0b # v4.0.0
with:
path: docs-site/site
@ -59,4 +59,4 @@ jobs:
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4.0.5
uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0

26
.github/workflows/release.yml vendored
View file

@ -411,7 +411,7 @@ jobs:
provenance:
name: Generate SLSA provenance
needs: [hash]
needs: [hash, release]
permissions:
actions: read
id-token: write
@ -419,28 +419,8 @@ jobs:
uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@f7dd8c54c2067bafc12ca7a55595d5ee9b75204a # v2.1.0
with:
base64-subjects: "${{ needs.hash.outputs.hashes }}"
upload-assets: false
upload-provenance:
name: Upload provenance to release
needs: [provenance, release]
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Download provenance artifact
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: ${{ needs.provenance.outputs.provenance-name }}
- name: Upload to release
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ needs.release.outputs.tag }}
PROVENANCE_FILE: ${{ needs.provenance.outputs.provenance-name }}
run: |
gh release upload "${TAG}" "${PROVENANCE_FILE}" \
--repo "${{ github.repository }}" \
--clobber
upload-assets: true
upload-tag-name: "${{ needs.release.outputs.tag }}"
# ──────────────── Publish Docker image ────────────────
publish-docker:

5
.gitignore vendored
View file

@ -7,6 +7,8 @@
*.json
!webserver/static/sample-report.json
!docs/access-map-viewer/sample-report.json
!testdata/parsers/context_verifier_golden.json
!testdata/parsers/scan_findings_baseline.json
!testdata/parsers/tree_sitter_capture_baseline.json
*.jsonl
*.bson
@ -17,7 +19,10 @@ logs/*
*.orig
*.rej
*.html
!testdata/html_vulnerable.html
!testdata/html_embedded_vulnerable.html
!docs/access-map-viewer/index.html
!docs-site/overrides/*.html
*.dot
fuzz/*
!fuzz/Cargo.toml

17
AGENTS.md
View file

@ -22,16 +22,16 @@ Key capabilities:
## Repository Structure
- `src/`: main binary source
- `src/cli/commands/`: CLI command implementations
- `src/validation/`: provider-specific credential validators
- `src/matcher/`: pattern matching engine
- `src/scanner/`: core scanning logic
- `src/parser/`: language-aware parsing (`tree-sitter`)
- `src/parser/`: language-aware context verification (lightweight lexers, `tl` for HTML, `cssparser` for CSS)
- `src/reporter/`: TOON/JSON/SARIF/HTML report generation
- `src/access_map/`: access mapping analysis
- `crates/kingfisher-core/`: shared types and core logic
- `crates/kingfisher-rules/`: rule loading and rule data
- `crates/kingfisher-rules/data/rules/`: YAML detection rules
- `crates/kingfisher-scanner/`: embeddable high-level scanning API
- `crates/kingfisher-scanner/src/validation/`: shared typed and raw credential validators
- `tests/`: integration/e2e tests
- `testdata/`: test fixtures
- `docs/`: user and developer docs
@ -81,18 +81,21 @@ Key capabilities:
- `use-mimalloc` (default)
- `use-jemalloc`
- `system-alloc`
- Validation modules live in `crates/kingfisher-scanner/src/validation/`; optional validation feature sets are defined in `crates/kingfisher-scanner/Cargo.toml` (e.g., `validation-aws`, `validation-gcp`, `validation-database`, `validation-all`).
- Validation modules live in `crates/kingfisher-scanner/src/validation/`; optional validation feature sets are defined in `crates/kingfisher-scanner/Cargo.toml` (e.g., `validation-raw`, `validation-aws`, `validation-gcp`, `validation-database`, `validation-all`).
## Validation and Revocation Policy
- Default rule: define validation logic in rule YAML (`validation:` block), not Rust code.
- Code-based validation in `crates/kingfisher-scanner/src/validation/` is an exception path for cases that cannot be expressed reliably in YAML alone (for example AWS, GCP, Coinbase, MongoDB, and similar complex/provider-specific flows).
- Default rule: define validation logic in rule YAML (`validation:` block), especially `Http` or `Grpc`, not Rust code.
- Typed validators are first-class schema variants (`AWS`, `AzureStorage`, `Coinbase`, `GCP`, `MongoDB`, `MySQL`, `Postgres`, `Jdbc`, `JWT`) for stable, reusable validation families.
- Raw validators use `validation: { type: Raw, content: <name> }` and are the ad-hoc exception path for provider-specific or protocol-specific validation that cannot be expressed reliably in YAML alone. Implement them in `crates/kingfisher-scanner/src/validation/raw.rs`.
- Treat Rust validation additions as rare; prefer extending YAML-based validation first.
- If a Rust exception path is required, prefer adding a raw validator before introducing a new typed validator. Add a new typed validator only when it represents a reusable schema-level validation family.
- Do not convert existing typed validators to `Raw` just for consistency.
- For rules that include validation, add a `revocation:` section whenever the third-party API safely supports revocation.
## Common Development Tasks
- Add a detection rule: follow the workflow below and validate with relevant tests.
- Add a CLI command: implement under `src/cli/commands/` and register in the CLI command wiring.
- Add a validator (rare exception path): implement in `crates/kingfisher-scanner/src/validation/` and wire feature flags/dependencies in `crates/kingfisher-scanner/Cargo.toml` only when YAML validation cannot express the required logic.
- Add a validator (rare exception path): implement it in `crates/kingfisher-scanner/src/validation/`, prefer `raw.rs` for one-off provider flows, and wire the narrowest feature/dependencies in `crates/kingfisher-scanner/Cargo.toml` only when YAML validation cannot express the required logic.
## Rule Authoring Workflow
Use this when creating or updating rules in `crates/kingfisher-rules/data/rules/`.
@ -105,7 +108,7 @@ Use this when creating or updating rules in `crates/kingfisher-rules/data/rules/
- `pattern_requirements` (e.g., `min_digits`, `min_uppercase`, `min_lowercase`, `min_special_chars`, `ignore_if_contains`) when format constraints are known.
- `pattern_requirements.checksum` when provider formats include check digits/signatures.
5. Add `validation` only when a reliable provider/API check exists.
6. Put validation in YAML by default; only use Rust validator logic for rare, justified exceptions.
6. Put validation in YAML by default. If YAML cannot express the check, use an existing typed validator or `type: Raw` exception path; add new Rust validator logic only for rare, justified cases.
7. Add `revocation` when the provider API supports safe revocation and the flow is well understood.
8. If a rule needs context from another match (for example ID + secret pair), use `depends_on_rule` and consider `visible: false` on the helper rule.
9. Verify locally:

6
CHANGELOG.md
View file

@ -2,6 +2,12 @@
All notable changes to this project will be documented in this file.
## [v1.95.0]
- Added 80+ built-in rules, bringing the bundled ruleset to 820 total. New coverage includes Amazon OAuth, Asaas, multiple Azure credential families, Bitrise, Canva, CockroachDB, eBay, Elastic, hCaptcha, Highnote, Lichess, MailerSend, Onfido, Paddle, Pangea, Persona, Pinterest, Proof, Rootly, Runpod, Telnyx, Thunderstore, Valtown, Volcengine, and more.
- Replaced tree-sitter with a lighter parser-based context verifier built from handwritten lexers plus `tl`/`cssparser`, preserving context-dependent matching while cutting about 19 MB from the release binary.
- Added a `validation: type: Raw` exception path for provider-specific checks, with new raw validators for Azure Batch, FTP, Kraken, LDAP, RabbitMQ, and Redis. Also added stable request-scoped template values plus new Liquid filters for HMAC-SHA384 hex output and timestamp generation.
- Expanded live validation coverage for several built-in rules, including Agora, Bitfinex, DocuSign, Dwolla, GitLab, KuCoin, RingCentral, Snowflake, Tableau, Trello, and Webex. Also tightened newly added helper regex to avoid high-match scan regressions, and made preflight-blocked raw validations report as skipped/not attempted instead of failed.
## [v1.94.0]
- Updated vendored `vectorscan-rs` from v0.0.5 (Vectorscan 5.4.11) to v0.0.6 (Vectorscan 5.4.12). The upstream crate now ships pre-extracted sources instead of a tarball+patch, and fixes the `cpu_native` feature flag. Local Windows and musl build patches have been re-applied.
- Added more built-in rules

718
Cargo.lock generated
View file

File diff suppressed because it is too large Load diff

27
Cargo.toml
View file

@ -48,7 +48,7 @@ http = "1.4"
[package]
name = "kingfisher"
version = "1.94.0"
version = "1.95.0"
description = "MongoDB's blazingly fast and accurate secret scanning and validation tool"
edition.workspace = true
rust-version.workspace = true
@ -168,28 +168,11 @@ reqwest-middleware = "0.5.1"
reqwest-middleware-octorust = { package = "reqwest-middleware", version = "0.4.2" }
tracing-subscriber = {version = "0.3.22", features = ["env-filter"] }
tracing-core = "0.1.35"
tree-sitter = "0.26.5"
aws-smithy-http-client = "1.1.10"
aws-smithy-runtime-api = "1.11.4"
aws-smithy-types = "1.4.4"
tree-sitter-bash = "0.25.1"
tree-sitter-c = "0.24.1"
tree-sitter-c-sharp = "0.23.1"
tree-sitter-cpp = "0.23.4"
tree-sitter-css = "0.25.0"
tree-sitter-go = "0.25.0"
tree-sitter-html = "0.23.2"
tree-sitter-java = "0.23.5"
tree-sitter-javascript = "0.25.0"
tree-sitter-php = "0.24.2"
tree-sitter-python = "0.25.0"
tree-sitter-ruby = "0.23.1"
tree-sitter-rust = "0.24.0"
tree-sitter-toml-ng = "0.7.0"
tree-sitter-typescript = "0.23.2"
tree-sitter-yaml = "0.7.2"
streaming-iterator = "0.1.9"
tree-sitter-regex = "0.25.0"
cssparser = { version = "0.37.0", default-features = false }
tl = "0.7.8"
tree_magic_mini = "3.2"
content_inspector = "0.2.4"
rustc-hash = "2.1.1"
@ -223,10 +206,10 @@ bloomfilter = "3.0.1"
uuid = "1.19.0"
rand = "0.10.0"
percent-encoding = "2.3.2"
self_update = { version = "0.43.1", default-features = false, features = ["reqwest", "rustls", "archive-tar", "archive-zip", "compression-flate2"] }
self_update = { version = "0.44.0", default-features = false, features = ["reqwest", "rustls", "archive-tar", "archive-zip", "compression-flate2"] }
semver = "1.0.27"
globset = "0.4.18"
jsonwebtoken = { version = "10.2.0", features = ["aws-lc-rs"] }
jsonwebtoken = { version = "10.3.0", features = ["aws-lc-rs"] }
ipnet = "2.11.0"
gouqi = { version = "0.20.0", features = ["async"] }
oci-client = { version = "0.16", default-features = false, features = ["rustls-tls"] }

20
README.md
View file

@ -7,7 +7,7 @@
<img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License" style="height: 24px;" />
</a>
<a href="https://github.com/mongodb/kingfisher">
<img src="https://img.shields.io/badge/Detection%20Rules-734-2ea043.svg" alt="Detection Rules" style="height: 24px;" />
<img src="https://img.shields.io/badge/Detection%20Rules-821-2ea043.svg" alt="Detection Rules" style="height: 24px;" />
</a>
<br>
<a href="https://github.com/mongodb/kingfisher/pkgs/container/kingfisher">
@ -17,7 +17,7 @@
Kingfisher is an open source secret scanner and **live secret validation** tool built in Rust.
It combines Intel's SIMD-accelerated regex engine (Hyperscan) with language-aware parsing to achieve high accuracy at massive scale, and **ships with 700+ built-in rules** to detect, **validate**, and triage leaked API keys, tokens, and credentials before they ever reach production.
It combines Intel's SIMD-accelerated regex engine (Hyperscan) with language-aware parsing to achieve high accuracy at massive scale, and **ships with 800+ built-in rules** to detect, **validate**, and triage leaked API keys, tokens, and credentials before they ever reach production.
Designed for offensive security engineers and blue-team defenders alike, Kingfisher helps you scan repositories, cloud storage, chat, docs, and CI pipelines to find and verify exposed secrets quickly.
@ -49,9 +49,9 @@ Kingfisher is a high-performance, open source secret detection tool for source c
</div>
### Performance, Accuracy, and 700+ Rules
### Performance, Accuracy, and 800+ Rules
- **Performance**: multithreaded, Hyperscan‑powered scanning built for huge codebases
- **Extensible rules**: 700+ built-in rules plus YAML-defined custom rules ([docs/RULES.md](/docs/RULES.md))
- **Extensible rules**: 800+ built-in rules plus YAML-defined custom rules ([docs/RULES.md](/docs/RULES.md))
- **Validate & Revoke**: live validation of discovered secrets, plus direct revocation for supported platforms (GitHub, GitLab, Slack, AWS, GCP, and more) ([docs/USAGE.md](/docs/USAGE.md))
- **Revocation support matrix**: current built-in revocation coverage across providers and rule IDs ([docs/REVOCATION_PROVIDERS.md](/docs/REVOCATION_PROVIDERS.md))
- **Blast Radius Mapping**: instantly map leaked keys to their effective cloud identities and exposed resources with `--access-map`. Supports 39 providers (see table below).
@ -345,7 +345,7 @@ gh attestation verify kingfisher-linux-x64.tgz --repo mongodb/kingfisher
# Detection Rules
Kingfisher ships with [700+ built-in rules](crates/kingfisher-rules/data/rules/) covering cloud keys, AI tokens, CI/CD secrets, database credentials, and SaaS API keys. Below is an overview — see the full list in [crates/kingfisher-rules/data/rules/](crates/kingfisher-rules/data/rules/):
Kingfisher ships with [800+ built-in rules](crates/kingfisher-rules/data/rules/) covering cloud keys, AI tokens, CI/CD secrets, database credentials, and SaaS API keys. Below is an overview — see the full list in [crates/kingfisher-rules/data/rules/](crates/kingfisher-rules/data/rules/):
| Category | What we catch |
|----------|---------------|
@ -362,7 +362,7 @@ Kingfisher ships with [700+ built-in rules](crates/kingfisher-rules/data/rules/)
## Write Custom Rules
Kingfisher ships with 700+ rules with HTTP and service‑specific validation checks (AWS, Azure, GCP, etc.) to confirm if a detected string is a live credential.
Kingfisher ships with 800+ rules with HTTP and service‑specific validation checks (AWS, Azure, GCP, etc.) to confirm if a detected string is a live credential.
However, you may want to add your own custom rules, or modify a detection to better suit your needs / environment.
@ -401,7 +401,7 @@ kingfisher scan /path/to/code
kingfisher scan ~/src/myrepo --no-validate
# Turbo mode: run as fast as possible by disabling Git commit metadata, Base64 decoding,
# MIME sniffing, language detection, and tree-sitter parsing
# MIME sniffing, language detection, and parser-based context verification
# (findings omit commit context, Base64-only matches, MIME type, and language metadata)
kingfisher scan ~/src/myrepo --turbo
@ -510,7 +510,7 @@ cat /path/to/file.py | kingfisher scan -
kingfisher scan /some/file --max-file-size 500
# Turbo mode: equivalent to --commit-metadata=false --no-base64 and disables MIME sniffing,
# language detection/tree-sitter parsing for maximum speed
# language detection/parser-based context verification for maximum speed
# No Git commit metadata (author, date, hash), Base64 decoding, MIME, or language metadata in findings
kingfisher scan /path/to/repo --turbo
@ -725,7 +725,7 @@ kingfisher scan /tmp/repo --branch feature-1 \
| [FINGERPRINT.md](docs/FINGERPRINT.md) | Understanding finding fingerprints and deduplication |
| [COMPARISON.md](docs/COMPARISON.md) | Benchmark results and performance comparisons |
| [PARSING.md](docs/PARSING.md) | Language-aware parsing details |
| [TREE_SITTER.md](docs/TREE_SITTER.md) | Tree-sitter scanning flow, verification gates, and fallback behavior |
| [CONTEXT_VERIFICATION.md](docs/CONTEXT_VERIFICATION.md) | Context-verification flow, gates, and parser backends |
# Library Usage
@ -751,7 +751,7 @@ Since then it has evolved far beyond that starting point, introducing live valid
- **Live validation** of detected secrets directly within rules
- **Hundreds of new built-in rules** and an expanded YAML rule schema
- **Baseline management** to suppress known findings over time
- **Tree-sitter parsing** layered on Hyperscan for language-aware detection
- **Parser-based context verification** layered on Hyperscan for language-aware detection
- **More scan targets** (GitLab, Bitbucket, Gitea, Jira, Confluence, Slack, Microsoft Teams, S3, GCS, Docker, Hugging Face, etc.)
- **Compressed Files**, **SQLite database**, and **Python bytecode (.pyc)** scanning support
- **New storage model** (in-memory + Bloom filter, replacing SQLite)

11
crates/kingfisher-rules/data/rules/AGENTS.md
View file

@ -30,6 +30,7 @@ Strongly recommended fields:
## Pattern Quality Rules
- Prefer specific anchors/prefixes and provider context over broad generic regex.
- Keep helper/context regex narrow. Avoid patterns that match generic URLs, hostnames, query params, or assignments without strong provider-specific constraints; broad helpers can create huge match counts and cause major memory/time regressions on large repos and git history.
- When the token format is generic or common-looking (for example bare 32-hex keys), prefer contextual patterns of the form: provider keyword -> short flexible gap -> key/secret label -> short flexible gap -> token. A good default is:
- `\b`
- provider identifier (for example `amplitude`, `azure`, `speech`, `translator`)
@ -57,8 +58,11 @@ Strongly recommended fields:
## Validation Policy (Important)
- Default: define validation logic in YAML under `validation:`.
- Do not move validation logic into Rust unless YAML cannot reliably express it.
- Code-backed validation types (for example AWS, GCP, Coinbase, MongoDB) are notable exceptions and should remain rare.
- For new rules, first attempt `Http`/`Grpc` YAML validation before considering exception paths.
- Typed validation kinds such as `AWS`, `AzureStorage`, `Coinbase`, `GCP`, `MongoDB`, `MySQL`, `Postgres`, `Jdbc`, and `JWT` are schema-level validator families. Use them when an existing typed validator already matches the problem.
- `validation: { type: Raw, content: <name> }` is the ad-hoc exception path for provider-specific or protocol-specific flows that cannot be expressed cleanly in YAML. Raw implementations live in `crates/kingfisher-scanner/src/validation/raw.rs`.
- When Rust validation is unavoidable for a one-off provider, prefer adding a raw validator instead of inventing a new typed validator.
- Do not convert existing typed validators to `Raw` just for consistency.
## Revocation Policy
- If a rule has validation and the provider API safely supports revocation, add `revocation:` in the same YAML rule.
@ -70,7 +74,7 @@ Strongly recommended fields:
1. Choose the target provider file (or add a new provider file if no suitable file exists).
2. Copy a structurally similar rule from this directory.
3. Implement/adjust `pattern`, `examples`, and filtering (`pattern_requirements`, `min_entropy`).
4. Add YAML `validation` (default path).
4. Add YAML `validation` (default path). Prefer `Http`/`Grpc`; if that fails, use an existing typed validator or `type: Raw` only when justified.
5. Add YAML `revocation` when supported.
6. Add `references` for token format/API behavior.
7. Verify locally (below).
@ -80,6 +84,9 @@ Strongly recommended fields:
- `cargo test -p kingfisher-rules`
- Broader regression check:
- `cargo test --workspace --all-targets`
- Match-volume check on a realistic large target:
- `kingfisher scan <large-repo-or-test-corpus> --rule-stats`
- Review unexpected high-match helper/generic rules before submitting.
- **Warning-free build**: `cargo check` (or `make darwin` / `make linux`) must produce zero warnings. Address all `dead_code`, `unused_*`, and other warnings before submitting. Use `#[allow(dead_code)]` on individual struct fields kept for deserialization completeness, and remove truly unused code.
- Behavioral check against sample content:
- `kingfisher scan ./testdata --rule <rule-family-or-id> --rule-stats`

59
crates/kingfisher-rules/data/rules/adobe.yml
View file

@ -70,7 +70,58 @@ rules:
examples:
- |
{
"client_credentials": {
"client_id": "a65b0146769d433a835f36660881db50",
"client_secret": "p8e-ibndcvsmAp9ZgPBZ606FSlYIZVlsZ-g5"
},
"adobe_client_credentials": {
"client_id": "a65b0146769d433a835f36660881db50",
"client_secret": "p8e-ibndcvsmAp9ZgPBZ606FSlYIZVlsZ-g5"
}
}
depends_on_rule:
- rule_id: "kingfisher.adobe.4"
variable: ADOBE_CLIENT_ID
validation:
type: Http
content:
request:
method: POST
url: https://ims-na1.adobelogin.com/ims/token/v3
headers:
Authorization: 'Basic {{ ADOBE_CLIENT_ID | append: ":" | append: TOKEN | b64enc }}'
Content-Type: application/x-www-form-urlencoded
Accept: application/json
body: 'code=invalid_code&grant_type=authorization_code'
response_matcher:
- report_response: true
- type: StatusMatch
status: [400]
- type: WordMatch
words:
- invalid_client
negative: true
# Revocation not added: Adobe documents revocation for access and refresh
# tokens, not for the OAuth client secret itself.
references:
- https://developer.adobe.com/developer-console/docs/guides/authentication/UserAuthentication/ims
- name: Adobe OAuth Client ID
id: kingfisher.adobe.4
pattern: |
(?xi)
\b
adobe
(?:.|[\n\r]){0,64}?
client_id
(?:.|[\n\r]){0,16}?
(
[a-f0-9]{32}
)
\b
min_entropy: 3.0
visible: false
examples:
- |
{
"adobe_client_credentials": {
"client_id": "a65b0146769d433a835f36660881db50",
"client_secret": "p8e-ibndcvsmAp9ZgPBZ606FSlYIZVlsZ-g5"
}
}

24
crates/kingfisher-rules/data/rules/agora.yml
View file

@ -47,6 +47,28 @@ rules:
examples:
- "agora.app_certificate=397a3af3db1950bdbd84f4e4ec18ebef"
- "agora.app_secret = \"127a3af3db1950b8dbd4fe440c28ebef\""
validation:
type: Http
content:
request:
method: GET
url: https://api.agora.io/dev/v1/projects
headers:
Accept: application/json
Authorization: "Basic {{ AGORA_ID | append: ':' | append: TOKEN | b64enc }}"
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: false
words:
- '"projects"'
- '"vendor_key"'
depends_on_rule:
- rule_id: kingfisher.agora.1
variable: AGORA_ID
references:
- https://docs.agora.io/en/voice-calling/reference/agora-console-rest-api
- https://docs.agora.io/en/rtc/restfulapi
- https://docs.agora.io/en/video-calling/reference/authentication-workflow

19
crates/kingfisher-rules/data/rules/amazonoauth.yml Normal file
View file

@ -0,0 +1,19 @@
rules:
- name: Login with Amazon OAuth Client ID
id: kingfisher.amazonoauth.1
pattern: |
(?x)
\b
(
amzn1\.application-oa2-client\.[a-f0-9]{20,40}
)
\b
pattern_requirements:
min_digits: 4
min_entropy: 3.0
confidence: high
categories: [api, key]
examples:
- 'AMAZON_CLIENT_ID=amzn1.application-oa2-client.1a2b3c4d5e6f7890abcdef1234567890'
references:
- https://developer.amazon.com/docs/login-with-amazon/authorization-code-grant.html

47
crates/kingfisher-rules/data/rules/asaas.yml Normal file
View file

@ -0,0 +1,47 @@
rules:
- name: Asaas API Token
id: kingfisher.asaas.1
pattern: |
(?x)
(
\$aact_(?:prod|hmlg)_[a-zA-Z0-9_-]{20,100}
)
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'ASAAS_API_KEY=$aact_prod_abcdefghijklmnop1234567890ABCDEF'
- 'api_token: $aact_hmlg_abcdefghijklmnop1234567890ABCDEF'
validation:
type: Http
content:
request:
method: GET
url: >
{%- if TOKEN contains "$aact_hmlg_" -%}
https://api-sandbox.asaas.com/v3/myAccount/commercialInfo/
{%- else -%}
https://api.asaas.com/v3/myAccount/commercialInfo/
{%- endif -%}
headers:
Accept: application/json
User-Agent: kingfisher
access_token: "{{ TOKEN }}"
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: true
words:
- '"object"'
- '"commercialInfo"'
# Revocation not added: Asaas documents key deletion in the dashboard and
# parent-driven sub-account key management, but not a self-revoke endpoint
# for the current access_token alone.
references:
- https://docs.asaas.com/docs/authentication-2
- https://docs.asaas.com/docs/change-the-name-of-a-business-subaccount-via-api

26
crates/kingfisher-rules/data/rules/asana.yml
View file

@ -41,6 +41,32 @@ rules:
examples:
- "asana :'20c2F0d03201af478ca1aBE9515A1A4FEfb'"
- ASANA_PAT = 1234567890abcdef1234567890abcdef12
depends_on_rule:
- rule_id: kingfisher.asana.1
variable: ASANA_CLIENT_ID
validation:
type: Http
content:
request:
method: POST
url: https://app.asana.com/-/oauth_token
headers:
Content-Type: application/x-www-form-urlencoded
Accept: application/json
body: >
grant_type=authorization_code&client_id={{ ASANA_CLIENT_ID | url_encode }}&client_secret={{ TOKEN | url_encode }}&redirect_uri={{ "https://example.com/oauth/callback" | url_encode }}&code=invalid_code
response_matcher:
- report_response: true
- type: StatusMatch
status: [400]
- type: WordMatch
words:
- invalid_client
negative: true
# Revocation not added: Asana's revoke endpoint deauthorizes refresh tokens,
# not OAuth client secrets.
references:
- https://developers.asana.com/docs/oauth
- name: Asana OAuth / Personal Access Token (Legacy)
id: kingfisher.asana.3

46
crates/kingfisher-rules/data/rules/azure.yml
View file

@ -70,6 +70,30 @@ rules:
- |
if __name__ == "__main__":
ado_pat = "iyfmob6xjrfmit67anxbot64umfx2clwx7dz5ynxi4q2z3uqegvq"
validation:
type: Http
content:
request:
method: GET
url: https://app.vssps.visualstudio.com/_apis/profile/profiles/me?api-version=7.1
headers:
Authorization: 'Basic {{ ":" | append: TOKEN | b64enc }}'
Accept: application/json
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: true
words:
- '"id"'
- '"displayName"'
# Revocation not added: Azure DevOps PAT lifecycle management is documented
# separately and is not a self-revoke flow driven solely by the PAT itself.
references:
- https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops
- https://learn.microsoft.com/en-us/rest/api/azure/devops/profile/profiles/get?view=azure-devops-rest-7.1
- name: Azure Container Registry URL
id: kingfisher.azure.4
pattern: |
@ -114,3 +138,25 @@ rules:
variable: ACR_USERNAME
references:
- https://learn.microsoft.com/en-us/azure/container-registry/container-registry-authentication
- name: Azure AD Client Secret (Microsoft Entra ID)
id: kingfisher.azure.6
pattern: |
(?x)
(?:^|['"\x60\s>=:(,])
(
[a-zA-Z0-9_~.]{3}
\d
Q~
[a-zA-Z0-9_~.\-]{31,34}
)
(?:$|['"\x60\s<),])
pattern_requirements:
min_digits: 1
min_entropy: 3.5
confidence: medium
examples:
- '"aBc4Q~xY9kLmNpQrStUvWxYz01234567890abcd"'
- 'AZURE_CLIENT_SECRET=xY14Q~abcdefghijklmnopqrstuvwxyz01234'
references:
- https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app

64
crates/kingfisher-rules/data/rules/azureapim.yml Normal file
View file

@ -0,0 +1,64 @@
rules:
- name: Azure API Management Subscription Key
id: kingfisher.azureapim.1
pattern: |
(?x)
\b
(?:
(?i:(?:apim|api[_\s-]*management)[_\s-]*(?:subscription[_\s-]*key|key))
|
(?i:Ocp-Apim-Subscription-Key)
)
(?:.|[\n\r]){0,16}?
(
[a-f0-9]{32}
)
\b
pattern_requirements:
min_digits: 2
min_lowercase: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'Ocp-Apim-Subscription-Key: 1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d'
- 'APIM_SUBSCRIPTION_KEY=abcdef0123456789abcdef0123456789'
validation:
type: Http
content:
request:
method: GET
url: "{{ APIM_URL }}"
headers:
Ocp-Apim-Subscription-Key: "{{ TOKEN }}"
response_is_html: true
response_matcher:
- report_response: true
- type: StatusMatch
status: [200, 201, 202, 204, 400, 404, 405]
- type: StatusMatch
status: [401, 403]
negative: true
depends_on_rule:
- rule_id: kingfisher.azureapim.2
variable: APIM_URL
references:
- https://learn.microsoft.com/en-us/azure/api-management/api-management-subscriptions
- name: Azure API Management Gateway URL
id: kingfisher.azureapim.2
pattern: |
(?xi)
\b
(
https://[a-z0-9-]+(?:\.developer)?\.azure-api\.net(?:/[^\s"'<>]{0,200})?
)
min_entropy: 1.0
confidence: medium
visible: false
examples:
- https://contoso.azure-api.net/echo
- APIM_URL=https://contoso.developer.azure-api.net/api
references:
- https://learn.microsoft.com/en-us/azure/api-management/api-management-subscriptions
- https://learn.microsoft.com/en-us/troubleshoot/azure/api-mgmt/availability/unauthorized-errors-invoke-apis

71
crates/kingfisher-rules/data/rules/azurebatch.yml Normal file
View file

@ -0,0 +1,71 @@
rules:
- name: Azure Batch Account Key
id: kingfisher.azurebatch.1
pattern: |
(?x)
\b
(?:
(?i:azure[_\s-]*batch[_\s-]*(?:key|account[_\s-]*key|access[_\s-]*key))
|
(?i:batch[_\s-]*account[_\s-]*key)
)
(?:.|[\n\r]){0,16}?
(
[A-Za-z0-9+/]{86}==
)
pattern_requirements:
min_digits: 2
min_uppercase: 2
min_lowercase: 2
min_special_chars: 1
min_entropy: 4.0
confidence: medium
categories: [api, key]
examples:
- 'AZURE_BATCH_KEY=oqb4TdY9T0hphvktd5fJnMiHuQqzVy1jd5sSuOpAbGkaoqTlrHl0BOJN2okcasinVLOJzfDbZo1L+ASt68RAhA=='
validation:
type: Http
content:
request:
method: GET
url: '{{ BATCH_URL }}/applications?api-version=2020-09-01.12.0'
headers:
Accept: application/json
Content-Type: application/json
Date: '{{ REQUEST_RFC1123_DATE }}'
Authorization: |
{%- assign host = BATCH_URL | split: "://" | last | split: "/" | first -%}
{%- assign account_name = host | split: "." | first -%}
{%- assign resource_path = "/" | append: account_name | append: "/applications" | downcase -%}
{%- assign string_to_sign = "GET\n\n\n\n\napplication/json\n" | append: REQUEST_RFC1123_DATE | append: "\n\n\n\n\n\n" | append: resource_path | append: "\napi-version:2020-09-01.12.0" -%}
{%- assign signature = string_to_sign | hmac_sha256_b64key: TOKEN -%}
SharedKey {{ account_name }}:{{ signature }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
depends_on_rule:
- rule_id: kingfisher.azurebatch.2
variable: BATCH_URL
references:
- https://learn.microsoft.com/en-us/azure/batch/batch-account-create-portal
- https://learn.microsoft.com/en-us/rest/api/batchservice/authenticate-requests-to-the-azure-batch-service
- name: Azure Batch Account Endpoint
id: kingfisher.azurebatch.2
pattern: |
(?xi)
\b
(
https://[a-z0-9-]+\.[a-z0-9-]+\.batch\.azure\.com
)
\b
min_entropy: 1.0
confidence: medium
visible: false
examples:
- BATCH_URL=https://mybatch.westus.batch.azure.com
- batchAccountUrl="https://contoso-prod.eastus.batch.azure.com"
references:
- https://learn.microsoft.com/en-us/rest/api/batchservice/authenticate-requests-to-the-azure-batch-service

46
crates/kingfisher-rules/data/rules/azurecognitive.yml Normal file
View file

@ -0,0 +1,46 @@
rules:
- name: Azure Cognitive Services / AI Services Key
id: kingfisher.azurecognitive.1
pattern: |
(?x)
\b
(?:
(?i:azure[_\s-]*(?:cognitive|ai)[_\s-]*(?:service|key))
|
(?i:cognitive[_\s-]*service[_\s-]*(?:key|secret|subscription))
|
(?i:
(?:azure[_\s-]*)?(?:anomaly[_\s-]*detector|computer[_\s-]*vision|content[_\s-]*moderator|content[_\s-]*safety
|custom[_\s-]*vision|face[_\s-]*(?:api)?|form[_\s-]*recognizer|document[_\s-]*intelligence
|immersive[_\s-]*reader|language[_\s-]*understanding|luis
|personalizer|qna[_\s-]*maker|text[_\s-]*analytics|video[_\s-]*indexer
|metrics[_\s-]*advisor|health[_\s-]*insights|cognitive[_\s-]*service|ai[_\s-]*service)
[_\s-]*(?:key|api[_\s-]*key|subscription[_\s-]*key|secret)
)
)
\b
(?:.|[\n\r]){0,16}?
(
[a-f0-9]{32}
)
\b
pattern_requirements:
min_digits: 2
min_lowercase: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- AZURE_COGNITIVE_SERVICE_KEY=1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d
- AZURE_COMPUTER_VISION_KEY=abcdef0123456789abcdef0123456789
- content_moderator_key="1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d"
- AZURE_FACE_KEY=abcdef0123456789abcdef0123456789
- AZURE_FORM_RECOGNIZER_KEY=1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d
- AZURE_LUIS_KEY=abcdef0123456789abcdef0123456789
- AZURE_QNA_MAKER_KEY=1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d
- AZURE_TEXT_ANALYTICS_KEY=abcdef0123456789abcdef0123456789
references:
- https://learn.microsoft.com/en-us/azure/ai-services/
- https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/
- https://learn.microsoft.com/en-us/azure/ai-services/content-moderator/
- https://learn.microsoft.com/en-us/azure/ai-services/luis/

20
crates/kingfisher-rules/data/rules/azurecommunication.yml Normal file
View file

@ -0,0 +1,20 @@
rules:
- name: Azure Communication Services Connection String
id: kingfisher.azurecommunication.1
pattern: |
(?x)
(?i:endpoint=https://(?:[a-z0-9-]+\.)?(?:communication|comm)\.azure\.com/;accesskey=)
(
[A-Za-z0-9+/]{40,90}={0,2}
)
pattern_requirements:
min_digits: 2
min_uppercase: 2
min_special_chars: 1
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'endpoint=https://myresource.communication.azure.com/;accesskey=AbCdEfGhIjKlMnOpQrStUvWxYz1234567890ABCDEFGHIJKLMNOPQRS+/=='
references:
- https://learn.microsoft.com/en-us/azure/communication-services/

77
crates/kingfisher-rules/data/rules/azurecosmosdb.yml Normal file
View file

@ -0,0 +1,77 @@
rules:
- name: Azure CosmosDB Account Key
id: kingfisher.azurecosmosdb.1
pattern: |
(?x)
\b
(?:
(?i:cosmos(?:db)?[_\s-]*(?:key|account[_\s-]*key|primary[_\s-]*key|secondary[_\s-]*key|master[_\s-]*key))
|
(?i:azure[_\s-]*cosmos(?:db)?[_\s-]*(?:key|account_key|primary_key|master_key))
|
(?i:documentdb(?:authkey|key))
)
(?:.|[\n\r]){0,16}?
(
[A-Za-z0-9+/]{86}==
)
pattern_requirements:
min_digits: 2
min_uppercase: 2
min_lowercase: 2
min_special_chars: 1
min_entropy: 4.0
confidence: medium
categories: [api, key]
examples:
- AZURE_COSMOSDB_KEY=oqb4TdY9T0hphvktd5fJnMiHuQqzVy1jd5sSuOpAbGkaoqTlrHl0BOJN2okcasinVLOJzfDbZo1L+ASt68RAhA==
- 'DocumentDbAuthKey=B/1EVX2Ui47X09tqU3GI/j+Nko9r5COPm0Hea9tfzitF9MQX9lZZiNO3tYQckWnt+rtlGIWS+sCx+AStkq8ZLg=='
references:
- https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-obtain-account-keys
- name: Azure CosmosDB Connection String
id: kingfisher.azurecosmosdb.2
pattern: |
(?x)
(?i:AccountEndpoint=(?P<COSMOS_ENDPOINT>https://[a-z0-9-]+\.documents\.azure\.com(?::\d+)?)/?;)
AccountKey=
(?P<secret>
(?P<TOKEN>
[A-Za-z0-9+/]{86}==
)
)
pattern_requirements:
min_digits: 2
min_uppercase: 2
min_special_chars: 1
min_entropy: 4.0
confidence: high
categories: [api, key]
examples:
- 'AccountEndpoint=https://myaccount.documents.azure.com:443;AccountKey=oqb4TdY9T0hphvktd5fJnMiHuQqzVy1jd5sSuOpAbGkaoqTlrHl0BOJN2okcasinVLOJzfDbZo1L+ASt68RAhA==;'
validation:
type: Http
content:
request:
method: GET
url: "{{ COSMOS_ENDPOINT }}/dbs"
headers:
Accept: application/json
x-ms-date: '{{ REQUEST_RFC1123_DATE | downcase }}'
x-ms-version: "2018-12-31"
Authorization: |
{%- assign x_ms_date = REQUEST_RFC1123_DATE | downcase -%}
{%- assign string_to_sign = "get\ndbs\n\n" | append: x_ms_date | append: "\n\n" -%}
{%- assign signature = string_to_sign | hmac_sha256_b64key: TOKEN | url_encode -%}
type=master&ver=1.0&sig={{ signature }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
words:
- '"Databases"'
references:
- https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-obtain-account-keys
- https://learn.microsoft.com/en-us/rest/api/cosmos-db/access-control-on-cosmosdb-resources

28
crates/kingfisher-rules/data/rules/azureeventgrid.yml Normal file
View file

@ -0,0 +1,28 @@
rules:
- name: Azure Event Grid Key
id: kingfisher.azureeventgrid.1
pattern: |
(?x)
\b
(?:
(?i:event[_\s-]*grid[_\s-]*(?:key|access[_\s-]*key|topic[_\s-]*key))
|
(?i:azure_event_grid[_\s-]*(?:key|access_key|topic_key))
|
(?i:aeg-sas-key)
)
(?:.|[\n\r]){0,16}?
(
[A-Za-z0-9+/]{40,50}={0,2}
)
pattern_requirements:
min_digits: 2
min_uppercase: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'AZURE_EVENT_GRID_KEY=AbCdEfGhIjKlMnOpQrStUvWxYz1234567890ABCDEFG=='
- 'aeg-sas-key: AbCdEfGhIjKlMnOpQrStUvWxYz1234567890ABCDEFG=='
references:
- https://learn.microsoft.com/en-us/azure/event-grid/

56
crates/kingfisher-rules/data/rules/azurefunctionkey.yml Normal file
View file

@ -0,0 +1,56 @@
rules:
- name: Azure Function Key in URL
id: kingfisher.azurefunctionkey.1
pattern: |
(?x)
(
(?i:https://[a-z0-9-]+\.azurewebsites\.net/api/)[a-zA-Z0-9_-]+
(?i:\?code=)[a-zA-Z0-9_/+=-]{20,100}
)
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'https://myfunc.azurewebsites.net/api/HttpTrigger1?code=AbCdEfGhIjKlMnOpQrStUvWxYz1234567890/+=='
validation:
type: Http
content:
request:
method: GET
url: "{{ TOKEN }}"
response_matcher:
- report_response: true
- type: StatusMatch
status: [200, 202, 204, 400, 404, 405]
- type: StatusMatch
status: [401, 403]
negative: true
references:
- https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-http-webhook-trigger
- name: Azure Function Master/Host Key
id: kingfisher.azurefunctionkey.2
pattern: |
(?x)
\b
(?:
(?i:azure[_\s-]*function[_\s-]*(?:key|master[_\s-]*key|host[_\s-]*key))
|
(?i:x-functions-key)
)
(?:.|[\n\r]){0,16}?
(
[a-zA-Z0-9_/+=-]{40,100}
)
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'AZURE_FUNCTION_KEY=AbCdEfGhIjKlMnOpQrStUvWxYz1234567890ABCDEFGH/+=='
- 'x-functions-key: AbCdEfGhIjKlMnOpQrStUvWxYz1234567890ABCDEFGH'
references:
- https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-http-webhook-trigger

22
crates/kingfisher-rules/data/rules/azurelogicapps.yml Normal file
View file

@ -0,0 +1,22 @@
rules:
- name: Azure Logic Apps SAS URL
id: kingfisher.azurelogicapps.1
pattern: |
(?x)
(
(?i:https://(?:[a-z0-9-]+\.)+logic\.azure\.com)
(?::\d+)?/workflows/[A-Fa-f0-9]+/triggers/[a-zA-Z0-9_-]+/paths/invoke
(?i:\?api-version=)[0-9-]+
(?i:&sp=)[%a-zA-Z0-9/]+
(?i:&sv=)[0-9.]+
(?i:&sig=)[a-zA-Z0-9_/+=-]+
)
pattern_requirements:
min_digits: 4
min_entropy: 3.0
confidence: high
categories: [api, key]
examples:
- 'https://prod-00.eastus.logic.azure.com/workflows/abcdef1234567890/triggers/manual/paths/invoke?api-version=2016-10-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=AbCdEfGhIjKlMnOpQrStUvWxYz123456'
references:
- https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-securing-a-logic-app

42
crates/kingfisher-rules/data/rules/azuremaps.yml Normal file
View file

@ -0,0 +1,42 @@
rules:
- name: Azure Maps Subscription Key
id: kingfisher.azuremaps.1
pattern: |
(?x)
\b
(?i:azure[_\s-]*maps[_\s-]*(?:key|subscription[_\s-]*key|api[_\s-]*key|secret))
(?:.|[\n\r]){0,16}?
(
[a-zA-Z0-9_-]{32,44}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- AZURE_MAPS_KEY=AbCdEfGhIjKlMnOpQrStUvWxYz123456
validation:
type: Http
content:
request:
method: GET
url: https://atlas.microsoft.com/geocode?api-version=2025-01-01&addressLine=15127%20NE%2024th%20Street%20Redmond%20WA&countryRegion=US&subscription-key={{ TOKEN }}
headers:
Accept: application/geo+json, application/json
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: true
words:
- '"FeatureCollection"'
- '"features"'
# Revocation not added: Azure Maps shared-key docs cover rotation and
# authentication, but I did not find a token self-revoke API.
references:
- https://learn.microsoft.com/en-us/azure/azure-maps/how-to-manage-authentication
- https://learn.microsoft.com/en-us/rest/api/maps/search/get-geocoding

28
crates/kingfisher-rules/data/rules/azuremixedreality.yml Normal file
View file

@ -0,0 +1,28 @@
rules:
- name: Azure Mixed Reality / Spatial Anchors Key
id: kingfisher.azuremixedreality.1
pattern: |
(?x)
\b
(?:
(?i:
(?:azure[_\s-]*)?(?:mixed[_\s-]*reality|spatial[_\s-]*anchors?|remote[_\s-]*rendering)
[_\s-]*(?:key|account[_\s-]*key|access[_\s-]*key)
)
)
(?:.|[\n\r]){0,16}?
(
[a-f0-9]{32}
)
\b
pattern_requirements:
min_digits: 2
min_lowercase: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'AZURE_SPATIAL_ANCHORS_KEY=1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d'
- 'AZURE_REMOTE_RENDERING_KEY=abcdef0123456789abcdef0123456789'
references:
- https://learn.microsoft.com/en-us/azure/spatial-anchors/

50
crates/kingfisher-rules/data/rules/azuresastoken.yml Normal file
View file

@ -0,0 +1,50 @@
rules:
- name: Azure SAS Token
id: kingfisher.azuresastoken.1
pattern: |
(?x)
(
(?i:(?:sv|SharedAccessSignature\s+sr))=[0-9]{4}-[0-9]{2}-[0-9]{2}
(?:&(?i:[a-z]{2,4})=[^&\s"']{1,200}){2,10}
(?i:&sig=)[a-zA-Z0-9%+/=]{20,100}
)
pattern_requirements:
min_digits: 4
min_entropy: 3.0
confidence: medium
categories: [api, key]
examples:
- 'sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-12-31&st=2024-01-01&spr=https&sig=AbCdEfGhIjKlMnOpQrStUvWxYz1234567890%2BABCDE%3D'
references:
- https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview
- name: Azure SAS Token in URL
id: kingfisher.azuresastoken.2
pattern: |
(?x)
(
(?i:https://[a-z0-9-]+\.(?:blob|queue|table|file|dfs)\.core\.windows\.net/)[^\s"']*
\?[^\s"']*(?i:sig=)[a-zA-Z0-9%+/=]{20,100}[^\s"']*
)
pattern_requirements:
min_digits: 4
min_entropy: 3.0
confidence: medium
categories: [api, key]
examples:
- 'https://mystorageaccount.blob.core.windows.net/mycontainer/myblob?sv=2021-06-08&st=2024-01-01&se=2024-12-31&sr=b&sp=r&sig=AbCdEfGhIjKlMnOp%2BQrStUvWxYz%3D'
validation:
type: Http
content:
request:
method: HEAD
url: "{{ TOKEN }}"
response_matcher:
- report_response: true
- type: StatusMatch
status: [200, 206, 404]
- type: StatusMatch
status: [401, 403]
negative: true
references:
- https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview

20
crates/kingfisher-rules/data/rules/azuresignalr.yml Normal file
View file

@ -0,0 +1,20 @@
rules:
- name: Azure SignalR Connection String
id: kingfisher.azuresignalr.1
pattern: |
(?x)
(?i:Endpoint=https://(?:[a-z0-9-]+\.service\.signalr\.net);AccessKey=)
(
[A-Za-z0-9+/]{40,90}={0,2}
)
pattern_requirements:
min_digits: 2
min_uppercase: 2
min_special_chars: 1
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'Endpoint=https://myservice.service.signalr.net;AccessKey=AbCdEfGhIjKlMnOpQrStUvWxYz1234567890ABCDEFGHIJKLMNOPQRS+/==;Version=1.0;'
references:
- https://learn.microsoft.com/en-us/azure/azure-signalr/

46
crates/kingfisher-rules/data/rules/azuresql.yml Normal file
View file

@ -0,0 +1,46 @@
rules:
- name: Azure SQL Connection String
id: kingfisher.azuresql.1
pattern: |
(?x)
(?i:Server=tcp:(?:[a-z0-9-]+\.database\.windows\.net),\d+;)
(?:.|[\n\r]){0,100}?
(?i:Password=)(
[^;'"]{8,128}
)
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
categories: [api, key]
examples:
- 'Server=tcp:myserver.database.windows.net,1433;Initial Catalog=mydb;Persist Security Info=False;User ID=admin;Password=MyP@ssw0rd!123;'
references:
- https://learn.microsoft.com/en-us/azure/azure-sql/database/connect-query-content-reference-guide
- name: Azure SQL Password Assignment
id: kingfisher.azuresql.2
pattern: |
(?x)
\b
(?:
(?i:azure[_\s-]*sql[_\s-]*password)
|
(?i:sql_admin_password)
|
(?i:mssql_sa_password)
)
\b
(?:.|[\n\r]){0,8}?
[=:]
\s*["']?
([^\s"']{8,128})
["']?
min_entropy: 3.0
confidence: medium
categories: [password]
examples:
- 'AZURE_SQL_PASSWORD=MyStr0ngP@ssword!'
- 'SQL_ADMIN_PASSWORD="Compl3x!Pass#2024"'
references:
- https://learn.microsoft.com/en-us/azure/azure-sql/database/connect-query-content-reference-guide

20
crates/kingfisher-rules/data/rules/azurewebpubsub.yml Normal file
View file

@ -0,0 +1,20 @@
rules:
- name: Azure Web PubSub Connection String
id: kingfisher.azurewebpubsub.1
pattern: |
(?x)
(?i:Endpoint=https://(?:[a-z0-9-]+\.webpubsub\.azure\.com);AccessKey=)
(
[A-Za-z0-9+/]{40,90}={0,2}
)
pattern_requirements:
min_digits: 2
min_uppercase: 2
min_special_chars: 1
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'Endpoint=https://myservice.webpubsub.azure.com;AccessKey=AbCdEfGhIjKlMnOpQrStUvWxYz1234567890ABCDEFGHIJKLMNOPQRS+/==;Version=1.0;'
references:
- https://learn.microsoft.com/en-us/azure/azure-web-pubsub/

29
crates/kingfisher-rules/data/rules/bitfinex.yml
View file

@ -47,9 +47,30 @@ rules:
examples:
- "bitfinex\nsecret = 8d7c3965318b8d20f7648dbda96fbfa23f4d1c449aa"
- "bitfinex\napi-secret = 1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1d2"
validation:
type: Http
content:
request:
method: POST
url: https://api.bitfinex.com/v2/auth/r/wallets
headers:
Accept: application/json
Content-Type: application/json
bfx-apikey: "{{ BITFINEX_KEY }}"
bfx-nonce: "{{ REQUEST_UNIX_MILLIS }}"
bfx-signature: |
{%- assign request_path = "/v2/auth/r/wallets" -%}
{%- assign request_body = "{}" -%}
{%- assign signature_payload = "/api" | append: request_path | append: REQUEST_UNIX_MILLIS | append: request_body -%}
{{ signature_payload | hmac_sha384_hex: TOKEN }}
body: "{}"
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
depends_on_rule:
- rule_id: kingfisher.bitfinex.1
variable: BITFINEX_KEY
references:
- https://docs.bitfinex.com/docs
- https://docs.bitfinex.com/docs/rest-auth
# No simple validation: Bitfinex REST API v2 uses HMAC-SHA384
# request signing with a nonce and payload. Cannot validate with
# a static Bearer/API-key style header.

35
crates/kingfisher-rules/data/rules/bitrise.yml Normal file
View file

@ -0,0 +1,35 @@
rules:
- name: Bitrise Personal Access Token
id: kingfisher.bitrise.1
pattern: |
(?x)
\b
(?i:bitrise)
(?:.|[\n\r]){0,24}?
(?i:token|pat|personal[_\s-]*access)
(?:.|[\n\r]){0,16}?
(
[a-zA-Z0-9_-]{60,120}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'BITRISE_TOKEN=AbCdEfGhIjKlMnOpQrStUvWxYz1234567890AbCdEfGhIjKlMnOpQrStUvWxYz12'
validation:
type: Http
content:
request:
method: GET
url: https://api.bitrise.io/v0.1/me
headers:
Authorization: '{{ TOKEN }}'
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://devcenter.bitrise.io/en/api/authenticating-with-the-bitrise-api.html

19
crates/kingfisher-rules/data/rules/blockprotocol.yml Normal file
View file

@ -0,0 +1,19 @@
rules:
- name: Block Protocol API Key
id: kingfisher.blockprotocol.1
pattern: |
(?x)
\b
(
b10ck5\.[a-zA-Z0-9]{28,36}\.[a-zA-Z0-9]{32,40}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'BLOCK_PROTOCOL_API_KEY=b10ck5.AbCdEfGhIjKlMnOpQrStUvWxYz1234.AbCdEfGhIjKlMnOpQrStUvWxYz12345678'
references:
- https://blockprotocol.org/docs/hub/api

14
crates/kingfisher-rules/data/rules/branchio.yml
View file

@ -45,6 +45,20 @@ rules:
- 'branch.init("key_test_plqYW3Aq9Xija1cobGMieipndBzO5y7J");'
references:
- https://help.branch.io/developers-hub/docs/deep-linking-api
- https://help.branch.io/apidocs/app-api
depends_on_rule:
- rule_id: kingfisher.branchio.3
variable: BRANCH_SECRET
validation:
type: Http
content:
request:
method: GET
url: "https://api2.branch.io/v1/app/{{ TOKEN }}?branch_secret={{ BRANCH_SECRET }}"
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- name: Branch.io Secret
id: kingfisher.branchio.3

20
crates/kingfisher-rules/data/rules/canva.yml Normal file
View file

@ -0,0 +1,20 @@
rules:
- name: Canva Connect API Client Secret
id: kingfisher.canva.1
pattern: |
(?x)
\b
(
cnvca[a-zA-Z0-9_-]{20,80}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'CANVA_CLIENT_SECRET=cnvcaAbCdEfGhIjKlMnOpQrStUvWxYz123456'
references:
- https://www.canva.dev/docs/connect/authentication/
- https://www.canva.dev/docs/connect/guidelines/security/

19
crates/kingfisher-rules/data/rules/cfxre.yml Normal file
View file

@ -0,0 +1,19 @@
rules:
- name: Cfx.re FiveM Server Key
id: kingfisher.cfxre.1
pattern: |
(?x)
\b
(
cfxk_[a-zA-Z0-9_-]{20,100}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'sv_licenseKey "cfxk_AbCdEfGhIjKlMnOpQrStUvWxYz1234567890_abcdef"'
references:
- https://docs.fivem.net/docs/server-manual/setting-up-a-server/

51
crates/kingfisher-rules/data/rules/cockroachlabs.yml Normal file
View file

@ -0,0 +1,51 @@
rules:
- name: CockroachDB Cloud API Key
id: kingfisher.cockroachlabs.1
pattern: |
(?x)
\b
(?:
(?i:cockroach(?:db)?(?:cloud)?)
(?:.|[\n\r]){0,24}?
(?i:api[_\s-]*key|secret|token)
|
(?i:CC_API_KEY)
)
(?:.|[\n\r]){0,16}?
(
[A-Z0-9_]{20,60}
)
\b
pattern_requirements:
min_digits: 2
min_uppercase: 4
min_entropy: 3.0
confidence: medium
categories: [api, key]
examples:
- 'COCKROACHDB_API_KEY=B81649_8F7D11A_92BCE13_56782D_C53'
validation:
type: Http
content:
request:
method: GET
url: https://cockroachlabs.cloud/api/v1/clusters?show_inactive=true
headers:
Authorization: Bearer {{ TOKEN }}
Accept: application/json
Cc-Version: "2024-09-16"
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: true
words:
- '"clusters"'
- '"pagination"'
# Revocation not added: the public Cloud API docs describe bearer-token
# authentication for service-account secret keys, but not a documented
# self-revocation endpoint for the current secret key value.
references:
- https://www.cockroachlabs.com/docs/cockroachcloud/cloud-api

24
crates/kingfisher-rules/data/rules/databricks.yml
View file

@ -22,6 +22,26 @@ rules:
- secret
references:
- https://docs.databricks.com/dev-tools/api/latest/authentication.html
- https://docs.databricks.com/en/dev-tools/auth/pat.html
validation:
type: Http
content:
request:
headers:
Authorization: Bearer {{ TOKEN }}
method: GET
response_matcher:
- report_response: true
- status:
- 200
type: StatusMatch
url: https://{{ DOMAIN }}/api/2.0/clusters/list
depends_on_rule:
- rule_id: "kingfisher.databricks.3"
variable: DOMAIN
# Revocation not added: Databricks PAT docs describe token creation and
# use, but I did not find a PAT-only self-revoke endpoint suitable for YAML
# revocation here.
- name: Databricks API Token
id: kingfisher.databricks.2
@ -51,7 +71,7 @@ rules:
type: StatusMatch
url: https://{{ DOMAIN }}/api/2.0/clusters/list
depends_on_rule:
- rule_id: "kingfisher.databricks.2"
- rule_id: "kingfisher.databricks.3"
variable: DOMAIN
- name: Databricks Domain
@ -83,4 +103,4 @@ rules:
references:
- https://docs.databricks.com/workspace/workspace-details.html
- https://docs.gcp.databricks.com/workspace/workspace-details.html
- https://docs.microsoft.com/en-us/azure/databricks/scenarios/what-is-azure-databricks
- https://docs.microsoft.com/en-us/azure/databricks/scenarios/what-is-azure-databricks

40
crates/kingfisher-rules/data/rules/docker.yml
View file

@ -45,4 +45,42 @@ rules:
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
status: [200]
- name: Docker Swarm Join Token
id: kingfisher.docker.2
pattern: |
(?x)
\b
(
SWMTKN-1-[a-z0-9]{50,60}-[a-z0-9]{24,30}
)
\b
pattern_requirements:
min_digits: 4
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'docker swarm join --token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-1awxwuwd3z9j1z3puu7rcgdbx 192.168.99.100:2377'
references:
- https://docs.docker.com/engine/swarm/join-nodes/
- name: Docker Swarm Unlock Key
id: kingfisher.docker.3
pattern: |
(?x)
\b
(
SWMKEY-1-[A-Za-z0-9+/]{40,50}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'docker swarm unlock --key SWMKEY-1-AbCdEfGhIjKlMnOpQrStUvWxYz1234567890ABCDEFG'
references:
- https://docs.docker.com/engine/swarm/swarm_manager_locking/

104
crates/kingfisher-rules/data/rules/docusign.yml
View file

@ -25,7 +25,107 @@ rules:
examples:
- "docusign.secret_key = 7a39ce6d-94cf-4bf6-9e9e-9213373c15f4"
- "docusign\nds_secret = 3d2f18c9-2075-4e78-834b-64f57f8757d0"
validation:
type: Http
content:
request:
method: POST
url: "https://{{ DOCUSIGN_AUTH_HOST }}/oauth/token"
headers:
Accept: application/json
Content-Type: application/x-www-form-urlencoded
body: >
grant_type=authorization_code&code=INVALID_AUTH_CODE&client_id={{ DOCUSIGN_CLIENT_ID | url_encode }}&client_secret={{ TOKEN | url_encode }}&redirect_uri={{ REDIRECT_URI | url_encode }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [400]
- type: WordMatch
match_all_words: false
words:
- invalid_grant
- invalid authorization code
- type: WordMatch
words:
- invalid_client
negative: true
depends_on_rule:
- rule_id: kingfisher.docusign.2
variable: DOCUSIGN_CLIENT_ID
- rule_id: kingfisher.docusign.3
variable: DOCUSIGN_AUTH_HOST
- rule_id: kingfisher.docusign.4
variable: REDIRECT_URI
references:
- https://developers.docusign.com/platform/auth/
- https://developers.docusign.com/platform/build-integration/
- name: DocuSign Integration Key
id: kingfisher.docusign.2
pattern: |
(?xi)
\b
docusign
(?:.|[\n\r]){0,64}?
(?:integration[_-]?key|client[_-]?id|app[_-]?id)\b
(?:.|[\n\r]){0,16}?
[=:"'\s]
['"]*
(
[a-f0-9]{8}-
[a-f0-9]{4}-
[a-f0-9]{4}-
[a-f0-9]{4}-
[a-f0-9]{12}
)
\b
pattern_requirements:
min_digits: 6
min_entropy: 3.0
confidence: medium
visible: false
examples:
- DOCUSIGN_CLIENT_ID=7a39ce6d-94cf-4bf6-9e9e-9213373c15f4
- 'docusign.integration_key = "3d2f18c9-2075-4e78-834b-64f57f8757d0"'
references:
- https://developers.docusign.com/platform/build-integration/
# No public validation endpoint: DocuSign OAuth secret keys cannot be
# validated without a full Authorization Code Grant flow.
- name: DocuSign Auth Host
id: kingfisher.docusign.3
pattern: |
(?xi)
\b
(
account(?:-d)?\.docusign\.com
)
\b
min_entropy: 1.0
confidence: medium
visible: false
examples:
- account.docusign.com
- account-d.docusign.com
references:
- https://developers.docusign.com/platform/auth/
- name: DocuSign Redirect URI
id: kingfisher.docusign.4
pattern: |
(?xi)
\b
docusign
(?:.|[\n\r]){0,64}?
(?:redirect[_-]?uri|oauth[_-]?redirect)\b
(?:.|[\n\r]){0,16}?
[=:"'\s]
(
https?://[^\s"'<>]{6,200}
)
min_entropy: 1.5
confidence: medium
visible: false
examples:
- DOCUSIGN_REDIRECT_URI=https://example.com/docusign/callback
- 'docusign.redirect_uri = "https://localhost:3000/oauth/docusign"'
references:
- https://developers.docusign.com/platform/auth/

35
crates/kingfisher-rules/data/rules/dropbox.yml
View file

@ -33,5 +33,40 @@ rules:
- '"account_id":'
- '"email":'
url: https://api.dropboxapi.com/2/users/get_current_account
references:
- https://www.dropbox.com/developers/documentation/http/documentation#auth
- name: Dropbox Long-Lived API Token
id: kingfisher.dropbox.2
pattern: |
(?x)
\b
(
[a-z0-9]{11}
AAAAAAAAAA
[a-z0-9\-_=]{43}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- 'ab3cd5ef7g9AAAAAAAAAAbcdefghij-klmnopqrstuvwxyz01234567890abcdef'
validation:
type: Http
content:
request:
headers:
Authorization: Bearer {{ TOKEN }}
method: POST
response_matcher:
- report_response: true
- match_all_words: true
type: WordMatch
words:
- '"account_id":'
- '"email":'
url: https://api.dropboxapi.com/2/users/get_current_account
references:
- https://www.dropbox.com/developers/documentation/http/documentation#auth

47
crates/kingfisher-rules/data/rules/dwolla.yml
View file

@ -47,7 +47,48 @@ rules:
examples:
- "dwolla secret = 4d1d407752bfd562bZ=9b7c21f8862fdfc57bc1e45\n"
- "dwolla\nclient_secret = 7e4e5297691a673cA=0c7d43b997ec1h8dcaf56\n"
validation:
type: Http
content:
request:
method: POST
url: "{{ DWOLLA_API_BASE }}/token"
headers:
Accept: application/json
Content-Type: application/x-www-form-urlencoded
Authorization: "Basic {{ CLIENT_ID | append: ':' | append: TOKEN | b64enc }}"
body: grant_type=client_credentials
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
words:
- '"access_token"'
depends_on_rule:
- rule_id: kingfisher.dwolla.1
variable: CLIENT_ID
- rule_id: kingfisher.dwolla.3
variable: DWOLLA_API_BASE
references:
- https://developers.dwolla.com/
# No simple validation: Dwolla OAuth2 requires both client_id and
# client_secret together for the token endpoint.
- https://developers.dwolla.com/docs/api-reference/tokens/create-an-application-access-token
- https://developers.dwolla.com/docs/balance/auth/application-access-tokens
- name: Dwolla API Base URL
id: kingfisher.dwolla.3
visible: false
pattern: |
(?xi)
\b
(
https://api(?:-sandbox)?\.dwolla\.com
)
\b
min_entropy: 1.0
confidence: medium
examples:
- DWOLLA_API_BASE=https://api.dwolla.com
- DWOLLA_BASE_URL="https://api-sandbox.dwolla.com"
references:
- https://developers.dwolla.com/docs/api-reference/tokens/create-an-application-access-token

58
crates/kingfisher-rules/data/rules/ebay.yml Normal file
View file

@ -0,0 +1,58 @@
rules:
- name: eBay Production Client ID
id: kingfisher.ebay.1
pattern: |
(?x)
\b
(
[a-zA-Z0-9_-]+-[a-zA-Z0-9_-]+-PRD-[a-f0-9]{8,12}-[a-f0-9]{8,12}
)
\b
pattern_requirements:
min_digits: 4
min_entropy: 3.0
confidence: medium
categories: [api, key]
examples:
- 'EBAY_CLIENT_ID=MyApp-MyApp-PRD-1a2b3c4d-567890ab'
references:
- https://developer.ebay.com/api-docs/static/oauth-credentials.html
- name: eBay Sandbox Client ID
id: kingfisher.ebay.2
pattern: |
(?x)
\b
(
[a-zA-Z0-9_-]+-[a-zA-Z0-9_-]+-SBX-[a-f0-9]{8,12}-[a-f0-9]{8,12}
)
\b
pattern_requirements:
min_digits: 4
min_entropy: 3.0
confidence: medium
categories: [api, key]
examples:
- 'EBAY_SANDBOX_CLIENT_ID=MyApp-MyApp-SBX-1a2b3c4d-567890ab'
references:
- https://developer.ebay.com/api-docs/static/oauth-credentials.html
- name: eBay Client Secret
id: kingfisher.ebay.3
pattern: |
(?x)
\b
(
(?:PRD|SBX)-[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4,12}
)
\b
pattern_requirements:
min_digits: 8
min_entropy: 3.0
confidence: medium
categories: [api, key]
examples:
- 'EBAY_CLIENT_SECRET=PRD-1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b'
- 'EBAY_SANDBOX_SECRET=SBX-1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b'
references:
- https://developer.ebay.com/api-docs/static/oauth-credentials.html

48
crates/kingfisher-rules/data/rules/elastic.yml Normal file
View file

@ -0,0 +1,48 @@
rules:
- name: Elastic Cloud API Key
id: kingfisher.elastic.1
pattern: |
(?x)
\b
(?:
(?i:elastic(?:[_\s-]*(?:search|cloud))?)
(?:.|[\n\r]){0,24}?
(?i:api[_\s-]*key|apikey)
|
(?i:EC_API_KEY)
)
(?:.|[\n\r]){0,16}?
(
[A-Za-z0-9+/=]{40,120}
)
pattern_requirements:
min_digits: 2
min_uppercase: 2
min_lowercase: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'ELASTIC_CLOUD_API_KEY=VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw=='
references:
- https://www.elastic.co/docs/deploy-manage/api-keys/elastic-cloud-api-keys
- name: Elasticsearch API Key with Prefix
id: kingfisher.elastic.2
pattern: |
(?x)
\b
(?i:Authorization:\s*ApiKey\s+)
(
[A-Za-z0-9+/=]{40,120}
)
pattern_requirements:
min_digits: 2
min_uppercase: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'Authorization: ApiKey VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw=='
references:
- https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html

23
crates/kingfisher-rules/data/rules/flutterwave.yml
View file

@ -38,6 +38,27 @@ rules:
- FLW_SECRET_KEY=FLWSECK_TEST-a514d8f1abd080db1502a144f22954dc-X
- 'Authorization: Bearer FLWSECK_TEST-5b1f0a33de9c41748c2a7e9b51d3c6af-X'
- seckey=FLWSECK-e6db11d1f8a6208de8cb2f94e293450e-X
validation:
type: Http
content:
request:
method: POST
url: https://idp.flutterwave.com/realms/flutterwave/protocol/openid-connect/token
headers:
Accept: application/json
Content-Type: application/x-www-form-urlencoded
body: >
client_id={{ CLIENT_ID | url_encode }}&client_secret={{ TOKEN | url_encode }}&grant_type=client_credentials
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
words:
- '"access_token"'
depends_on_rule:
- rule_id: kingfisher.flutterwave.1
variable: CLIENT_ID
references:
- https://developer.flutterwave.com/docs/authentication
- https://developer.flutterwave.com/v2.0/reference/api-request-and-response-standards

21
crates/kingfisher-rules/data/rules/ftp.yml
View file

@ -4,12 +4,16 @@ rules:
pattern: |
(?xi)
\b
ftps?://
[^:@\s]{1,64}
:
([^@\s]{6,128})
@
[^\s/"']{4,128}
(?P<TOKEN>
(?P<FTP_SCHEME>ftps?)://
(?P<FTP_USERNAME>[^:@\s]{1,64})
:
(?P<FTP_PASSWORD>[^@\s]{6,128})
@
(?P<FTP_HOST>[^\s/"':]{4,128})
(?::(?P<FTP_PORT>\d{2,5}))?
(?:/[^\s"']*)?
)
pattern_requirements:
min_digits: 2
min_entropy: 2.5
@ -17,6 +21,9 @@ rules:
examples:
- "ftp://johndoe:pg9stqu2018@files.example.edu.cn"
- "BACKUP_URL=ftps://backupuser:S5ec4rePassWord@ftp.corp.example.com"
validation:
type: Raw
content: ftp
references:
- https://datatracker.ietf.org/doc/html/rfc959
# No public validation endpoint: FTP servers are instance-specific.
- https://datatracker.ietf.org/doc/html/rfc4217

282
crates/kingfisher-rules/data/rules/gitlab.yml
View file

@ -191,3 +191,285 @@ rules:
- type: StatusMatch
status: [204]
url: https://gitlab.com/api/v4/personal_access_tokens/self
- name: GitLab CI/CD Job Token
id: kingfisher.gitlab.5
pattern: |
(?x)
\b
(
glcbt-
[0-9a-zA-Z]{1,5}
_
[0-9a-zA-Z_-]{20}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- glcbt-ab12_xY9kLmNpQrStUvWxYz01
- 'CI_JOB_TOKEN=glcbt-a1b2c_3dEfGhIjKlMnOpQrStUv'
references:
- https://docs.gitlab.com/ci/jobs/ci_job_token/
- https://docs.gitlab.com/api/jobs/
validation:
type: Http
content:
request:
method: GET
url: https://gitlab.com/api/v4/job
headers:
JOB-TOKEN: '{{ TOKEN }}'
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: true
words:
- '"id"'
- '"status"'
# Revocation not added: CI/CD job tokens are short-lived and automatically
# invalidated when the job finishes.
- name: GitLab Deploy Token
id: kingfisher.gitlab.6
pattern: |
(?x)
\b
(
gldt-
[0-9a-zA-Z_-]{20}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- gldt-xY9kLmNpQrStUvWxYz01
- 'DEPLOY_TOKEN=gldt-3dEfGhIjK4MnOpQrStUv'
references:
- https://docs.gitlab.com/user/project/deploy_tokens/
- name: GitLab Feature Flag Client Token
id: kingfisher.gitlab.7
pattern: |
(?x)
\b
(
glffct-
[0-9a-zA-Z_-]{20}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- glffct-xY9kLmNpQrStUvWxYz01
references:
- https://docs.gitlab.com/operations/feature_flags/
- name: GitLab Feed Token
id: kingfisher.gitlab.8
pattern: |
(?x)
\b
(
glft-
[0-9a-zA-Z_-]{20}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- glft-xY9kLmNpQrStUvWxYz01
references:
- https://docs.gitlab.com/user/profile/contributions_calendar/#feed-token
- name: GitLab Incoming Mail Token
id: kingfisher.gitlab.9
pattern: |
(?x)
\b
(
glimt-
[0-9a-zA-Z_-]{25}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- glimt-xY9kLmNpQrStUvWxYz0123456
references:
- https://docs.gitlab.com/administration/incoming_email/
- name: GitLab Kubernetes Agent Token
id: kingfisher.gitlab.10
pattern: |
(?x)
\b
(
glagent-
[0-9a-zA-Z_-]{50}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- glagent-xY9kLmNpQrStUvWxYz01234567890abcdefghijklmnopqrstu
references:
- https://docs.gitlab.com/user/clusters/agent/
- name: GitLab OAuth Application Secret
id: kingfisher.gitlab.11
pattern: |
(?x)
\b
(
gloas-
[0-9a-zA-Z_-]{64}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- gloas-xY9kLmNpQrStUvWxYz01234567890abcdefghijklmnopqrstuvwxyz012345678
references:
- https://docs.gitlab.com/integration/oauth_provider/
- name: GitLab Runner Authentication Token
id: kingfisher.gitlab.12
pattern: |
(?x)
\b
(
glrt-
[0-9a-zA-Z_-]{20}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- glrt-xY9kLmNpQrStUvWxYz01
- |
gitlab-runner register \
--url https://gitlab.com \
--token glrt-3dEfGhIjK4MnOpQrStUv
references:
- https://docs.gitlab.com/runner/register/
validation:
type: Http
content:
request:
method: POST
headers:
Content-Type: application/x-www-form-urlencoded
Accept: application/json
body: token={{ TOKEN }}
response_matcher:
- report_response: true
- type: StatusMatch
status: 200
- type: WordMatch
words:
- '"token is missing"'
- '"403 Forbidden"'
negative: true
url: https://gitlab.com/api/v4/runners/verify
- name: GitLab Runner Authentication Token - Routable Format
id: kingfisher.gitlab.13
pattern: |
(?x)
\b
(
glrt-t
\d
_
[0-9a-zA-Z_-]{27,300}
\.
[0-9a-z]{2}
[0-9a-z]{7}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 4.0
confidence: medium
examples:
- glrt-t1_xY9kLmNpQrStUvWxYz01234567890.01abc1234
references:
- https://docs.gitlab.com/runner/register/
validation:
type: Http
content:
request:
method: POST
headers:
Content-Type: application/x-www-form-urlencoded
Accept: application/json
body: token={{ TOKEN }}
response_matcher:
- report_response: true
- type: StatusMatch
status: 200
- type: WordMatch
words:
- '"token is missing"'
- '"403 Forbidden"'
negative: true
url: https://gitlab.com/api/v4/runners/verify
- name: GitLab SCIM Token
id: kingfisher.gitlab.14
pattern: |
(?x)
\b
(
glsoat-
[0-9a-zA-Z_-]{20}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- glsoat-xY9kLmNpQrStUvWxYz01
references:
- https://docs.gitlab.com/api/scim/
- name: GitLab Session Cookie
id: kingfisher.gitlab.15
pattern: |
(?x)
(?:^|[;\s])
_gitlab_session=
(
[0-9a-f]{32}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
examples:
- '_gitlab_session=a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6'
- 'Cookie: _gitlab_session=0f1e2d3c4b5a69788796a5b4c3d2e1f0'
references:
- https://docs.gitlab.com/

90
crates/kingfisher-rules/data/rules/google.yml
View file

@ -23,6 +23,32 @@ rules:
confidence: medium
examples:
- 'const CLIENTSECRET = "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"'
depends_on_rule:
- rule_id: "kingfisher.google.1"
variable: GOOGLE_CLIENT_ID
validation:
type: Http
content:
request:
method: POST
url: https://oauth2.googleapis.com/token
headers:
Content-Type: application/x-www-form-urlencoded
Accept: application/json
body: >
code=invalid_code&client_id={{ GOOGLE_CLIENT_ID | url_encode }}&client_secret={{ TOKEN | url_encode }}&redirect_uri={{ "https://example.com/oauth/callback" | url_encode }}&grant_type=authorization_code
response_matcher:
- report_response: true
- type: StatusMatch
status: [400]
- type: WordMatch
words:
- invalid_client
negative: true
# Revocation not added: Google's OAuth revocation endpoint revokes tokens,
# not client secrets.
references:
- https://developers.google.com/identity/protocols/oauth2/web-server
- name: Google OAuth Client Secret
id: kingfisher.google.3
@ -36,6 +62,32 @@ rules:
examples:
- " //$google_client_secret = 'fnhqAakzWrX-mtFQ4PRdMoy0';"
- " 'clientSecret' : 'Ufvuj-d6alhwGKvvLh_8Nq0K'"
depends_on_rule:
- rule_id: "kingfisher.google.1"
variable: GOOGLE_CLIENT_ID
validation:
type: Http
content:
request:
method: POST
url: https://oauth2.googleapis.com/token
headers:
Content-Type: application/x-www-form-urlencoded
Accept: application/json
body: >
code=invalid_code&client_id={{ GOOGLE_CLIENT_ID | url_encode }}&client_secret={{ TOKEN | url_encode }}&redirect_uri={{ "https://example.com/oauth/callback" | url_encode }}&grant_type=authorization_code
response_matcher:
- report_response: true
- type: StatusMatch
status: [400]
- type: WordMatch
words:
- invalid_client
negative: true
# Revocation not added: Google's OAuth revocation endpoint revokes tokens,
# not client secrets.
references:
- https://developers.google.com/identity/protocols/oauth2/web-server
- name: Google OAuth Access Token
id: kingfisher.google.4
@ -61,6 +113,42 @@ rules:
- |
-- Clear login if it's a new connection.
--propertyTable.access_token = 'ya29.Ci_UA7aEsvT6-oVI8f96kvB6i8oO13WgdZUviLaCVtpEPYZqhQcQycR-u2X9xtmYGA'
validation:
type: Http
content:
request:
method: GET
url: https://www.googleapis.com/oauth2/v3/tokeninfo?access_token={{ TOKEN | url_encode }}
headers:
Accept: application/json
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: true
words:
- '"aud"'
- '"expires_in"'
revocation:
type: Http
content:
request:
method: POST
url: https://oauth2.googleapis.com/revoke
headers:
Content-Type: application/x-www-form-urlencoded
Accept: application/json
body: token={{ TOKEN | url_encode }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://developers.google.com/identity/openid-connect/openid-connect
- https://developers.google.com/data-portability/user-guide/quickstart
- https://developers.google.com/identity/protocols/oauth2/web-server
- name: Google OAuth Credentials
id: kingfisher.google.6
@ -118,4 +206,4 @@ rules:
match_all_words: true
words:
- '"models"'
- '"name"'
- '"name"'

17
crates/kingfisher-rules/data/rules/googleoauth2.yml
View file

@ -30,5 +30,20 @@ rules:
- type: WordMatch
words:
- '"email":'
revocation:
type: Http
content:
request:
method: POST
url: https://oauth2.googleapis.com/revoke
headers:
Content-Type: application/x-www-form-urlencoded
Accept: application/json
body: token={{ TOKEN | url_encode }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://developers.google.com/identity/protocols/oauth2
- https://developers.google.com/identity/protocols/oauth2
- https://developers.google.com/identity/protocols/oauth2/web-server

24
crates/kingfisher-rules/data/rules/hcaptcha.yml Normal file
View file

@ -0,0 +1,24 @@
rules:
- name: hCaptcha Site Verify Secret Key
id: kingfisher.hcaptcha.1
pattern: |
(?x)
\b
(
(?:
0x[a-fA-F0-9]{40}
|
ES_[a-fA-F0-9]{32}
)
)
\b
pattern_requirements:
min_digits: 4
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'HCAPTCHA_SECRET=0x0000000000000000000000000000000000000000'
- 'hcaptcha_secret: ES_abcdef1234567890abcdef1234567890'
references:
- https://docs.hcaptcha.com/

53
crates/kingfisher-rules/data/rules/highnote.yml Normal file
View file

@ -0,0 +1,53 @@
rules:
- name: Highnote API Key
id: kingfisher.highnote.1
pattern: |
(?x)
\b
(?i:highnote)
(?:.|[\n\r]){0,24}?
\b
(
(?:sk|rk)_(?:live|test)_[a-zA-Z0-9]{20,60}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'HIGHNOTE_API_KEY=sk_live_AbCdEfGhIjKlMnOpQrStUvWxYz1234'
- 'highnote_key: rk_test_AbCdEfGhIjKlMnOpQrStUvWxYz1234'
validation:
type: Http
content:
request:
method: POST
url: >
{%- if TOKEN contains "_test_" -%}
https://api.us.test.highnote.com/graphql
{%- else -%}
https://api.us.highnote.com/graphql
{%- endif -%}
headers:
Authorization: "Basic {{ TOKEN | b64enc }}"
Content-Type: application/json
Accept: application/json
body: '{"query":"query { ping }"}'
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: true
words:
- '"data"'
- '"ping"'
- '"pong"'
# Revocation not added: the public Highnote docs I found describe API key
# usage and rotation guidance, but not an API endpoint to revoke the
# current key directly.
references:
- https://docs.highnote.com/docs/developers/api/using-the-api

38
crates/kingfisher-rules/data/rules/hop.yml Normal file
View file

@ -0,0 +1,38 @@
rules:
- name: HOP Project Token
id: kingfisher.hop.1
pattern: |
(?x)
\b
(
ptk_[a-zA-Z0-9_-]{20,80}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'HOP_TOKEN=ptk_AbCdEfGhIjKlMnOpQrStUvWxYz123456'
references:
- https://docs.hop.io/reference/rest-api
- name: HOP Personal Access Token
id: kingfisher.hop.2
pattern: |
(?x)
(?:^|['"\x60\s>=:(,])
(
hop_pat_[a-zA-Z0-9_-]{20,80}
)
(?:$|['"\x60\s<),])
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'HOP_PAT="hop_pat_AbCdEfGhIjKlMnOpQrStUvWxYz123456"'
references:
- https://docs.hop.io/reference/rest-api

19
crates/kingfisher-rules/data/rules/iterative.yml Normal file
View file

@ -0,0 +1,19 @@
rules:
- name: Iterative DVC Studio Access Token
id: kingfisher.iterative.1
pattern: |
(?x)
\b
(
isat_[a-zA-Z0-9_-]{20,80}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'DVC_STUDIO_TOKEN=isat_AbCdEfGhIjKlMnOpQrStUvWxYz123456'
references:
- https://dvc.org/doc/command-reference/studio/token

39
crates/kingfisher-rules/data/rules/kraken.yml
View file

@ -26,6 +26,45 @@ rules:
examples:
- KRAKEN_API_SECRET=dGhpcy1sb29rcy1saWtlLWEtYmFzZTY0LWtyYWtlbi1zZWNyZXQtdGhhdC1pcy1sb25nLWVub3VnaA==
- kraken_secret="Aq1Bq2Cr3Ds4Et5Fu6Gv7Hw8Ix9Jy0Kz1La2Mb3Nc4Od5Pe6Qf7Rg8Sh9Ti0Uj1Vk2Wm3Xn4Yo5Za6Bc7"
validation:
type: Raw
content: kraken
depends_on_rule:
- rule_id: kingfisher.kraken.2
variable: KRAKEN_API_KEY
references:
- https://docs.kraken.com/api/docs/guides/spot-rest-auth/
- https://docs.kraken.com/api/docs/rest-api/get-account-balance/
- name: Kraken API Key
id: kingfisher.kraken.2
pattern: |
(?xi)
\b
kraken
(?:.|[\n\r]){0,32}?
(?:
api[_-]?key |
key |
public[_-]?key
)
(?:.|[\n\r]){0,12}?
(
[A-Za-z0-9]{16,64}
)
\b
pattern_requirements:
min_digits: 2
min_uppercase: 1
min_lowercase: 4
ignore_if_contains:
- your_api_key
- xxxxxx
min_entropy: 3.0
confidence: medium
visible: false
examples:
- KRAKEN_API_KEY=Ab12Cd34Ef56Gh78Ij90Kl12Mn34Op56
- 'kraken_key: 5A6b7C8d9E0f1G2h3I4J5K6L7M8N9P0Q'
references:
- https://docs.kraken.com/api/docs/guides/spot-rest-auth/

57
crates/kingfisher-rules/data/rules/kucoin.yml
View file

@ -57,6 +57,63 @@ rules:
examples:
- KUCOIN_API_SECRET=7d70f6c7-42e9-4261-8a8d-8ca2d5028d4f
- 'kucoin_secret: a1b2c3d4-e5f6-7890-abcd-ef1234567890'
validation:
type: Http
content:
request:
method: GET
url: https://api.kucoin.com/api/v1/accounts
headers:
Accept: application/json
Content-Type: application/json
KC-API-KEY: "{{ KUCOIN_KEY }}"
KC-API-TIMESTAMP: "{{ REQUEST_UNIX_MILLIS }}"
KC-API-KEY-VERSION: "2"
KC-API-PASSPHRASE: '{%- assign passphrase = KUCOIN_PASSPHRASE | hmac_sha256: TOKEN -%}{{ passphrase }}'
KC-API-SIGN: '{%- assign prehash = REQUEST_UNIX_MILLIS | append: "GET" | append: "/api/v1/accounts" -%}{{ prehash | hmac_sha256: TOKEN }}'
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: false
words:
- '"data"'
- '"code":"200000"'
depends_on_rule:
- rule_id: kingfisher.kucoin.1
variable: KUCOIN_KEY
- rule_id: kingfisher.kucoin.3
variable: KUCOIN_PASSPHRASE
references:
- https://www.kucoin.com/docs-new/authentication
- name: KuCoin API Passphrase
id: kingfisher.kucoin.3
pattern: |
(?xi)
\b
kucoin
(?:.|[\n\r]){0,32}?
(?:
api[_-]?passphrase |
passphrase
)
(?:.|[\n\r]){0,12}?
(
[A-Za-z0-9!@\#$%^&*()_+=./:-]{6,64}
)
\b
pattern_requirements:
ignore_if_contains:
- your_passphrase
- xxxxxx
min_entropy: 2.5
confidence: medium
visible: false
examples:
- KUCOIN_API_PASSPHRASE=my-strong-passphrase
- 'kucoin_passphrase: S3cur3Passphrase123'
references:
- https://www.kucoin.com/docs-new/authentication

11
crates/kingfisher-rules/data/rules/langfuse.yml
View file

@ -2,10 +2,10 @@ rules:
- name: Langfuse Secret Key
id: kingfisher.langfuse.1
pattern: |
(?xi)
(?x)
\b
(
sk-lf-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}
sk-lf-[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}
)
\b
pattern_requirements:
@ -42,10 +42,10 @@ rules:
- name: Langfuse Public Key
id: kingfisher.langfuse.2
pattern: |
(?xi)
(?x)
\b
(
pk-lf-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}
pk-lf-[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}
)
\b
pattern_requirements:
@ -57,9 +57,6 @@ rules:
examples:
- pk-lf-a1b2c3d4-e5f6-7890-abcd-ef1234567890
- 'LANGFUSE_PUBLIC_KEY="pk-lf-9f8e7d6c-5b4a-3210-fedc-ba0987654321"'
negative_examples:
- pk-lf-test
- pk-lf-
references:
- https://langfuse.com/docs/sdk/typescript
- https://langfuse.com/docs/get-started

33
crates/kingfisher-rules/data/rules/ldap.yml
View file

@ -1,4 +1,33 @@
rules:
- name: LDAP Bind URI Credentials
id: kingfisher.ldap.2
pattern: |
(?xi)
\b
(?P<TOKEN>
(?P<LDAP_SCHEME>ldaps?)://
(?P<LDAP_BIND_DN>[^:@\s]{1,128})
:
(?P<LDAP_PASSWORD>[^@\s]{6,128})
@
(?P<LDAP_HOST>[^\s/"':]{4,128})
(?::(?P<LDAP_PORT>\d{2,5}))?
(?:/[^\s"']*)?
)
pattern_requirements:
min_digits: 1
min_entropy: 2.5
confidence: medium
examples:
- ldap://cn=admin,dc=example,dc=com:Tr0ub4dor%263!@ldap.example.com:389
- ldaps://uid=svc-reader,ou=people,dc=corp,dc=example,dc=com:S3cur3BindPass@directory.corp.example.com
validation:
type: Raw
content: ldap
references:
- https://datatracker.ietf.org/doc/html/rfc4511
- https://datatracker.ietf.org/doc/html/rfc4516
- name: LDAP Credentials
id: kingfisher.ldap.1
pattern: |
@ -22,6 +51,4 @@ rules:
- "ldap_pwd = 'Tr0ub4dor&3!'"
- "LDAP_PASSWORD=s3cur3P@ssw0rd\n"
references:
- https://tools.ietf.org/html/rfc2251
# No public validation endpoint: LDAP servers are self-hosted;
# the host, port, and DN are instance-specific.
- https://datatracker.ietf.org/doc/html/rfc4511

31
crates/kingfisher-rules/data/rules/lichess.yml Normal file
View file

@ -0,0 +1,31 @@
rules:
- name: Lichess Personal Access Token
id: kingfisher.lichess.1
pattern: |
(?x)
\b
(
lip_[a-zA-Z0-9_]{16,60}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'LICHESS_TOKEN=lip_AbCdEfGhIjKlMnOpQr12'
validation:
type: Http
content:
request:
method: GET
url: https://lichess.org/api/account
headers:
Authorization: Bearer {{ TOKEN }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://lichess.org/api

21
crates/kingfisher-rules/data/rules/localstack.yml Normal file
View file

@ -0,0 +1,21 @@
rules:
- name: LocalStack Simulated AWS Access Key
id: kingfisher.localstack.1
pattern: |
(?x)
\b
(
(?:LSIA|LKIA)[A-Z0-9]{16,}
)
\b
pattern_requirements:
min_digits: 2
min_uppercase: 4
min_entropy: 3.0
confidence: medium
categories: [api, key]
examples:
- 'AWS_ACCESS_KEY_ID=LSIAQAAAAAAVNCBMPN59'
- 'aws_access_key=LKIAQAAAAAAVNCBMPN59'
references:
- https://docs.localstack.cloud/aws/capabilities/config/credentials/

32
crates/kingfisher-rules/data/rules/mailersend.yml Normal file
View file

@ -0,0 +1,32 @@
rules:
- name: MailerSend API Token
id: kingfisher.mailersend.1
pattern: |
(?x)
\b
(
mlsn\.[a-zA-Z0-9]{30,100}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'MAILERSEND_API_TOKEN=mlsn.AbCdEfGhIjKlMnOpQrStUvWxYz1234567890AbCdEfGhIj'
validation:
type: Http
content:
request:
method: GET
url: https://api.mailersend.com/v1/api-quota
headers:
Authorization: Bearer {{ TOKEN }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://www.mailersend.com/help/managing-api-tokens
- https://developers.mailersend.com/

32
crates/kingfisher-rules/data/rules/onfido.yml Normal file
View file

@ -0,0 +1,32 @@
rules:
- name: Onfido API Token
id: kingfisher.onfido.1
pattern: |
(?x)
\b
(
api_(?:live|sandbox)\.[a-zA-Z0-9_-]{20,80}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'ONFIDO_API_TOKEN=api_live.AbCdEfGhIjKlMnOpQrStUvWxYz123456'
- 'onfido_token: api_sandbox.AbCdEfGhIjKlMnOpQrStUvWxYz123456'
validation:
type: Http
content:
request:
method: GET
url: https://api.eu.onfido.com/v3.6/ping
headers:
Authorization: Token token={{ TOKEN }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://documentation.identity.entrust.com/api/latest/

19
crates/kingfisher-rules/data/rules/openvsx.yml Normal file
View file

@ -0,0 +1,19 @@
rules:
- name: OpenVSX Access Token
id: kingfisher.openvsx.1
pattern: |
(?x)
\b
(
(?:ovsxat|ovsxp)_[a-zA-Z0-9_-]{20,80}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'OVSX_PAT=ovsxat_AbCdEfGhIjKlMnOpQrStUvWxYz123456'
references:
- https://github.com/eclipse/openvsx/wiki/Publishing-Extensions

34
crates/kingfisher-rules/data/rules/paddle.yml Normal file
View file

@ -0,0 +1,34 @@
rules:
- name: Paddle API Key
id: kingfisher.paddle.1
pattern: |
(?x)
\b
(
pdl_(?:live|sdbx)_apikey_[a-z0-9_]{30,60}
)
\b
pattern_requirements:
min_digits: 2
min_lowercase: 4
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'PADDLE_API_KEY=pdl_live_apikey_01gtgztp8f4kek3yd4g1wrksa3_q6tgtjyvoiz7ldtxt65bx7_aqo'
- 'paddle_key: pdl_sdbx_apikey_01h9fjk2z4qqw8n3m7xr5bc6y1_p3rstuvwxyz'
validation:
type: Http
content:
request:
method: GET
url: https://api.paddle.com/event-types
headers:
Authorization: Bearer {{ TOKEN }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://developer.paddle.com/api-reference/about/api-keys
- https://developer.paddle.com/api-reference/about/authentication

34
crates/kingfisher-rules/data/rules/pangea.yml Normal file
View file

@ -0,0 +1,34 @@
rules:
- name: Pangea Service Token
id: kingfisher.pangea.1
pattern: |
(?x)
\b
(
pts_[a-z0-9]{20,60}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'PANGEA_TOKEN=pts_gqmqvvxk4yhirapuhw6bs7nswu'
- 'pangea_token: pts_hpbc3klkkq54tigu4osc5eygthxps6vf'
validation:
type: Http
content:
request:
method: POST
url: https://audit.aws.us.pangea.cloud/v1/log
headers:
Authorization: Bearer {{ TOKEN }}
Content-Type: application/json
body: '{"event":{"message":"test"}}'
response_matcher:
- report_response: true
- type: StatusMatch
status: [200, 401, 403]
references:
- https://pangea.cloud/docs/

34
crates/kingfisher-rules/data/rules/persona.yml Normal file
View file

@ -0,0 +1,34 @@
rules:
- name: Persona API Key
id: kingfisher.persona.1
pattern: |
(?x)
\b
(
persona_(?:production|sandbox)_[a-z0-9_-]{20,80}
)
\b
pattern_requirements:
min_digits: 2
min_lowercase: 4
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'PERSONA_API_KEY=persona_production_abc123def456ghi789jkl012mno345pqr'
- 'api_key: persona_sandbox_abc123def456ghi789jkl012mno345pqr'
validation:
type: Http
content:
request:
method: GET
url: https://withpersona.com/api/v1/accounts
headers:
Authorization: Bearer {{ TOKEN }}
Persona-Version: "2023-01-05"
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://docs.withpersona.com/api-keys

50
crates/kingfisher-rules/data/rules/pinterest.yml Normal file
View file

@ -0,0 +1,50 @@
rules:
- name: Pinterest Access Token
id: kingfisher.pinterest.1
pattern: |
(?x)
\b
(
pina_[a-zA-Z0-9_-]{20,200}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'PINTEREST_ACCESS_TOKEN=pina_AbCdEfGhIjKlMnOpQrStUvWxYz1234567890'
validation:
type: Http
content:
request:
method: GET
url: https://api.pinterest.com/v5/user_account
headers:
Authorization: Bearer {{ TOKEN }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://developers.pinterest.com/docs/api/v5/
- name: Pinterest Refresh Token
id: kingfisher.pinterest.2
pattern: |
(?x)
\b
(
pinr_[a-zA-Z0-9._-]{20,500}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'PINTEREST_REFRESH_TOKEN=pinr_eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.abc123'
references:
- https://developers.pinterest.com/docs/api/v5/oauth-token/

20
crates/kingfisher-rules/data/rules/polar.yml Normal file
View file

@ -0,0 +1,20 @@
rules:
- name: Polar Personal Access Token
id: kingfisher.polar.1
pattern: |
(?x)
\b
(
polar_(?:at|oat|rt)_[a-zA-Z0-9_-]{20,100}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'POLAR_TOKEN=polar_at_AbCdEfGhIjKlMnOpQrStUvWxYz1234'
- 'polar_org_token: polar_oat_AbCdEfGhIjKlMnOpQrStUvWx12'
references:
- https://docs.polar.sh/api/authentication

16
crates/kingfisher-rules/data/rules/posthog.yml
View file

@ -57,6 +57,22 @@ rules:
examples:
- "pha_XgrXUnvwyoPLmjwHES5lc8scZUtheBpa1QV1qmssutB"
- "pha_35kHVLA1E068nvrwUTgabkh8xvGGTpSpsVjGcpVNfis"
validation:
type: Http
content:
request:
method: GET
url: https://app.posthog.com/api/users/@me/
headers:
Authorization: "Bearer {{ TOKEN }}"
Content-Type: "application/json"
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
# Revocation not added: I did not find a documented token self-revoke
# endpoint for OAuth access tokens in the public PostHog API docs.
references:
- https://posthog.com/docs/api
- https://github.com/PostHog/posthog/blob/e408aac5debe02b39a6a67cfd028f16a2ca7bc90/posthog/models/utils.py#L260-L290

49
crates/kingfisher-rules/data/rules/proof.yml Normal file
View file

@ -0,0 +1,49 @@
rules:
- name: Proof API Key
id: kingfisher.proof.1
pattern: |
(?x)
\b
(
prf_(?:(?:cli_)?test_)?[a-zA-Z0-9_-]{20,80}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'PROOF_API_KEY=prf_AbCdEfGhIjKlMnOpQrStUvWxYz123456'
- 'proof_key: prf_test_AbCdEfGhIjKlMnOpQrStUvWxYz123456'
- 'proof_key: prf_cli_AbCdEfGhIjKlMnOpQrStUvWxYz123456'
- 'proof_key: prf_cli_test_AbCdEfGhIjKlMnOpQrStUvWxYz123456'
validation:
type: Http
content:
request:
method: POST
url: >
{%- if TOKEN contains "_test_" -%}
https://api.fairfax.proof.com/v1/transactions
{%- else -%}
https://api.proof.com/v1/transactions
{%- endif -%}
headers:
ApiKey: "{{ TOKEN }}"
Content-Type: application/json
Accept: application/json
body: '{}'
response_matcher:
- report_response: true
- type: StatusMatch
status: [422]
- type: WordMatch
words:
- signer
# Revocation not added: the public Proof docs describe dashboard key
# management and secret-scanning guidance, but not a self-revoke API.
references:
- https://dev.proof.com/docs/api-keys
- https://dev.proof.com/docs/environments
- https://dev.proof.com/reference/createtransaction

23
crates/kingfisher-rules/data/rules/rabbitmq.yml
View file

@ -3,22 +3,25 @@ rules:
id: kingfisher.rabbitmq.1
pattern: |
(?xi)
(?:
amqps?
(?P<TOKEN>
(?P<RABBITMQ_SCHEME>amqps?)
:\/\/
(?P<RABBITMQ_USERNAME>[\S]{3,50})
:
(?P<RABBITMQ_PASSWORD>[\S]{3,50})
@
(?P<RABBITMQ_HOST>[-.%\w]+)
(?::(?P<RABBITMQ_PORT>\d{2,5}))?
(?:\/(?P<RABBITMQ_VHOST>[-.%\w\/]+))?
)
:\/\/
[\S]{3,50}
:
(
[\S]{3,50}
)
@
[-.%\w\/:]+
\b
min_entropy: 3.5
confidence: medium
examples:
- amqp://user:password@rabbitmq.example.com/queue
- amqps://admin:3eCa3P@192.168.1.10:5671/vhost
validation:
type: Raw
content: rabbitmq
references:
- https://www.rabbitmq.com/uri-spec.html

21
crates/kingfisher-rules/data/rules/rainforestpay.yml Normal file
View file

@ -0,0 +1,21 @@
rules:
- name: Rainforest Pay API Key
id: kingfisher.rainforestpay.1
pattern: |
(?x)
\b
(
(?:sbx_)?apikey_[a-f0-9]{64}
)
\b
pattern_requirements:
min_digits: 4
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'RAINFOREST_API_KEY=apikey_1ad1c535b0c0093e7b9bf093d7e3444cd0e2ddefab36199216f555c3efa65d63'
- 'api_key: sbx_apikey_2bd2c646e7e1194f8c9cf194e8f4555de1f3eefbbc472003276666d4efb76e74'
references:
- https://docs.rainforestpay.com/docs/api-keys
- https://docs.rainforestpay.com/reference/authentication

19
crates/kingfisher-rules/data/rules/redis.yml
View file

@ -5,13 +5,15 @@ rules:
# Host supports hostnames, IPv4, and IPv6 in brackets
pattern: |
(?xi)
(?: redis | rediss | redis\+sentinel ) ://
(?: (?P<username>[a-zA-Z0-9%;._~!$&'()*+,;=-]*)
:
)?
(?P<password>[a-zA-Z0-9%;._~!$&'()*+,;:=/+-]{8,})
@ (?P<host>(?:\[[0-9a-fA-F:.]+\]|[a-zA-Z0-9_.-]{1,})) (?: :(?P<port>\d{1,5}))?
(?: / (?P<db>\d{1,2}))?
(?P<TOKEN>
(?: redis | rediss | redis\+sentinel ) ://
(?: (?P<username>[a-zA-Z0-9%;._~!$&'()*+,;=-]*)
:
)?
(?P<password>[a-zA-Z0-9%;._~!$&'()*+,;:=/+-]{8,})
@ (?P<host>(?:\[[0-9a-fA-F:.]+\]|[a-zA-Z0-9_.-]{1,})) (?: :(?P<port>\d{1,5}))?
(?: / (?P<db>\d{1,2}))?
)
\b
pattern_requirements:
@ -42,6 +44,9 @@ rules:
- https://redis.io/docs/latest/develop/clients/redis-py/connect/
- https://redis.io/docs/latest/commands/auth/
- https://github.com/redis/redis-py/blob/master/redis/client.py
validation:
type: Raw
content: redis
- id: kingfisher.redis.2
name: Python Redis Client Debug Output

74
crates/kingfisher-rules/data/rules/ringcentral.yml
View file

@ -51,5 +51,77 @@ rules:
- 'RINGCENTRAL_CLIENT_SECRET="xY9zW8vU7tS6rQ5pO4nM3l"'
negative_examples:
- 'RINGCENTRAL_URL="https://platform.ringcentral.com"'
validation:
type: Http
content:
request:
method: POST
url: "{{ RINGCENTRAL_BASE_URL }}/restapi/oauth/token"
headers:
Accept: application/json
Content-Type: application/x-www-form-urlencoded
Authorization: "Basic {{ CLIENT_ID | append: ':' | append: TOKEN | b64enc }}"
body: >
grant_type=authorization_code&code=INVALID_AUTH_CODE&redirect_uri={{ REDIRECT_URI | url_encode }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [400]
- type: WordMatch
match_all_words: false
words:
- invalid_grant
- authentication_error
- type: WordMatch
words:
- invalid_client
negative: true
depends_on_rule:
- rule_id: kingfisher.ringcentral.1
variable: CLIENT_ID
- rule_id: kingfisher.ringcentral.3
variable: RINGCENTRAL_BASE_URL
- rule_id: kingfisher.ringcentral.4
variable: REDIRECT_URI
references:
- https://developers.ringcentral.com/api-reference/
- https://developers.ringcentral.com/guide/authentication/auth-code-flow
- name: RingCentral OAuth Base URL
id: kingfisher.ringcentral.3
pattern: |
(?xi)
\b
(
https://platform(?:\.devtest)?\.ringcentral\.com
)
\b
min_entropy: 1.0
confidence: medium
visible: false
examples:
- RINGCENTRAL_BASE_URL=https://platform.ringcentral.com
- RINGCENTRAL_SANDBOX_URL=https://platform.devtest.ringcentral.com
references:
- https://developers.ringcentral.com/guide/authentication/auth-code-flow
- name: RingCentral Redirect URI
id: kingfisher.ringcentral.4
pattern: |
(?xi)
\b
ring.?central
(?:.|[\n\r]){0,64}?
(?:redirect[_-]?uri|oauth[_-]?redirect)\b
(?:.|[\n\r]){0,16}?
[=:"'\s]
(
https?://[^\s"'<>]{6,200}
)
min_entropy: 1.5
confidence: medium
visible: false
examples:
- RINGCENTRAL_REDIRECT_URI=https://example.com/ringcentral/callback
- 'ringcentral.redirect_uri = "https://localhost:8080/oauth/ringcentral"'
references:
- https://developers.ringcentral.com/guide/authentication/auth-code-flow

31
crates/kingfisher-rules/data/rules/rootly.yml Normal file
View file

@ -0,0 +1,31 @@
rules:
- name: Rootly API Key
id: kingfisher.rootly.1
pattern: |
(?x)
\b
(
rootly_[a-f0-9]{64}
)
\b
pattern_requirements:
min_digits: 4
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'ROOTLY_API_KEY=rootly_abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890'
validation:
type: Http
content:
request:
method: GET
url: https://api.rootly.com/v1/users/me
headers:
Authorization: Bearer {{ TOKEN }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://rootly.com/api

36
crates/kingfisher-rules/data/rules/runpod.yml Normal file
View file

@ -0,0 +1,36 @@
rules:
- name: RunPod API Key
id: kingfisher.runpod.1
pattern: |
(?x)
\b
(
rpa_[a-zA-Z0-9]{20,60}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'RUNPOD_API_KEY=rpa_ABC123DEF456GHI789JKL012MNO345PQR678'
validation:
type: Http
content:
request:
method: POST
url: https://api.runpod.io/graphql
headers:
Authorization: Bearer {{ TOKEN }}
Content-Type: application/json
body: '{"query":"{ myself { id } }"}'
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: WordMatch
match_all_words: true
words: ['"myself"']
references:
- https://docs.runpod.io/get-started/api-keys

73
crates/kingfisher-rules/data/rules/snowflake.yml
View file

@ -24,3 +24,76 @@ rules:
- https://docs.snowflake.com/en/
- https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api
# Snowflake credentials are endpoint-specific; no public REST endpoint for standalone validation.
- name: Snowflake Programmatic Access Token
id: kingfisher.snowflake.2
pattern: |
(?x)
\b
(?:
(?i:snowflake[_\s-]*(?:programmatic[_\s-]*)?(?:access[_\s-]*)?token)
|
(?i:SF_TOKEN)
)
\b
(?:.|[\n\r]){0,16}?
[=:]
\s*["']?
(
[a-zA-Z0-9_-]{100,500}
)
["']?
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'SNOWFLAKE_TOKEN=AbCdEfGhIjKlMnOpQrStUvWxYz1234567890AbCdEfGhIjKlMnOpQrStUvWxYz1234567890AbCdEfGhIjKlMnOpQrStUvWxYz12'
validation:
type: Http
content:
request:
method: POST
url: "https://{{ SNOWFLAKE_HOST }}/api/v2/statements"
headers:
Accept: application/json
Content-Type: application/json
Authorization: "Bearer {{ TOKEN }}"
X-Snowflake-Authorization-Token-Type: PROGRAMMATIC_ACCESS_TOKEN
body: '{"statement":"select 1","timeout":5}'
response_matcher:
- report_response: true
- type: StatusMatch
status: [200, 202]
- type: JsonValid
- type: WordMatch
match_all_words: false
words:
- '"statementHandle"'
- '"resultSetMetaData"'
depends_on_rule:
- rule_id: kingfisher.snowflake.3
variable: SNOWFLAKE_HOST
references:
- https://docs.snowflake.com/en/user-guide/programmatic-access-tokens
- https://docs.snowflake.com/en/developer-guide/sql-api/submitting-requests
- name: Snowflake Account Host
id: kingfisher.snowflake.3
pattern: |
(?xi)
\b
(
[a-z0-9_-]+(?:\.[a-z0-9_-]+)*\.snowflakecomputing\.com
)
\b
min_entropy: 1.0
confidence: medium
visible: false
examples:
- account = "xy12345.us-east-1.snowflakecomputing.com"
- host=acme-prod.eu-west-1.aws.snowflakecomputing.com
references:
- https://docs.snowflake.com/en/user-guide/programmatic-access-tokens
- https://docs.snowflake.com/en/developer-guide/sql-api/submitting-requests

91
crates/kingfisher-rules/data/rules/tableau.yml
View file

@ -13,11 +13,12 @@ rules:
X-Tableau-Auth
(?:.|[\n\r]){0,16}?
)
(
[A-Za-z0-9+/]{12,24}
(?:
(?P<TABLEAU_PAT_NAME>[A-Za-z0-9+/]{12,24}
(?:={1,2})?
)
:
[A-Za-z0-9+/=_-]{24,48}
(?P<TOKEN>[A-Za-z0-9+/=_-]{24,48})
)
pattern_requirements:
min_digits: 2
@ -28,7 +29,87 @@ rules:
examples:
- "tableau_auth = TSC.PersonalAccessTokenAuth('prod_svc', 'WLQKWBs1TnuBx4G7gIzz/w==:yDwZ74EWDPIgU6cSlz8RDJHp7CV2rtFP', 'companysite')"
- 'curl -H "X-Tableau-Auth:oJzK8bqwPTnmSl1/E2+aXw==:ZvTsRqFmKpWuLdNhYcBjXiGe" https://tableau.example.com/api/3.17/sites'
validation:
type: Http
content:
request:
method: POST
url: "{{ TABLEAU_SERVER }}/api/3.28/auth/signin"
headers:
Accept: application/json
Content-Type: application/json
body: >
{"credentials":{"personalAccessTokenName":"{{ TABLEAU_PAT_NAME }}","personalAccessTokenSecret":"{{ TOKEN }}","site":{"contentUrl":"{{ TABLEAU_SITE | default: "" }}"}}}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: false
words:
- '"token"'
- '"site"'
depends_on_rule:
- rule_id: kingfisher.tableau.2
variable: TABLEAU_SERVER
- rule_id: kingfisher.tableau.3
variable: TABLEAU_SITE
references:
- https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_ref.htm
- https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_ref_authentication.htm
- https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm
# Tableau PATs are instance-specific; no universal public endpoint for standalone validation.
- name: Tableau Server URL
id: kingfisher.tableau.2
pattern: |
(?xi)
\b
(
https://(?:
(?:[a-z0-9-]+\.)?online\.tableau\.com
|
(?:[a-z0-9-]+\.)*tableau(?:\.[a-z0-9-]+)+
)
)
(?:
/api/\d+\.\d+
)?
(?:
/[^\s"'<>]{0,120}
)?
min_entropy: 1.5
confidence: medium
visible: false
examples:
- https://tableau.example.com
- https://10ax.online.tableau.com
- server="https://analytics.tableau.example.com"
references:
- https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_ref_authentication.htm
- name: Tableau Site Content URL
id: kingfisher.tableau.3
pattern: |
(?xi)
\b
(?:
tableau[_-]?(?:site|content[_-]?url)
|
tableau
(?:.|[\n\r]){0,48}?
(?:site|content[_-]?url)
)
(?:.|[\n\r]){0,12}?
[=:"'\s]
(
[A-Za-z0-9._-]{1,64}
)
\b
min_entropy: 1.0
confidence: medium
visible: false
examples:
- tableau_site=companysite
- tableau_content_url="default"
references:
- https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_ref_authentication.htm

31
crates/kingfisher-rules/data/rules/telnyx.yml Normal file
View file

@ -0,0 +1,31 @@
rules:
- name: Telnyx API V2 Key
id: kingfisher.telnyx.1
pattern: |
(?x)
\b
(
KEY[0-9A-Za-z_-]{55}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'TELNYX_API_KEY=KEYabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRS'
validation:
type: Http
content:
request:
method: GET
url: https://api.telnyx.com/v2/balance
headers:
Authorization: Bearer {{ TOKEN }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
references:
- https://developers.telnyx.com/development/api-fundamentals/authentication

19
crates/kingfisher-rules/data/rules/thunderstore.yml Normal file
View file

@ -0,0 +1,19 @@
rules:
- name: Thunderstore API Token
id: kingfisher.thunderstore.1
pattern: |
(?x)
\b
(
tss_[a-zA-Z0-9_-]{20,80}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: medium
categories: [api, key]
examples:
- 'THUNDERSTORE_TOKEN=tss_AbCdEfGhIjKlMnOpQrStUvWxYz123456'
references:
- https://thunderstore.io/api/docs/

54
crates/kingfisher-rules/data/rules/trello.yml
View file

@ -27,5 +27,59 @@ rules:
examples:
- TRELLO_TOKEN=0a1b2c3d4e5f6g7h8i9j0k1l2m3n4p5q
- trello_access_token="Ab12Cd34Ef56Gh78Ij90Kl12Mn34Op56"
validation:
type: Http
content:
request:
method: GET
url: "https://api.trello.com/1/members/me?key={{ TRELLO_KEY | url_encode }}&token={{ TOKEN | url_encode }}"
headers:
Accept: application/json
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: false
words:
- '"id"'
- '"username"'
depends_on_rule:
- rule_id: kingfisher.trello.2
variable: TRELLO_KEY
references:
- https://developer.atlassian.com/cloud/trello/guides/rest-api/api-introduction/
- https://developer.atlassian.com/cloud/trello/guides/rest-api/authorization/
- name: Trello API Key
id: kingfisher.trello.2
visible: false
pattern: |
(?xi)
\b
trello
(?:.|[\n\r]){0,32}?
(?:
api[_-]?key |
app[_-]?key |
key
)
(?:.|[\n\r]){0,12}?
(
[A-Za-z0-9]{32}
)
\b
pattern_requirements:
min_digits: 2
min_lowercase: 6
ignore_if_contains:
- yourkey
- placeholder
min_entropy: 3.1
confidence: medium
examples:
- TRELLO_KEY=0a1b2c3d4e5f6g7h8i9j0k1l2m3n4p5q
- trello_api_key="Ab12Cd34Ef56Gh78Ij90Kl12Mn34Op56"
references:
- https://developer.atlassian.com/cloud/trello/guides/rest-api/authorization/

24
crates/kingfisher-rules/data/rules/ubidots.yml
View file

@ -16,7 +16,25 @@ rules:
examples:
- "API_KEY = \"BBUS-kDvT2Vrm6JThnHZvgzNyO2K7DAHdWs12abc\""
- "UBIDOTS_TOKEN=BBUS-AbCdEfGhIjKlMnOpQrStUvWxYz0123456"
validation:
type: Http
content:
request:
method: GET
url: https://industrial.api.ubidots.com/api/v2.0/devices
headers:
Accept: application/json
X-Auth-Token: "{{ TOKEN }}"
response_matcher:
- report_response: true
- type: StatusMatch
status: [200]
- type: JsonValid
- type: WordMatch
match_all_words: false
words:
- '"count"'
- '"results"'
references:
- https://docs.ubidots.com/v1.6/reference/authentication
# No API validation available: Ubidots API keys generate tokens but cannot
# themselves be validated against a public endpoint.
- https://docs.ubidots.com/reference/authentication
- https://docs.ubidots.com/reference/get-devices

19
crates/kingfisher-rules/data/rules/valtown.yml Normal file
View file

@ -0,0 +1,19 @@
rules:
- name: Val Town API Token
id: kingfisher.valtown.1
pattern: |
(?x)
\b
(
vtwn_[a-zA-Z0-9_-]{20,80}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.5
confidence: high
categories: [api, key]
examples:
- 'VALTOWN_TOKEN=vtwn_AbCdEfGhIjKlMnOpQrStUvWxYz123456'
references:
- https://docs.val.town/api/authentication/

19
crates/kingfisher-rules/data/rules/volcengine.yml Normal file
View file

@ -0,0 +1,19 @@
rules:
- name: VolcEngine Access Key ID
id: kingfisher.volcengine.1
pattern: |
(?x)
\b
(
AKLT[a-zA-Z0-9_-]{16,60}
)
\b
pattern_requirements:
min_digits: 2
min_entropy: 3.0
confidence: medium
categories: [api, key]
examples:
- 'VOLCENGINE_ACCESS_KEY=AKLTabcdefghijklmnop1234567890'
references:
- https://www.volcengine.com/docs/6291/65568

53
crates/kingfisher-rules/data/rules/webex.yml
View file

@ -47,6 +47,57 @@ rules:
examples:
- "webex.secret = 8ab9b3c77035e1121e2d7d64529749682b3ce5b93dc1f1e6677f0800dcf00d1e"
- "webex\nclient_secret=1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b"
validation:
type: Http
content:
request:
method: POST
url: https://webexapis.com/v1/access_token
headers:
Accept: application/json
Content-Type: application/x-www-form-urlencoded
body: >
grant_type=authorization_code&client_id={{ CLIENT_ID | url_encode }}&client_secret={{ TOKEN | url_encode }}&code=INVALID_AUTH_CODE&redirect_uri={{ REDIRECT_URI | url_encode }}
response_matcher:
- report_response: true
- type: StatusMatch
status: [400]
- type: WordMatch
match_all_words: false
words:
- invalid_grant
- Invalid authorization code
- type: WordMatch
words:
- invalid_client
negative: true
depends_on_rule:
- rule_id: kingfisher.webex.1
variable: CLIENT_ID
- rule_id: kingfisher.webex.3
variable: REDIRECT_URI
references:
- https://developer.webex.com/docs/platform-introduction
- https://developer.webex.com/create/docs/authentication
- https://developer.webex.com/docs/integrations
- name: Webex Redirect URI
id: kingfisher.webex.3
pattern: |
(?xi)
\b
webex
(?:.|[\n\r]){0,64}?
(?:redirect[_-]?uri|oauth[_-]?redirect)\b
(?:.|[\n\r]){0,16}?
[=:"'\s]
(
https?://[^\s"'<>]{6,200}
)
min_entropy: 1.5
confidence: medium
visible: false
examples:
- WEBEX_REDIRECT_URI=https://example.com/webex/callback
- 'webex.redirect_uri = "https://localhost:3000/oauth/webex"'
references:
- https://developer.webex.com/create/docs/authentication

101
crates/kingfisher-rules/src/liquid_filters.rs
View file

@ -12,7 +12,10 @@ use percent_encoding::{utf8_percent_encode, NON_ALPHANUMERIC};
use rand::{distr::Alphanumeric, RngExt};
use sha1::Sha1;
use sha2::{Digest, Sha256, Sha384};
use time::{format_description::well_known::Iso8601, OffsetDateTime};
use time::{
format_description::well_known::{Iso8601, Rfc2822},
OffsetDateTime,
};
use uuid::Uuid;
// -----------------------------------------------------------------------------
@ -297,6 +300,42 @@ impl Filter for HmacSha384Filter {
}
}
#[derive(Clone, ParseFilter, FilterReflection, Default)]
#[filter(
name = "hmac_sha384_hex",
description = "HMAC-SHA384 - returns lowercase hex.",
parameters(Hmac384Args),
parsed(HmacSha384HexFilter)
)]
pub struct HmacSha384Hex;
#[derive(Debug, FromFilterParameters, Display_filter)]
#[name = "hmac_sha384_hex"]
struct HmacSha384HexFilter {
#[parameters]
args: Hmac384Args,
}
impl Filter for HmacSha384HexFilter {
fn evaluate(&self, input: &dyn ValueView, runtime: &dyn Runtime) -> Result<Value> {
use std::fmt::Write as _;
let args = self.args.evaluate(runtime)?;
let key = args.key.to_kstr();
let mut mac = Hmac::<Sha384>::new_from_slice(key.as_bytes()).unwrap();
mac.update(input.to_kstr().as_bytes());
let bytes = mac.finalize().into_bytes();
let mut hex = String::with_capacity(bytes.len() * 2);
for byte in bytes {
let _ = write!(&mut hex, "{byte:02x}");
}
Ok(Value::scalar(hex))
}
}
// ── random_string ────────────────────────────────
#[derive(Debug, FilterParameters)]
struct RandomStringArgs {
@ -903,6 +942,15 @@ static_filter!(
}
);
// {{ "" | unix_timestamp_ms }}
static_filter!(
/// Current Unix epoch milliseconds.
UnixTimestampMsFilter, "unix_timestamp_ms",
|_input: &dyn ValueView| -> i64 {
(OffsetDateTime::now_utc().unix_timestamp_nanos() / 1_000_000) as i64
}
);
// {{ "" | iso_timestamp_no_frac }}
static_filter!(
/// Current ISO-8601 timestamp (UTC) with no fractional seconds.
@ -933,6 +981,21 @@ static_filter!(
}
);
// {{ "" | rfc1123_date }}
static_filter!(
/// Current RFC-1123 timestamp in GMT.
Rfc1123DateFilter, "rfc1123_date",
|_input: &dyn ValueView| -> String {
let rendered = OffsetDateTime::now_utc()
.format(&Rfc2822)
.unwrap_or_else(|_| "Thu, 01 Jan 1970 00:00:00 +0000".into());
rendered
.strip_suffix(" +0000")
.map(|prefix| format!("{prefix} GMT"))
.unwrap_or(rendered)
}
);
// -----------------------------------------------------------------------------
// Request Uniqueness
// -----------------------------------------------------------------------------
@ -953,8 +1016,10 @@ pub fn register_all(builder: liquid::ParserBuilder) -> liquid::ParserBuilder {
.filter(UrlEncodeFilter::default())
.filter(JsonEscapeFilter::default())
.filter(UnixTimestampFilter::default())
.filter(UnixTimestampMsFilter::default())
.filter(IsoTimestampFilter::default())
.filter(IsoTimestampNoFracFilter::default())
.filter(Rfc1123DateFilter::default())
.filter(UuidFilter::default())
.filter(JwtHeaderFilter::default())
.filter(B64EncFilter::default())
@ -974,6 +1039,7 @@ pub fn register_all(builder: liquid::ParserBuilder) -> liquid::ParserBuilder {
.filter(HmacSha256B64Key::default())
.filter(HmacSha1::default())
.filter(HmacSha384::default())
.filter(HmacSha384Hex::default())
}
#[cfg(test)]
@ -1148,6 +1214,24 @@ mod tests {
assert_eq!(render(r#"{{ "payload" | hmac_sha384: "topsecret" }}"#), expect);
}
#[test]
fn hmac_sha384_hex_filter() {
use std::fmt::Write as _;
let key = b"topsecret";
let data = b"payload";
let mut mac = Hmac::<Sha384>::new_from_slice(key).unwrap();
mac.update(data);
let bytes = mac.finalize().into_bytes();
let mut expect = String::with_capacity(bytes.len() * 2);
for byte in bytes {
let _ = write!(&mut expect, "{byte:02x}");
}
assert_eq!(render(r#"{{ "payload" | hmac_sha384_hex: "topsecret" }}"#), expect);
}
// -------------------------------------------------------------------------
// Random string
// -------------------------------------------------------------------------
@ -1174,6 +1258,13 @@ mod tests {
assert!((now - tmpl_val).abs() < 5, "timestamp differs by >5 s");
}
#[test]
fn unix_timestamp_ms_filter_is_nowish() {
let tmpl_val: i64 = render(r#"{{ "" | unix_timestamp_ms }}"#).parse().unwrap();
let now = (OffsetDateTime::now_utc().unix_timestamp_nanos() / 1_000_000) as i64;
assert!((now - tmpl_val).abs() < 5_000, "timestamp differs by >5 s");
}
#[test]
fn iso_timestamp_filter_parses() {
let out = render(r#"{{ "" | iso_timestamp }}"#);
@ -1192,6 +1283,14 @@ mod tests {
let v = render(r#"{{ "" | uuid }}"#);
assert!(uuid_re.is_match(&v));
}
#[test]
fn rfc1123_date_filter_format() {
let out = render(r#"{{ "" | rfc1123_date }}"#);
assert!(out.ends_with(" GMT"), "unexpected RFC-1123 date: {out}");
let normalized = out.replace(" GMT", " +0000");
assert!(OffsetDateTime::parse(&normalized, &Rfc2822).is_ok());
}
// -------------------------------------------------------------------------
// Replace filter
// -------------------------------------------------------------------------

94
crates/kingfisher-rules/src/rules_database.rs
View file

@ -60,21 +60,7 @@ impl RulesDatabase {
let mut reason_codes: Vec<&'static str> = Vec::new();
let has_self_identifying_prefix = [
"ccipat_",
"xoxb-",
"xoxa-",
"xoxp-",
"xapp-",
"ghp_",
"github_pat_",
"sk_live_",
"sk_test_",
"ltai",
"akia",
]
.iter()
.any(|m| normalized.contains(m));
let has_self_identifying_prefix = has_self_identifying_shape(&normalized);
if has_self_identifying_prefix {
reason_codes.push("self_identifying_prefix");
return RuleMatchProfile {
@ -307,6 +293,33 @@ impl RulesDatabase {
}
}
fn has_self_identifying_shape(normalized_pattern: &str) -> bool {
let literal_markers = [
"ccipat_",
"xapp-",
"ghp_",
"github_pat_",
"sk_live_",
"sk_test_",
"ltai",
"akia",
"aizasy",
"pypi-ageichlwas5vcmc",
"https://hooks\\.slack\\.com/services/",
];
literal_markers.iter().any(|needle| normalized_pattern.contains(needle))
|| normalized_pattern.contains("xox[pbarose]")
|| normalized_pattern.contains("xoxe\\.xox[bparose]-")
|| normalized_pattern.contains("xoxe-\\d-")
|| (normalized_pattern.contains("-----begin\\s")
&& normalized_pattern.contains("private\\skey")
&& normalized_pattern.contains("-----end\\s"))
|| (normalized_pattern.contains("-----begin\\ ")
&& normalized_pattern.contains("private\\ key")
&& normalized_pattern.contains("-----end\\ "))
}
fn has_generic_token_class(normalized_pattern: &str) -> bool {
[
"[a-za-z0-9]{",
@ -436,6 +449,57 @@ mod test_rule_match_profiles {
assert!(profile.reason_codes.contains(&"self_identifying_prefix"));
}
#[test]
fn classifies_google_api_key_rule_as_self_identifying() {
let rule = mk_rule("kingfisher.google.7", r"(?xi)\b(AIzaSy[A-Za-z0-9_-]{33})");
let profile = RulesDatabase::classify_rule_profile(&rule);
assert_eq!(profile.kind, RuleDetectionProfileKind::SelfIdentifying);
}
#[test]
fn classifies_slack_token_charclass_rule_as_self_identifying() {
let rule = mk_rule(
"kingfisher.slack.2",
r"(?xi)\b(xox[pbarose][-0-9]{0,3}-[0-9a-z]{6,15}-[0-9a-z]{6,15}-[-0-9a-z]{6,66})\b",
);
let profile = RulesDatabase::classify_rule_profile(&rule);
assert_eq!(profile.kind, RuleDetectionProfileKind::SelfIdentifying);
}
#[test]
fn classifies_slack_webhook_rule_as_self_identifying() {
let rule = mk_rule(
"kingfisher.slack.4",
r"(?xi)\b(https://hooks\.slack\.com/services/T[a-z0-9_-]{8,12}/B[a-z0-9_-]{8,12}/[a-z0-9_-]{20,30})",
);
let profile = RulesDatabase::classify_rule_profile(&rule);
assert_eq!(profile.kind, RuleDetectionProfileKind::SelfIdentifying);
}
#[test]
fn classifies_pypi_token_rule_as_self_identifying() {
let rule = mk_rule("kingfisher.pypi.1", r"(?x)(pypi-AgEIcHlwaS5vcmc[A-Za-z0-9_-]{50,})\b");
let profile = RulesDatabase::classify_rule_profile(&rule);
assert_eq!(profile.kind, RuleDetectionProfileKind::SelfIdentifying);
}
#[test]
fn classifies_private_key_envelope_rules_as_self_identifying() {
let rule = mk_rule(
"kingfisher.privkey.2",
r"(?xims)(-----BEGIN\s(?:RSA|PGP|DSA|OPENSSH|ENCRYPTED|EC)?\s{0,1}PRIVATE\sKEY-----[a-z0-9 /+=\r\n\\n]{32,}?-----END\s(?:RSA|PGP|DSA|OPENSSH|ENCRYPTED|EC)?\s{0,1}PRIVATE\sKEY-----)",
);
let profile = RulesDatabase::classify_rule_profile(&rule);
assert_eq!(profile.kind, RuleDetectionProfileKind::SelfIdentifying);
let pem_rule = mk_rule(
"kingfisher.pem.1",
r#"(?x)-----BEGIN\ .{0,20}\ ?PRIVATE\ KEY\ ?.{0,20}-----\s*((?:[a-zA-Z0-9+/=\s"',]|\\r|\\n){50,})\s*-----END\ .{0,20}\ ?PRIVATE\ KEY\ ?.{0,20}-----"#,
);
let pem_profile = RulesDatabase::classify_rule_profile(&pem_rule);
assert_eq!(pem_profile.kind, RuleDetectionProfileKind::SelfIdentifying);
}
#[test]
fn classifies_context_dependent_generic_rule() {
let rule = mk_rule(

34
crates/kingfisher-scanner/Cargo.toml
View file

@ -24,6 +24,22 @@ validation-http = [
"dep:liquid-core",
"dep:quick-xml",
"dep:sha1",
"dep:time",
]
# Provider/protocol-specific validation flows that need custom network logic.
validation-raw = [
"validation-http",
"dep:chrono",
"dep:hmac",
"dep:sha2",
"dep:hex",
"dep:url",
"dep:percent-encoding",
"dep:rustls",
"dep:rustls-native-certs",
"dep:tokio-rustls",
"dep:ldap3",
]
# AWS credential validation
@ -94,6 +110,7 @@ validation-database = [
# All validation features
validation-all = [
"validation",
"validation-raw",
"validation-aws",
"validation-azure",
"validation-coinbase",
@ -153,11 +170,12 @@ tracing.workspace = true
reqwest = { version = "0.12", default-features = false, features = [
"json", "gzip", "brotli", "deflate", "stream", "rustls-tls", "rustls-tls-native-roots", "multipart"
], optional = true }
tokio = { version = "1.48", features = ["net", "time", "sync"], optional = true }
tokio = { version = "1.51", features = ["net", "time", "sync", "io-util"], optional = true }
liquid = { version = "0.26", optional = true }
liquid-core = { version = "0.26", optional = true }
quick-xml = { version = "0.39", features = ["serde", "serialize"], optional = true }
sha1 = { workspace = true, optional = true }
time = { workspace = true, optional = true }
chrono = { version = "0.4.42", optional = true }
hmac = { workspace = true, optional = true }
sha2 = { workspace = true, optional = true }
@ -165,7 +183,7 @@ pem = { version = "3.0.6", optional = true }
percent-encoding = { workspace = true, optional = true }
ring = { version = "0.17", optional = true }
jsonwebtoken = { version = "10.2.0", features = ["aws-lc-rs"], optional = true }
jsonwebtoken = { version = "10.3.0", features = ["aws-lc-rs"], optional = true }
p256 = { version = "0.13.2", optional = true }
ed25519-dalek = { version = "2.2", features = ["pkcs8"], optional = true }
hex = { workspace = true, optional = true }
@ -176,7 +194,7 @@ tokio-postgres = { version = "0.7", default-features = false, features = ["runti
tokio-postgres-rustls = { version = "0.13.0", optional = true }
rustls = { version = "0.23.35", optional = true }
rustls-native-certs = { version = "0.8.2", optional = true }
tokio-rustls = { version = "0.26.4", optional = true }
# AWS validation
aws-config = { version = "1.8.14", default-features = false, features = ["default-https-client", "rt-tokio"], optional = true }
aws-credential-types = { version = "1.2.12", optional = true }
@ -190,7 +208,15 @@ base32 = { version = "0.5", optional = true }
byteorder = { version = "1.5", optional = true }
rand = { version = "0.10", optional = true }
[target.'cfg(all(windows, target_arch = "aarch64"))'.dependencies]
# ldap3's rustls backend still pulls ring 0.16, which fails to build on Windows ARM64.
# Use the platform TLS backend there to keep the raw LDAP validator available.
ldap3 = { version = "0.11.5", default-features = false, features = ["tls-native"], optional = true }
[target.'cfg(not(all(windows, target_arch = "aarch64")))'.dependencies]
ldap3 = { version = "0.11.5", default-features = false, features = ["tls-rustls"], optional = true }
[dev-dependencies]
pretty_assertions = "1.4"
tempfile = "3.23"
tokio = { version = "1.48", features = ["macros", "rt"] }
tokio = { version = "1.51", features = ["macros", "rt"] }

39
crates/kingfisher-scanner/src/scanner.rs
View file

@ -14,6 +14,8 @@ use crate::finding::{Finding, FindingLocation};
use crate::primitives;
use crate::scanner_pool::ScannerPool;
const RAW_MATCH_LOOKBACK: usize = 64 * 1024;
/// Configuration options for the scanner.
#[derive(Debug, Clone)]
pub struct ScannerConfig {
@ -26,7 +28,7 @@ pub struct ScannerConfig {
/// Override the minimum entropy threshold for all rules.
pub min_entropy_override: Option<f32>,
/// Language hint for tree-sitter parsing (e.g., "python", "javascript").
/// Language hint for parser-based context verification (e.g., "python", "javascript").
pub language_hint: Option<String>,
/// Whether to redact secrets in findings.
@ -167,9 +169,14 @@ impl Scanner {
// Process matches through regex
let mut findings = Vec::new();
let mut seen_matches: FxHashSet<u64> = FxHashSet::default();
let mut previous_spans: FxHashMap<usize, Vec<OffsetSpan>> = FxHashMap::default();
let mut seen_raw_match_ends: FxHashSet<(usize, usize)> = FxHashSet::default();
let mut previous_full_spans: FxHashMap<usize, Vec<OffsetSpan>> = FxHashMap::default();
for (rule_id, start, end) in raw_matches.into_iter().rev() {
let _ = start; // Block-mode Vectorscan reports `from` as 0 unless SOM is enabled.
if !seen_raw_match_ends.insert((rule_id, end)) {
continue;
}
let rule = match self.rules_db.get_rule(rule_id) {
Some(r) => r,
None => continue,
@ -180,16 +187,18 @@ impl Scanner {
Err(_) => continue,
};
let current_span = OffsetSpan::from_range(start..end);
// Check for overlapping spans
if !primitives::record_match(&mut previous_spans, rule_id, current_span) {
continue;
}
let haystack = &bytes[start..end];
let scan_start = end.saturating_sub(RAW_MATCH_LOOKBACK);
let haystack = &bytes[scan_start..end];
for captures in anchored_regex.captures_iter(haystack) {
let full_capture = captures.get(0).unwrap();
let full_capture_span = OffsetSpan::from_range(
(scan_start + full_capture.start())..(scan_start + full_capture.end()),
);
if !primitives::record_match(&mut previous_full_spans, rule_id, full_capture_span) {
continue;
}
// Get the primary secret value
let secret_capture = primitives::find_secret_capture(&anchored_regex, &captures);
let secret_bytes = secret_capture.as_bytes();
@ -203,20 +212,20 @@ impl Scanner {
}
// Compute match key for dedup
let offset_start = scan_start + secret_capture.start();
let offset_end = scan_start + secret_capture.end();
let match_key = primitives::compute_match_key(
secret_bytes,
rule.id().as_bytes(),
start + secret_capture.start(),
start + secret_capture.end(),
offset_start,
offset_end,
);
if !seen_matches.insert(match_key) {
continue;
}
// Build the finding
let offset_span = OffsetSpan::from_range(
(start + secret_capture.start())..(start + secret_capture.end()),
);
let offset_span = OffsetSpan::from_range(offset_start..offset_end);
let source_span = loc_mapping.get_source_span(&offset_span);
let secret = if self.config.redact_secrets {

101
crates/kingfisher-scanner/src/validation/http_validation.rs
View file

@ -3,6 +3,7 @@ use std::{collections::BTreeMap, future::Future, net::IpAddr, str::FromStr, time
use anyhow::{anyhow, Error, Result};
use http::StatusCode;
use liquid::Object;
use liquid_core::Value;
use quick_xml::de::from_str as xml_from_str;
use reqwest::{
header,
@ -11,6 +12,7 @@ use reqwest::{
};
use serde::de::IgnoredAny;
use sha1::{Digest, Sha1};
use time::{format_description::well_known::Rfc2822, OffsetDateTime};
use tokio::{net::lookup_host, time::sleep};
use tracing::debug;
@ -34,12 +36,11 @@ use kingfisher_rules::ResponseMatcher;
/// Build a deterministic cache key from the immutable parts of an HTTP request.
pub fn generate_http_cache_key_parts(
method: &str,
url: &Url,
url: &str,
headers: &BTreeMap<String, String>,
body: Option<&str>,
) -> String {
let method = method.to_uppercase();
let url = url.as_str();
let mut hasher = Sha1::new();
hasher.update(method.as_bytes());
@ -68,6 +69,52 @@ pub fn parse_http_method(method_str: &str) -> Result<Method, String> {
Method::from_str(method_str).map_err(|_| format!("Invalid HTTP method: {}", method_str))
}
fn format_rfc1123(now: OffsetDateTime) -> String {
let rendered =
now.format(&Rfc2822).unwrap_or_else(|_| "Thu, 01 Jan 1970 00:00:00 +0000".to_string());
rendered.strip_suffix(" +0000").map(|prefix| format!("{prefix} GMT")).unwrap_or(rendered)
}
pub fn is_auto_provided_request_var(var: &str) -> bool {
matches!(var, "REQUEST_RFC1123_DATE" | "REQUEST_UNIX_MILLIS")
}
/// Clone `globals` and add stable request-scoped values for templated request rendering.
///
/// These values are computed once so the same generated timestamp can be reused across the URL,
/// headers, body, and multipart parts of a single request.
pub fn with_request_template_globals(globals: &Object) -> Object {
let mut out = globals.clone();
let now = OffsetDateTime::now_utc();
if !out.contains_key("REQUEST_RFC1123_DATE") {
out.insert("REQUEST_RFC1123_DATE".into(), Value::scalar(format_rfc1123(now)));
}
if !out.contains_key("REQUEST_UNIX_MILLIS") {
out.insert(
"REQUEST_UNIX_MILLIS".into(),
Value::scalar((now.unix_timestamp_nanos() / 1_000_000).to_string()),
);
}
out
}
/// Clone `globals` and add stable placeholder values for request-scoped template vars that
/// would otherwise make HTTP validation cache keys vary per execution.
pub fn with_cache_key_template_globals(globals: &Object) -> Object {
let mut out = globals.clone();
if !out.contains_key("REQUEST_RFC1123_DATE") {
out.insert("REQUEST_RFC1123_DATE".into(), Value::scalar("REQUEST_RFC1123_DATE"));
}
if !out.contains_key("REQUEST_UNIX_MILLIS") {
out.insert("REQUEST_UNIX_MILLIS".into(), Value::scalar("REQUEST_UNIX_MILLIS"));
}
out
}
/// Build a reqwest RequestBuilder using the provided parameters.
pub fn build_request_builder(
client: &Client,
@ -566,7 +613,57 @@ pub async fn check_url_resolvable_safe(url: &Url) -> Result<(), Box<dyn std::err
#[cfg(test)]
mod tests {
use super::*;
use liquid_core::ValueView;
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use time::OffsetDateTime;
#[test]
fn request_template_globals_add_stable_values() {
let globals = Object::new();
let rendered = with_request_template_globals(&globals);
let date = rendered.get("REQUEST_RFC1123_DATE").unwrap().to_kstr().to_string();
let millis = rendered.get("REQUEST_UNIX_MILLIS").unwrap().to_kstr().to_string();
assert!(date.ends_with(" GMT"), "unexpected date format: {date}");
assert!(OffsetDateTime::parse(&date.replace(" GMT", " +0000"), &Rfc2822).is_ok());
let millis_val: i128 = millis.parse().unwrap();
assert!(millis_val > 0);
}
#[test]
fn request_template_globals_preserve_explicit_overrides() {
let mut globals = Object::new();
globals.insert("REQUEST_RFC1123_DATE".into(), Value::scalar("custom-date"));
globals.insert("REQUEST_UNIX_MILLIS".into(), Value::scalar("123"));
let rendered = with_request_template_globals(&globals);
assert_eq!(rendered.get("REQUEST_RFC1123_DATE").unwrap().to_kstr(), "custom-date");
assert_eq!(rendered.get("REQUEST_UNIX_MILLIS").unwrap().to_kstr(), "123");
}
#[test]
fn cache_key_template_globals_use_stable_placeholders() {
let globals = Object::new();
let rendered = with_cache_key_template_globals(&globals);
assert_eq!(rendered.get("REQUEST_RFC1123_DATE").unwrap().to_kstr(), "REQUEST_RFC1123_DATE");
assert_eq!(rendered.get("REQUEST_UNIX_MILLIS").unwrap().to_kstr(), "REQUEST_UNIX_MILLIS");
}
#[test]
fn cache_key_template_globals_preserve_explicit_overrides() {
let mut globals = Object::new();
globals.insert("REQUEST_RFC1123_DATE".into(), Value::scalar("custom-date"));
globals.insert("REQUEST_UNIX_MILLIS".into(), Value::scalar("123"));
let rendered = with_cache_key_template_globals(&globals);
assert_eq!(rendered.get("REQUEST_RFC1123_DATE").unwrap().to_kstr(), "custom-date");
assert_eq!(rendered.get("REQUEST_UNIX_MILLIS").unwrap().to_kstr(), "123");
}
#[test]
fn rejects_ipv4_loopback() {

10
crates/kingfisher-scanner/src/validation/mod.rs
View file

@ -20,6 +20,8 @@
//! - **Azure**: Azure Storage credential validation (requires `validation-azure` feature)
//! - **Databases**: MongoDB, MySQL, Postgres, JDBC (requires `validation-database` feature)
//! - **JWT**: JWT token validation (requires `validation-jwt` feature)
//! - **Raw**: provider/protocol-specific validators that need custom logic
//! (requires `validation-raw` feature)
mod utils;
mod validation_body;
@ -54,6 +56,9 @@ pub mod mysql;
#[cfg(feature = "validation-database")]
pub mod postgres;
#[cfg(feature = "validation-raw")]
pub mod raw;
// Re-exports
pub use utils::{find_closest_variable, process_captures};
pub use validation_body::{as_str, clone_as_string, from_string, ValidationResponseBody};
@ -62,9 +67,12 @@ pub use validation_body::{as_str, clone_as_string, from_string, ValidationRespon
pub use http_validation::{
build_request_builder, check_url_resolvable, generate_http_cache_key_parts, is_ssrf_safe_ip,
parse_http_method, process_headers, retry_multipart_request, retry_request, validate_response,
SsrfBlockedError,
with_request_template_globals, SsrfBlockedError,
};
#[cfg(feature = "validation-raw")]
pub use raw::{required_vars as raw_required_vars, validate_raw, RawValidationOutcome};
#[cfg(feature = "validation-http")]
#[allow(deprecated)]
pub use http_validation::check_url_resolvable_safe;

639
crates/kingfisher-scanner/src/validation/raw.rs Normal file
View file

@ -0,0 +1,639 @@
//! Provider-specific raw validators for secret formats that need custom protocol logic.
use std::{
collections::BTreeSet,
sync::{Arc, OnceLock},
time::{Duration, SystemTime, UNIX_EPOCH},
};
use anyhow::{anyhow, Context, Result};
use base64::{engine::general_purpose::STANDARD as B64, Engine};
use hmac::{digest::KeyInit, Hmac, Mac};
use http::StatusCode;
use ldap3::LdapConnSettings;
use liquid::Object;
use liquid_core::ValueView;
use once_cell::sync::OnceCell;
use percent_encoding::percent_decode_str;
use reqwest::Client;
use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier};
use rustls::crypto::{ring, verify_tls12_signature, verify_tls13_signature, CryptoProvider};
use rustls::pki_types::{CertificateDer, ServerName, UnixTime};
use rustls::{ClientConfig, DigitallySignedStruct, RootCertStore, SignatureScheme};
use sha2::{Digest, Sha256, Sha512};
use tokio::{
io::{AsyncBufReadExt, AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt, BufStream},
net::TcpStream,
time::timeout,
};
use tokio_rustls::TlsConnector;
use url::Url;
use crate::validation::http_validation::check_url_resolvable;
pub struct RawValidationOutcome {
pub valid: bool,
pub status: StatusCode,
pub body: String,
}
static INIT_PROVIDER: OnceCell<()> = OnceCell::new();
static LAX_PROVIDER: OnceLock<Arc<CryptoProvider>> = OnceLock::new();
fn ensure_crypto_provider() {
INIT_PROVIDER.get_or_init(|| {
let _ = CryptoProvider::install_default(ring::default_provider());
});
}
#[derive(Debug)]
struct LaxCertVerifier(Arc<CryptoProvider>);
impl ServerCertVerifier for LaxCertVerifier {
fn verify_server_cert(
&self,
_end_entity: &CertificateDer<'_>,
_intermediates: &[CertificateDer<'_>],
_server_name: &ServerName<'_>,
_ocsp_response: &[u8],
_now: UnixTime,
) -> std::result::Result<ServerCertVerified, rustls::Error> {
Ok(ServerCertVerified::assertion())
}
fn verify_tls12_signature(
&self,
message: &[u8],
cert: &CertificateDer<'_>,
dss: &DigitallySignedStruct,
) -> std::result::Result<HandshakeSignatureValid, rustls::Error> {
verify_tls12_signature(message, cert, dss, &self.0.signature_verification_algorithms)
}
fn verify_tls13_signature(
&self,
message: &[u8],
cert: &CertificateDer<'_>,
dss: &DigitallySignedStruct,
) -> std::result::Result<HandshakeSignatureValid, rustls::Error> {
verify_tls13_signature(message, cert, dss, &self.0.signature_verification_algorithms)
}
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
self.0.signature_verification_algorithms.supported_schemes()
}
}
pub fn required_vars(kind: &str) -> BTreeSet<String> {
let mut vars = BTreeSet::new();
vars.insert("TOKEN".to_string());
match kind {
"azurebatch" => {
vars.insert("BATCH_URL".to_string());
}
"kraken" => {
vars.insert("KRAKEN_API_KEY".to_string());
}
_ => {}
}
vars
}
pub async fn validate_raw(
kind: &str,
globals: &Object,
client: &Client,
use_lax_tls: bool,
allow_internal_ips: bool,
) -> Result<RawValidationOutcome> {
if let Some(url) = raw_validation_target_url(kind, globals)? {
if let Err(e) = check_url_resolvable(&url, allow_internal_ips).await {
return Ok(RawValidationOutcome {
valid: false,
status: StatusCode::PRECONDITION_REQUIRED,
body: format!(
"Validation skipped - raw validation target blocked or not resolvable: {e}"
),
});
}
}
match kind {
"azurebatch" => validate_azure_batch(globals, client).await,
"ftp" => validate_ftp(globals, use_lax_tls).await,
"kraken" => validate_kraken(globals, client).await,
"ldap" => validate_ldap(globals, use_lax_tls).await,
"rabbitmq" => validate_rabbitmq(globals, use_lax_tls).await,
"redis" => validate_redis(globals, use_lax_tls).await,
other => Ok(RawValidationOutcome {
valid: false,
status: StatusCode::NOT_IMPLEMENTED,
body: format!("Raw validator `{other}` is not implemented."),
}),
}
}
fn raw_validation_target_url(kind: &str, globals: &Object) -> Result<Option<Url>> {
match kind {
"azurebatch" => string_var(globals, "BATCH_URL")
.map(|s| Url::parse(&s).context("invalid BATCH_URL"))
.transpose(),
"ftp" | "ldap" | "rabbitmq" | "redis" => string_var(globals, "TOKEN")
.map(|s| Url::parse(&s).context("invalid raw validation URI"))
.transpose(),
_ => Ok(None),
}
}
fn string_var(globals: &Object, name: &str) -> Option<String> {
globals.get(name).map(|v| v.to_kstr().to_string()).filter(|s| !s.is_empty())
}
fn decode_userinfo(input: &str) -> String {
percent_decode_str(input).decode_utf8_lossy().to_string()
}
fn current_unix_millis() -> String {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_else(|_| Duration::from_millis(0))
.as_millis()
.to_string()
}
fn rfc1123_now() -> String {
chrono::Utc::now().format("%a, %d %b %Y %H:%M:%S GMT").to_string()
}
fn build_root_store() -> Result<RootCertStore> {
let mut roots = RootCertStore::empty();
let native = rustls_native_certs::load_native_certs();
for cert in native.certs {
roots.add(cert).map_err(|e| anyhow!("failed to add native root cert: {e:?}"))?;
}
Ok(roots)
}
fn lax_provider() -> Arc<CryptoProvider> {
LAX_PROVIDER.get_or_init(|| Arc::new(ring::default_provider())).clone()
}
fn tls_connector(use_lax_tls: bool) -> Result<TlsConnector> {
let cfg = if use_lax_tls {
ensure_crypto_provider();
ClientConfig::builder()
.dangerous()
.with_custom_certificate_verifier(Arc::new(LaxCertVerifier(lax_provider())))
.with_no_client_auth()
} else {
ClientConfig::builder().with_root_certificates(build_root_store()?).with_no_client_auth()
};
Ok(TlsConnector::from(Arc::new(cfg)))
}
trait AsyncStream: AsyncRead + AsyncWrite + Unpin + Send {}
impl<T> AsyncStream for T where T: AsyncRead + AsyncWrite + Unpin + Send {}
type DynStream = Box<dyn AsyncStream>;
async fn connect_plain(host: &str, port: u16) -> Result<DynStream> {
let stream = timeout(Duration::from_secs(10), TcpStream::connect((host, port)))
.await
.context("connection timed out")??;
Ok(Box::new(stream))
}
async fn connect_tls(host: &str, port: u16, use_lax_tls: bool) -> Result<DynStream> {
let stream = timeout(Duration::from_secs(10), TcpStream::connect((host, port)))
.await
.context("connection timed out")??;
let server_name =
ServerName::try_from(host.to_string()).map_err(|_| anyhow!("invalid TLS host: {host}"))?;
let tls =
timeout(Duration::from_secs(10), tls_connector(use_lax_tls)?.connect(server_name, stream))
.await
.context("TLS handshake timed out")??;
Ok(Box::new(tls))
}
async fn connect_from_url(
url: &Url,
tls_default_port: u16,
plain_default_port: u16,
use_lax_tls: bool,
) -> Result<DynStream> {
let host = url.host_str().ok_or_else(|| anyhow!("URL is missing host"))?;
let tls = matches!(url.scheme(), "ftps" | "amqps" | "rediss" | "ldaps");
let port = url.port().unwrap_or(if tls { tls_default_port } else { plain_default_port });
if tls {
connect_tls(host, port, use_lax_tls).await
} else {
connect_plain(host, port).await
}
}
async fn validate_azure_batch(globals: &Object, client: &Client) -> Result<RawValidationOutcome> {
let endpoint = string_var(globals, "BATCH_URL").ok_or_else(|| anyhow!("missing BATCH_URL"))?;
let account_key = string_var(globals, "TOKEN").ok_or_else(|| anyhow!("missing TOKEN"))?;
let parsed = Url::parse(&endpoint).context("invalid BATCH_URL")?;
let host = parsed.host_str().ok_or_else(|| anyhow!("BATCH_URL is missing host"))?;
let account_name = host
.split('.')
.next()
.filter(|s| !s.is_empty())
.ok_or_else(|| anyhow!("failed to derive Batch account name from host"))?;
let api_version = "2020-09-01.12.0";
let url = format!("{endpoint}/applications?api-version={api_version}");
let date = rfc1123_now();
let string_to_sign = format!(
"GET\n\n\n\n\napplication/json\n{}\n\n\n\n\n\n{}\napi-version:{}",
date,
format!("/{account_name}/applications").to_lowercase(),
api_version
);
let key = B64.decode(account_key.as_bytes()).context("Azure Batch key is not valid base64")?;
let mut mac = <Hmac<Sha256> as KeyInit>::new_from_slice(&key)
.map_err(|e| anyhow!("invalid HMAC key: {e}"))?;
mac.update(string_to_sign.as_bytes());
let signature = B64.encode(mac.finalize().into_bytes());
let resp = client
.get(&url)
.header("Content-Type", "application/json")
.header("Date", &date)
.header("Authorization", format!("SharedKey {account_name}:{signature}"))
.send()
.await
.context("Azure Batch validation request failed")?;
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
let valid = status == StatusCode::OK;
Ok(RawValidationOutcome { valid, status, body })
}
async fn validate_ftp(globals: &Object, use_lax_tls: bool) -> Result<RawValidationOutcome> {
let token = string_var(globals, "TOKEN").ok_or_else(|| anyhow!("missing TOKEN"))?;
let url = Url::parse(&token).context("invalid FTP URI")?;
let host = url.host_str().ok_or_else(|| anyhow!("FTP URI is missing host"))?;
let username = decode_userinfo(url.username());
let password =
decode_userinfo(url.password().ok_or_else(|| anyhow!("FTP URI is missing password"))?);
let scheme = url.scheme().to_ascii_lowercase();
let mut stream = if scheme == "ftp" {
BufStream::new(connect_plain(host, url.port().unwrap_or(21)).await?)
} else {
let port = url.port().unwrap_or(990);
if url.port().unwrap_or(990) == 990 {
BufStream::new(connect_tls(host, port, use_lax_tls).await?)
} else {
let tcp = timeout(Duration::from_secs(10), TcpStream::connect((host, port)))
.await
.context("connection timed out")??;
let mut plain = BufStream::new(tcp);
let _ = read_ftp_reply(&mut plain).await?;
plain.write_all(b"AUTH TLS\r\n").await?;
plain.flush().await?;
let (code, auth_body) = read_ftp_reply(&mut plain).await?;
if code != 234 {
return Ok(RawValidationOutcome {
valid: false,
status: StatusCode::UNAUTHORIZED,
body: auth_body,
});
}
let tcp = plain.into_inner();
let server_name = ServerName::try_from(host.to_string())
.map_err(|_| anyhow!("invalid TLS host: {host}"))?;
let tls = timeout(
Duration::from_secs(10),
tls_connector(use_lax_tls)?.connect(server_name, tcp),
)
.await
.context("TLS handshake timed out")??;
BufStream::new(Box::new(tls) as DynStream)
}
};
let _ = read_ftp_reply(&mut stream).await?;
stream.write_all(format!("USER {username}\r\n").as_bytes()).await?;
stream.flush().await?;
let (user_code, user_body) = read_ftp_reply(&mut stream).await?;
if user_code == 230 {
return Ok(RawValidationOutcome { valid: true, status: StatusCode::OK, body: user_body });
}
if user_code != 331 {
return Ok(RawValidationOutcome {
valid: false,
status: StatusCode::UNAUTHORIZED,
body: user_body,
});
}
stream.write_all(format!("PASS {password}\r\n").as_bytes()).await?;
stream.flush().await?;
let (pass_code, pass_body) = read_ftp_reply(&mut stream).await?;
let _ = stream.write_all(b"QUIT\r\n").await;
let _ = stream.flush().await;
Ok(RawValidationOutcome {
valid: pass_code == 230,
status: if pass_code == 230 { StatusCode::OK } else { StatusCode::UNAUTHORIZED },
body: pass_body,
})
}
async fn read_ftp_reply<S>(stream: &mut BufStream<S>) -> Result<(u16, String)>
where
S: AsyncRead + AsyncWrite + Unpin,
{
let mut body = String::new();
let mut code_prefix: Option<String> = None;
loop {
let mut line = String::new();
let read = timeout(Duration::from_secs(10), stream.read_line(&mut line))
.await
.context("FTP server did not reply in time")??;
if read == 0 {
return Err(anyhow!("FTP server closed the connection"));
}
body.push_str(&line);
let trimmed = line.trim_end_matches(['\r', '\n']);
if trimmed.len() < 4 {
continue;
}
let code = &trimmed[0..3];
if !code.chars().all(|c| c.is_ascii_digit()) {
continue;
}
match trimmed.as_bytes()[3] {
b' ' => return Ok((code.parse().unwrap_or(0), body)),
b'-' => {
code_prefix = Some(code.to_string());
}
_ => {}
}
if let Some(prefix) = &code_prefix {
if trimmed.starts_with(prefix) && trimmed.as_bytes()[3] == b' ' {
return Ok((code.parse().unwrap_or(0), body));
}
}
}
}
async fn validate_kraken(globals: &Object, client: &Client) -> Result<RawValidationOutcome> {
let api_key =
string_var(globals, "KRAKEN_API_KEY").ok_or_else(|| anyhow!("missing KRAKEN_API_KEY"))?;
let api_secret = string_var(globals, "TOKEN").ok_or_else(|| anyhow!("missing TOKEN"))?;
let secret = B64.decode(api_secret.as_bytes()).context("Kraken secret is not valid base64")?;
let nonce = current_unix_millis();
let body = format!("nonce={nonce}");
let mut sha = Sha256::new();
sha.update(format!("{nonce}{body}").as_bytes());
let shasum = sha.finalize();
let path = "/0/private/Balance";
let mut mac = <Hmac<Sha512> as KeyInit>::new_from_slice(&secret)
.map_err(|e| anyhow!("invalid HMAC key: {e}"))?;
let mut payload = Vec::with_capacity(path.len() + shasum.len());
payload.extend_from_slice(path.as_bytes());
payload.extend_from_slice(&shasum);
mac.update(&payload);
let signature = B64.encode(mac.finalize().into_bytes());
let resp = client
.post(format!("https://api.kraken.com{path}"))
.header("Content-Type", "application/x-www-form-urlencoded")
.header("API-Key", api_key)
.header("API-Sign", signature)
.body(body)
.send()
.await
.context("Kraken validation request failed")?;
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
let valid = status == StatusCode::OK && body.contains(r#""error":[]"#);
Ok(RawValidationOutcome { valid, status, body })
}
async fn validate_ldap(globals: &Object, use_lax_tls: bool) -> Result<RawValidationOutcome> {
let token = string_var(globals, "TOKEN").ok_or_else(|| anyhow!("missing TOKEN"))?;
let url = Url::parse(&token).context("invalid LDAP URI")?;
let scheme = url.scheme().to_ascii_lowercase();
let host = url.host_str().ok_or_else(|| anyhow!("LDAP URI is missing host"))?;
let port = url.port().unwrap_or(if scheme == "ldaps" { 636 } else { 389 });
let bind_dn = if let Some(bind_dn) = string_var(globals, "LDAP_BIND_DN") {
bind_dn
} else {
decode_userinfo(url.username())
};
let password = if let Some(password) = string_var(globals, "LDAP_PASSWORD") {
password
} else {
decode_userinfo(url.password().ok_or_else(|| anyhow!("LDAP URI is missing password"))?)
};
let ldap_url = format!("{scheme}://{host}:{port}");
let settings = LdapConnSettings::new().set_no_tls_verify(use_lax_tls);
let (conn, mut ldap) = ldap3::LdapConnAsync::with_settings(settings, &ldap_url)
.await
.with_context(|| format!("failed to connect to LDAP server {ldap_url}"))?;
ldap3::drive!(conn);
let bind_result = ldap.simple_bind(&bind_dn, &password).await;
let _ = ldap.unbind().await;
match bind_result {
Ok(res) => match res.success() {
Ok(_) => Ok(RawValidationOutcome {
valid: true,
status: StatusCode::OK,
body: "LDAP bind succeeded.".to_string(),
}),
Err(err) => Ok(RawValidationOutcome {
valid: false,
status: StatusCode::UNAUTHORIZED,
body: err.to_string(),
}),
},
Err(err) => Ok(RawValidationOutcome {
valid: false,
status: StatusCode::BAD_GATEWAY,
body: err.to_string(),
}),
}
}
async fn validate_rabbitmq(globals: &Object, use_lax_tls: bool) -> Result<RawValidationOutcome> {
let token = string_var(globals, "TOKEN").ok_or_else(|| anyhow!("missing TOKEN"))?;
let url = Url::parse(&token).context("invalid AMQP URI")?;
let _host = url.host_str().ok_or_else(|| anyhow!("AMQP URI is missing host"))?;
let username = decode_userinfo(url.username());
let password =
decode_userinfo(url.password().ok_or_else(|| anyhow!("AMQP URI is missing password"))?);
let mut stream = connect_from_url(&url, 5671, 5672, use_lax_tls).await?;
timeout(Duration::from_secs(10), stream.write_all(b"AMQP\x00\x00\x09\x01"))
.await
.context("failed to write AMQP protocol header")??;
timeout(Duration::from_secs(10), stream.flush()).await.context("flush timed out")??;
let (_, _, start_payload) = read_amqp_frame(&mut stream).await?;
let (class_id, method_id) = amqp_method_ids(&start_payload)?;
if class_id != 10 || method_id != 10 {
return Ok(RawValidationOutcome {
valid: false,
status: StatusCode::BAD_GATEWAY,
body: format!("unexpected AMQP frame {class_id}.{method_id}"),
});
}
let start_ok = build_amqp_start_ok_frame(&username, &password);
timeout(Duration::from_secs(10), stream.write_all(&start_ok))
.await
.context("failed to write AMQP start-ok frame")??;
timeout(Duration::from_secs(10), stream.flush()).await.context("flush timed out")??;
let (_, _, next_payload) = read_amqp_frame(&mut stream).await?;
let (class_id, method_id) = amqp_method_ids(&next_payload)?;
let valid = class_id == 10 && method_id == 30;
Ok(RawValidationOutcome {
valid,
status: if valid { StatusCode::OK } else { StatusCode::UNAUTHORIZED },
body: format!("received AMQP method frame {class_id}.{method_id}"),
})
}
fn build_amqp_start_ok_frame(username: &str, password: &str) -> Vec<u8> {
let mut payload = Vec::new();
payload.extend_from_slice(&10u16.to_be_bytes());
payload.extend_from_slice(&11u16.to_be_bytes());
payload.extend_from_slice(&0u32.to_be_bytes()); // empty client properties table
payload.extend_from_slice(&(5u32).to_be_bytes());
payload.extend_from_slice(b"PLAIN");
let mut response = Vec::with_capacity(username.len() + password.len() + 2);
response.push(0);
response.extend_from_slice(username.as_bytes());
response.push(0);
response.extend_from_slice(password.as_bytes());
payload.extend_from_slice(&(response.len() as u32).to_be_bytes());
payload.extend_from_slice(&response);
payload.extend_from_slice(&(5u32).to_be_bytes());
payload.extend_from_slice(b"en_US");
let mut frame = Vec::with_capacity(payload.len() + 8);
frame.push(1); // method frame
frame.extend_from_slice(&0u16.to_be_bytes());
frame.extend_from_slice(&(payload.len() as u32).to_be_bytes());
frame.extend_from_slice(&payload);
frame.push(0xCE);
frame
}
async fn read_amqp_frame(stream: &mut DynStream) -> Result<(u8, u16, Vec<u8>)> {
let mut header = [0u8; 7];
timeout(Duration::from_secs(10), stream.read_exact(&mut header))
.await
.context("timed out while reading AMQP frame header")??;
let frame_type = header[0];
let channel = u16::from_be_bytes([header[1], header[2]]);
let size = u32::from_be_bytes([header[3], header[4], header[5], header[6]]) as usize;
let mut payload = vec![0u8; size];
timeout(Duration::from_secs(10), stream.read_exact(&mut payload))
.await
.context("timed out while reading AMQP frame payload")??;
let mut end = [0u8; 1];
timeout(Duration::from_secs(10), stream.read_exact(&mut end))
.await
.context("timed out while reading AMQP frame terminator")??;
if end[0] != 0xCE {
return Err(anyhow!("invalid AMQP frame terminator"));
}
Ok((frame_type, channel, payload))
}
fn amqp_method_ids(payload: &[u8]) -> Result<(u16, u16)> {
if payload.len() < 4 {
return Err(anyhow!("AMQP payload too short"));
}
Ok((u16::from_be_bytes([payload[0], payload[1]]), u16::from_be_bytes([payload[2], payload[3]])))
}
async fn validate_redis(globals: &Object, use_lax_tls: bool) -> Result<RawValidationOutcome> {
let token = string_var(globals, "TOKEN").ok_or_else(|| anyhow!("missing TOKEN"))?;
let url = Url::parse(&token).context("invalid Redis URI")?;
let username = if let Some(username) = string_var(globals, "USERNAME") {
username
} else if !url.username().is_empty() {
decode_userinfo(url.username())
} else {
String::new()
};
let password = if let Some(password) = string_var(globals, "PASSWORD") {
password
} else {
decode_userinfo(url.password().ok_or_else(|| anyhow!("Redis URI is missing password"))?)
};
let mut stream = BufStream::new(connect_from_url(&url, 6380, 6379, use_lax_tls).await?);
let auth_cmd = if username.is_empty() {
format!("*2\r\n$4\r\nAUTH\r\n${}\r\n{}\r\n", password.len(), password)
} else {
format!(
"*3\r\n$4\r\nAUTH\r\n${}\r\n{}\r\n${}\r\n{}\r\n",
username.len(),
username,
password.len(),
password
)
};
stream.write_all(auth_cmd.as_bytes()).await?;
stream.flush().await?;
let auth_reply = read_resp_line(&mut stream).await?;
if !auth_reply.starts_with("+OK") {
return Ok(RawValidationOutcome {
valid: false,
status: StatusCode::UNAUTHORIZED,
body: auth_reply,
});
}
stream.write_all(b"*1\r\n$4\r\nPING\r\n").await?;
stream.flush().await?;
let ping_reply = read_resp_line(&mut stream).await?;
Ok(RawValidationOutcome {
valid: ping_reply.starts_with("+PONG"),
status: if ping_reply.starts_with("+PONG") {
StatusCode::OK
} else {
StatusCode::UNAUTHORIZED
},
body: ping_reply,
})
}
async fn read_resp_line<S>(stream: &mut BufStream<S>) -> Result<String>
where
S: AsyncRead + AsyncWrite + Unpin,
{
let mut line = String::new();
timeout(Duration::from_secs(10), stream.read_line(&mut line))
.await
.context("Redis server did not reply in time")??;
Ok(line)
}

58
data/default/rule_cleanup/count_rules.py
View file

@ -27,8 +27,8 @@ DEFAULT_RULES_DIR = (
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description=(
"Count total rules and detector rules. "
"Detector rules are rules that do not "
"Count total rules and standalone detector rules. "
"Standalone detector rules are rules that do not "
"declare depends_on_rule."
)
)
@ -38,6 +38,14 @@ def parse_args() -> argparse.Namespace:
default=DEFAULT_RULES_DIR,
help="Directory containing rule YAML files (default: %(default)s)",
)
parser.add_argument(
"--list-validators",
action="store_true",
help=(
"Print the IDs of standalone detectors with and "
"without a validator"
),
)
return parser.parse_args()
@ -59,6 +67,14 @@ def iter_rule_entries(path: Path) -> list[dict]:
return entries
def rule_identifier(rule: dict, path: Path, index: int) -> str:
if isinstance(rule.get("id"), str) and rule["id"].strip():
return rule["id"]
if isinstance(rule.get("name"), str) and rule["name"].strip():
return rule["name"]
return f"{path.stem}#{index}"
def main() -> int:
args = parse_args()
rules_dir = args.rules_dir.resolve()
@ -74,6 +90,8 @@ def main() -> int:
total_rules = 0
dependent_rules = 0
standalone_with_validator: list[str] = []
standalone_without_validator: list[str] = []
for path in rule_files:
try:
@ -86,14 +104,44 @@ def main() -> int:
dependent_rules += sum(
1 for rule in rules if rule.get("depends_on_rule")
)
for index, rule in enumerate(rules, start=1):
if rule.get("depends_on_rule"):
continue
detector_rules = total_rules - dependent_rules
identifier = rule_identifier(rule, path, index)
if rule.get("validation"):
standalone_with_validator.append(identifier)
else:
standalone_without_validator.append(identifier)
standalone_detector_rules = total_rules - dependent_rules
print(f"Rules directory: {rules_dir}")
print(f"Rule files: {len(rule_files)}")
print(f"Total rules: {total_rules}")
print(f"Dependent rules: {dependent_rules}")
print(f"Detectors: {detector_rules}")
print(f"Standalone detectors: {standalone_detector_rules}")
print(
"Standalone detectors with validator: "
f"{len(standalone_with_validator)}"
)
print(
"Standalone detectors without validator: "
f"{len(standalone_without_validator)}"
)
if args.list_validators:
print(
"\nStandalone detectors with validator "
f"({len(standalone_with_validator)}):"
)
for name in standalone_with_validator:
print(f" {name}")
print(
"\nStandalone detectors without validator "
f"({len(standalone_without_validator)}):"
)
for name in standalone_without_validator:
print(f" {name}")
return 0

BIN
docs-site/docs/assets/images/binary-size-comparison.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 35 KiB

3
docs-site/docs/assets/stylesheets/extra.css
View file

@ -99,6 +99,7 @@
max-width: 700px;
margin: 0 auto 2rem;
color: var(--md-default-fg-color--light);
font-size: 1rem;
line-height: 1.6;
}
@ -127,11 +128,13 @@
.kf-feature h3 {
margin-top: 0;
font-size: 1.3rem;
color: var(--md-primary-fg-color);
}
.kf-feature p {
color: var(--md-default-fg-color--light);
font-size: 0.85rem;
line-height: 1.6;
margin-bottom: 0;
}

6
docs-site/docs/changelog.md
View file

@ -7,6 +7,12 @@ description: "Kingfisher release history: new features, rules, bug fixes, and im
All notable changes to this project will be documented in this file.
## [v1.95.0]
- Added 80+ built-in rules, bringing the bundled ruleset to 820 total. New coverage includes Amazon OAuth, Asaas, multiple Azure credential families, Bitrise, Canva, CockroachDB, eBay, Elastic, hCaptcha, Highnote, Lichess, MailerSend, Onfido, Paddle, Pangea, Persona, Pinterest, Proof, Rootly, Runpod, Telnyx, Thunderstore, Valtown, Volcengine, and more.
- Replaced tree-sitter with a lighter parser-based context verifier built from handwritten lexers plus `tl`/`cssparser`, preserving context-dependent matching while cutting about 19 MB from the release binary.
- Added a `validation: type: Raw` exception path for provider-specific checks, with new raw validators for Azure Batch, FTP, Kraken, LDAP, RabbitMQ, and Redis. Also added stable request-scoped template values plus new Liquid filters for HMAC-SHA384 hex output and timestamp generation.
- Expanded live validation coverage for several built-in rules, including Agora, Bitfinex, DocuSign, Dwolla, GitLab, KuCoin, RingCentral, Snowflake, Tableau, Trello, and Webex. Also tightened newly added helper regex to avoid high-match scan regressions, and made preflight-blocked raw validations report as skipped/not attempted instead of failed.
## [v1.94.0]
- Updated vendored `vectorscan-rs` from v0.0.5 (Vectorscan 5.4.11) to v0.0.6 (Vectorscan 5.4.12). The upstream crate now ships pre-extracted sources instead of a tarball+patch, and fixes the `cpu_native` feature flag. Local Windows and musl build patches have been re-applied.
- Added more built-in rules

49
docs-site/docs/features/parsing.md
View file

@ -5,43 +5,50 @@ description: "Language-aware secret detection using tree-sitter parsing for 13+
# Kingfisher Source Code Parsing
Kingfisher leverages tree-sitter as an extra layer of analysis when scanning source files written in supported programming languages. In practice, after its initial regex-based scan (powered by Vectorscan/Hyperscan), Kingfisher can run a targeted verification pass for context-dependent rules.
Kingfisher uses a parser-based context verifier as a second pass on supported source files. After its initial regex scan (powered by Vectorscan/Hyperscan), it extracts assignment-style snippets from code and configuration files to confirm that generic keyword+token matches appear in plausible contexts.
If so, it creates a Checker (see below) that uses tree‐sitter to parse the file and run language‐specific queries. This additional pass refines the detection by capturing more structured patterns—such as secret-like tokens—that might be obscured or spread over code constructs.
The implementation favors lightweight extractors over full AST parsing:
- **Handwritten lexers** for common programming and config languages — comment-aware stripping followed by regex-based `key = value` extraction
- **`tl`** for HTML — attribute values, element text, and embedded `<script>` / `<style>` delegation
- **`cssparser`** for CSS — declaration parsing via Mozilla’s CSS tokenizer
> **History:** Earlier versions used tree-sitter with 17 statically-linked
> grammar crates. This added ~20 MB to the binary and required building a
> full syntax tree just to extract assignment pairs. The current lexer-based
> approach achieves the same extraction quality with near-zero binary overhead
> and no external grammar dependencies.
## How It’s Called
In the scanning phase (in the Matcher's implementation), Kingfisher does the following:
In the scanning phase (in the Matcher’s implementation), Kingfisher does the following:
- **Primary Regex Pass:** Kingfisher always scans the full blob with Vectorscan/Hyperscan first.
- **Candidate Selection:** Findings from rules classified as context-dependent become tree-sitter verification candidates.
- **Language Detection:** If a language string is provided (for example from metadata or extension), the code calls a helper (such as `get_language_and_queries`) to retrieve the corresponding tree-sitter language and queries.
- **Checker Creation:** With those values, a `Checker` is instantiated with the target language and query map.
- **Parsing and Querying:** The Checker retrieves a thread-local parser (to avoid recreating it on every call), sets language, parses source, and runs queries to extract structured snippets (for example `key = value` pairs).
- **Verification Decision:** Candidate findings are kept only if parser-extracted context verifies the matched secret. If tree-sitter is unavailable, fallback behavior is profile-driven (for strict generic keyword+token rules, findings are suppressed).
*(See the implementation details in the parser module – for example, the `modify_regex` function in the Checker, and the conditional tree‐sitter call in Matcher::scan_blob)*
- **Candidate Selection:** Findings from rules classified as context-dependent become parser-verification candidates.
- **Language Detection:** If a language string is provided (for example from metadata or extension), the code maps it to a supported parser backend.
- **Parsing and Querying:** The parser streams normalized snippets such as `key = value` without materializing a full syntax tree.
- **Verification Decision:** Candidate findings are kept only if parser-extracted context verifies the matched secret.
## Supported Languages
The design supports many common source code languages. The Language enum (defined in the parser module) includes variants for:
- **Scripting:** Bash, Python, Ruby, PHP
- **Compiled languages:** C, C++, C#, Rust, Java
- **Web-related languages:** CSS, HTML, JavaScript, TypeScript, YAML, Toml
- **Others:** Go, and even a generic “Regex” mode
- **Scripting:** Bash, Python, Ruby, PHP
- **Compiled languages:** C, C++, C#, Rust, Java
- **Web-related languages:** CSS, HTML, JavaScript, TypeScript, YAML, TOML
- **Others:** Go
Each variant maps to its corresponding tree‐sitter language through the `get_ts_language()` method.
## When Context Verification Is Not Called
## When Tree‐sitter Is Not Called
Context verification is skipped in certain cases:
Tree‐sitter won’t be invoked in certain cases:
- **No Language Identified:** If the file isn’t recognized as belonging to one of the supported languages or no language hint is provided, the Checker isn’t even constructed.
- **Non-source Files:** Binary files or files that aren’t expected to contain code (or aren’t extracted from archives) bypass tree‐sitter parsing.
- **Fallback on Errors:** If tree‐sitter parsing fails (e.g. due to malformed code or other errors), Kingfisher will fall back on its regex/Vectorscan matches without the additional tree‐sitter insights.
- **No Language Identified:** If the file isn’t recognized as belonging to one of the supported languages or no language hint is provided, the context verifier isn’t even constructed.
- **Non-source Files:** Binary files or files that aren’t expected to contain code (or aren’t extracted from archives) bypass parser-based context verification.
- **Large Blobs:** Files larger than 2 MiB skip context verification to avoid spending time on generated or minified content.
- **Verification Errors:** If extraction fails, context-dependent matches are suppressed instead of falling back to raw regex hits.
## Summary
In essence, Kingfisher’s use of tree‐sitter is conditional and complementary. It is called only when the scanned file is a source code file written in a supported language, and its role is to enrich the scanning results by leveraging the syntax tree and language-specific queries. When files are non-source, binary, or if no language is provided, tree‐sitter is not invoked, and Kingfisher relies solely on its regex-based detection.
Parser-based context verification is conditional and complementary. It is called only when the scanned file is a supported source or config file, and its role is to reduce noisy context-dependent findings by checking them against extracted code/config structure.
This layered approach helps improve the accuracy of secret detection while maintaining high performance.

11
docs-site/docs/reference/architecture.md
View file

@ -64,7 +64,7 @@ flowchart LR
subgraph Engines[Engines]
Vector[vectorscan]
ScanPool[scanner pool]
Tree[tree-sitter]
Context["context verifier"]
Liquid[Liquid templates]
end
@ -94,7 +94,7 @@ flowchart LR
ScannerLib --> Validate
Match --> Vector --> ScanPool
Match --> Tree
Match --> Context
Validate --> Liquid
Validate --> APIs
@ -112,10 +112,10 @@ flowchart LR
- `src/scanner/runner.rs`: the orchestration hub for `scan`, including repo enumeration, clone streaming, artifact fetching, validation setup, sequential or parallel scan execution (threshold: >10 git repos triggers parallel mode), reporting, and summary generation.
- `src/scanner/*`: input enumeration (`enumerate.rs`), repository handling and artifact fetching (`repos.rs`), blob processing (`processing.rs`), validation coordination (`validation.rs`), scan summaries (`summary.rs`), Docker image scanning (`docker.rs`), and utilities (`util.rs`).
- `src/matcher/*`: the main detection engine (`mod.rs`), including vectorscan callbacks, regex helpers, Base64 discovery (`base64_decode.rs`), capture group handling (`captures.rs`), dedup support (`dedup.rs`), filtering (`filter.rs`), and finding fingerprinting (`fingerprint.rs`).
- `src/parser.rs`: tree-sitter integration for language-aware parsing, supporting 17+ languages (Bash, C, C#, C++, CSS, Go, HTML, Java, JavaScript, PHP, Python, Ruby, Rust, TOML, TypeScript, YAML, and regex).
- `src/parser.rs` and `src/parser/*`: parser-based context verification for language-aware matching, with handwritten lexers plus lightweight HTML and CSS parsers.
- `src/scanner_pool.rs`: thread-local vectorscan `BlockScanner` pool, providing safe reuse of compiled pattern databases across scan threads.
- `src/reporter.rs` and `src/reporter/*`: report rendering for pretty, JSON, BSON, TOON, SARIF, and HTML outputs, plus the data model used by the viewer.
- `src/direct_validate.rs`: direct validation of a known secret without going through pattern matching. Supports HTTP, AWS, Azure, GCP, JDBC, MongoDB, MySQL, PostgreSQL, JWT, and Coinbase validators, with Liquid template integration for custom validation logic.
- `src/direct_validate.rs`: direct validation of a known secret without going through pattern matching. Supports HTTP, gRPC, plus schema-level typed validators such as AWS, AzureStorage, GCP, JDBC, MongoDB, MySQL, PostgreSQL, JWT, and Coinbase, and delegates ad-hoc `Raw` validators to `crates/kingfisher-scanner/src/validation/raw.rs`.
- `src/direct_revoke.rs`: direct revocation of a known secret without going through the scan pipeline. Uses Liquid templates for revocation configurations and supports multi-step HTTP revocation flows.
- `src/access_map.rs` and `src/access_map/*`: standalone blast-radius mapping with 24 provider implementations including AWS, Azure, GCP, GitHub, GitLab, Slack, Bitbucket, Gitea, Hugging Face, Buildkite, Anthropic, OpenAI, and more.
@ -123,8 +123,9 @@ flowchart LR
- The main CLI scan path is implemented primarily in the application modules under `src/`, not in `kingfisher-scanner`.
- `kingfisher-scanner` is still important: it provides the embeddable scanner API plus shared validation and primitive functionality reused by the application.
- The shared validation layer in `crates/kingfisher-scanner/src/validation/` contains both reusable typed validator families and the `Raw` exception-path validators used by rule YAML.
- Direct `validate`, `revoke`, and standalone `access-map` are sibling command paths. They are not downstream stages of `FindingsStore`.
- Reporting is downstream from the datastore, which lets Kingfisher emit multiple output formats and drive the local viewer from the same finding set.
- The matching layer is intentionally hybrid: vectorscan provides high-throughput SIMD-accelerated pattern detection, while regex helpers, Base64 support, and tree-sitter verification improve accuracy and reduce false positives.
- The matching layer is intentionally hybrid: vectorscan provides high-throughput SIMD-accelerated pattern detection, while regex helpers, Base64 support, and parser-based context verification improve accuracy and reduce false positives.
- `FindingsStore` uses an in-memory store with a Bloom filter for deduplication, replacing the earlier SQLite-based storage model.
- Validation and revocation templates are rendered via Liquid, allowing rule authors to define HTTP request sequences, variable extraction, and multi-step flows in YAML without touching Rust code.

89
docs-site/docs/reference/comparison.md
View file

@ -8,17 +8,17 @@ description: "Benchmark results comparing Kingfisher performance against Truffle
## Runtime Comparison (seconds)
*Lower runtimes are better.*
| Repository | Kingfisher Runtime | TruffleHog Runtime | GitLeaks Runtime | detect-secrets Runtime |
|------------|--------------------|--------------------|------------------|------------------------|
| croc | 2.64 | 10.36 | 3.10 | 0.16 |
| rails | 8.75 | 24.19 | 24.24 | 0.48 |
| ruby | 22.93 | 132.68 | 61.37 | 0.79 |
| gitlab | 135.41 | 325.93 | 350.84 | 5.04 |
| django | 6.91 | 227.63 | 59.50 | 0.61 |
| lucene | 15.62 | 89.11 | 76.24 | 0.66 |
| mongodb | 25.37 | 174.93 | 175.80 | 2.74 |
| linux | 205.19 | 597.51 | 548.96 | 5.49 |
| typescript | 64.99 | 183.04 | 232.34 | 4.23 |
| Repository | Kingfisher Runtime | TruffleHog Runtime | GitLeaks Runtime |
|------------|--------------------|--------------------|------------------|
| croc | 2.64 | 10.36 | 3.10 |
| rails | 8.75 | 24.19 | 24.24 |
| ruby | 22.93 | 132.68 | 61.37 |
| gitlab | 135.41 | 325.93 | 350.84 |
| django | 6.91 | 227.63 | 59.50 |
| lucene | 15.62 | 89.11 | 76.24 |
| mongodb | 25.37 | 174.93 | 175.80 |
| linux | 205.19 | 597.51 | 548.96 |
| typescript | 64.99 | 183.04 | 232.34 |
<p align="center">
<img src="../assets/images/runtime-comparison.png" alt="Kingfisher Runtime Comparison" style="vertical-align: center;" />
@ -28,37 +28,52 @@ description: "Benchmark results comparing Kingfisher performance against Truffle
Note: For GitLeaks and detect-secrets, validated/verified counts are not available.
| Repository | Kingfisher Validated | TruffleHog Verified | GitLeaks Verified | detect-secrets Verified |
|------------|----------------------|---------------------|-------------------|-------------------------|
| croc | 0 | 0 | 0 | 0 |
| rails | 0 | 0 | 0 | 0 |
| ruby | 0 | 0 | 0 | 0 |
| gitlab | 6 | 6 | 0 | 0 |
| django | 0 | 0 | 0 | 0 |
| lucene | 0 | 0 | 0 | 0 |
| mongodb | 0 | 0 | 0 | 0 |
| linux | 0 | 0 | 0 | 0 |
| typescript | 0 | 0 | 0 | 0 |
| Repository | Kingfisher Validated | TruffleHog Verified | GitLeaks Verified |
|------------|----------------------|---------------------|-------------------|
| croc | 0 | 0 | 0 |
| rails | 0 | 0 | 0 |
| ruby | 0 | 0 | 0 |
| gitlab | **6** | **6** | 0 |
| django | 0 | 0 | 0 |
| lucene | 0 | 0 | 0 |
| mongodb | 0 | 0 | 0 |
| linux | 0 | 0 | 0 |
| typescript | 0 | 0 | 0 |
### Network Requests Comparison
*'Network Requests' shows the total number of HTTP calls made during a scan. Since Gitleaks and detect‑secrets don’t validate secrets, they never make any network requests.*
| Repository | Kingfisher Network Requests | TruffleHog Network Requests | GitLeaks Network Requests | detect-secrets Network Requests |
|------------|-----------------------------|-----------------------------|---------------------------|----------------------------------|
| croc | 0 | 17 | 0 | 0 |
| rails | 1 | 25 | 0 | 0 |
| ruby | 3 | 33 | 0 | 0 |
| gitlab | 17 | 15624 | 0 | 0 |
| django | 0 | 66 | 0 | 0 |
| lucene | 0 | 116 | 0 | 0 |
| mongodb | 1 | 191 | 0 | 0 |
| linux | 0 | 287 | 0 | 0 |
| typescript | 0 | 10 | 0 | 0 |
| Repository | Kingfisher Network Requests | TruffleHog Network Requests | GitLeaks Network Requests |
|------------|-----------------------------|-----------------------------|---------------------------|
| croc | 0 | 17 | 0 |
| rails | 1 | 25 | 0 |
| ruby | 3 | 33 | 0 |
| gitlab | 17 | **15624** | 0 |
| django | 0 | 66 | 0 |
| lucene | 0 | 116 | 0 |
| mongodb | 1 | 191 | 0 |
| linux | 0 | 287 | 0 |
| typescript | 0 | 10 | 0 |
*Lower runtimes are better. Validated/Verified counts are reported where available. 'Network Requests' indicates the number of HTTP requests made during scanning.*
OS: darwin
Architecture: arm64
CPU Cores: 16
RAM: 48.00 GB
### Binary Size Comparison (macOS arm64)
| Tool | Version | Binary Size |
|------|---------|-------------|
| Gitleaks | 8.30.0 | 14.5 MB |
| **Kingfisher** | **1.95.0** | **32.8 MB** |
| TruffleHog | 3.94.2 | 160.3 MB |
*Smaller binaries are easier to distribute, deploy in CI, and embed in container images*
<p align="center">
<img src="./binary-size-comparison.png" alt="Binary Size Comparison" />
</p>
## Benchmark Environment
OS: darwin
Architecture: arm64
CPU Cores: 16
RAM: 48.00 GB

16
docs-site/docs/reference/library.md
View file

@ -39,7 +39,13 @@ The `kingfisher-scanner` crate supports optional validation features:
| ------- | ----------- |
| `validation` | Core validation support (includes HTTP validation) |
| `validation-http` | HTTP-based validation for API tokens |
| `validation-raw` | Provider/protocol-specific raw validation flows for `validation: type: Raw` rules |
| `validation-aws` | AWS credential validation via STS GetCallerIdentity |
| `validation-azure` | Azure storage credential validation |
| `validation-coinbase` | Coinbase credential validation |
| `validation-gcp` | GCP credential validation |
| `validation-jwt` | JWT validation |
| `validation-database` | MongoDB, MySQL, PostgreSQL, and JDBC validation |
| `validation-all` | Enable all validation features |
## Quick Start
@ -262,7 +268,7 @@ flowchart TD
### Loading Builtin Rules
Kingfisher comes with 700+ builtin rules for common secret types:
Kingfisher comes with 800+ builtin rules for common secret types:
```rust
use kingfisher_rules::{get_builtin_rules, Confidence};
@ -727,9 +733,17 @@ kingfisher-scanner = { git = "https://github.com/mongodb/kingfisher", features =
| ------- | ----------- |
| `validation` | Core validation support with HTTP validation |
| `validation-http` | HTTP-based validation for API tokens |
| `validation-raw` | Provider/protocol-specific raw validation flows for `validation: type: Raw` rules |
| `validation-aws` | AWS credential validation via STS |
| `validation-azure` | Azure storage credential validation |
| `validation-coinbase` | Coinbase credential validation |
| `validation-gcp` | GCP credential validation |
| `validation-jwt` | JWT validation |
| `validation-database` | MongoDB, MySQL, PostgreSQL, and JDBC validation |
| `validation-all` | Enable all validation features |
`validation: type: Raw` is the ad-hoc validator path for provider-specific or protocol-specific checks that are not generic enough to become schema-level validator families. Typed validators such as `AWS`, `GCP`, `MongoDB`, and `JWT` remain separate validator kinds in the rule schema.
### HTTP Validation Example
```rust

750
docs-site/docs/rules/builtin-rules.md
View file

File diff suppressed because it is too large Load diff

35
docs-site/docs/rules/overview.md
View file

@ -171,9 +171,29 @@ revocation:
| visible | false to hide non‑secret captures (e.g. IDs) |
| depends_on_rule | Chain rules: use captures from one rule in another's validation |
| pattern_requirements | Require character types and/or exclude placeholder words from matches |
| validation | Configure HTTP, AWS, GCP, etc. checks to verify live validity |
| validation | Configure `Http`, `Grpc`, typed validators (`AWS`, `GCP`, etc.), or `Raw` exception-path checks to verify live validity |
| revocation | Configure HTTP, AWS, or multi-step revocation for a detected secret |
## Validation Types
Kingfisher supports three validation buckets:
1. `Http` and `Grpc`: YAML-native validation flows. Prefer these first.
2. Typed validators: schema-level validation families already modeled in the rule schema, such as `AWS`, `AzureStorage`, `Coinbase`, `GCP`, `MongoDB`, `MySQL`, `Postgres`, `Jdbc`, and `JWT`.
3. Raw validators: provider-specific or protocol-specific exception paths dispatched through `validation: type: Raw`.
Raw validation looks like this:
```yaml
validation:
type: Raw
content: kraken
```
Use `Raw` only when the provider check cannot be expressed reliably with `Http` or `Grpc` and does not justify a new reusable validator family. Raw validator implementations live in `crates/kingfisher-scanner/src/validation/raw.rs`.
Typed validators are safer and more reusable because the validator kind is part of the schema. `Raw` validators are string-dispatched and fail at runtime if the `content` name is unknown. If you need a Rust-backed exception path for one provider, prefer `Raw`; reserve new typed validators for stable validation families that can be reused across rules.
## gRPC Validation (Grpc)
Some services (notably CLI/SDK control planes) are **gRPC-only**. For these, `validation: type: Http`
@ -473,6 +493,7 @@ Below is the complete list of Liquid filters available in Kingfisher, along with
| `hmac_sha1` | `key` (string) | Computes HMAC-SHA1 over the input, returns Base64-encoded result. | `{{ TOKEN \| hmac_sha1: "secret-key" }}` |
| `hmac_sha256` | `key` (string) | Computes HMAC-SHA256 over the input, returns Base64-encoded result. | `{{ TOKEN \| hmac_sha256: "secret-key" }}` |
| `hmac_sha384` | `key` (string) | Computes HMAC-SHA384 over the input, returns Base64-encoded result. | `{{ TOKEN \| hmac_sha384: "secret-key" }}` |
| `hmac_sha384_hex` | `key` (string) | Computes HMAC-SHA384 over the input, returns lowercase hexadecimal output. | `{{ TOKEN \| hmac_sha384_hex: "secret-key" }}` |
| `hmac_sha256_b64key` | `key` (string, base64-encoded) | Decodes the key from Base64 to raw bytes, then computes HMAC-SHA256. Returns Base64. Use for Azure SAS and other protocols where the signing key is base64-encoded. | `{{ to_sign \| hmac_sha256_b64key: TOKEN }}` |
| `random_string` | `len` (integer, optional) | Generates a cryptographically-secure random alphanumeric string of the specified length (default: 32). | `{{ "" \| random_string: 16 }}` |
| `prefix` | `len` (integer, optional) | Returns the first `len` characters from the string (default: full). | `{{ TOKEN \| prefix: 6 }}` |
@ -481,8 +502,10 @@ Below is the complete list of Liquid filters available in Kingfisher, along with
| `url_encode` | – | Percent-encodes the input according to RFC 3986. | `{{ TOKEN \| url_encode }}` |
| `json_escape` | – | Escapes special characters so a string can be safely injected into JSON contexts. | `{{ TOKEN \| json_escape }}` |
| `unix_timestamp` | – | Returns the current Unix epoch time in seconds (UTC). | `{{ "" \| unix_timestamp }}` |
| `unix_timestamp_ms` | – | Returns the current Unix epoch time in milliseconds (UTC). | `{{ "" \| unix_timestamp_ms }}` |
| `iso_timestamp` | – | Returns the current UTC timestamp in full ISO-8601 format (may include fractional seconds). | `{{ "" \| iso_timestamp }}` |
| `iso_timestamp_no_frac` | – | Current ISO-8601 timestamp (UTC) **without** fractional seconds. | `{{ "" \| iso_timestamp_no_frac }}` |
| `rfc1123_date` | – | Returns the current RFC-1123 timestamp in GMT. | `{{ "" \| rfc1123_date }}` |
| `uuid` | – | Generates a random UUIDv4 string. | `{{ "" \| uuid }}` |
| `jwt_header` | – | Builds a minimal JWT header JSON (`{"typ":"JWT","alg":…}`) and Base64URL-encodes it. | `{{ "HS256" \| jwt_header }}` |
| `replace` | `from` (string), `to` (string) | Replaces every occurrence of `from` with `to` in the input string. | `{{ "hello world" \| replace: "world", "mars" }}` |
@ -497,6 +520,11 @@ Authorization: Basic {{ "api:" | append: TOKEN | b64enc }}
```
**Runtime Values:** Filters like unix_timestamp and uuid are evaluated at runtime, enabling nonces, timestamps, and unique IDs in your requests.
**Stable Request Values:** HTTP and gRPC validation requests also expose stable per-request template variables. Use these when the same generated value must appear in multiple places within one request. Currently:
- `REQUEST_RFC1123_DATE`
- `REQUEST_UNIX_MILLIS`
### How depends_on_rule Works
- **Dependency Declaration:**
@ -743,7 +771,7 @@ When writing custom rules, consider the following best practices:
1. **Multi-line Regex:** Write your regex patterns over multiple lines for clarity. Use the `(?x)` flag to enable free-spacing mode.
2. **Optimize for Performance:** Structure your regex to minimize backtracking. Use non-capturing groups where possible and keep the pattern as concise as possible.
3. **Validation Integration:** Define a `validation` section if you want to verify the detected secret. You can use Liquid templating to insert dynamic values—use the unnamed capture as `TOKEN` and any named captures in uppercase.
3. **Validation Integration:** Define a `validation` section if you want to verify the detected secret. Prefer `Http` or `Grpc`; use an existing typed validator when the rule matches a supported validator family; use `Raw` only for rare provider-specific exception paths. You can use Liquid templating to insert dynamic values where supported. Use the unnamed capture as `TOKEN` and any named captures in uppercase.
4. **Revocation Integration:** Define a `revocation` section if you want to revoke a detected secret. It uses the same HTTP request format and template variables as `validation`.
5. **Test with Examples:** Always include examples that should match and, optionally, negative examples to ensure your rule behaves as expected.
@ -920,4 +948,5 @@ rules:
words: ['"Arn"']
depends_on_rule:
- rule_id: kingfisher.alibabacloud.1
variable: AKID```
variable: AKID
```

Some files were not shown because too many files have changed in this diff Show more