kingfisher/AGENTS.md

AGENTS.md

Guidance for coding agents working in this repository.

Scope

• Applies to the entire repository rooted at this file.
• If a deeper AGENTS.md exists, that file takes precedence for its subtree.

Project

Kingfisher is a Rust secret scanner, live credential validator, revocation helper, and access-map tool. It scans repositories, git history, local files, archives, cloud storage, source-host artifacts, Docker images, and collaboration-platform exports.

Key Paths

• src/: main CLI binary and application code.
• src/cli/commands/: CLI command definitions and wiring.
• src/scanner/: scan orchestration, input enumeration, repository/artifact fetching, validation phase.
• src/matcher/: pattern matching, captures, filtering, deduplication.
• src/reporter/: TOON, JSON, JSONL, SARIF, BSON, HTML, and pretty output.
• src/access_map/: blast-radius and permission mapping.
• crates/kingfisher-core/: shared core types.
• crates/kingfisher-rules/: rule schema, rule loading, and bundled rule data.
• crates/kingfisher-rules/data/rules/: YAML detection rules.
• crates/kingfisher-scanner/: embeddable scanning API and shared validators.
• tests/ and testdata/: integration tests and fixtures.
• docs/, docs/viewer/, docs-site/: docs, report viewer assets, and generated MkDocs site.
• vendor/vectorscan-rs/: vendored Vectorscan bindings.

Toolchain

• Workspace minimum Rust version is 1.94 in Cargo.toml; make check-rust enforces >= 1.94.1 for build targets.
• Rust formatting is defined by rustfmt.toml (max_width = 100, 4 spaces, Unix newlines, reordered imports).
• Build scripts assume bash with set -eu -o pipefail.
• Windows Makefile targets expect MSYS2 with pacman.

Common Commands

• Build: cargo build
• Release build: cargo build --release
• Preferred test wrapper: make tests
• Direct tests: cargo test --workspace --all-targets
• Nextest: cargo nextest run --workspace --all-targets
• Format: cargo fmt --all
• Lint: cargo clippy --workspace --all-targets -- -D warnings
• Clean: make clean

Workflow Expectations

• Keep edits minimal, targeted, and consistent with touched code.
• Do not revert user-authored or unrelated in-progress changes.
• Prefer clear fixes over broad refactors unless requested.
• Run the narrowest relevant tests first; run broader checks when practical.
• If a validation/build command cannot be run, state exactly what was skipped and why.
• Prefer kingfisher scan --format toon for agent/LLM workflows; use pretty only when human-interactive output is explicitly desired.
• After markdown/doc changes, verify local documentation links when practical.
• After docs-site/ source changes, rebuild with docs-site/.venv/bin/mkdocs build -f docs-site/mkdocs.yml when practical so generated output stays in sync.

Architecture Notes

• Detection rules are YAML-driven and loaded from crates/kingfisher-rules/data/rules/.
• Allocator feature flags live in root Cargo.toml: use-mimalloc default, use-jemalloc, and system-alloc.
• Optional validator feature sets live in crates/kingfisher-scanner/Cargo.toml.
• Validation modules live primarily in crates/kingfisher-scanner/src/validation/ and src/validation.rs.

Validation And Revocation Policy

• Default to YAML validation (validation:), especially Http or Grpc; do not add Rust validation unless YAML cannot express the flow reliably.
• Typed validators are schema-level reusable families: AWS, AzureStorage, Coinbase, GCP, MongoDB, MySQL, Postgres, Jdbc, and JWT.
• Raw validators use validation: { type: Raw, content: <name> } and are implemented in crates/kingfisher-scanner/src/validation/raw.rs for provider-specific exceptions.
• If Rust validation is unavoidable, prefer a raw validator before introducing a new typed validator.
• Do not convert existing typed validators to Raw for consistency alone.
• For rules with validation, add revocation: when the third-party API safely supports revocation.

Rule Authoring

Use this when creating or updating rules in crates/kingfisher-rules/data/rules/.

1. Read docs/RULES.md before non-trivial rule/schema work.
2. Start from a nearby provider-family rule and preserve the existing YAML style.
3. Use a stable kingfisher.<provider>... rule id and set confidence: medium.
4. Write a valid Hyperscan/Vectorscan regex. Lookahead and lookbehind are not supported.
5. Start pattern with (?x), use one unnamed capture around the secret for {{ TOKEN }}, and use non-capturing groups for structure.
6. Prefer specific token formats and provider context; avoid broad generic patterns.
7. Use min_entropy, pattern_requirements, ignore_if_contains, and checksum requirements when format constraints are known.
8. Include examples that must match.
9. Use depends_on_rule for multi-part credentials; consider visible: false for helper rules.
10. Add YAML validation and revocation only when reliable and safe.

Rule Verification

• Rule crate: cargo test -p kingfisher-rules
• Rule syntax/check path: kingfisher rules check --rules-path crates/kingfisher-rules/data/rules/<file>.yml --load-builtins=false --no-update-check
• Scan fixture/corpus: kingfisher scan ./testdata --rule <rule-family-or-id> --rule-stats
• Validator check: kingfisher validate --rule <rule-id> <token-or-secret>
• Broad regression when practical: cargo test --workspace --all-targets

Common Tasks

• Add a detection rule: follow the rule authoring and verification sections above.
• Add a CLI command: implement under src/cli/commands/ and register it in CLI wiring.
• Add a validator: prefer YAML first; if Rust is required, use raw.rs and the narrowest feature/dependency wiring.
• Update docs-site rule counts: run uv run '/Users/mickg/src/kingfisher/data/default/rule_cleanup/count_rules.py', update docs-site/overrides/ and docs-site/mkdocs.yml, then rebuild the docs site when practical.

Docs Pointers

• Usage: docs/USAGE.md, docs/ADVANCED.md, docs/INTEGRATIONS.md
• Rules: docs/RULES.md
• Architecture: docs/ARCHITECTURE.md, docs/ACCESS_MAP.md
• Deployment/install: docs/INSTALLATION.md, docs/DEPLOYMENT.md, docs/PYPI.md
• Library API: docs/LIBRARY.md