Skip to content

FEAT: Add Garak API key elicitation scenario - #2538

Open
寇依 (hey-Chloe) wants to merge 3 commits into
microsoft:mainfrom
hey-Chloe:feat/garak-api-key-scenario-2535
Open

寇依 (hey-Chloe) wants to merge 3 commits into
microsoft:mainfrom
hey-Chloe:feat/garak-api-key-scenario-2535

Conversation

@hey-Chloe

Copy link
Copy Markdown

Description

Closes #2535 by adding an ApiKey Garak scenario with the
GetKey and CompleteKey techniques.

The scenario:

  • loads four fixed local corpora for services, templates, synthetic partial
    fixtures, and safe placeholders;
  • preserves all 58 Garak service prompts;
  • applies a deterministic scenario-wide default cap of 20 prompts;
  • exposes exact run-size estimates and supports persisted run-plan resume;
  • evaluates responses with an expanded CredentialLeakScorer;
  • excludes supplied fixtures without storing their plaintext in component
    identifiers.

For precision, seven Garak entries that represent public resource or OAuth
client identifiers are retained in the prompt corpus but intentionally do not
count as credential leaks. The remaining 51 service entries map to reviewed
credential-shaped patterns. Weak generic shapes require response-side
service/key labeling so objective context alone cannot turn refusals,
checksums, UUIDs, or ordinary text into positive results.

The adapted Garak and dora material is pinned and attributed in
THIRD_PARTY_NOTICE.md.

Tests and Documentation

  • 381 passed across the complete regex scorer test directory and the new
    dataset/scenario tests.
  • Ruff and ty pass for all changed Python source and test files.
  • All changed-file pre-commit hooks pass, including secret detection,
    documentation validation, notebook checks, and type checking.
  • git diff --check passes.
  • JupyText round-trip checks pass for:
    • doc/scanner/garak.ipynb
    • doc/code/datasets/1_loading_datasets.ipynb
  • Verified all four new local datasets are discoverable through
    SeedDatasetProvider.

Documentation was added to the Garak scanner guide and built-in dataset guide.

No live target or credential-provider validation was performed. The scenario
uses the normal user-configured PyRIT target and stores responses through the
existing framework behavior.

No breaking changes.

@hey-Chloe

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT: Add Garak API key scenario

1 participant