Skip to content

Add metrics dashboard: scorer quality and benchmark leaderboard - #2667

Draft
hannahwestra25 wants to merge 6 commits into
microsoft:mainfrom
hannahwestra25:hannahwestra25-metrics-dashboard
Draft

hannahwestra25 wants to merge 6 commits into
microsoft:mainfrom
hannahwestra25:hannahwestra25-metrics-dashboard

Conversation

@hannahwestra25

@hannahwestra25 hannahwestra25 commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

What this adds

Metrics dashboard (doc/dashboard/) — a new top-level docs section:

  • 0_dashboard.md: landing page explaining what's here, what's planned, and how to refresh the data.
  • 1_scorer_quality.py/.ipynb: an Objective Scorer Leaderboard (accuracy/F1/precision/recall) and a Harm Scorer Leaderboard (MAE, Krippendorff's alpha), rendered from the existing committed registries under pyrit/datasets/scorer_evals/. No new dependencies — reads local JSONL only.
  • 2_benchmark_leaderboard.py/.ipynb: a Technique / Adversarial Model Leaderboard (success rate, N, success/failure/error/undetermined counts), rendered from the new committed store below.
  • Both pages registered in doc/myst.yml's top-level TOC, after scanner.

Benchmark exporter identity fields + store (build_scripts/export_adversarial_benchmark_result.py):

  • Every technique-metrics row now carries objective_target, objective_scorer, and dataset identity fields (added to JSON/CSV/TXT output).
  • New --update-benchmark-store [--benchmark-store-path] flag that upserts rows into a new committed store, pyrit/datasets/benchmark_results/adversarial_benchmark_metrics.jsonl, keyed on (technique, adversarial_model, objective_target, objective_scorer, dataset). This is what the new Benchmark Leaderboard page reads.

Demo data behind the Benchmark Leaderboard: three rows from one real, live AdversarialBenchmark scenario run — role_play_video_game, context_compliance, and red_teaming techniques, one adversarial target, harmbench dataset, --max-dataset-size 1. This is genuine exporter/scenario output, not synthetic data (see caveat below).

Explicitly out of scope for this PR

  • No ADO pipeline/YAML changes.
  • No native objective-target sweep (a leaderboard comparing target models against a fixed adversarial model) — the exporter's identity fields support it, but building it is future work.
  • No shared PR-opening credential automation.
  • No CI automation that runs the benchmark scenario or refreshes the store automatically — both dashboard pages are refreshed manually today (documented in 0_dashboard.md). The existing manually-triggered .azuredevops/adversarial-benchmark.yml pipeline could in principle drive this, but today its export step never passes --update-benchmark-store, and it has no step to regenerate notebook outputs or to commit/push/open a PR (it has no GitHub write credentials). Wiring that up is a good follow-up, best done as its own change rather than bundled here — it also touches the same YAML file Improve adversarial benchmark dataset and scoring #2551 is actively modifying.

Known limitations

  • Demo-scale data, not a rigorous benchmark. The Benchmark Leaderboard's data comes from a single small demo run (1 dataset item per technique, so N=1 per row) using one adversarial-model identity that happens to share a deployment with the objective target (no independently-configured ADVERSARIAL_CHAT_* target was available in this environment). Treat the page as a working demonstration of the mechanism, not a statistically meaningful or cross-model comparison. Broadening this into a larger, regularly-refreshed sweep across real independent adversarial models is future work (see the page's "Note on scope" section).
  • The exporter logic (_objective_identity, _dataset_identity, _build_technique_metrics, _upsert_benchmark_metrics) is covered by unit tests using hand-constructed ScenarioResult/AttackResult/TargetIdentifier/ScorerIdentifier objects, and has now also been exercised end-to-end against the real scenario result described above — so both the isolated logic and the full memory-to-JSONL path are validated against genuine PyRIT output.

Testing

  • pytest tests/unit/build_scripts/test_export_adversarial_benchmark_result.py — 12 new tests, all passing.
  • pytest tests/unit/build_scripts tests/unit/common tests/unit/datasets — no regressions (268 + 5069 passed, 2 pre-existing skips).
  • ruff check / ruff format --check — clean.
  • python -m build_scripts.validate_docs — 163 TOC references validated, no orphaned files.
  • pre-commit run against all changed files — clean, aside from two confirmed pre-existing, environment-specific issues unrelated to this change: check-added-large-files fails against any file (including already-committed ones) due to a git check-attr subprocess error in this sandbox; nbstripout reports "modified" on every run even though a before/after hash comparison shows the file is byte-identical.
  • Ran a real, live AdversarialBenchmark scenario end-to-end (pyrit_scan runexport_adversarial_benchmark_result.py --update-benchmark-store → the new dashboard page) to validate the full pipeline against genuine data rather than mocks.

…fields

Adds a new doc/dashboard/ section with a Scorer Quality page (Objective
Scorer Leaderboard + Harm Scorer Leaderboard), rendered from the existing
committed pyrit/datasets/scorer_evals/ registries.

Extends build_scripts/export_adversarial_benchmark_result.py to attach
objective_target/objective_scorer/dataset identity fields to every
technique-metrics row, and adds an optional --update-benchmark-store flag
that upserts rows into a new committed JSONL store
(pyrit/datasets/benchmark_results/adversarial_benchmark_metrics.jsonl),
keyed on (technique, adversarial_model, objective_target, objective_scorer,
dataset).

Scope, intentionally:
- No ADO pipeline/YAML changes.
- No benchmark leaderboard dashboard page yet - the exporter change is
  groundwork for a future PR once that page and real exported data exist.
  --update-benchmark-store has been unit-tested with hand-constructed
  ScenarioResult/AttackResult/identifier objects (verified field-for-field
  against source) but not yet exercised against a real AdversarialBenchmark
  scenario run, since that requires live credentialed targets and has no
  consumer yet.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@hannahwestra25
hannahwestra25 marked this pull request as draft September 15, 2026 18:46
- doc/dashboard/2_benchmark_leaderboard.py/.ipynb: new page rendering a
  technique / adversarial-model leaderboard (success rate, N, and
  success/failure/error/undetermined counts) from
  pyrit/datasets/benchmark_results/adversarial_benchmark_metrics.jsonl.
- pyrit/datasets/benchmark_results/adversarial_benchmark_metrics.jsonl:
  new store populated with 3 rows from one real, live AdversarialBenchmark
  scenario run (role_play_video_game, context_compliance, red_teaming;
  one adversarial target; harmbench; --max-dataset-size 1) via
  export_adversarial_benchmark_result.py --update-benchmark-store. This
  is genuine exporter output, not synthetic/mocked data.
- doc/myst.yml: register the new page under the dashboard TOC.
- doc/dashboard/0_dashboard.md: move Benchmark Leaderboard from
  "planned" to "here today", document its refresh procedure, and note
  the future objective-target robustness leaderboard as the next step.

Demo-scale caveat: N=1 per technique and a single adversarial-model
identity (mirroring the objective target's Azure deployment, since no
independent ADVERSARIAL_CHAT_* target was available). This validates
the exporter/store/page mechanism end-to-end for real, but is not yet a
statistically meaningful or cross-model benchmark.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@hannahwestra25 hannahwestra25 changed the title Add scorer-quality metrics dashboard and benchmark exporter identity fields Add metrics dashboard: scorer quality and benchmark leaderboard Sep 15, 2026
hannahwestra25 and others added 4 commits September 15, 2026 17:00
Both the Scorer Quality and Benchmark Leaderboard notebooks now render
their tables via a shared render_leaderboard_card() helper (dark
GitHub-style card CSS, static HTML/no JS) instead of plain
print(df.to_string()), matching the dashboard mockups.

Also fixes two bugs found while wiring this up:
- 2_benchmark_leaderboard.py: the render call was nested inside an
  if/else block, so IPython's last-expression auto-display never
  fired and the card silently rendered nothing. Now wrapped in an
  explicit display(...) call.
- _format_cell in both notebooks treated percent columns as
  percentages only when the underlying value was a Python float, but
  pandas infers int64 for an all-whole-number JSON column (e.g.
  success_rate values of 0.0/1.0), so percents rendered as literal
  "1"/"0". percent_columns membership is now authoritative regardless
  of the pandas-inferred dtype.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Scorer Quality and Benchmark Leaderboard pages were .ipynb/.py
jupytext pairs whose rendered output showed a large wall of raw Python
source (imports, CSS strings, helper functions) above each leaderboard
table -- readers had to scroll past implementation details to see the
actual metrics.

Replace both with plain MyST Markdown pages that {include} pre-rendered
HTML fragments instead:

- build_scripts/generate_dashboard_html.py: new script that renders the
  same three leaderboard cards (Objective Scorer, Harm Scorer, Benchmark)
  the notebooks used to produce, and writes each as a standalone HTML
  fragment under doc/dashboard/_generated/. Faithfully ports the existing
  rendering logic (dark card CSS, percent-formatting fix, empty-state
  fallback for when the benchmark store doesn't exist yet).
- doc/dashboard/1_scorer_quality.md, 2_benchmark_leaderboard.md: new
  pages carrying the same prose as the old notebooks, with
  `{include}` directives in place of code cells.
- doc/dashboard/_generated/*.html: the generated fragments, committed
  like any other generated-and-checked-in artifact (comparable to the
  notebooks' own committed outputs today).
- doc/dashboard/0_dashboard.md: updated links (.ipynb -> .md) and
  rewrote "Refreshing the data" to describe running
  python -m build_scripts.generate_dashboard_html instead of
  re-executing a notebook.
- doc/myst.yml: point the TOC at the new .md files.
- Deleted the old 1_scorer_quality.py/.ipynb and
  2_benchmark_leaderboard.py/.ipynb pairs.

CI never executes notebooks today (jupyter-book build --all --html,
no --execute flag), and .github/instructions/docs.instructions.md
only applies to .py/.ipynb pairs, so this needed no CI changes --
pre-generate once, commit the static HTML, and the site build just
serves it, exactly like it already does for notebook outputs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…n script

- resolve_objective_identity() in adversarial.py derives stable
  (objective_target, objective_scorer) display labels, shared by the
  scenario's own result-reuse cache and the exporter.
- AdversarialBenchmark can now recognize combinations already present in a
  committed benchmark_store_path and skip re-running them.
- export_adversarial_benchmark_result.py gains --update-benchmark-store to
  upsert technique-metrics rows (keyed on technique/adversarial_model/
  objective_target/objective_scorer/dataset) into the committed JSONL store.
- New build_scripts/run_adversarial_benchmark.py: a re-runnable wrapper that
  skips combinations already cached (in the store or live memory), so adding
  a new adversarial target only executes the new combinations.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- New build_scripts/import_adversarial_benchmark_snapshot.py upserts an
  externally-produced technique-metrics.json (e.g. downloaded from an Azure
  DevOps pipeline artifact) into the committed benchmark metrics store,
  reusing the same upsert_benchmark_metrics() as the local exporter.
- Supports --default-objective-target/--default-objective-scorer/
  --default-dataset to backfill identity fields on rows produced by an
  exporter version that predates them, without overwriting a field a row
  already has.
- Ingested the 6 real technique-metrics rows from Azure DevOps build 16150
  (a preview run of PR microsoft#2551's not-yet-merged dataset/pipeline updates),
  backfilling objective_target=openai_chat and dataset=adversarial_benchmark_v1
  per the run's actual pipeline parameters; objective_scorer is <unknown>
  since it wasn't captured by that pipeline version.
- Documented the provenance of these rows and the new ingestion path in the
  dashboard docs, and regenerated the benchmark leaderboard HTML fragment.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
hannahwestra25 added a commit that referenced this pull request Sep 16, 2026
Combines PR #2551 benchmark-store/pipeline automation with PR #2667 metrics dashboard for an end-to-end hackathon demo run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant