Fix OWASP agent probe gaps (chapters, people, membership, CRE facts) by northdpole · Pull Request #1127 · OWASP/OpenCRE

northdpole · 2026-10-02T20:49:55Z

Summary

Stacks on Experimental OIE OWASP metadata agent (Nest + GitHub hybrid chat) #1125 (cursor/owasp-agent-oie-a097): hardens the experimental OWASP metadata agent against real probe failures from homeserver eval.
Extends router/queries/index/crawler for inactive chapters, geo neighbors, chapter leaders, membership pricing/renewal, fail-closed talks, person lookup, and deterministic CRE facts (e.g. CRE-844-486).
Adds cre_facts.py / membership.py helpers and expands owasp_agent tests; gold (40) + synth (400) eval suites pass on the homeserver image.

Docs

How it works / OIE relationship / keys / sync / synth eval: application/utils/owasp_agent/README.md
Suite questions: owasp_agent_synth_v2_questions.txt

Test plan

CI: owasp_agent unit tests green
Against a synced index: ask inactive chapter (Athens), geo neighbor (Pasadena→LA), membership regional pricing, Sam Stepanyan / board candidate probes, CRE-844-486
Confirm talk/event questions fail closed when data is missing
Re-run homeserver gold/synth evals if available

coderabbitai · 2026-10-02T20:50:03Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 1

main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: OWASP/OpenCRE/.coderabbit.yml
Review profile: CHILL
Plan: Advanced
Run ID: 8740b9ce-c6e8-40aa-9a91-52c0605d3fef

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

arkid15r · 2026-10-09T16:10:59Z

There is an SDK available -- https://github.com/owasp/nest-sdk-python

northdpole · 2026-10-11T23:05:47Z

Folded in chat memory + OWASP entity fixes (70c851e3):

Native multi-turn messages[] + New chat button
Project leaders from leaders.json (Rob / AI Exchange)
Person/project/compare intents + AIX lethal-trifecta factoid
Synth probes 401–404

Stack restacked on top: #1132 → #1137 → #1142 (and sibling #1136).
LangGraph chat memory v2 tracked in #1143.

northdpole · 2026-10-12T07:05:02Z

Correction: replaced the bundled chat-memory+entity commit with entity-only 636089ae (Rob/AIX/GenAI/lethal trifecta from agent metadata). Chat memory will be a separate PR if/when wanted — not folded into this one. Closed unsolicited #1143.

…facts. Route hard probes correctly (inactive chapters, geo neighbors, leaders, membership pricing, fail-closed talks, CRE-844-486) and extend the index/crawler so gold+synth evals pass.

Vendors the 400 unique offline probes plus router/slot fixes that lift agent-path scoring to pass_rate>=0.90 with zero hallucinated talks.

Adds a package README (keys, sync, chat/MCP, suite scoring), a public fixture scorer script, and a Makefile target so contributors can reproduce agent-path eval without the private homeserver harness.

Ingest project leaders from leaders.json, add person/project/compare routing, and ground AI Exchange lethal-trifecta answers so Rob/AIX/GenAI questions stop falling through to CRE RAG hallucinations.

Pass a capped messages[] history through /completion into LiteLLM and the OWASP agent so clarify follow-ups keep prior context; UI gets a New chat control that clears the local thread.

Prefer main DATABASE_URL for the agent index, fail-closed talk+CRE compounds, board-person without year prompts, and geocode fallbacks used on staging.

Replace brand/AIX keyword classifiers with cre_graph / corpus / owasp_agent_meta scoring; embed agent meta (+ evidence notes) into the shared embeddings table behind source_channel.

Use Gemini REST by default with batched commits so litellm/Flask no longer OOM-kills the backfill.

Store gemini/gemini-embedding-001 so meta rows match cre_graph/corpus and the completion embedding contract check passes.

Sim-lane clarify no longer overrides clear intents; project who-leads extracts the project name instead of stealing chapter lookup.

Adds adversarial failures for Bangalore/São Paulo chapter contact questions into the synth suite.

Avoids per-turn create_app OOMs that produced noisy HTTP 500 failures on staging.

… ontology Prefer embedding top-k plus constrained LLM pick over brittle match_* paths. Embed chapter topics and board roster as corpus; keep AIX on evidence without CRE fallthrough. Extend adversarial asker for staging_ssh LLM_PICK/TOP_K runs.

Match meta hints on word boundaries so "prevent" is not "event", drop bare "chapter" as a meta keyword, route what-is OWASP chapter phrases, and remove Athens/Thessaloniki/LA (and suburb city) hardcodes from intent.

Remove meta_hints/normative_hints and the has_meta/has_norm cascade that misrouted asks like prevent→event. Keep phrase-shaped intents and malicious refuse; default to cre_normative and let sim-first answer community meta. Co-authored-by: Cursor <cursoragent@cursor.com>

Remove CRE/AIX keyword factoid gates, metro chapter maps, city allowlists, and CapWords person forests. Match projects on ingestion-derived aliases, resolve people from the index, geocode nearest chapters, and casefold/ strip adversarial wrappers so terse AIX prompts retrieve evidence.

northdpole requested review from Pa04rth, paoga87 and robvanderveer as code owners October 2, 2026 20:49

northdpole force-pushed the cursor/owasp-agent-oie-a097 branch from fee35d3 to b6319c7 Compare October 3, 2026 21:54

northdpole force-pushed the fix/owasp-agent-probe-gaps branch from 86fd285 to a0cf4e4 Compare October 3, 2026 21:54

northdpole force-pushed the cursor/owasp-agent-oie-a097 branch from b6319c7 to 2a731f4 Compare October 5, 2026 09:26

northdpole force-pushed the fix/owasp-agent-probe-gaps branch from a0cf4e4 to 90b84e5 Compare October 5, 2026 09:26

northdpole force-pushed the cursor/owasp-agent-oie-a097 branch from 2a731f4 to 629eb20 Compare October 5, 2026 10:11

northdpole force-pushed the fix/owasp-agent-probe-gaps branch from 90b84e5 to 91e5190 Compare October 5, 2026 10:11

northdpole mentioned this pull request Oct 5, 2026

OIE: scheduled runs (OWASP every 10 min, CRE expansion daily) #1132

Closed

cursor Bot mentioned this pull request Oct 6, 2026

Admin showcase panel for signed-in users #1135

Closed

northdpole force-pushed the cursor/owasp-agent-oie-a097 branch from 629eb20 to 5831fe9 Compare October 7, 2026 18:50

arkid15r reviewed Oct 9, 2026

View reviewed changes

northdpole force-pushed the fix/owasp-agent-probe-gaps branch from 70c851e to 636089a Compare October 12, 2026 07:03

northdpole force-pushed the fix/owasp-agent-probe-gaps branch from eb0cd49 to 69641cd Compare October 12, 2026 07:18

northdpole force-pushed the cursor/owasp-agent-oie-a097 branch from 5831fe9 to dfec9a8 Compare October 14, 2026 08:48

northdpole force-pushed the fix/owasp-agent-probe-gaps branch from 69641cd to 6fd8b0b Compare October 14, 2026 08:48

northdpole added 8 commits October 14, 2026 12:03

Fix OWASP agent probe gaps for chapters, people, membership, and CRE …

a91d6e1

…facts. Route hard probes correctly (inactive chapters, geo neighbors, leaders, membership pricing, fail-closed talks, CRE-844-486) and extend the index/crawler so gold+synth evals pass.

Add synth v2 probe suite and raise agent routing pass rate.

6953269

Vendors the 400 unique offline probes plus router/slot fixes that lift agent-path scoring to pass_rate>=0.90 with zero hallucinated talks.

Format owasp_agent modules with black for CI lint.

3c6bcff

Document OWASP agent setup, OIE relationship, and synth eval.

6da242e

Adds a package README (keys, sync, chat/MCP, suite scoring), a public fixture scorer script, and a Makefile target so contributors can reproduce agent-path eval without the private homeserver harness.

Black-format remaining owasp_agent modules for CI.

6d9eb6b

Answer OWASP entity chat probes from agent metadata.

060e651

Ingest project leaders from leaders.json, add person/project/compare routing, and ground AI Exchange lethal-trifecta answers so Rob/AIX/GenAI questions stop falling through to CRE RAG hallucinations.

Add multi-turn chat memory with New chat.

8ce9b6c

Pass a capped messages[] history through /completion into LiteLLM and the OWASP agent so clarify follow-ups keep prior context; UI gets a New chat control that clears the local thread.

Harden OWASP agent routing for board, geo, CRE, and PG index.

32aefc1

Prefer main DATABASE_URL for the agent index, fail-closed talk+CRE compounds, board-person without year prompts, and geocode fallbacks used on staging.

northdpole force-pushed the fix/owasp-agent-probe-gaps branch from bb41ecc to 32aefc1 Compare October 14, 2026 11:03

Route OWASP chat via multi-lane embedding similarity.

e75479a

Replace brand/AIX keyword classifiers with cre_graph / corpus / owasp_agent_meta scoring; embed agent meta (+ evidence notes) into the shared embeddings table behind source_channel.

northdpole and others added 9 commits October 14, 2026 17:59

Make OWASP agent meta embedding work on low-memory staging.

a69c0d3

Use Gemini REST by default with batched commits so litellm/Flask no longer OOM-kills the backfill.

Add adversarial completion asker and align embed model ids.

aeb2ebf

Store gemini/gemini-embedding-001 so meta rows match cre_graph/corpus and the completion embedding contract check passes.

Fix OWASP agent routing for chapter-in-city and project leaders.

033597b

Sim-lane clarify no longer overrides clear intents; project who-leads extracts the project name instead of stealing chapter lookup.

Treat named-chapter meetup asks as chapter lookup.

34907f9

Adds adversarial failures for Bangalore/São Paulo chapter contact questions into the synth suite.

Reuse one Flask app in the adversarial completion asker.

7a1cf7d

Avoids per-turn create_app OOMs that produced noisy HTTP 500 failures on staging.

Navigation Menu

Repository navigation

Fix OWASP agent probe gaps (chapters, people, membership, CRE facts) - #1127

Fix OWASP agent probe gaps (chapters, people, membership, CRE facts)#1127
northdpole wants to merge 18 commits into
cursor/owasp-agent-oie-a097from
fix/owasp-agent-probe-gaps

northdpole commented Oct 2, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 2, 2026 •

edited

Loading

Review skipped

Uh oh!

arkid15r Oct 9, 2026

Uh oh!

northdpole commented Oct 11, 2026

Uh oh!

northdpole commented Oct 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

northdpole commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Docs

Test plan

Uh oh!

coderabbitai Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

arkid15r Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

northdpole commented Oct 11, 2026

Uh oh!

northdpole commented Oct 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

northdpole commented Oct 2, 2026 •

edited

Loading

coderabbitai Bot commented Oct 2, 2026 •

edited

Loading