feat(feedback): thumbs up/down on answers (MCP + chat) with an answer_log table by northdpole · Pull Request #1162 · OWASP/OpenCRE

northdpole · 2026-10-20T08:48:58Z

Adds thumbs up/down on answers so staging can collect labelled data.

answer_log table (+ migration d1e2f3a4b5c6, which joins the two existing heads) and recording of ratings.
MCP: ask returns an answer_id; new rate_answer tool.
Chat: completion returns answer_id; POST /rest/v1/answer_feedback; GET /admin/feedback. Chat answers are logged only when a deployment opts in (ANSWER_LOG_CHAT=1).
Chat UI: thumbs under answers, storage notice when answers are logged, and the "not stored" privacy paragraph is hidden when answers are stored.
Test: alembic_single_head_test fails when the migration chain has more than one head.

Stacked on #1158 (order: #1142, #1155, #1159, #1158, this). Needs the migration run and ANSWER_LOG_CHAT=1 set on staging. ask now writes one row per question, so this should be deployed after measurement runs.

Tested: answer_feedback_test, alembic_single_head_test, mcp_ask_test pass; Black clean. Frontend tests run in CI.

🤖 Generated with Claude Code

coderabbitai · 2026-10-20T08:49:06Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 1

main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

Source mode: Auto
Repository: OWASP/OpenCRE
Branch: feat/answer-feedback
File: .coderabbit.yml
Commit: 2f4e740
Layers: Repository YAML
Review profile: CHILL
Plan: Advanced
Run ID: 5c623119-28a6-4c73-b284-bd9ef7302da2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…turn probe Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…messages (background, not a question); rewrite stays for retrieval Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…list; ungrounded free-form answers are replaced by an honest refusal (manual-test items 11 and 15) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…box (off by default) replaces New chat Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…PROJECTS (3 best projects each contribute sources) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…x capped results at 40) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…t chunks to the candidate pool Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…h in the references instead of reordering sources Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… default) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ast question and answer Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…t references) The OWASP agent answers bypassed the CRE lookup, so control questions no longer showed their closest CRE. route_question now adds it to the references and citations. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… so [n] stays the only link form Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…es in citation URLs Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… Related before child-of on CRE pages Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… the shown near-ties Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…file path, show 2 CREs and at most 5 resources - paths.py: is_asset_path (image/asset-folder READMEs, vendored gems, site scaffolding) is dropped at ingestion; low-value paths (old versions, tests, changelogs) rank slightly lower and cite last - answers cite at most OWASP_MAX_RESOURCES=5 resources; featured projects only fill spare slots - closest CREs: 2 (was 3) - scripts/prune_noise_chunks.py removes already-indexed noise chunks (dry run by default) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

NO_LOGIN=0 is a truthy string, so staging (NO_LOGIN=0) got a random per-process secret key. Any worker restart invalidated session cookies, losing the OAuth state and sending beta testers back to /auth/login in a loop. Only NO_LOGIN=1 now randomizes the key, and SECRET_KEY is preferred over GOOGLE_CLIENT_SECRET. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…gleton The singleton held the pending state in process memory, so concurrent logins overwrote each other and a callback on another worker saw no state and looped. State now comes from the user's session cookie. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

The per-login flow built in the callback had no code_verifier, so Google rejected the token exchange (invalid_grant) and the callback returned 500. Store the verifier in the session at login and pass it to fetch_token. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

/auth/user was gated by login_required (admin). Signed-in users with only the user role got 403, the SPA read that as anonymous and sent them back through login in a loop. It only reports who is signed in, so user_required fits. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Named admins may sign in from outside LOGIN_ALLOWED_DOMAINS; /rest/v1/auth/*, /me and /my-api-keys are never browser-cached; /auth/callback no longer 500s on a replayed code, a consent denial or a missing state; chatbot redirects to login only on 401. (MCP half lives on feat/ext-integration.) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

northdpole force-pushed the feat/answer-feedback branch 2 times, most recently from b40939d to 957841c Compare October 20, 2026 08:51

northdpole force-pushed the feat/ingest-comprehension branch from c306180 to 6b2c573 Compare October 20, 2026 09:21

northdpole force-pushed the feat/answer-feedback branch 14 times, most recently from 76307f2 to dc9eb78 Compare October 21, 2026 20:05

northdpole force-pushed the feat/ingest-comprehension branch from b6ded96 to 0a34ae0 Compare October 21, 2026 21:09

northdpole force-pushed the feat/answer-feedback branch from dc9eb78 to c672c15 Compare October 21, 2026 21:09

northdpole force-pushed the feat/ingest-comprehension branch from 0a34ae0 to a6573ed Compare October 21, 2026 23:09

northdpole force-pushed the feat/answer-feedback branch 2 times, most recently from 2556722 to 6acc47d Compare October 22, 2026 01:29

northdpole force-pushed the feat/ingest-comprehension branch from c4b23d4 to 594e1d0 Compare October 22, 2026 07:59

northdpole force-pushed the feat/answer-feedback branch 2 times, most recently from 640e7c4 to f082dd3 Compare October 22, 2026 11:09

northdpole marked this pull request as ready for review October 22, 2026 11:17

northdpole requested review from Pa04rth, paoga87 and robvanderveer as code owners October 22, 2026 11:17

northdpole and others added 9 commits October 25, 2026 10:31

test(chat): scripted multi-turn conversations in CI and a live multi-…

b545ddb

…turn probe Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(chat): earlier turns reach the answer call as native multi-turn …

bc0e9a7

…messages (background, not a question); rewrite stays for retrieval Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(chat): [n] citation markers match the numbered, clickable source …

9e8c38c

…list; ungrounded free-form answers are replaced by an honest refusal (manual-test items 11 and 15) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(chat): new prompt is a fresh chat; 'Continue conversation' check…

9d22c24

…box (off by default) replaces New chat Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(comprehension): admin-editable project authority weights and TOP…

7f94586

…PROJECTS (3 best projects each contribute sources) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(comprehension): raise hnsw.ef_search for wide dense queries (inde…

f0d1edf

…x capped results at 40) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(comprehension): weighted-up projects always contribute their bes…

a1f08ca

…t chunks to the candidate pool Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(comprehension): authority above 1 features a project's best matc…

99b19ba

…h in the references instead of reordering sources Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(admin): apostrophe broke the Authority tab description string

7335acc

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

northdpole force-pushed the feat/ingest-comprehension branch from 5406406 to 4c92743 Compare October 25, 2026 10:35

northdpole force-pushed the feat/answer-feedback branch from 7c49a5d to 7335acc Compare October 25, 2026 10:35

northdpole and others added 9 commits October 25, 2026 10:49

docs(oie): document max_repos semantics (0 = no cap, omitted = capped…

ec70f2f

… default) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(chat): move 'Continue conversation' next to Ask; send only the l…

f2d8fec

…ast question and answer Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(chat): strip copied markdown links and [[n]] markers from answers…

0fa1a94

… so [n] stays the only link form Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(chat): strip links whose URLs contain spaces; percent-encode spac…

107e67d

…es in citation URLs Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(deps): keep litellm below 1.107 (reported malicious release)

10a238c

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(chat): CRE references link to /cre/<id>; list near-tie CREs; show…

00c4e47

… Related before child-of on CRE pages Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(chat): CRE names that share a keyword with the question rank into…

cd19478

… the shown near-ties Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This was referenced Oct 28, 2026

feat(ops): alerting and health checks (OPS-2,3,4,6,8) and thumbs-down export #1178

Open

feat(comprehension): external sources contract (EXT-1) #1179

Open

feat(ext): EXT-1 integration (external sites + YouTube): contains #1179-#1182 + retrieval #1183

Draft

feat(chat): ask for a short comment inline after a thumbs down

cfed581

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

northdpole mentioned this pull request Oct 29, 2026

feat(mcp): Google sign-in (OAuth) for the remote MCP server #1188

Open

northdpole and others added 5 commits October 30, 2026 14:06

Navigation Menu

Repository navigation

feat(feedback): thumbs up/down on answers (MCP + chat) with an answer_log table - #1162

feat(feedback): thumbs up/down on answers (MCP + chat) with an answer_log table#1162
northdpole wants to merge 46 commits into
feat/ingest-comprehensionfrom
feat/answer-feedback

northdpole commented Oct 20, 2026

Uh oh!

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

Review skipped

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

northdpole commented Oct 20, 2026

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading