Repository navigation
Conversation
The Stop hook filtered spans with (confidence or hallucination_score or 0.0) >= min_confidence, so a span without a score counted as 0.0 and was dropped once --min-confidence was above 0. In --llm-model mode neither the judge path nor the native v2 generative path produces a score, so every flagged span was dropped and the hook reported a clean answer. Keep unscored spans, the same rule as BaseDetector._filter_spans_by_confidence, and document in --help and the integration README that the threshold applies to scored spans only. Closes KRLabsOrg#117 Signed-off-by: Clannad47 <clannad47@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #117
Summary
The Claude Code Stop hook reported a hallucinated answer as clean whenever
--min-confidencewas above0.0in--llm-modelmode.check_answer.pyfiltered spans with(s.get("confidence", s.get("hallucination_score")) or 0.0) >= args.min_confidence,so a span without a score counted as
0.0and was dropped. No span carries a score in--llm-modelmode: the judge path buildsLLMDetectorwithoutinclude_reasoning, so noconfidencekey is produced, and the native v2 path (KRLabsOrg/lettucedect-v2-qwen-2b)builds spans through
gen.spans_to_offsets, which copies onlycategory,subcategoryand
explanation.main()then returned0— the exit code that means "nothing toreport" — so the agent was never asked to revise.
--api-urland--model-pathmodes areunaffected.
This change keeps unscored spans, the same rule as
BaseDetector._filter_spans_by_confidence(lettucedetect/detectors/base.py:91-96).check_answer.py: keep a span whose score is missing. The post-filter stays local onpurpose:
LettuceClient.detect_spanstakes no threshold, and API spans usehallucination_scorerather thanconfidence, so forwardingmin_confidenceinto thedetectors is not a drop-in for the API mode.
--min-confidencehelp text and the integration README flag table now say the thresholdapplies to scored spans only.
[Unreleased]gains aFixedentry.tests/test_claude_code_hook_pytest.py:test_min_confidence_keeps_unscored_spans— two unscored spans,--min-confidence 0.5,expects exit 2 and both spans in the report (the scenario from Stop-hook --min-confidence drops every --llm-model span (missing confidence treated as 0.0), reporting hallucinated answers as clean #117).
test_min_confidence_mixes_scored_and_unscored— scored low / unscored / scored high inone result: the low-confidence span is dropped, the other two are kept.
test_llm_mode_unscored_spans_reported_above_threshold— end-to-end--llm-modelwith anative-path span shape (no
confidence),--min-confidence 0.5, expects exit 2.The existing
test_min_confidence_filters(scored span below threshold → exit 0) stillpasses unchanged.
Related issue
Closes #117
Type of change
Testing
ruff format --check lettucedetect/ lettucedetect_api/ tests/—69 files already formattedruff check lettucedetect/ lettucedetect_api/ tests/ --extend-exclude lettucedetect/integrations/—All checks passed!python -m pytest—210 passedon Linux (Ubuntu 24.04, Python 3.12, thepython-tests.ymlplatform). OnWindows the 8 tests in
tests/test_benchmark_detectors_pytest.pycannot importscripts/benchmark_detectors.py,which does
import resource(Unix-only); those 8 pass on Linux, and the remaining 202 pass on Windows.--llm-model KRLabsOrg/lettucedect-v2-qwen-2b --llm-base-url http://localhost:8001/v1 --min-confidence 0.5→ exit 2 with the span report on stderr.Checklist
Rights & sign-off (required)
distributed under the repository's MIT license
(see CONTRIBUTING).