Skip to content

fix: keep unscored spans in the Claude Code hook --min-confidence filter - #124

Open
Clannad47 wants to merge 1 commit into
KRLabsOrg:mainfrom
Clannad47:fix/claude-code-hook-min-confidence-unscored-spans
Open

Clannad47 wants to merge 1 commit into
KRLabsOrg:mainfrom
Clannad47:fix/claude-code-hook-min-confidence-unscored-spans

Conversation

@Clannad47

Copy link
Copy Markdown

Closes #117

Summary

The Claude Code Stop hook reported a hallucinated answer as clean whenever
--min-confidence was above 0.0 in --llm-model mode.

check_answer.py filtered spans with
(s.get("confidence", s.get("hallucination_score")) or 0.0) >= args.min_confidence,
so a span without a score counted as 0.0 and was dropped. No span carries a score in
--llm-model mode: the judge path builds LLMDetector without include_reasoning, so no
confidence key is produced, and the native v2 path (KRLabsOrg/lettucedect-v2-qwen-2b)
builds spans through gen.spans_to_offsets, which copies only category, subcategory
and explanation. main() then returned 0 — the exit code that means "nothing to
report" — so the agent was never asked to revise. --api-url and --model-path modes are
unaffected.

This change keeps unscored spans, the same rule as
BaseDetector._filter_spans_by_confidence (lettucedetect/detectors/base.py:91-96).

  • check_answer.py: keep a span whose score is missing. The post-filter stays local on
    purpose: LettuceClient.detect_spans takes no threshold, and API spans use
    hallucination_score rather than confidence, so forwarding min_confidence into the
    detectors is not a drop-in for the API mode.
  • --min-confidence help text and the integration README flag table now say the threshold
    applies to scored spans only.
  • CHANGELOG [Unreleased] gains a Fixed entry.
  • Three regression tests.

tests/test_claude_code_hook_pytest.py:

The existing test_min_confidence_filters (scored span below threshold → exit 0) still
passes unchanged.

Related issue

Closes #117

Type of change

  • Bug fix
  • Feature
  • Documentation
  • Tests
  • Refactor or maintenance

Testing

  • ruff format --check lettucedetect/ lettucedetect_api/ tests/ — 69 files already formatted
  • ruff check lettucedetect/ lettucedetect_api/ tests/ --extend-exclude lettucedetect/integrations/ — All checks passed!
  • python -m pytest — 210 passed on Linux (Ubuntu 24.04, Python 3.12, the python-tests.yml platform). On
    Windows the 8 tests in tests/test_benchmark_detectors_pytest.py cannot import scripts/benchmark_detectors.py,
    which does import resource (Unix-only); those 8 pass on Linux, and the remaining 202 pass on Windows.
  • Other: ran the hook in a real subprocess with a stubbed detector, --llm-model KRLabsOrg/lettucedect-v2-qwen-2b --llm-base-url http://localhost:8001/v1 --min-confidence 0.5 → exit 2 with the span report on stderr.

Checklist

  • I kept the PR focused on one change.
  • I added or updated tests/docs when needed.
  • I checked that no secrets, API keys, or credentials are included.

Rights & sign-off (required)

  • I certify that I have the right to submit this code and that it may be
    distributed under the repository's MIT license
    (see CONTRIBUTING).

The Stop hook filtered spans with
(confidence or hallucination_score or 0.0) >= min_confidence, so a span
without a score counted as 0.0 and was dropped once --min-confidence was
above 0. In --llm-model mode neither the judge path nor the native v2
generative path produces a score, so every flagged span was dropped and
the hook reported a clean answer.

Keep unscored spans, the same rule as
BaseDetector._filter_spans_by_confidence, and document in --help and the
integration README that the threshold applies to scored spans only.

Closes KRLabsOrg#117

Signed-off-by: Clannad47 <clannad47@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop-hook --min-confidence drops every --llm-model span (missing confidence treated as 0.0), reporting hallucinated answers as clean

1 participant