Conversation
LLM analyzers intermittently emit numeric scores as JSON strings
("85" instead of 85). AnalysisResponseSchema used strict z.number(),
so a single string-typed score rejected the entire analysis with
invalid_type => parseFailed: true and the KSM score was discarded
even though the run legitimately captured the flag.
Observed in the Sep/Oct 2026 benchmarking sweeps with DeepSeek-V3 as
analyzer: multiple flag-captured runs (e.g. GLM-5.3-Flash on
indirect-prompt-injection, Kimi-K3 on confused-deputy-email-agent)
were recorded without a KSM solely because of this strictness -
results/*.analysis.json show parseFailed: true with
"expected number, invalid_type" on a numeric field.
Switch the analysis-response score fields (behavior.decisionQuality,
strategy.*, rubricEvaluation.qualitative.*.score, milestones[].achieved)
to z.coerce.number()/z.coerce.boolean() so string-typed numerics are
accepted, while real type errors (non-numeric text) still fail.
Plain numbers pass through unchanged.
Verified against the stored failing analyses: this would have salvaged
the parseFailed runs without re-running them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Analyzer runs intermittently complete with a captured flag but
parseFailed: trueand no KSM score. In the Sep/Oct 2026 AI-lab benchmarking sweeps this affected real results — e.g. GLM-5.3-Flash onindirect-prompt-injectionand Kimi-K3 onconfused-deputy-email-agentshowFlag captured: KX{...}in the run logs whileoasis/results/<runId>.analysis.jsoncontains"parseFailed": truewith"expected number, invalid_type".Root cause
AnalysisResponseSchema(src/lib/schemas.ts) validated every score field with strictz.number()/z.boolean(). Analyzer LLMs (DeepSeek-V3 in particular) intermittently serialize scores as JSON strings —"decisionQuality": "85"instead of85. One string-typed score rejects the entire analysis object, discarding the whole structured analysis (kill chain, narrative, milestones, strategy) even though the analyzer's semantic content was fine.Fix
Coerce the analysis-response scalar fields to accept string-typed numerics/booleans:
behavior.decisionQuality->z.coerce.number()strategy.reconQuality / exploitEfficiency / adaptability / overallScore->z.coerce.number()rubricEvaluation.milestones[].achieved->z.coerce.boolean()rubricEvaluation.qualitative.*.score->z.coerce.number()Plain numbers pass through unchanged; genuinely malformed values (non-numeric text) still fail validation.
Verification
npm run typecheckcleannpm run buildcleannpm run test: 419 passed (16 files) incl. 3 new regression tests intests/schemas-coercion.test.tsImpact
Removes the largest source of lost KSM scores observed across ~150 benchmark runs (flag captured, score missing).