Skip to content

fix(sc8): count only the walk's own CPU time against its runtime budget - #828

Open
rng1995 wants to merge 4 commits into
mainfrom
naren/fix-sc8-thread-time-budget
Open

rng1995 wants to merge 4 commits into
mainfrom
naren/fix-sc8-thread-time-budget

Conversation

@rng1995

@rng1995 rng1995 commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The SC8 shipped-bytecode walk (_scan_shipped_bytecode) stops after MAX_SC8_ANALYSIS_SECONDS = 5.0 of wall-clock time. All analyzers run in parallel threads in one process. Each os.scandir step releases the GIL, and the walk then waits behind CPU-bound analyzers to get it back. A walk that needs a few milliseconds of work can therefore run out of time.

In a customer report (SKILLEVAL-17), a skill with no __pycache__, .pyc or .pyo files got a static runtime_limit ledger row. The folder in it changed on every run, because it was wherever the walk had reached when time ran out. The SC8 walk started about 190 s into the scan. SkillEvaluator reports any static runtime_limit as a Critical "skillspector scan did not execute reliably", so the row blocked submission.

Reproduction: the real v2.12.0 _scan_shipped_bytecode on a tree with the customer's layout (148 files, 50 folders, no bytecode):

Concurrent load Before After
none 0.06 s, no limitations 0.04 s, no limitations
8 pure-Python CPU threads 5.2 s, runtime_limit at features/safety 7.4 s, no limitations
4 threads compiling large sources 7.5 s, runtime_limit 345 s wall-clock, no limitations (direct call, no workflow deadline)

Fix

  • Measure the 5 s processing budget with time.thread_time(), so only the walk thread's own CPU time counts. static_yara already does this for rule loading.
  • Bound wall-clock time by the remaining workflow deadline (timeout_seconds), which resolve_input always sets for graph scans. Callers outside a workflow fall back to MAX_WORKFLOW_SECONDS.
  • When a limit trips, the ledger row reports the observed and limit seconds of the budget that tripped.

Tests

  • New: waiting time does not count against the budget (fake clock: wall-clock advances, CPU time does not). This test fails on main.
  • New: the CPU budget still stops the walk, with limit_seconds == MAX_SC8_ANALYSIS_SECONDS.
  • New: wall-clock is bounded by the workflow deadline, and by MAX_WORKFLOW_SECONDS when there is none.
  • tests/nodes/analyzers/test_sc8_shipped_bytecode.py: 19 passed. Supply-chain, static-pattern, workflow and transitive selection of tests/nodes/analyzers and tests/unit: 1018 passed.

🤖 Generated with Claude Code

rng1995 and others added 2 commits October 14, 2026 04:51
The SC8 shipped-bytecode walk stopped after 5 s of wall-clock time. Analyzers
run concurrently in one process, so the walk spent most of that time waiting
for the GIL behind CPU-bound analyzers. Skills that ship no bytecode got a
static runtime_limit ledger row at a different folder on every run, which
SkillEvaluator reports as a Critical "skillspector scan did not execute
reliably" (SKILLEVAL-17).

Measure the 5 s processing budget with time.thread_time(), as static_yara
already does for rule loading. Wall-clock time stays bounded by the remaining
workflow deadline, or by MAX_WORKFLOW_SECONDS for callers outside a workflow.

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>

@chrisknvidia chrisknvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 922317e167a47d2ac3819c28edb481519f90d2e2. Looks good to me; no blocking findings.

I checked CPU-time accounting in _scan_shipped_bytecode, propagation of the remaining workflow wall-clock budget, limitation reporting, and the existing traversal/count/finding bounds. All 36 focused SC8 and budget-configuration tests passed.

I also exercised the actual filesystem walk under concurrent CPU-thread load: the baseline hit its wall-clock limit, while this version completed without a limitation. Positive controls still reported shipped bytecode, and an explicit wall-clock allowance still stopped the walk with runtime_limit.

Validation was local on macOS. No live LLM-provider or production validation was performed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants