Skip to content

Preserve safe central review policy traceback locations in sidecar diagnostics #2122

Open

Description

@seonghobae

Observed on 2026-09-12 in late-life-anxiety-reanalysis PR10 head a1cd5bc6783c6510dfcf937f523c733366e82213, Noema run34700409452/job103571267389. Sidecar exits before healthz. Uploaded artifact10299608783 (SHA256 deeeec486758661e03a193daad597eeb1dc4cbde40fb078ab2d8ce3510b83fd7) contains an empty preflight JSON and:

discovery_diagnostics_complete
unexpected_exception type=scripts.ci.contextual_orchestrator_review_policy.PolicyError frame=unknown
omitted_unstructured_lines=13

Confirmed against actual trusted source fb17ef5: scripts/ci/sanitize_contextual_orchestrator_sidecar_stream.py only retains contextual_orchestrator package frames. Central policy/launcher frames are discarded. This prevents locating the failing validation guard; it does not prove the underlying provider cause or link it causally to #2119. Raw provider/exception messages must remain excluded.

An isolated minimal candidate below admits only the two canonical review policy/launcher filenames, preserves current hosted request-ID diagnostics, and emits relative paths only. Baseline loses policy/launcher location; candidate passes 8 Unix/Windows path cases including CO parity, non-allowlisted rejection, and secret canaries. All 11 existing sanitizer tests extracted unchanged from the same hosted test source pass (pytest 3.46s). This is targeted sanitizer verification, not the entire runtime suite or hosted integration. No owner source, permissions, credentials, or serving policy was modified; candidate is not applied/released.

Patch SHA256: 4007325f34e4f5660cd2ae997cdf1ba0efb3673ebdac05e24cc7ad0538ad1462.

--- a/scripts/ci/sanitize_contextual_orchestrator_sidecar_stream.py
+++ b/scripts/ci/sanitize_contextual_orchestrator_sidecar_stream.py
@@ -79,13 +79,14 @@
 # Python traceback anatomy. The orchestrator's generic request handler
 # (``server.py`` ``except Exception: traceback.print_exc(); _send_error(500,
 # "internal_error", ...)``) prints one traceback per unhandled exception, so the
-# exception *type* and the innermost ``contextual_orchestrator`` frame are the
+# exception *type* and the innermost allowlisted orchestrator or review-bootstrap frame are the
 # only evidence of what escaped. Frame lines are indented; the terminal line
 # (``Type: message``) starts at column 0. Only the bounded type identifier and
 # the package-relative frame are re-emitted -- never the message, which can
 # carry provider bodies or credentials.
 _TRACEBACK_FRAME = re.compile(
-    r'^\s+File ".*?[/\\]contextual_orchestrator[/\\](?P<module>[A-Za-z0-9_][A-Za-z0-9_/\\]*\.py)", '
+    r'^\s+File ".*?[/\\](?P<module>contextual_orchestrator[/\\][A-Za-z0-9_][A-Za-z0-9_/\\]*\.py|'
+    r'scripts[/\\]ci[/\\]contextual_orchestrator_review_(?:policy|launcher)\.py)", '
     r"line (?P<line>\d+), in (?P<function>[A-Za-z0-9_<>]{1,80})$"
 )
 _TRACEBACK_TERMINAL = re.compile(
@@ -214,8 +215,8 @@
     terminal ``Type: message`` line (emitting ``unexpected_exception type=...
     frame=...``), at the next header or allowlisted line, or at end of stream
     (``type=unknown``). Indented lines inside it are frames and source echoes:
-    consumed, not counted as omitted, and only a ``contextual_orchestrator``
-    frame's package path, line and function are retained. Any other column-0
+    consumed, not counted as omitted, and only an allowlisted orchestrator or review-bootstrap
+    frame's relative path, line and function are retained. Any other column-0
     line closes the traceback and is classified like every other line.
     """
     omitted = 0
@@ -235,7 +236,7 @@
             frame_match = _TRACEBACK_FRAME.match(line.rstrip("\n"))
             if frame_match is not None:
                 module = frame_match.group("module").replace("\\", "/")
-                frame = f"contextual_orchestrator/{module}:{frame_match.group('line')}:{frame_match.group('function')}"
+                frame = f"{module}:{frame_match.group('line')}:{frame_match.group('function')}"
                 continue
             if line[:1].isspace():
                 continue

Owner: central review/bootstrap sanitizer maintainer. Please coordinate this narrow delta with the active writer rather than duplicating request telemetry changes. Next action: adopt/review the allowlisted-frame patch against current owner head, run owner tests and then inspect the next authorized private-review artifact. Acceptance: policy/launcher failures retain safe relative frame and exception type, credentials/raw messages remain absent, existing CO diagnostics remain intact, and actual failing policy guard is identified and repaired separately. Internal investigation/adoption checkpoint: 2026-09-13 (active review outage; not a promised deployment date). No automatic rerun or bypass requested. Consumer PR10 remains subject to its independent statistical/provenance findings.

Activity

  1. seonghobae commented on Sep 12, 2026

    @seonghobae
    ContributorAuthor

    Canonical repair is implemented in clean Draft successor #2124. TDD RED head ebb21495267987befd4a88970f03de490326f361 reproduced frame=unknown; current head e9acd81064408f0c49b72d7a617c747c96f21244 retains only the policy/launcher relative frames across Unix/Windows fixtures, preserves the existing contextual-orchestrator frame, rejects neighboring scripts/ci/untrusted.py, and suppresses secret canaries. Focused exact-head verification passed 4 traceback + 7 bounded-line cases. Fresh hosted Security/Python Security/Semgrep/CodeQL runs are queued, so this issue remains open and the source fix is not yet claimed integrated or released.

  2. seonghobae commented on Sep 13, 2026

    @seonghobae
    ContributorAuthor

    Root cause behind the PolicyError this issue could not locate is now isolated and repaired in #2146: _load_zdr_endpoints keyed the OpenRouter ZDR feed by the display model_name instead of the model_id slug (since 17052a7, #1360), so no live-feed route ever matched and every --require-zdr (private/internal) target failed closed with "no attested ZDR model route is available". Offline reproduction used the consumer strix artifact's discovery JSON (60 free rows) plus the live feed: exit 1 on main, exit 0 with three attested inclusionai/ling-3.0-flash-*:free routes after the fix.

    With this PR's frame retention, the sanitized line would have read frame=scripts/ci/contextual_orchestrator_review_policy.py:371:build_zdr_prioritized_catalog. Separate read-only review findings on #2124 are on that PR (the copied launch_sidecar.py basename is never admitted by the launcher allowlist).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: highHigh-priority or P1 work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions