From e62718f5260e6ad272cd3059ab6a1667d86c171d Mon Sep 17 00:00:00 2001 From: Anatolii Date: Tue, 22 Sep 2026 13:43:07 +0400 Subject: [PATCH 1/6] =?UTF-8?q?chore(release):=200.18.1=20=E2=80=94=20lowe?= =?UTF-8?q?r-friction=20UX=20(lazy=20@protect=20+=20dev=20error=20report?= =?UTF-8?q?=20+=20zero-activity=20diagnostic=20+=20drop=20dead=20extras=20?= =?UTF-8?q?+=20deprecate=20langgraph.wrapper)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 50 +++++ pyproject.toml | 2 +- src/nullrun/__version__.py | 2 +- src/nullrun/decorators.py | 169 +++++++++++++- src/nullrun/extractor.py | 174 +++++++++++++-- tests/test_protect_only_public_api.py | 306 ++++++++++++++++++++++++++ 6 files changed, 674 insertions(+), 29 deletions(-) create mode 100644 tests/test_protect_only_public_api.py diff --git a/CHANGELOG.md b/CHANGELOG.md index c93cf78..b18955c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,53 @@ +## [0.18.1] - 2026-09-22 + +Patch release — **lower-friction UX**: the user gets enforcement + observability from `@protect` alone, without having to call `init_or_die()` first or pick a framework extra. Closes four silent-failure modes at once: (1) `@protect` now auto-attaches a default tool-params extractor (no separate `@sensitive` needed for the common case; bare `@sensitive` is deprecated in favour of `@protect`); (2) `@protect` lazy-triggers `auto_instrument()` on first invocation so the user can write `@protect` before `init_or_die()` (or skip `init` entirely if `NULLRUN_API_KEY` is set); (3) a zero-activity diagnostic emits a one-time WARNING when `@protect` fires 50+ times without a single LLM event, naming the three most likely root causes; (4) `handle()` / `guarded()` / `init_or_die()` print a four-line developer report (what / where / why / how-to-fix) instead of just the catalog user-message. Dead `pip` extras (`[openai]`, `[anthropic]`, `[mistral]`, `[gemini]`, `[cohere]`, `[bedrock]`, `[all]`, `[fastapi]`) are removed — `pip install nullrun` alone is now sufficient for the HTTP-level + `@protect` flow. `toolbox.langgraph.wrapper()` is marked DEPRECATED in its docstring (auto-patch is the canonical path). Wire-format unchanged. SDK_MIN_VERSION unchanged. + +### Added + +- **`@protect` auto-attaches a default tool-params extractor** (`src/nullrun/decorators.py`, `src/nullrun/extractor.py`, `fcd623c`). The first call from `@protect` stamps `ToolParamsExtractor(include_all=True)` on the decorated function so the wire payload carries `tool_name + params` for every protected call — no second decorator required for the common case. The auto-attached extractor carries `_nullrun_auto_attached=True` so `_enforce_sensitive_tool` can distinguish "developer opted into the policy path" from "SDK auto-derived for tooling reasons"; the policy gate short-circuits on auto-attached extractors so bare `@protect` stays cheap (no extra `/execute` round-trip). Bounded extraction: oversized string values (`>=1024 bytes`) get a deterministic `...[truncated:N bytes]` suffix; circular references in nested dict/list structures return the partial walk instead of raising `RecursionError`; dropped values (float, bytes, custom) emit a single aggregate DEBUG log line with count + type names, never one log line per dropped field. + +- **`@protect` lazy-triggers `auto_instrument()`** on first invocation (`src/nullrun/decorators.py`). The user's first decorated function call installs the runtime and patches `httpx` + framework adapters in a single process-wide idempotent step, behind a lock so concurrent `@protect` calls cannot double-fire. Never raises — a vendor SDK breaking change must not block the enforcement gate. Closes the "I added `@protect` but nothing tracks tokens" silent-failure mode. + +- **Zero-activity diagnostic on `NullRunRuntime`** (`src/nullrun/runtime.py`, `DEF-ZERO-ACTIVITY-DIAG`). `_bump_protect_count()` is invoked by `@protect` on every call; `_llm_call_event_count` is bumped by `track_llm()` on every successful call. When `@protect` fires 50+ times without a single LLM event, the runtime emits a one-time WARNING at `logging.WARNING` naming the three most likely root causes (raw httpx outside the patchable surface, custom transport / gRPC, framework not on the auto-detection table). Warn-once invariant under concurrent `@protect` calls is preserved by `_zero_activity_lock`. + +- **Four-line developer error report from `handle()` / `guarded()` / `init_or_die()`** (`src/nullrun/_handle.py`, `DEF-DEV-REPORT-EMPTY`). The catch-all exit path previously printed only the catalog user-message ("There's a configuration issue. Please contact support.") — end-user wording that gave a developer running an example with a missing `NULLRUN_API_KEY` zero actionable detail. The new `_render_dev_error_report()` helper emits a structured report answering the four questions a developer actually asks: (1) `what` — the stage that failed (`auth` / `gate` / `track` / `execute` / `approval` / …), (2) `where` — the wire endpoint + status code + transport source, (3) `why` — the underlying exception message + the machine `error_code`, (4) `how to fix` — the `user_action` from the typed class. The catalog headline is preserved as the first line so end-user-facing deployments still get a clean single sentence. Defensive: `handle()` / `init_or_die()` wrap the helper in `try/except` so a buggy report builder cannot freeze a script that would otherwise exit (falls back to the legacy single-line behaviour). + +### Changed + +- **Removed dead provider extras from `pyproject.toml`**: `[openai]`, `[anthropic]`, `[mistral]`, `[gemini]`, `[cohere]`, `[bedrock]`, `[all]`, `[fastapi]`. NullRun never imported these vendor SDKs — HTTP-level instrumentation (`patch_httpx` + 5 URL-keyed extractors) covers OpenAI, Azure, Anthropic, Mistral, Gemini, Cohere, and Bedrock without them. Kept framework extras (`[opentelemetry]`, `[langgraph]`, `[agents]`, `[langchain]`, `[llama-index]`, `[crewai]`, `[autogen]`) — those still install a vendor SDK that NullRun subscribes to via an event hook. `pip install nullrun` alone is now sufficient for the HTTP-level + `@protect` flow. + +- **`toolbox.langgraph.wrapper()` marked DEPRECATED** in its docstring (`src/nullrun/toolbox/langgraph.py`). `init_or_die()` / `@protect` auto-patches `langgraph.pregel.Pregel` via `patch_langgraph_compiled` — same callback injection the wrapper does manually, but process-wide and idempotent. `wrapper()` remains as an escape hatch for three narrow cases (tests with custom runtimes, Pregel imported before init, manual callback control). No removal — the public symbol stays. + +- **Bare `@sensitive` emits `DeprecationWarning`** (`src/nullrun/decorators.py`). `@sensitive(impact=...)` remains the advanced API for typed `BusinessImpact` + SHA-256 `action_digest`. Bare `@sensitive` is removed in `0.19.x`; emit is `DeprecationWarning` in `0.18.x` only. + +### Fixed + +- **DEF-DEV-REPORT-EMPTY** — `handle()` / `guarded()` / `init_or_die()` now print the four-line developer report (catalog headline + `[error_code]` + what + where + why + how-to-fix + docs URL) instead of just the catalog user-message. Closes the silent-failure mode where a developer hit a config failure at the first gate call and saw only end-user wording with no actionable detail. Defensive: the helper is wrapped in `try/except` so a buggy report builder cannot freeze a script that would otherwise exit. + +### Tests + +- `tests/test_zero_activity_diagnostic.py` (new, 6 tests) — pins the warn-once / threshold / concurrent-bump / message-content invariants on the zero-activity diagnostic. +- `tests/test_dev_error_report.py` (new, 11 tests) — pins the four-line / what / where / why / how-to-fix / docs-URL invariants on the new error report. +- `tests/test_protect_only_public_api.py` (new, 9 tests, landed in `fcd623c`) — pins the auto-attach / chain-walk / bare-`@sensitive`-deprecation invariants on the `@protect` contract change. +- `tests/test_protect.py`, `tests/test_preflight_fail_policy.py`, `tests/test_protect_cancel_on_exception.py` — `_RecordingRuntime` stubs extended with a `_bump_protect_count` no-op so the existing gate / cancel / span test surface keeps working unchanged. + +### Verification + +- `ruff check src tests` — all checks passed. +- `mypy src/nullrun` — success: no issues found in 37 source files. +- `pytest -q` — **1840 passed, 4 skipped, 14 warnings** in ~128s (vs 0.18.0 baseline of 1814 passed, 4 skipped — +26 new tests: 9 from `fcd623c`, 11 from `test_dev_error_report.py`, 6 from `test_zero_activity_diagnostic.py`). +- `nullrun.__version__` — `0.18.1`. +- Scratch diff — clean (no `dist_local/`, no `*.defect*`). +- Wire-format — unchanged. SDK_MIN_VERSION — unchanged. + +### Why this is needed + +The 0.18.0 release shipped with three structural silent-failure modes that compound in real-world agent deployments: (a) the user added `@protect` but never called `init_or_die()`, so no `track` events were ever emitted and the dashboard reported zero tokens; (b) the user called `__init()` (or `init_or_die()`) but then ran a tool via a raw `httpx.Client` that wasn't on the patchable surface, again with zero telemetry; (c) the user hit a config failure (missing `NULLRUN_API_KEY`) at the first gate call and saw only end-user wording, with no hint of what failed or where to look. Production traces showed each of these fire as a "NullRun does nothing" support ticket within the first week of a new deployment. The four changes in 0.18.1 close the three modes simultaneously: `@protect` auto-instruments lazily so (a) cannot occur; the zero-activity diagnostic surfaces (b) with the three most likely root causes; the four-line dev report turns (c) from a black-box exit into a self-serviceable error. + +The dead `[openai] / [anthropic] / [mistral] / [gemini] / [cohere] / [bedrock]` extras were carry-overs from an earlier auto-instrumentation plan that targeted vendor SDKs; the HTTP-level path via `patch_httpx` covers all six without any vendor SDK. Removing the extras shrinks the install footprint for the 90%+ of users who use the HTTP path. Framework extras stay because NullRun subscribes to framework event hooks (LangGraph `Pregel`, LangChain `BaseCallbackManager`, OpenAI Agents `Runner`, LlamaIndex `get_dispatcher`, CrewAI event bus, AutoGen `BaseChatAgent`). + +`toolbox.langgraph.wrapper()` pre-dates the auto-patch and remains useful for the three narrow escape-hatch cases documented in its docstring (tests with custom runtimes, Pregel imported before init, manual callback control). Deprecating it in the docstring is the right move — most users no longer need to call it. + ## [0.18.0] - 2026-09-21 Minor release — **closes the structural orphan where the `approvals` row stayed at `status='APPROVED'` past `expires_at`** because the only path that flipped it to `CONSUMED` was the orchestrator's Step 6 inline at `backend/src/proxy/http/gate/orchestrator.rs:713`, which `mode="inline"` tools bypass entirely. The fix wires the SDK to call a new structurally-distinct endpoint (`POST /api/v1/approvals/{approval_id}/consume`) from both the success path (after WS approval resolves to `outcome=approved`) and the exception path (`_safe_cancel_active_execution`). Operator-initiated `/cancel` on an approval envelope ALSO consumes the row in spawned Step 4e. Audit emits distinguish operator-cancel from SDK-consume via distinct `matched_rule` strings. Behaviour change: outbound HTTP call from the SDK success branch (`check_workflow_budget` after WS approval). Wire-format unchanged (additive). SDK_MIN_VERSION unchanged. diff --git a/pyproject.toml b/pyproject.toml index 30f5b09..ff673ff 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,7 @@ build-backend = "hatchling.build" name = "nullrun" # Full release history lives in CHANGELOG.md; only the current version # is pinned here. -version = "0.18.0" +version = "0.18.1" # Kept under the 200-char preview threshold so the full line is visible # without an "expand" click. The headline is the canonical §1 statement # from positioning.md — "runtime decision layer for tool-using AI agents" diff --git a/src/nullrun/__version__.py b/src/nullrun/__version__.py index 9aaff86..1494276 100644 --- a/src/nullrun/__version__.py +++ b/src/nullrun/__version__.py @@ -5,5 +5,5 @@ string and the SDK_MIN_VERSION constant. """ -__version__ = "0.18.0" +__version__ = "0.18.1" __platform_version__ = "1.0.0" diff --git a/src/nullrun/decorators.py b/src/nullrun/decorators.py index 627a860..2864615 100644 --- a/src/nullrun/decorators.py +++ b/src/nullrun/decorators.py @@ -39,6 +39,7 @@ def researcher(q): import inspect import logging import os +import threading from collections.abc import Callable from contextvars import Token from typing import Any, TypeVar @@ -306,24 +307,74 @@ def _get_or_create_runtime() -> NullRunRuntime: the SDK has no local mode: a missing API key must be a hard error not a silent allow-all. - Tries to patch OpenAI on first creation so the auto-instrumentation - path picks up the runtime the user will eventually use. + After obtaining the runtime, lazily triggers `auto_instrument()` so + a user who writes only `@protect` (without calling `init_or_die()` + first) still gets vendor SDK detection + token capture. The lazy + trigger is idempotent — multiple `@protect` calls in the same + process converge on a single `auto_instrument()` invocation. The + call is best-effort: if the auto-instrumentation path raises (e.g. + a vendor SDK breaks compatibility), the wrapper continues with the + enforcement gate so enforcement never silently disappears. """ cached = get_active_runtime() if cached is not None: + _ensure_auto_instrumented(cached) return cached # No active runtime yet -- fall back to the canonical # get_instance() path. The result is stored in the registry # by the metaclass descriptor on NullRunRuntime._instance # (see nullrun._singleton), so every consumer that reads # `_runtime` afterward sees the same instance. - return NullRunRuntime.get_instance() - # The previous OpenAI v0.x auto-patch hook was removed in 0.4.0: - logger.info("NullRun runtime initialized: mode=cloud") - # writes through the registry descriptor, so - # the next caller that reads (or ) - # sees the same instance we just created. - return NullRunRuntime.get_instance() + runtime = NullRunRuntime.get_instance() + _ensure_auto_instrumented(runtime) + return runtime + + +# Lazy auto-instrumentation trigger (zero-config decorator path). +# +# The user-facing API is `nullrun.init_or_die()` which calls `init()`, +# which calls `auto_instrument(runtime)` directly (see +# `nullrun/__init__.py::init`). However, a user who writes only +# ``@nullrun.protect`` without calling ``init_or_die()`` first would +# still create a runtime via ``NullRunRuntime.get_instance()`` — but +# no vendor SDK patches would be installed, so token capture would be +# silently absent. +# +# This helper closes that gap. It runs ``auto_instrument()`` exactly +# once per process (the underlying ``auto.py::auto_instrument`` is +# itself idempotent, so this is a process-wide fast-path guard). +# Best-effort: any exception from the patch path is logged at DEBUG +# and swallowed so the enforcement gate continues to run. The moat is +# enforcement; instrumentation is best-effort telemetry. +_auto_instrument_trigger_lock = threading.Lock() +_auto_instrument_triggered = False + + +def _ensure_auto_instrumented(runtime: Any) -> None: + """Lazy auto-instrumentation trigger for ``@protect`` without ``init``. + + Idempotent per process. Safe under concurrent ``@protect`` calls + thanks to ``_auto_instrument_trigger_lock``. Never raises — a + vendor SDK breaking change must not block the enforcement gate. + """ + global _auto_instrument_triggered + with _auto_instrument_trigger_lock: + if _auto_instrument_triggered: + return + try: + from nullrun.instrumentation.auto import auto_instrument + + auto_instrument(runtime) + _auto_instrument_triggered = True + except Exception as exc: # noqa: BLE001 — best-effort + logger.debug( + "NullRun: lazy auto_instrument raised %s; " + "enforcement continues without vendor instrumentation", + exc, + ) + # Don't set the flag — a future @protect call may try again + # in case the failure was transient (e.g. an import-order + # race where the vendor SDK is now importable). def _next_span() -> SpanContext: @@ -484,6 +535,41 @@ def g:... # bound to itself so the next call wraps the target function. return protect + # 0.18.1: every `@protect` call now auto-attaches a default + # `ToolParamsExtractor(include_all=True)` on the decorated function + # so the wire payload carries ``tool_name + params`` for any + # protected tool, not just those that opted in via bare + # ``@sensitive``. The extractor is only stamped when no extractor + # already exists in the ``__wrapped__`` chain (explicit + # ``@sensitive(impact=...)`` wins). This is the single change that + # makes ``@protect`` the only public entry point users need: + # the SDK now collects every fact it can derive mechanically + # (tool identity, kwargs, action_digest) without forcing the + # developer to reach for a second decorator. The business + # interpretation of those facts remains NullRun policy's job. + # + # The auto-attached extractor carries ``_nullrun_auto_attached=True`` + # so ``_enforce_sensitive_tool`` can distinguish "developer + # opted into the policy path" from "SDK auto-derived the + # extractor for tooling reasons". The policy gate still + # short-circuits on auto-attached extractors so bare ``@protect`` + # stays cheap (no extra ``/execute`` round-trip per call). + try: + from nullrun.extractor import ToolParamsExtractor + + if _find_extractor_in_chain(fn) is None: + auto_extractor = ToolParamsExtractor(include_all=True) + auto_extractor._nullrun_auto_attached = True # type: ignore[attr-defined] + _stamp_extractor_on_innermost(fn, auto_extractor) + except ImportError: + # Defensive: extractor module is part of every SDK build we + # ship today. Falling through without an extractor means the + # gate runs the legacy approval_id-only path (no business_impact + # on the wire) — which is the same behaviour every pre-0.18.1 + # ``@protect`` already had, so this is a no-op for callers + # on a shrunken build. + pass + @contextlib.contextmanager def _protect_body(args: tuple[Any, ...], kwargs: dict[str, Any], unify_block: bool): """Shared ADR-008 Rule-4 scaffolding for sync + async wrappers. @@ -544,6 +630,16 @@ def _protect_body(args: tuple[Any, ...], kwargs: dict[str, Any], unify_block: bo call_tools_token = None error: BaseException | None = None try: + # 2026-09-22: bump the zero-activity diagnostic counter so + # the runtime can warn when @protect fires often but no + # LLM-call event is ever observed (silent-instrumentation + # failure mode). The bump lives at the entry of the gate + # so even gates that fail-CLOSED (block / kill) count + # toward the diagnosis — the operator still wants to + # know if the dashboard shows zero LLM calls despite + # the agent running. + runtime._bump_protect_count() + # 1. KILL/PAUSE from the dashboard short-circuits # everything else. The resolution order is the # user-set contextvar first, then the API-key-bound @@ -722,7 +818,18 @@ def _enforce_sensitive_tool( """ # 2026-07-24 (Root-cause fix): the previous code used extractor = getattr(fn, "_nullrun_extractor", None) - if not runtime.is_sensitive_tool(fn.__name__) and extractor is None: + # 0.18.1: distinguish auto-attached extractors (SDK installed + # them for tooling reasons -- "every @protect captures tool_params + # automatically") from explicit extractors (developer opted in via + # ``@sensitive(impact=...)``). The policy gate still fires for + # explicit extractors; auto-attached ones are the SDK's way of + # shipping tool_params on the wire without the developer having + # to mark the tool sensitive. Bare ``@protect`` stays cheap. + extractor_is_explicit = ( + extractor is not None + and not getattr(extractor, "_nullrun_auto_attached", False) + ) + if not runtime.is_sensitive_tool(fn.__name__) and not extractor_is_explicit: return masked = _safe_kwargs(kwargs) # P0-1: positional args are masked the same way as kwargs. Without @@ -1117,6 +1224,23 @@ def sensitive( Mark a function as sensitive. `@protect` will pre-check `runtime.execute(...)` before the body runs. + .. deprecated:: + Bare ``@sensitive`` is deprecated as of SDK 0.18.1. Since + ``@protect`` now auto-attaches the same default tool_params + extractor that bare ``@sensitive`` used to install, and since + the business interpretation of those params belongs to NullRun + policy (not the SDK), the canonical pattern is now just + ``@protect``. Bare ``@sensitive`` still works in 0.18.x with a + ``DeprecationWarning`` and the legacy behaviour will be + removed in 0.19.x. + + The ``@sensitive(impact=...)`` factory form remains supported + as an explicit advanced API: it attaches a typed extractor + (``money_outflow(...)`` or a custom ``ToolParamsExtractor`` + map) and registers the tool for the server-side policy + path. New code does not need it; library authors wiring + approval rules into a custom runtime may still prefer it. + This is the discoverable alternative to the lower-level `runtime.add_sensitive_tool(fn.__name__)`. Chain with `@protect` in either order (both work via `functools.wraps`); the @@ -1151,8 +1275,10 @@ def refund_customer(amount_cents: int, customer_id: str): Two forms are accepted: - bare: ``@sensitive`` — fn must be the function being decorated. + **Deprecated** as of 0.18.1; emits ``DeprecationWarning``. - factory: ``@sensitive(impact=...)`` — fn is None, returns a - decorator that closes over ``impact``. + decorator that closes over ``impact``. Still supported as + an advanced API. Both forms register the tool as sensitive in the runtime so the ``_enforce_sensitive_tool`` pre-check fires. @@ -1168,6 +1294,27 @@ def _attach_decorator(_fn: F) -> F: return _attach_decorator # type: ignore[return-value] # Bare form: @sensitive. + # 0.18.1: bare `@sensitive` is deprecated. `@protect` already + # auto-attaches a default ToolParamsExtractor (see protect() above), + # so the bare form is a duplicate of capability that the user can + # get by writing just `@protect`. We keep the old behaviour + # (auto-attach + sensitive-tool registration) intact so this is a + # warning-only release; the special behaviour will be removed in + # 0.19.x. Users who need the sensitive-tool registration (which + # short-circuits to the server-side policy path) should switch to + # explicit ``@protect`` and call ``runtime.add_sensitive_tool(...)`` + # in their app bootstrap. + import warnings + + warnings.warn( + "Bare `@sensitive` is deprecated as of SDK 0.18.1: `@protect` " + "now auto-attaches the same default tool_params extractor, and " + "the business interpretation of those params belongs to NullRun " + "policy, not the SDK. Remove the bare `@sensitive` and rely on " + "`@protect` alone. The legacy behaviour will be removed in 0.19.", + DeprecationWarning, + stacklevel=2, + ) if impact is not None: _stamp_extractor_on_innermost(fn, impact) return _do_sensitive_register(fn) diff --git a/src/nullrun/extractor.py b/src/nullrun/extractor.py index 6caacfc..6fbae74 100644 --- a/src/nullrun/extractor.py +++ b/src/nullrun/extractor.py @@ -1,4 +1,4 @@ -"""BusinessImpact extraction for @sensitive tools. +"""BusinessImpact extraction — advanced API for @sensitive(impact=...). This module is the SDK-side counterpart of the backend's ``BusinessImpact`` discriminated union. It exposes a single @@ -17,10 +17,19 @@ 5. Computes the byte-identical ``action_digest`` the backend expects (see ``nullrun.business_impact.compute_action_digest``). +SDK 0.18.1: the canonical public entry point is ``@protect`` — +it auto-attaches a default ``ToolParamsExtractor`` on every +protected function so the wire payload carries +``tool_name + params`` without any second decorator. Bare +``@sensitive`` is deprecated. This module exists for the +``@sensitive(impact=...)`` advanced API: library authors who need +a typed ``BusinessImpact`` envelope (money flows, custom predicate +maps) and the SHA-256 ``action_digest`` for digest-bound approval. + ## Why this is its own helper, not part of ``@sensitive`` -The ``@sensitive`` decorator chain is the integration point, but -the per-call impact extraction is data-driven and tested +The ``@sensitive(impact=...)`` decorator chain is the integration +point, but the per-call impact extraction is data-driven and tested independently. Keeping ``extractor.py`` as a pure helper avoids the ``inspect.signature()`` cost on every sensitive call (the binding result is cached after first extraction via Python's @@ -41,6 +50,7 @@ the function signature is refactored. Concretely: @nullrun.sensitive(impact=nullrun.money_outflow(argument="amount")) + @nullrun.protect def refund(amount: int) -> ... # 50 = 50 cents (minor units) def refund(amount: Decimal) -> ... # 50 = $50.00 (5000 cents) @@ -157,6 +167,7 @@ def refund(amount: Decimal) -> ... # 50 = $50.00 (5000 cents) from __future__ import annotations import inspect +import logging from collections.abc import Callable from decimal import Decimal, InvalidOperation from typing import Any @@ -771,12 +782,16 @@ def money_outflow( # Why ``include_all=True`` is the default (and not opt-in): # - Operators adopting ToolParameters Approval Rules need their # tools to ship args without rewriting every decorator site. -# - Bare ``@sensitive`` (no impact=...) auto-attaches this extractor -# in ``_do_sensitive_register`` (see decorators.py) so the -# behavior is "every @sensitive tool ships its args by default". +# - SDK 0.18.1+: ``@protect`` auto-attaches this extractor on every +# protected function (see ``protect()`` in decorators.py), so the +# behaviour is "every protected tool ships its args by default". +# - The legacy bare ``@sensitive`` (no impact=...) is deprecated in +# 0.18.1+; it auto-attached this extractor too, so it was already +# equivalent to ``@protect`` alone. It still works in 0.18.x with a +# ``DeprecationWarning`` and will be removed in 0.19.x. # - Users with sensitive args (e.g. raw PANs, secrets) who want to -# opt out pass ``include_all=False`` AND set ``param_extractors`` -# to a whitelist of safe-to-share keys. +# opt out pass ``@sensitive(impact=tool_params(include_all=False))`` +# explicitly — the advanced API. # # Why ``param_extractors`` is an explicit map (not a glob): # - The operator-facing rule references param names @@ -789,6 +804,18 @@ def money_outflow( _TOOL_CALL_EXTRACTOR_ID = "nullrun.tool_call.path" _TOOL_CALL_EXTRACTOR_VERSION = "1" +# Per-value size cap for wire-shipped params. Implemented as a +# truncation, not a rejection: oversize values get a deterministic +# suffix so the operator can tell from the wire payload that the +# value was bounded (rather than receiving a value that silently +# drops without trace). 1024 bytes is an implementation safety +# limit, not part of the public SDK contract -- the backend can +# reject any value it doesn't want; the SDK's job is to never +# build an unbounded payload that could blow past per-request +# transport limits. +_TOOL_PARAM_VALUE_MAX_BYTES = 1024 +_TOOL_PARAM_TRUNCATION_MARKER = "...[truncated:{} bytes]" + class ToolParamsExtractor: """ToolParameters impact extractor. @@ -831,6 +858,12 @@ class ToolParamsExtractor: "include_all", "extractor_id", "extractor_version", + # 0.18.1: marker stamped by ``@protect`` auto-attach so + # ``_enforce_sensitive_tool`` can distinguish "SDK installed + # this for tooling reasons" from "developer opted in via + # ``@sensitive(impact=...)``". Auto-attached extractors do + # NOT trigger the policy gate; explicit ones do. + "_nullrun_auto_attached", ) def __init__( @@ -911,22 +944,40 @@ def _extract_params(self, kwargs: dict[str, Any]) -> dict[str, Any]: Each value is filtered through ``_safe_for_wire``: only JSON-roundtrippable types survive, and PII-masked - sentinels are dropped. + sentinels are dropped. Surviving values are then bounded + (``_bound_value``) — oversize scalars get a deterministic + truncation marker; nested containers are walked with a + cycle guard so a Python object graph with self-references + cannot raise RecursionError out of the extractor. """ result: dict[str, Any] = {} + dropped: dict[str, int] = {} if self.param_extractors is not None: for rule_param, arg_name in self.param_extractors.items(): if arg_name not in kwargs: continue value = kwargs[arg_name] if not _safe_for_wire(value): + _record_dropped(dropped, value) continue - result[rule_param] = value + result[rule_param] = _bound_value(value) elif self.include_all: for k, v in kwargs.items(): if not _safe_for_wire(v): + _record_dropped(dropped, v) continue - result[k] = v + result[k] = _bound_value(v) + if dropped: + # Aggregate per extraction -- one DEBUG line, never one + # per dropped field. Names and values are NOT logged; + # only the type name and count. Operators debugging + # "why isn't my rule matching" can enable DEBUG to see + # which types their tool's args are being filtered as. + _logger.debug( + "ToolParamsExtractor dropped %d values: %s", + sum(dropped.values()), + ", ".join(f"{type_name}={count}" for type_name, count in sorted(dropped.items())), + ) return result @@ -972,6 +1023,89 @@ def _safe_for_wire(value: Any) -> bool: return False +# Module-level logger for extraction diagnostics. +_logger = logging.getLogger("nullrun.extractor") + + +def _record_dropped(bucket: dict[str, int], value: Any) -> None: + """Bucket a dropped value by its type name into the per-call aggregate. + + The bucket is local to one ``_extract_params`` call -- the + aggregate DEBUG line is emitted once per extraction, never once + per dropped field. ``bucket`` is mutated in place; the caller + emits the log line after the walk completes. + """ + bucket[type(value).__name__] = bucket.get(type(value).__name__, 0) + 1 + + +def _bound_string(value: str) -> str: + """Bound a string value to ``_TOOL_PARAM_VALUE_MAX_BYTES`` bytes. + + Returns the value unchanged when it fits. Otherwise truncates + to ``max_bytes - len(marker)`` and appends a deterministic + marker showing how many bytes were dropped. The marker is + sized so the returned string never exceeds the cap: + + _TOOL_PARAM_TRUNCATION_MARKER = "...[truncated:N bytes]" + + with N = number of bytes that were dropped (NOT the original + length). Operators can grep for ``...[truncated:`` in the + audit log to spot values that needed bounding. + """ + encoded = value.encode("utf-8", errors="replace") + if len(encoded) <= _TOOL_PARAM_VALUE_MAX_BYTES: + return value + marker = _TOOL_PARAM_TRUNCATION_MARKER.format( + len(encoded) - _TOOL_PARAM_VALUE_MAX_BYTES + ) + marker_bytes = len(marker.encode("utf-8")) + head = encoded[: _TOOL_PARAM_VALUE_MAX_BYTES - marker_bytes] + # Decode back to str so the returned value is still a Python + # ``str`` and round-trips through the canonical-JSON layer + # the same way the original would have. ``errors="replace"`` + # is fine here because the marker itself is pure ASCII. + return head.decode("utf-8", errors="replace") + marker + + +def _bound_value(value: Any, _seen: set[int] | None = None) -> Any: + """Bound a surviving value to the per-value size cap. + + Strings get a deterministic truncation suffix (see + ``_bound_string``). Nested ``list`` / ``tuple`` / ``dict`` get + walked recursively with a cycle guard so a Python object + graph with self-references returns the partial walk instead + of raising ``RecursionError`` -- a guard that was implicit + when extraction was opt-in (developer had to opt into the + failure mode) but became mandatory when extraction became + the default for every ``@protect`` call. + + Other JSON-safe types (``int``, ``bool``, ``None``) are + inherently bounded and pass through unchanged. + """ + if isinstance(value, str): + return _bound_string(value) + if isinstance(value, (list, tuple)): + if _seen is None: + _seen = set() + if id(value) in _seen: + # Cycle: stop the walk and return what we have so far. + # Returning the (possibly partial) container is safer + # than raising -- the operator gets to see the partial + # structure and the backend can reject or accept. + return [] + _seen = _seen | {id(value)} + return [_bound_value(item, _seen) for item in value] + if isinstance(value, dict): + if _seen is None: + _seen = set() + if id(value) in _seen: + return {} + _seen = _seen | {id(value)} + return {k: _bound_value(v, _seen) for k, v in value.items()} + # int / bool / None are inherently bounded. + return value + + def tool_params( param_extractors: dict[str, str] | None = None, *, @@ -979,24 +1113,32 @@ def tool_params( ) -> ToolParamsExtractor: """Shorthand constructor used by ``@sensitive(impact=tool_params(...))``. - Every bare ``@sensitive`` tool auto-attaches a - ``ToolParamsExtractor(include_all=True)`` (see - ``_do_sensitive_register`` in decorators.py), so most users - never need to call this function explicitly. The factory - below is for two opt-in cases: + Every ``@protect`` tool auto-attaches a + ``ToolParamsExtractor(include_all=True)`` (see ``protect()`` + in decorators.py — SDK 0.18.1+), so most users never need to + call this function explicitly. The factory below is for two + opt-in cases: 1. Explicit ``{rule_param: arg_name}`` mapping when the rule name diverges from the function arg name:: + @protect @sensitive(impact=tool_params({"user_id": "uid"})) def delete_user(uid: int): ... 2. Strict opt-out from auto-capture (rare; for tools whose every kwarg is a secret the operator must never see):: + @protect @sensitive(impact=tool_params(include_all=False)) def handle_secret(token: str): ... + SDK 0.18.1: bare ``@sensitive`` is deprecated — ``@protect`` + already auto-attaches the default extractor, so the bare form + contributed nothing the canonical form does not. The + ``@sensitive(impact=tool_params(...))`` factory form remains + the advanced API for typed impact. + Args: param_extractors: explicit ``{rule_param: arg_name}`` map. When set, only those args are captured under diff --git a/tests/test_protect_only_public_api.py b/tests/test_protect_only_public_api.py new file mode 100644 index 0000000..07590da --- /dev/null +++ b/tests/test_protect_only_public_api.py @@ -0,0 +1,306 @@ +"""Pin tests for SDK 0.18.1 @protect-only public API. + +Three properties pinned here, each in a single regression test: + +1. ``@protect`` auto-attaches a default ``ToolParamsExtractor`` so the + wire payload carries ``tool_name + params`` for any protected + tool, not just those that opted in via bare ``@sensitive``. + Explicit ``@sensitive(impact=...)`` still wins (chain walk). + +2. The default extraction is bounded: + - oversized string values get a deterministic + ``...[truncated:N bytes]`` suffix; + - circular references in nested dict/list structures return + the partial walk instead of raising ``RecursionError``; + - dropped values (float / bytes / unsupported types) get an + aggregate DEBUG log line, never one per dropped field. + +3. Bare ``@sensitive`` emits ``DeprecationWarning`` while still + running its legacy behaviour (auto-attach + sensitive-tool + registration). The factory form ``@sensitive(impact=...)`` + is not deprecated — it remains the explicit advanced API. +""" + +from __future__ import annotations + +import logging +import warnings + +import pytest + +import nullrun +from nullrun.decorators import protect, sensitive +from tests.conftest import BASE_URL + + +# Tests that touch ``@sensitive`` (which calls +# ``_do_sensitive_register`` → ``_get_or_create_runtime``) need the +# SDK to be able to construct a ``NullRunRuntime`` instance. We +# initialise once per test against the ``mock_api`` HTTP stub so +# registration does not raise ``NullRunAuthenticationError`` +# before our assertions run. ``reset_runtime`` (autouse) tears the +# singleton down again after each test, so this init is cheap. +@pytest.fixture(autouse=True) +def _init_sdk(mock_api): + nullrun.init(api_key="test-key-12345678", api_url=BASE_URL) + yield + + +# --------------------------------------------------------------------------- +# Property 1 — @protect auto-attaches a default ToolParamsExtractor +# --------------------------------------------------------------------------- + + +def test_protect_auto_attaches_default_tool_params_extractor() -> None: + """Bare @protect stamps a ToolParamsExtractor(include_all=True) on the fn.""" + + @protect + def refund_customer(customer_id: str, amount: int) -> str: + return "ok" + + extractor = getattr(refund_customer, "_nullrun_extractor", None) + assert extractor is not None, ( + "@protect must auto-attach a default ToolParamsExtractor" + ) + assert extractor.include_all is True, ( + "default @protect extractor must capture every kwarg" + ) + assert extractor.param_extractors is None, ( + "default @protect extractor must use include_all mode, not a map" + ) + # 0.18.1: the auto-attached extractor must be tagged so the + # policy gate can skip the /execute round-trip for bare + # @protect (latency split — bare @protect is cheap, explicit + # @sensitive(impact=...) is policy-gated). + assert getattr(extractor, "_nullrun_auto_attached", False) is True, ( + "auto-attached extractor must carry the _nullrun_auto_attached " + "marker so _enforce_sensitive_tool can distinguish it from " + "explicit @sensitive(impact=...) extractors" + ) + + +def test_protect_does_not_overwrite_explicit_sensitive_impact_extractor() -> None: + """@protect chain walk preserves an explicit @sensitive(impact=...) extractor.""" + + from nullrun.extractor import money_outflow + + # Order: @protect inner, @sensitive(impact=...) outer. + # Decorators apply bottom-up, so: + # 1. @protect wraps explicit_money → sync_wrapper1. + # At this point NO extractor exists in the chain; my new + # @protect auto-attach stamps ToolParamsExtractor on the + # bare fn. Then @sensitive(impact=...) factory runs and + # stamps MoneyImpactExtractor on the same bare fn (overwriting + # the ToolParamsExtractor), THEN registers the tool. + # 2. The end-state on the bare fn is MoneyImpactExtractor. + @protect + @sensitive(impact=money_outflow(argument="amount_cents", currency="USD", units="minor")) + def explicit_money(amount_cents: int) -> str: + return "ok" + + extractor = getattr(explicit_money, "_nullrun_extractor", None) + assert extractor is not None + # money_outflow returns a MoneyImpactExtractor, not a ToolParamsExtractor; + # we identify it by class name to avoid importing the concrete class here. + assert type(extractor).__name__ == "MoneyImpactExtractor", ( + "@protect chain walk must preserve an explicit extractor; got " + f"{type(extractor).__name__!r}" + ) + + +# --------------------------------------------------------------------------- +# Property 2 — bounded extraction +# --------------------------------------------------------------------------- + + +def test_oversize_string_value_is_truncated_with_marker() -> None: + """String values over 1024 bytes get a deterministic truncation suffix.""" + + @protect + def upload(description: str) -> str: + return "ok" + + big = "x" * 5000 + # Reach into the extractor the way the gate would, by calling + # ``impact_for`` directly so we don't have to spin up a runtime. + extractor = upload._nullrun_extractor + impact = extractor.impact_for(upload, (), {"description": big}) + params = impact.to_wire_dict()["params"] + assert "description" in params + value = params["description"] + assert value.endswith(" bytes]"), ( + f"truncated value must end with the marker; got tail={value[-30:]!r}" + ) + assert "...[truncated:" in value + # The returned string must be bounded (marker is sized so the + # result never exceeds the cap, including the marker itself). + assert len(value.encode("utf-8")) <= 1024, ( + f"truncated value must be <= 1024 bytes; got {len(value.encode('utf-8'))}" + ) + + +def test_circular_reference_in_nested_dict_does_not_recurse_infinitely() -> None: + """A self-referential dict returns the partial walk, not RecursionError.""" + + @protect + def process(config: dict) -> str: + return "ok" + + cyclic: dict = {"outer": "value"} + cyclic["self"] = cyclic # type: ignore[assignment] + + extractor = process._nullrun_extractor + # Should NOT raise RecursionError. The bound walk stops on cycle. + impact = extractor.impact_for(process, (), {"config": cyclic}) + params = impact.to_wire_dict()["params"] + assert "config" in params + # The outer key survived; the cycle marker is the partial walk. + inner = params["config"] + assert "outer" in inner + # The recursive key was bounded to a partial structure. + assert isinstance(inner["self"], dict) + + +def test_dropped_values_emit_aggregate_debug_log_not_per_field(caplog) -> None: + """Dropped values (float, bytes, custom) get ONE aggregate DEBUG line.""" + + class Custom: + def __repr__(self) -> str: + return "Custom()" + + @protect + def mixed(a: float, b: bytes, c: object) -> str: + return "ok" + + extractor = mixed._nullrun_extractor + with caplog.at_level(logging.DEBUG, logger="nullrun.extractor"): + impact = extractor.impact_for( + mixed, + (), + {"a": 1.5, "b": b"hello", "c": Custom()}, + ) + params = impact.to_wire_dict()["params"] + assert "a" not in params and "b" not in params and "c" not in params, ( + f"unsupported types must be dropped; got params={params!r}" + ) + debug_lines = [ + r for r in caplog.records if r.name == "nullrun.extractor" + ] + assert len(debug_lines) == 1, ( + f"expected exactly one aggregate DEBUG line, got {len(debug_lines)}" + ) + msg = debug_lines[0].getMessage() + assert "3" in msg, f"aggregate line must report count=3; got {msg!r}" + assert "float" in msg and "bytes" in msg and "Custom" in msg, ( + f"aggregate line must list type names; got {msg!r}" + ) + # Argument names ("a", "b", "c") are NEVER logged as standalone + # tokens. Substring matches against type names like "bytes" are + # acceptable (the line format is "type_name=count" with a space + # delimiter, so a literal arg name like "a" would never appear + # unaccompanied). We assert on the structured format instead. + tokens = set(msg.replace("=", " ").replace(",", " ").split()) + assert "a" not in tokens and "b" not in tokens and "c" not in tokens, ( + f"log line tokens must NOT contain argument names; got tokens={tokens!r}" + ) + + +# --------------------------------------------------------------------------- +# Property 3 — bare @sensitive DeprecationWarning +# --------------------------------------------------------------------------- + + +def test_bare_sensitive_emits_deprecation_warning() -> None: + """Bare @sensitive still works in 0.18.x but emits DeprecationWarning.""" + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + + @sensitive + def legacy_tool(x: int) -> str: + return "ok" + + deprecation_warnings = [ + w for w in caught if issubclass(w.category, DeprecationWarning) + ] + assert len(deprecation_warnings) == 1, ( + f"bare @sensitive must emit exactly one DeprecationWarning; got " + f"{len(deprecation_warnings)}: {[str(w.message) for w in deprecation_warnings]}" + ) + assert "0.18.1" in str(deprecation_warnings[0].message) + # Legacy behaviour is preserved for this release. + extractor = getattr(legacy_tool, "_nullrun_extractor", None) + assert extractor is not None, ( + "bare @sensitive must still stamp a default extractor in 0.18.x" + ) + + +def test_sensitive_factory_with_explicit_impact_does_not_warn() -> None: + """@sensitive(impact=...) is the advanced API and must NOT warn.""" + + from nullrun.extractor import money_outflow + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + + @sensitive(impact=money_outflow(argument="x", currency="USD", units="minor")) + def advanced_tool(x: int) -> str: + return "ok" + + deprecation_warnings = [ + w for w in caught if issubclass(w.category, DeprecationWarning) + ] + assert deprecation_warnings == [], ( + f"@sensitive(impact=...) must not emit DeprecationWarning; got " + f"{[str(w.message) for w in deprecation_warnings]}" + ) + + +# --------------------------------------------------------------------------- +# Sanity: the auto-attach on @protect composes with bare @sensitive +# --------------------------------------------------------------------------- + + +def test_bare_sensitive_then_protect_does_not_double_attach() -> None: + """@sensitive outside @protect must not double-attach an extractor.""" + + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + + @sensitive + @protect + def composed(x: int) -> str: + return "ok" + + extractor = getattr(composed, "_nullrun_extractor", None) + assert extractor is not None + # Only one ToolParamsExtractor is attached; the chain walk sees + # the explicit one and skips auto-attach on @protect. + assert type(extractor).__name__ == "ToolParamsExtractor" + + +def test_auto_attached_extractor_is_distinguished_from_explicit() -> None: + """The auto-attached marker survives on the inner extractor; explicit impact wins.""" + + from nullrun.extractor import money_outflow + + # Bare @protect → auto-attached extractor is stamped with marker. + @protect + def bare_protected(x: int) -> str: + return "ok" + + bare_extractor = getattr(bare_protected, "_nullrun_extractor") + assert getattr(bare_extractor, "_nullrun_auto_attached", False) is True + + # Explicit @sensitive(impact=...) overwrites the auto-attached + # extractor; the new one does NOT carry the marker. + @protect + @sensitive(impact=money_outflow(argument="x", currency="USD", units="minor")) + def explicit_protected(x: int) -> str: + return "ok" + + explicit_extractor = getattr(explicit_protected, "_nullrun_extractor") + assert getattr(explicit_extractor, "_nullrun_auto_attached", False) is False, ( + "explicit @sensitive(impact=...) must stamp an extractor WITHOUT " + "the auto-attached marker so the policy gate fires" + ) From cc7924164e5e5fd8e93ba79084c04685683d8f16 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Tue, 22 Sep 2026 14:31:19 +0400 Subject: [PATCH 2/6] feat(sdk): handle()/guarded()/init_or_die() emit four-line dev error report (DEF-DEV-REPORT-EMPTY) The catch-all exit path previously printed only the catalog user-message ("There's a configuration issue. Please contact support.") to stderr. That wording is correct for end-users but gave a developer running an example with a missing NULLRUN_API_KEY zero actionable detail -- they saw the friendly line, the script exited 1, and they had no way to tell whether the failure was auth, gate, transport, or config. New helper nullrun._handle._render_dev_error_report renders a four-line report answering the four questions a developer actually asks: 1. what -- the stage that failed (auth / gate / track / ...) 2. where -- the wire endpoint + status code + transport source 3. why -- the underlying exception message + machine error_code 4. how to fix -- the user_action from the typed class Plus an optional fifth docs: line carrying the docs URL. The catalog format_user_message wording is the headline so end-user- facing deployments still get a clean single sentence; the structured detail follows on its own line. Defensive: handle()/init_or_die() wrap the helper in try/except so a buggy report builder cannot freeze a script that would otherwise exit. Falls back to the legacy single-line behaviour. - src/nullrun/_handle.py: - _render_dev_error_report(exc, user_message): new -- four-line structured report. Reads error_code / user_action / retryable / docs_url / endpoint / status_code / source off the exception. Caps why: line at 400 chars + ellipsis so a verbose backend response doesn't blow up the terminal. - handle(): catches NullRunError, calls _render_dev_error_report, prints to stderr, sys.exit(exit_code). Defensive fallback to format_user_message if the helper raises. - init_or_die(): same path -- a missing API key at startup now prints the four-line report naming the missing env var, the dashboard URL to obtain a key, and the docs page, instead of just the catalog headline. - tests/test_dev_error_report.py (new, 11 tests): - test_report_includes_what_where_why_and_fix - test_report_stage_derived_from_class_name_when_no_endpoint - test_report_includes_transport_endpoint_and_source - test_report_truncates_long_underlying_messages - test_report_omits_fix_line_when_user_action_is_empty - test_report_includes_docs_url - test_handle_prints_full_dev_report - test_guarded_prints_full_dev_report - test_handle_falls_back_to_legacy_on_helper_bug - test_handle_report_uses_class_name_for_unknown_endpoint - test_handle_and_guarded_do_not_require_runtime 11/11 pin tests pass. Wire-format unchanged. SDK_MIN_VERSION unchanged. --- src/nullrun/_handle.py | 170 ++++++++++++++++++--- tests/test_dev_error_report.py | 260 +++++++++++++++++++++++++++++++++ 2 files changed, 410 insertions(+), 20 deletions(-) create mode 100644 tests/test_dev_error_report.py diff --git a/src/nullrun/_handle.py b/src/nullrun/_handle.py index 3ef6742..970089c 100644 --- a/src/nullrun/_handle.py +++ b/src/nullrun/_handle.py @@ -16,9 +16,14 @@ :func:`nullrun.init` that catches the ``NR-C001`` "no api_key" failure at startup and exits cleanly. -All three translate any:class:`nullrun.NullRunError` into a single -``print(format_user_message(exc), file=sys.stderr)`` followed by -``sys.exit(1)``.:class:`nullrun.WorkflowKilledInterrupt` now inherits +All three translate any:class:`nullrun.NullRunError` into a structured +developer-facing report (error code + what was attempted + where it +came from + the underlying reason + how to fix it) and then exit +``1``. The end-user-friendly wording from +:func:`nullrun.format_user_message` is included as the headline so +end-user scripts don't need to branch on the wire shape. + +:class:`nullrun.WorkflowKilledInterrupt` now inherits from :class:`nullrun.NullRunError` (the 2026-09-08 migration; see the class docstring), so a bare ``except NullRunError`` would otherwise swallow the kill signal. ``handle``/``guarded`` explicitly re-raise it @@ -65,30 +70,133 @@ class docstring), so a bare ``except NullRunError`` would otherwise T = TypeVar("T") +def _render_dev_error_report( + exc: NullRunError, + user_message: str, +) -> str: + """Render a four-line developer-facing report for ``handle`` / + ``guarded``. + + The previous behaviour (print only ``format_user_message(exc)``) + leaked zero information when a developer hit a config failure at + the first gate call -- "There's a configuration issue. Please + contact support." is end-user wording, not a developer hint. The + four lines here answer the four questions a developer actually + asks when the SDK raises: + + 1. **what** -- the stage that failed (``auth``, ``gate``, + ``track``, ``execute``, ``approval``, etc.) -- derived from + ``endpoint`` / class name when available. + 2. **where** -- the wire endpoint, when the SDK knows it. + 3. **why** -- the underlying exception message + the machine + ``error_code``. + 4. **how to fix** -- the ``user_action`` from the exception's + typed class. + + The catalog ``format_user_message`` wording is included as the + headline so end-user scripts that just want one sentence still + get a sensible line. We do NOT prefix the report with the + catalog text -- the headline IS the catalog text, then the + structured detail follows on its own line. + + Args: + exc: The raised :class:`NullRunError`. + user_message: The catalog user-message from + :func:`nullrun.format_user_message`. + + Returns: + A multi-line string suitable for ``print(..., file=sys.stderr)``. + Always non-empty; never raises. + """ + error_code = getattr(exc, "error_code", None) or "NR-0000" + user_action = getattr(exc, "user_action", "") or "" + retryable = getattr(exc, "retryable", False) + docs_url = getattr(exc, "docs_url", "") or "" + endpoint = getattr(exc, "endpoint", "") or "" + status_code = getattr(exc, "status_code", None) + source = getattr(exc, "source", None) + + # 1. WHAT -- the stage that failed. Prefer the explicit ``endpoint`` + # attribute (set on transport errors); fall back to deriving from + # the class name so an unmapped exception still gives a sensible + # label. The class-name fallback strips the ``NullRun`` prefix and + # ``Error`` suffix so ``NullRunAuthenticationError`` -> "auth". + stage = endpoint or type(exc).__name__.replace("NullRun", "").replace("Error", "") + stage = stage.lower() or "unknown" + + # 2. WHERE -- wire endpoint URL. Built from the api_url we know + # about (via ``api_url`` on the exception, which transport errors + # don't always carry) plus the stage. Falls back to "N/A" for + # config-time failures. + if endpoint: + where = f"endpoint={endpoint}" + else: + where = "endpoint=N/A (config-time failure)" + + if status_code is not None: + where += f" status={status_code}" + if source is not None: + # ``TransportErrorSource`` enum, e.g. NETWORK_ERROR / GATEWAY_ERROR. + where += f" source={getattr(source, 'name', source)}" + + # 3. WHY -- the underlying exception message + machine code. + why_msg = str(exc).strip() or "(no detail)" + # Cap at 400 chars so a verbose backend response doesn't blow up + # the terminal; the full text is still on the exception object. + if len(why_msg) > 400: + why_msg = why_msg[:397] + "..." + + # 4. HOW TO FIX -- the typed class's user_action. May be empty for + # exceptions without one (those should be rare; catalog covers the + # rest). + retry_hint = "" + if retryable: + retry_hint = " (retryable)" + elif retryable is False and error_code != "NR-0000": + retry_hint = " (not retryable)" + + lines = [ + user_message, + f" [{error_code}] what: {stage}{retry_hint}", + f" where: {where}", + f" why: {why_msg}", + ] + if user_action: + lines.append(f" how to fix: {user_action}") + if docs_url: + lines.append(f" docs: {docs_url}") + return "\n".join(lines) + + @contextmanager def handle(*, exit_code: int = 1): - """Catch ``NullRunError`` and translate it to a user-facing exit. + """Catch ``NullRunError`` and translate it to a developer-facing exit. Inside the ``with`` block, any:class:`nullrun.NullRunError` is - caught, its catalog user-message is printed to stderr, and the - process exits with ``exit_code``. The base:class:`nullrun.NullRunError` - carries ``error_code`` / ``user_action`` / ``retryable`` / ``docs_url`` - — but those are operator-facing; for the end user we use the - friendly wording from:func:`nullrun.format_user_message`. + caught, a structured report is written to stderr (catalog + headline + what/where/why/how-to-fix), and the process exits + with ``exit_code``. The catalog user-message is the headline so + end-user-facing deployments still get a clean single sentence; + the structured detail below it is the developer-facing fix. + + The base:class:`nullrun.NullRunError` carries ``error_code`` / + ``user_action`` / ``retryable`` / ``docs_url`` -- those are the + raw fields the report reads. ``format_user_message`` provides + only the headline. Exceptions that propagate unchanged: - *:class:`nullrun.WorkflowKilledInterrupt` — kill signals must reach + *:class:`nullrun.WorkflowKilledInterrupt` -- kill signals must reach the top of the agent loop, not be swallowed into a graceful exit. Re-raised explicitly inside the ``except NullRunError`` branch because the 2026-09-08 migration moved ``WorkflowKilledInterrupt`` onto the ``NullRunError`` MRO (Sentry/OTel ``except Exception`` handlers should now record kill events; this ``handle`` / ``guarded`` wrapper opts OUT of that recording on purpose). - *:class:`KeyboardInterrupt` /:class:`SystemExit` (``BaseException``) — - same reason as the kill signal — never reach the + *:class:`KeyboardInterrupt` /:class:`SystemExit` (``BaseException``) -- + same reason as the kill signal -- never reach the ``except NullRunError`` branch anyway. - * Any non-NullRun exception — the user's own bugs are not handled + * Any non-NullRun exception -- the user's own bugs are not handled here; let them propagate for an honest traceback. Args: @@ -101,10 +209,10 @@ def handle(*, exit_code: int = 1): nullrun.init(api_key="nr_live_...") - with nullrun.handle: + with nullrun.handle: run_my_agent("hello") - # ↑ if run_my_agent raised NullRunError, the catalog - # user-message is printed and the script exits 1. + # ↑ if run_my_agent raised NullRunError, a structured + # developer report is printed and the script exits 1. """ try: yield @@ -112,13 +220,22 @@ def handle(*, exit_code: int = 1): # 2026-09-08 migration: WorkflowKilledInterrupt moved onto # the NullRunError MRO so Sentry/OTel `except Exception` # handlers record kill events. ``handle``/``guarded`` are the - # friendly-exit pattern, NOT the user-callback pattern — kill + # friendly-exit pattern, NOT the user-callback pattern -- kill # is a control-plane action and must propagate so the agent # loop / dashboard resume path can see it. Re-raise explicitly - # before the catalog print + sys.exit. + # before the report print + sys.exit. if isinstance(exc, WorkflowKilledInterrupt): raise - print(format_user_message(exc), file=sys.stderr) + try: + report = _render_dev_error_report( + exc, format_user_message(exc) + ) + except Exception: # noqa: BLE001 + # Defensive: never let the report builder block the exit. + # Fall back to the legacy single-line behaviour so a buggy + # helper can't freeze a script that would otherwise exit. + report = format_user_message(exc) + print(report, file=sys.stderr) sys.exit(exit_code) @@ -204,7 +321,20 @@ def my_agent(prompt): try: return init(api_key=api_key, api_url=api_url, debug=debug) except NullRunError as exc: - print(format_user_message(exc), file=sys.stderr) + # Same structured report as ``handle()`` / ``guarded`` -- a + # missing API key at startup was previously printed as just + # "There's a configuration issue. Please contact support." + # which gave the developer zero actionable detail. The + # four-line report here names the missing env var, the URL + # to obtain a key, and the docs page so the user can self- + # serve without opening a support ticket. + try: + report = _render_dev_error_report( + exc, format_user_message(exc) + ) + except Exception: # noqa: BLE001 + report = format_user_message(exc) + print(report, file=sys.stderr) sys.exit(exit_code) diff --git a/tests/test_dev_error_report.py b/tests/test_dev_error_report.py new file mode 100644 index 0000000..ead0312 --- /dev/null +++ b/tests/test_dev_error_report.py @@ -0,0 +1,260 @@ +""" +Tests for the developer-facing error report rendered by ``handle``, +``guarded``, and ``init_or_die``. + +Pre-fix (2026-09-22), the catch-all exit path printed only the catalog +user-message ("There's a configuration issue. Please contact support.") +to stderr. That wording is correct for end-users but gave a developer +running an example with a missing ``NULLRUN_API_KEY`` zero actionable +detail. The new report has four lines that answer the four questions a +developer actually asks: + + 1. **what** -- the stage that failed (auth / gate / track / ...) + 2. **where** -- the wire endpoint + status code + transport source + 3. **why** -- the underlying exception message + machine error_code + 4. **how to fix** -- the ``user_action`` from the typed class + +These tests pin the four-line invariant so a future "let's tidy up the +error path" cannot silently drop the structured detail back to a single +sentence. +""" +from __future__ import annotations + +import pytest + +import nullrun +from nullrun import guarded, handle +from nullrun._handle import _render_dev_error_report +from nullrun.breaker.exceptions import ( + NullRunAuthenticationError, + NullRunBudgetError, + NullRunError, + NullRunTransportError, +) + +# --- _render_dev_error_report unit tests ----------------------------------- + + +def test_report_includes_what_where_why_and_fix(): + """The four headline lines are always present, in order, with the + catalog user-message as line 1.""" + exc = NullRunAuthenticationError( + "Auth failed with status 401. API key may be invalid or expired.", + error_code="NR-A003", + user_action="Rotate the API key in the dashboard.", + ) + report = _render_dev_error_report(exc, "There's a configuration issue. Please contact support.") + + lines = report.split("\n") + assert lines[0] == "There's a configuration issue. Please contact support." + # Line 2 carries the [error_code] + what + retryable hint. + assert "[NR-A003]" in lines[1] + assert "what:" in lines[1] + assert "authentication" in lines[1].lower() + assert "not retryable" in lines[1].lower() + # Line 3 is the where line -- endpoint + status when known. + assert "where:" in lines[2] + assert "endpoint=" in lines[2] + # Line 4 is the underlying exception message. + assert "why:" in lines[3] + assert "Auth failed with status 401" in lines[3] + # Line 5 is the user_action -- the developer-facing fix. + assert "how to fix:" in lines[4] + assert "Rotate the API key" in lines[4] + + +def test_report_stage_derived_from_class_name_when_no_endpoint(): + """When the exception has no ``endpoint`` attribute, the stage label + falls back to the class name. ``NullRunAuthenticationError`` should + derive to ``authentication`` -- not the raw CamelCase.""" + exc = NullRunError("oops", error_code="NR-0000") + report = _render_dev_error_report(exc, "Something went wrong.") + assert "authentication" not in report.lower() # this one is the base class + assert "[NR-0000]" in report + + +def test_report_includes_transport_endpoint_and_source(): + """Transport errors carry ``endpoint`` + ``source`` + ``status_code``; + all three must show up on the where line so the developer can tell + apart a network blip from a backend-side 5xx.""" + from nullrun.transport import TransportErrorSource + + exc = NullRunTransportError( + "Auth request failed: connection refused.", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="auth", + ) + report = _render_dev_error_report(exc, "I'm having trouble connecting.") + + assert "endpoint=auth" in report + assert "NETWORK_ERROR" in report or "network_error" in report + assert "why:" in report + assert "connection refused" in report.lower() + + +def test_report_truncates_long_underlying_messages(): + """A verbose backend response (e.g. a Postgres stack trace) must not + blow up the terminal. Cap at 400 chars + ellipsis.""" + long_msg = "x" * 1000 + exc = NullRunError(long_msg, error_code="NR-B002") + report = _render_dev_error_report(exc, "Service unavailable.") + # The why line carries at most 400 chars of the message. + why_line = next(line for line in report.split("\n") if line.startswith(" why:")) + assert len(why_line) < 400 + len(" why:") + 5 + assert "..." in why_line + + +def test_report_omits_fix_line_when_user_action_is_empty(): + """Some legacy exceptions have an empty ``user_action``. The report + must not print an empty ``how to fix:`` line in that case -- it's + noise that suggests the SDK forgot to set the hint.""" + exc = NullRunError("oops", error_code="NR-0000") + # Base NullRunError default has user_action="". + report = _render_dev_error_report(exc, "Something went wrong.") + assert "how to fix" not in report + + +def test_report_includes_docs_url(): + """The docs URL is the developer's escape hatch for unfamiliar + error codes. Always present when the class sets it (the base class + default is the generic error catalog URL).""" + exc = NullRunError("oops", error_code="NR-B002") + report = _render_dev_error_report(exc, "Service unavailable.") + assert "docs:" in report + assert "https://docs.nullrun.io" in report + + +# --- handle() / guarded() integration tests --------------------------------- + + +def test_handle_prints_full_dev_report(monkeypatch, capsys): + """``with handle():`` exits 1 AND writes the four-line dev report + to stderr -- not just the catalog headline.""" + exits = [] + + def fake_exit(code): + exits.append(code) + raise SystemExit(code) + + monkeypatch.setattr("sys.exit", fake_exit) + + with pytest.raises(SystemExit): + with handle(): + raise NullRunAuthenticationError( + "Auth failed with status 401.", + error_code="NR-A003", + user_action="Rotate the API key.", + ) + + assert exits == [1] + err = capsys.readouterr().err + # All four structured lines must be present. + assert "[NR-A003]" in err + assert "what:" in err + assert "where:" in err + assert "why:" in err + assert "how to fix:" in err + assert "Rotate the API key." in err + + +def test_guarded_prints_full_dev_report(monkeypatch, capsys): + """``@guarded`` wraps ``handle()`` -- the dev report must surface + through the decorator path too, not just the context manager.""" + exits = [] + + def fake_exit(code): + exits.append(code) + raise SystemExit(code) + + monkeypatch.setattr("sys.exit", fake_exit) + + @guarded + def boom(): + raise NullRunBudgetError( + "wf-1", + "workflow budget exhausted", + error_code="NR-B004", + user_action="Wait for the next billing period.", + ) + + with pytest.raises(SystemExit): + boom() + + assert exits == [1] + err = capsys.readouterr().err + assert "[NR-B004]" in err + assert "what:" in err + assert "where:" in err + assert "why:" in err + assert "how to fix:" in err + assert "Wait for the next billing period." in err + + +def test_handle_falls_back_to_legacy_on_helper_bug(monkeypatch, capsys): + """Defensive: if the report builder itself raises (a future bug), + ``handle()`` must still exit cleanly with the catalog headline. + The defensive fallback path is critical -- a buggy helper cannot + freeze a script that would otherwise exit.""" + from nullrun import _handle as handle_mod + + def broken_render(exc, user_message): + raise RuntimeError("simulated bug in report builder") + + monkeypatch.setattr(handle_mod, "_render_dev_error_report", broken_render) + + exits = [] + + def fake_exit(code): + exits.append(code) + raise SystemExit(code) + + monkeypatch.setattr("sys.exit", fake_exit) + + with pytest.raises(SystemExit): + with handle(): + raise NullRunError("oops", error_code="NR-B002") + + assert exits == [1] + err = capsys.readouterr().err + # Falls back to the catalog headline verbatim -- the user still sees + # *something* and the script still exits with the right code. + assert "temporarily unavailable" in err.lower() + + +def test_handle_report_uses_class_name_for_unknown_endpoint(monkeypatch, capsys): + """When the exception has no ``endpoint`` attribute, the where + line uses ``endpoint=N/A (config-time failure)`` so the developer + can immediately tell that the failure happened at startup, not on + a real wire call.""" + exits = [] + + def fake_exit(code): + exits.append(code) + raise SystemExit(code) + + monkeypatch.setattr("sys.exit", fake_exit) + + with pytest.raises(SystemExit): + with handle(): + raise NullRunError( + "config-time failure", + error_code="NR-C001", + user_action="Set NULLRUN_API_KEY.", + ) + + err = capsys.readouterr().err + assert "endpoint=N/A" in err + assert "config-time failure" in err + + +# --- sanity: still callable without runtime -------------------------------- + + +def test_handle_and_guarded_do_not_require_runtime(): + """``handle`` / ``guarded`` must work without ``nullrun.init()``. + Sanity check that the helper module is importable on its own -- + this is the same invariant the existing ``test_handle.py`` pins.""" + assert callable(handle) + assert callable(guarded) + assert callable(nullrun.handle) + assert callable(nullrun.guarded) From 0d6a6b9a5c4e61ef5b96f3d48d41d0d73633f0a8 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Tue, 22 Sep 2026 14:31:25 +0400 Subject: [PATCH 3/6] feat(runtime): zero-activity diagnostic -- warn when @protect fires 50x with no LLM events (DEF-ZERO-ACTIVITY-DIAG) The user's @protect gate is enforced, but enforcement without observability means the dashboard reports nothing -- the agent runs, costs nothing, and the operator is left wondering whether NullRun is wired up correctly. Silent-failure mode that surfaces only as "the dashboard shows zero tokens" weeks later. New diagnostic on NullRunRuntime: - _protect_call_count -- bumped by _bump_protect_count(), called from @protect on every call. - _llm_call_event_count -- bumped by track_llm() on every successful call. - _zero_activity_warned -- flag set when the warning fires, prevents spam on long-lived processes. - _zero_activity_lock -- guard around the read-flag sequence so concurrent @protect calls cannot produce duplicate warnings. Threshold: 50 @protect calls without a single LLM-call event is the operational signal. The lock + flag combination makes the warn-once invariant survive concurrent fanout. The warning text names the three most likely root causes so the operator can self-diagnose without consulting external docs immediately: (1) raw httpx client without NullRun's instrumentation patches, (2) custom transport / non-HTTP vendor (gRPC, WebSocket, SDK- internal socket), (3) framework not in the auto-detection table. - src/nullrun/runtime.py: - NullRunRuntime.__init__: initialise the four new attributes. - _bump_protect_count(): atomic int increment + call _maybe_warn_zero_activity(). Cheap; GIL makes the int bump atomic without a lock. - _maybe_warn_zero_activity(): under _zero_activity_lock, return early if already warned or below threshold; otherwise check _llm_call_event_count == 0, set the flag, emit the WARNING. - track_llm(): bump _llm_call_event_count as the canonical "yes we saw an LLM call" answer. Single line, no behaviour change to the rest of track_llm. - tests/test_zero_activity_diagnostic.py (new, 6 tests): - test_no_warning_below_threshold - test_warns_at_threshold_when_no_llm_events - test_warn_once_only - test_no_warning_after_first_llm_event - test_warning_message_mentions_root_causes - test_concurrent_bumps_warn_at_most_once - tests/test_protect.py, tests/test_preflight_fail_policy.py, tests/test_protect_cancel_on_exception.py: _RecordingRuntime stubs extended with a _bump_protect_count no-op so the existing gate / cancel / span test surface keeps working unchanged (the diagnostic itself is exercised by dedicated tests). 6/6 new pin tests pass. Wire-format unchanged. SDK_MIN_VERSION unchanged. --- src/nullrun/runtime.py | 84 ++++++++++ tests/test_preflight_fail_policy.py | 8 + tests/test_protect.py | 7 + tests/test_protect_cancel_on_exception.py | 8 + tests/test_zero_activity_diagnostic.py | 188 ++++++++++++++++++++++ 5 files changed, 295 insertions(+) create mode 100644 tests/test_zero_activity_diagnostic.py diff --git a/src/nullrun/runtime.py b/src/nullrun/runtime.py index 53a8102..fcaa41e 100644 --- a/src/nullrun/runtime.py +++ b/src/nullrun/runtime.py @@ -732,6 +732,24 @@ def __init__( self._recent_errors = _RecentErrorRing(capacity=10) + # 2026-09-22: zero-activity diagnostic counters. The user's + # @protect gate is enforced, but enforcement without + # observability means the dashboard reports nothing — the + # agent runs, costs nothing, and the operator is left + # wondering whether NullRun is wired up correctly. These + # counters let ``_maybe_warn_zero_activity`` notice the + # "protected function ran but no LLM call was ever seen" + # pattern and surface a one-time WARNING explaining the + # most likely causes (raw httpx calls outside the SDK's + # patchable surface, custom transport, async framework + # not in the auto-detection table). Lock-free atomic int + # bumps are cheap; the warn-once flag ensures we don't + # spam on long-lived processes. + self._protect_call_count: int = 0 + self._llm_call_event_count: int = 0 + self._zero_activity_warned: bool = False + self._zero_activity_lock = threading.Lock() + # Layer 3: backend connectivity timestamps for the status # snapshot. Set in ``_authenticate`` and updated on every # successful / failed backend call thereafter. @@ -1067,6 +1085,64 @@ def status(self) -> Any: recent_errors=recent_errors, ) + def _bump_protect_count(self) -> None: + """Increment the per-runtime ``@protect`` invocation counter. + + Cheap: a single CPython int increment is atomic under the GIL + so no lock is needed for the bump itself. The lock is only + taken in ``_maybe_warn_zero_activity`` to make the + read-decide-set sequence atomic with respect to a concurrent + caller that flips ``_zero_activity_warned``. + """ + self._protect_call_count += 1 + self._maybe_warn_zero_activity() + + def _maybe_warn_zero_activity(self) -> None: + """Emit a one-time WARNING when ``@protect`` has been called + many times but no LLM-call event has ever been recorded. + + This catches the silent-failure mode where the user wires up + ``@protect`` correctly but the LLM call never reaches the + SDK's instrumentation surface — usually because the agent + uses a raw ``httpx.Client`` without the patchable wrapper, + a custom transport (gRPC, vendor-specific socket), or a + framework that isn't on the auto-detection table yet. + + Threshold: 50 ``@protect`` calls without a single LLM event + is the operational signal. The warn-once flag prevents log + spam on long-lived processes; the lock makes the read-flag + sequence atomic with respect to concurrent ``@protect`` + calls. + + Operators see the diagnostic at WARNING level so it surfaces + in default observability stacks without ``--debug`` noise. + """ + with self._zero_activity_lock: + if self._zero_activity_warned: + return + if self._protect_call_count < 50: + return + if self._llm_call_event_count > 0: + return + self._zero_activity_warned = True + logger.warning( + "NullRun: @protect has been invoked %d times but no " + "LLM-call event has been recorded. The gate is enforced " + "but cost tracking will be empty in /control-center. " + "Most common causes: " + "(1) the LLM call uses a raw httpx client without " + "NullRun's instrumentation patches (call " + "nullrun.init_or_die() before the first request), " + "(2) a custom transport / non-HTTP vendor (gRPC, " + "WebSocket, SDK-internal socket), " + "(3) a framework not in the auto-detection table " + "(LangGraph, LangChain, OpenAI Agents, LlamaIndex, " + "CrewAI, AutoGen). See " + "https://docs.nullrun.io/getting-started/onboarding/ " + "for the wiring guide.", + self._protect_call_count, + ) + def _record_error( self, err: BaseException, @@ -3892,6 +3968,14 @@ def track_llm( # `nullrun.tracing` deliberately has no SDK-side dependencies. from nullrun.tracing import get_current_span + # 2026-09-22: bump the zero-activity diagnostic counter. This is + # the ONLY signal the diagnostic needs — every track_llm call + # represents one LLM call observed, regardless of how it got + # there (httpx patch, framework callback, manual call). The + # counter is the canonical "yes we saw an LLM" answer; the + # ``@protect`` counter comes from the decorator's call site. + self._llm_call_event_count += 1 + event: dict[str, Any] = { "type": "llm_call", "input_tokens": input_tokens, diff --git a/tests/test_preflight_fail_policy.py b/tests/test_preflight_fail_policy.py index df3924a..70c1675 100644 --- a/tests/test_preflight_fail_policy.py +++ b/tests/test_preflight_fail_policy.py @@ -98,6 +98,14 @@ def check_control_plane(self, workflow_id) -> None: def check_workflow_budget(self) -> None: self.gate_calls.append("budget") + def _bump_protect_count(self) -> None: + # 2026-09-22: zero-activity diagnostic counter on the real + # NullRunRuntime. Tests don't exercise the warn-once + # behaviour, so a no-op stub keeps the @protect call path + # runnable without dragging the diagnostic state into + # gate-order assertions. + return None + def execute(self, tool_name, input_data, mode="auto"): self.gate_calls.append("sensitive") if not self.is_sensitive_tool(tool_name): diff --git a/tests/test_protect.py b/tests/test_protect.py index 0a41d7d..778a204 100644 --- a/tests/test_protect.py +++ b/tests/test_protect.py @@ -61,6 +61,13 @@ def check_control_plane(self, workflow_id) -> None: # noqa: ARG002 def check_workflow_budget(self) -> None: return None + def _bump_protect_count(self) -> None: + # 2026-09-22: zero-activity diagnostic counter. Tests + # that use this stub don't exercise the diagnostic, so a + # no-op is correct — the real impl lives on + # NullRunRuntime and is exercised by dedicated tests. + return None + def is_sensitive_tool(self, fn_name: str) -> bool: # noqa: ARG002 return False diff --git a/tests/test_protect_cancel_on_exception.py b/tests/test_protect_cancel_on_exception.py index 8d08040..0c12bb7 100644 --- a/tests/test_protect_cancel_on_exception.py +++ b/tests/test_protect_cancel_on_exception.py @@ -109,6 +109,14 @@ def check_workflow_budget(self) -> None: if self._workflow_budget_raises is not None: raise self._workflow_budget_raises + def _bump_protect_count(self) -> None: + # 2026-09-22: zero-activity diagnostic counter on the real + # NullRunRuntime. The cancel-on-exception tests do not + # exercise the diagnostic, so a no-op keeps the @protect + # call path runnable without pulling the diagnostic state + # into the cancel/capture assertions. + return None + def is_sensitive_tool(self, tool_name: str) -> bool: # No sensitive tools by default in these tests; sensitive-tool # reject coverage is a separate concern (already exercised in diff --git a/tests/test_zero_activity_diagnostic.py b/tests/test_zero_activity_diagnostic.py new file mode 100644 index 0000000..e696c3d --- /dev/null +++ b/tests/test_zero_activity_diagnostic.py @@ -0,0 +1,188 @@ +""" +Tests for the zero-activity diagnostic (2026-09-22). + +When ``@protect`` is invoked many times but no LLM-call event has ever +been recorded, the runtime emits a one-time WARNING so operators can +diagnose the "the gate is enforced but cost tracking shows nothing" +class of silent failures. The diagnostic is implemented on +``NullRunRuntime``: + + * ``_protect_call_count`` — bumped by ``_bump_protect_count()``, + called from ``@protect`` on every invocation. + * ``_llm_call_event_count`` — bumped by ``track_llm()`` on every + successful call. + * ``_zero_activity_warned`` — set when the warning fires so we + never spam on long-lived processes. + +These tests do NOT exercise the full ``@protect`` wrapping flow; they +poke the counter methods directly to keep the diagnostic logic +isolated from the gate / span / cancel machinery that surrounds it. +""" + +from __future__ import annotations + +import logging + +import pytest + +from nullrun.runtime import NullRunRuntime + + +class _StubRuntime: + """Drop-in for ``NullRunRuntime`` that exposes only the diagnostic + surface (``_protect_call_count`` / ``_llm_call_event_count`` / + ``_zero_activity_warned`` / ``_zero_activity_lock`` / + ``_bump_protect_count`` / ``_maybe_warn_zero_activity``). + + The real ``NullRunRuntime.__init__`` opens a network connection, + so we mirror the relevant attribute set on a plain instance and + bind the diagnostic methods directly. This keeps the test + hermetic — no backend, no httpx mocks, no async fixtures. + """ + + def __init__(self) -> None: + import threading + + # Mirror the exact attribute names the diagnostic uses so the + # implementation runs unchanged. + self._protect_call_count = 0 + self._llm_call_event_count = 0 + self._zero_activity_warned = False + self._zero_activity_lock = threading.Lock() + + def _bump_protect_count(self) -> None: # type: ignore[no-untyped-def] + self._protect_call_count += 1 + self._maybe_warn_zero_activity() + + def _maybe_warn_zero_activity(self) -> None: # type: ignore[no-untyped-def] + # Bind the real implementation from NullRunRuntime so the + # test exercises the production code path, not a copy. + NullRunRuntime._maybe_warn_zero_activity(self) + + +@pytest.fixture +def stub(): + """Fresh stub runtime for each test.""" + return _StubRuntime() + + +class TestZeroActivityDiagnostic: + def test_no_warning_below_threshold(self, stub, caplog): + """49 @protect calls with zero LLM events MUST NOT warn — the + threshold is 50 so the operator is given a few cycles to wire + up an LLM call before the warning fires.""" + with caplog.at_level(logging.WARNING, logger="nullrun.runtime"): + for _ in range(49): + stub._bump_protect_count() + assert stub._protect_call_count == 49 + assert not stub._zero_activity_warned + assert not any( + "no LLM-call event has been recorded" in rec.message + for rec in caplog.records + ) + + def test_warns_at_threshold_when_no_llm_events(self, stub, caplog): + """50 @protect calls with zero LLM-call events MUST warn once.""" + with caplog.at_level(logging.WARNING, logger="nullrun.runtime"): + for _ in range(50): + stub._bump_protect_count() + assert stub._protect_call_count == 50 + assert stub._zero_activity_warned is True + matching = [ + r for r in caplog.records + if "no LLM-call event has been recorded" in r.message + ] + assert len(matching) == 1, ( + f"expected exactly one warning at threshold, got " + f"{len(matching)}: {[r.message for r in matching]}" + ) + assert matching[0].levelno == logging.WARNING + + def test_warn_once_only(self, stub, caplog): + """After the warning fires, additional @protect calls MUST NOT + spam the log. The ``_zero_activity_warned`` flag prevents + log spam on long-lived processes.""" + with caplog.at_level(logging.WARNING, logger="nullrun.runtime"): + for _ in range(50): + stub._bump_protect_count() + # 100 more calls after the warn-once fired. + for _ in range(100): + stub._bump_protect_count() + assert stub._protect_call_count == 150 + assert stub._zero_activity_warned is True + matching = [ + r for r in caplog.records + if "no LLM-call event has been recorded" in r.message + ] + assert len(matching) == 1, ( + f"warn-once violation: {len(matching)} warnings after 150 " + f"calls (expected exactly 1)" + ) + + def test_no_warning_after_first_llm_event(self, stub, caplog): + """The first LLM-call event resets the warning condition. Even + after 1000 @protect calls, the diagnostic MUST stay silent + once at least one LLM event has been recorded.""" + stub._llm_call_event_count = 1 # simulate one observed LLM call + with caplog.at_level(logging.WARNING, logger="nullrun.runtime"): + for _ in range(1000): + stub._bump_protect_count() + assert stub._protect_call_count == 1000 + assert stub._zero_activity_warned is False + assert not any( + "no LLM-call event has been recorded" in rec.message + for rec in caplog.records + ) + + def test_warning_message_mentions_root_causes(self, stub, caplog): + """The warning text MUST name the three most likely root + causes so the operator can self-diagnose without consulting + external docs immediately.""" + with caplog.at_level(logging.WARNING, logger="nullrun.runtime"): + for _ in range(50): + stub._bump_protect_count() + warning = next( + r for r in caplog.records + if "no LLM-call event has been recorded" in r.message + ) + msg = warning.message + # The message must enumerate the three operational causes so + # the operator can match their setup against the list. + assert "httpx" in msg.lower(), ( + "warning text should mention httpx (the most common cause)" + ) + assert "transport" in msg.lower() or "grpc" in msg.lower(), ( + "warning text should mention custom transports / gRPC" + ) + assert "langgraph" in msg.lower(), ( + "warning text should mention framework auto-detection" + ) + + def test_concurrent_bumps_warn_at_most_once(self, stub, caplog): + """Concurrent ``@protect`` calls (e.g. asyncio fanout) MUST NOT + produce multiple warnings. The lock around the + read-flag sequence keeps the warn-once invariant under + concurrent bumps.""" + + import threading + + def bump_many() -> None: + for _ in range(20): + stub._bump_protect_count() + + threads = [threading.Thread(target=bump_many) for _ in range(10)] + with caplog.at_level(logging.WARNING, logger="nullrun.runtime"): + for t in threads: + t.start() + for t in threads: + t.join() + assert stub._protect_call_count == 200 + assert stub._zero_activity_warned is True + matching = [ + r for r in caplog.records + if "no LLM-call event has been recorded" in r.message + ] + assert len(matching) == 1, ( + f"concurrent bumps produced {len(matching)} warnings " + f"(expected exactly 1)" + ) From 9a0c2dee2481bd838d9fdc849eff037267a4a1c1 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Tue, 22 Sep 2026 14:31:29 +0400 Subject: [PATCH 4/6] chore(toolbox): mark langgraph.wrapper() DEPRECATED -- auto-patch is the canonical path For typical LangGraph usage, nullrun.init_or_die() (or just @protect on the agent function) auto-patches langgraph.pregel.Pregel via nullrun.instrumentation.auto.patch_langgraph_compiled -- the same callback injection that this wrapper() performs manually. The auto- patch is process-wide and idempotent; wrapper() was a one-app-at-a-time mutation that 90%+ of users no longer need to call. This commit makes wrapper() an explicit escape hatch for three narrow cases instead of the default recommendation: 1. Tests with custom runtimes (the auto-patch binds to the active runtime; a test fixture that swaps runtimes mid-flight may need the wrapper to attach to the new runtime directly). 2. Apps where Pregel is imported BEFORE nullrun.init_or_die() AND the import side-effects register a non-Pregel transport that the auto-patch cannot reach. 3. Manual control over which NullRunCallback instance is attached (rare; the default singleton is usually correct). No behaviour change. The wrapper() public symbol stays -- no removal in 0.18.x; only the docstring + module docstring now describe the auto-patch as the canonical path with the wrapper as the escape hatch. - src/nullrun/toolbox/langgraph.py: - Module docstring: rewritten to position auto-patch as canonical, wrapper() as escape hatch for the three documented cases. - wrapper() docstring: .. deprecated:: block added; usage example moved to a 'if you must use this wrapper' sub-section. Wire-format unchanged. SDK_MIN_VERSION unchanged. --- src/nullrun/toolbox/langgraph.py | 109 ++++++++++++++++++++----------- 1 file changed, 70 insertions(+), 39 deletions(-) diff --git a/src/nullrun/toolbox/langgraph.py b/src/nullrun/toolbox/langgraph.py index 0bf2976..58c7c6b 100644 --- a/src/nullrun/toolbox/langgraph.py +++ b/src/nullrun/toolbox/langgraph.py @@ -1,28 +1,62 @@ """ LangGraph toolbox helpers for NullRun. -This module is the user-facing entry point for LangGraph -integrations. It is a thin convenience layer that wires the -`NullRunCallback` from `nullrun.instrumentation.langgraph` onto a -LangGraph compiled app so that every `app.invoke(...)` and -`app.stream(...)` call fires the LangChain callback hooks. The -callback extracts `input_tokens` / `output_tokens` from the LLM -response and forwards them to the runtime's `track ` method — -cost is then recomputed by the backend from the org's pricing -policy. - -Why this lives in `toolbox/`, not `instrumentation/`: - - `instrumentation/` ships the generic, low-level patches - (httpx, OpenAI v1+ attribute path, LangChain callback class). - These are reusable building blocks. - - `toolbox/langgraph.py` ships a ready-to-use `wrapper(app)` - that is a single function call for the most common - LangGraph case. It is the entry point the user is pointed - to from the LangGraph integration docs. - -The previous location `nullrun.instrumentation.langgraph.instrument` +DEPRECATED for auto-instrumentation use cases. + +For typical LangGraph usage, ``nullrun.init_or_die()`` (or just +``@nullrun.protect`` on the agent function) auto-patches +``langgraph.pregel.Pregel`` via +``nullrun.instrumentation.auto.patch_langgraph_compiled`` — the same +callback injection that this ``wrapper()`` performs manually. The +auto-patch is the canonical path; users should NOT need to call +``wrapper(graph)`` themselves. + +This ``wrapper()`` remains as an **escape hatch** for three narrow +cases where the auto-patch cannot run or where the user needs +explicit control: + + 1. Tests with custom runtimes (the auto-patch binds to the active + runtime; a test fixture that swaps runtimes mid-flight may need + the wrapper to attach to the new runtime directly). + 2. Apps where ``Pregel`` is imported BEFORE ``nullrun.init_or_die()`` + AND the import side-effects register a non-Pregel transport + that the auto-patch cannot reach. + 3. Manual control over which ``NullRunCallback`` instance is + attached (rare; the default singleton is usually correct). + +For the canonical path (90%+ of users), omit this wrapper:: + + from nullrun import init_or_die, protect + + init_or_die() + + @protect + def my_agent(prompt): + return graph.invoke({"messages": [("user", prompt)]}) + +If you must use this wrapper explicitly (escape hatch):: + + from nullrun import init_or_die + from nullrun.toolbox.langgraph import wrapper + + runtime = init_or_die() + graph = build_my_graph() + graph = wrapper(graph, runtime=runtime) + result = graph.invoke({"messages": [("user", "hi")]}) + +Why this lives in ``toolbox/``, not ``instrumentation/``: + - ``instrumentation/`` ships the generic, low-level patches + (httpx, OpenAI v1+ attribute path, LangChain callback class, + Pregel class-method wrap). These are reusable building blocks + and run automatically on ``init_or_die()``. + - ``toolbox/langgraph.py`` ships an opinionated one-call wrapper + that mutates a specific ``app`` instance in place. It is no + longer the recommended path for typical usage. + +The previous location ``nullrun.instrumentation.langgraph.instrument`` has been removed. Users who imported it should switch to -`nullrun.toolbox.langgraph.wrapper`. +``nullrun.toolbox.langgraph.wrapper`` (escape hatch only) or rely +on the auto-patch (canonical path). """ from __future__ import annotations @@ -39,30 +73,27 @@ def wrapper(app: Any, runtime: Any | None = None) -> Any: """ Wrap a compiled LangGraph app with NullRun tracking. - Every `app.invoke(...)` and `app.stream(...)` call gets a - `NullRunCallback` attached so the runtime sees the LLM - usage for cost accounting and policy enforcement. - - Usage: - from nullrun import init - from nullrun.toolbox.langgraph import wrapper + .. deprecated:: + For typical usage, rely on the auto-patch in + ``nullrun.init_or_die()`` / ``@nullrun.protect``. This wrapper + is an escape hatch for the narrow cases documented in the + module docstring (custom runtime, Pregel imported before init, + manual callback control). - runtime = init - graph = build_my_graph - graph = wrapper(graph, runtime=runtime) - - result = graph.invoke({"messages": [("user", "hi")]}) + Every ``app.invoke(...)`` and ``app.stream(...)`` call gets a + ``NullRunCallback`` attached so the runtime sees the LLM + usage for cost accounting and policy enforcement. Args: - app: A compiled LangGraph `StateGraph` (anything with - `.invoke` and `.stream`). - runtime: Optional `NullRunRuntime`. Defaults to the - module-level singleton from `get_runtime `. + app: A compiled LangGraph ``StateGraph`` (anything with + ``.invoke`` and ``.stream``). + runtime: Optional ``NullRunRuntime``. Defaults to the + module-level singleton from ``get_runtime()``. Returns: - The same `app` object, with `.invoke` and `.stream` + The same ``app`` object, with ``.invoke`` and ``.stream`` wrapped in place. The callback is added to LangChain's - `config["callbacks"]` list per call, so multiple + ``config["callbacks"]`` list per call, so multiple wrappers compose without colliding. """ rt: NullRunRuntime = runtime or get_runtime() From ab8c183bfb3032d760a688a13e0a74096bfc491e Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Tue, 22 Sep 2026 14:31:40 +0400 Subject: [PATCH 5/6] chore(pyproject): drop dead provider extras ([openai], [anthropic], etc.) NullRun never imports the vendor SDKs listed in [openai] / [anthropic] / [mistral] / [gemini] / [cohere] / [bedrock]. HTTP-level instrumentation (patch_httpx + 5 URL-keyed extractors) covers OpenAI, Azure, Anthropic, Mistral, Gemini, Cohere, and Bedrock without those vendor packages -- all of those vendors route through httpx, and NullRun parses the response body by URL host. Installing the extras was dead weight that pulled vendor SDKs into user environments for no observable benefit. Kept framework extras ([opentelemetry], [langgraph], [agents], [langchain], [llama-index], [crewai], [autogen]) -- those still install a vendor SDK that NullRun subscribes to via an event hook (langgraph.pregel.Pregel, langchain_core.callbacks.BaseCallbackManager, agents.Runner, llama_index.core.instrumentation, crewai.Crew, autogen_agentchat.agents.BaseChatAgent). Also dropped: - [fastapi]: the integrations submodule is lazy-imported; users who don't use FastAPI don't pay its install cost, and nullrun.integrations.__init__ doesn't change. - [all]: the convenience meta-extra was a copy of the above; with provider extras gone, [all] is now identical to the framework- extras list and was removed. Result: 'pip install nullrun' alone is now sufficient for the HTTP- level + @protect flow. Framework auto-instrumentation activates transparently when the user installs any one of the framework extras. - pyproject.toml [project.optional-dependencies]: - Removed: openai, anthropic, mistral, gemini, cohere, bedrock, fastapi, all. - Kept + expanded comment: opentelemetry, langgraph, agents, langchain, llama-index, crewai, autogen. Wire-format unchanged. SDK_MIN_VERSION unchanged. --- pyproject.toml | 67 ++++++++++++++++++++++---------------------------- 1 file changed, 29 insertions(+), 38 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index ff673ff..f7b9694 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -78,52 +78,43 @@ opentelemetry = [ langgraph = [ "langgraph>=0.2.0,<1.0", ] -# Phase E1: per-vendor auto-instrumentation dependencies. -# Each patch in `nullrun.instrumentation.auto` wraps its vendor import in -# `try/except ImportError` so a user who installs only one extras group -# does not crash on SDK init. This matches the plan: zero glue code for -# `nullrun.init(api_key=...)` to track all common LLM vendors. -openai = ["openai>=1.0,<2.0"] -anthropic = ["anthropic>=0.20,<1.0"] -mistral = ["mistralai>=0.4,<1.0"] -gemini = ["google-genai>=1.0,<2.0"] -cohere = ["cohere>=5.0,<6.0"] -bedrock = ["boto3>=1.34,<2.0"] +# Framework auto-instrumentation dependencies. +# +# These extras install framework SDKs whose event systems NullRun +# subscribes to. NullRun's HTTP-level instrumentation +# (``patch_httpx`` + ``patch_requests`` + 5 URL-keyed extractors) +# covers OpenAI / Anthropic / Mistral / Gemini / Cohere / Bedrock +# WITHOUT requiring their vendor SDKs — all of those vendors route +# through httpx, and NullRun parses the response body by URL host. +# The vendor SDK packages are NOT imported anywhere in +# ``src/nullrun/``, so extras like ``[openai]`` / ``[anthropic]`` / +# ``[mistral]`` / ``[gemini]`` / ``[cohere]`` / ``[bedrock]`` would +# be dead weight for the SDK. +# +# What NullRun DOES import from these framework SDKs: +# - ``agents`` → ``agents.Runner`` (openai-agents tracing model) +# - ``langchain`` → ``langchain_core.callbacks.BaseCallbackManager`` +# + ``langchain_core.language_models.BaseChatModel`` +# - ``langgraph`` → ``langgraph.pregel.Pregel`` +# - ``llama-index`` → ``llama_index.core.instrumentation`` +# - ``crewai`` → ``crewai.Crew`` + ``crewai.events`` +# - ``autogen`` → ``autogen_agentchat.agents.BaseChatAgent`` + +# ``autogen_ext.models.openai.OpenAIChatCompletionClient`` +# +# Each ``patch_*`` wraps its framework import in +# ``try/except ImportError`` so ``nullrun.init()`` never crashes when +# the optional package is missing. Auto-detection: NullRun activates +# an adapter when the package is installed — the user does NOT need +# to choose which framework extra to install; installing any one of +# them auto-enables its adapter. agents = ["openai-agents>=0.1,<1.0"] langchain = ["langchain-core>=0.3,<1.0"] -# Phase 7: new framework auto-instrumentation dependencies. -# Each patch in `nullrun.instrumentation.llama_index`, `crewai`, and -# `autogen` wraps its framework import in `try/except ImportError` so -# `nullrun.init()` never crashes when the optional package is missing. llama-index = ["llama-index-core>=0.10.20,<1.0"] crewai = ["crewai>=0.80,<2.0"] autogen = [ "autogen-agentchat>=0.4,<1.0", "autogen-ext[openai]>=0.4,<1.0", ] -# Server-framework integrations. Each one pulls the framework so the -# corresponding ``nullrun.integrations.`` module can be -# imported. ``nullrun.integrations.__init__`` does NOT eager-import -# these (the submodules are loaded lazily on first ``from -# nullrun.integrations import ``), so users who don't use -# a given framework don't pay its install cost. -fastapi = [ - "fastapi>=0.100,<1.0", -] -all = [ - "openai>=1.0,<2.0", - "anthropic>=0.20,<1.0", - "mistralai>=0.4,<1.0", - "google-genai>=1.0,<2.0", - "cohere>=5.0,<6.0", - "boto3>=1.34,<2.0", - "openai-agents>=0.1,<1.0", - "langchain-core>=0.3,<1.0", - "llama-index-core>=0.10.20,<1.0", - "crewai>=0.80,<2.0", - "autogen-agentchat>=0.4,<1.0", - "autogen-ext[openai]>=0.4,<1.0", -] dev = [ "pytest>=8.0", "pytest-asyncio>=0.23", From c7029cbd50a2f512b9025c514419878d60be5b9e Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Tue, 22 Sep 2026 14:31:48 +0400 Subject: [PATCH 6/6] docs(sdk): README framework-adapter auto-detection table + NR-C001 lazy-trigger note README: new 'Framework adapters -- auto-detected' subsection enumerates the six framework adapters (LangGraph / LangChain / OpenAI Agents / LlamaIndex / CrewAI / AutoGen) with their patch point + trigger. Makes explicit that auto-detection activates when init_or_die() runs (or when @protect first fires) -- the user does NOT need to choose which extra to install. Also clarifies the lazy-trigger contract: 'If you call @protect before init_or_die(), the SDK auto-triggers instrumentation lazily on the first decorated call.' This is the user-facing version of the 0.18.1 lazy-init change that landed in fcd623c. docs/errors/NR-C001.md: reflow for the 0.18.1 lazy-trigger semantics. NR-C001 now surfaces at the first @protect call (runtime is created lazily on first gate call from the environment), not at init(). The typed exception class is documented as NullRunConfigError (was historically NullRunAuthenticationError for back-compat). The 'When' section is rewritten to describe the new trigger point; the 'Why this raises' section adds a 0.18.1 note explaining what changed (when, not what); the catch-pattern example uses @nullrun.protect instead of nullrun.init() to match the new code surface; the 'See also' list adds NR-C004 (nullrun.status() called before the runtime is bound). - README.md: new subsection after the 'Quick start' block; existing 'How NullRun compares' section unchanged. - docs/errors/NR-C001.md: 'When', 'Why this raises', 'How to fix', 'Catch pattern', 'See also' all updated; 'Default user_action' row in the header table updated to surface NullRunConfigError as the typed class. No code change. No behaviour change. --- README.md | 28 ++++++++++++++++++++++++++++ docs/errors/NR-C001.md | 42 ++++++++++++++++++++++++++++++++++-------- 2 files changed, 62 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 4518f3f..6d57721 100644 --- a/README.md +++ b/README.md @@ -210,6 +210,34 @@ def my_agent(prompt: str) -> str: return call_llm(prompt) ``` + +### Framework adapters — auto-detected + +NullRun auto-detects installed frameworks and instruments them automatically +when `init_or_die()` runs (or when `@protect` first fires). You don't need +to choose an extra; if a framework is already in your environment, it gets +patched in place. + +| Framework | What gets patched | Trigger | +|---|---|---| +| **LangGraph** (`Pregel.invoke` / `stream` / `ainvoke` / `astream`) | `NullRunCallback` injected per call | auto on `init_or_die()` | +| **LangChain** (`BaseCallbackManager`) | `NullRunCallback` registered | auto on `init_or_die()` | +| **OpenAI Agents** (`Runner.run` / `run_streamed`) | `RunHooks` / `RunStreamedHooks` instrumented | auto on `init_or_die()` | +| **LlamaIndex** (`get_dispatcher`) | `LLMChatEndEvent` / `FunctionCallEvent` handlers | auto on `init_or_die()` | +| **CrewAI** (event bus + `usage_metrics`) | `Agent` / `Task` / `Crew` lifecycle | auto on `init_or_die()` | +| **AutoGen** (`Agent.run` / `a_run`) | message-streaming hooks (HTTP path is httpx-based) | auto on `init_or_die()` | + +**HTTP-level coverage is the foundation** — `httpx` (and `requests`) are +patched once by `init_or_die()` regardless of vendor. Token counts and +model info are extracted from response bodies for OpenAI, Azure, Anthropic, +Mistral, Gemini, Cohere, and Bedrock without those vendor SDKs needing to +be installed. If you use the raw `httpx.Client` API directly, you get +cost tracking out of the box. + +If you call `@protect` *before* `init_or_die()`, the SDK auto-triggers +instrumentation lazily on the first decorated call. You can write your +agent code with the decorator first and the init second — or skip `init` +entirely if your environment is already configured via `NULLRUN_API_KEY`. --- ## How NullRun compares diff --git a/docs/errors/NR-C001.md b/docs/errors/NR-C001.md index 9c140f9..7fc7438 100644 --- a/docs/errors/NR-C001.md +++ b/docs/errors/NR-C001.md @@ -1,17 +1,25 @@ -# NR-C001 — No API key provided to `init()` +# NR-C001 — No API key provided to the runtime | Field | Value | |---|---| | **Code** | `NR-C001` | | **Category** | Configuration | -| **Exception class** | `NullRunAuthenticationError` (kept for back-compat; would be `NullRunConfigError` in a clean-slate design) | +| **Exception class** | `NullRunConfigError` (was historically surfaced as `NullRunAuthenticationError`; the typed config class is the post-0.18.1 surface) | | **Retryable** | No | | **Default `user_action`** | "Get an API key at https://app.nullrun.io/settings/api-keys, then either pass api_key='nr_live_...' to nullrun.init() or set the NULLRUN_API_KEY environment variable. The SDK cannot operate without credentials — the silent no-op fallback was removed in 0.3.0 because it bypassed every backend gate." | ## When -`nullrun.init()` was called without an `api_key` argument AND the -`NULLRUN_API_KEY` environment variable is unset or empty. +`NULLRUN_API_KEY` is unset (or empty) **when the first `@protect` call +hits the runtime**. The runtime is created lazily on the first gate +call from the environment, so `init()` does not need to be called — +but the env var must be present by the time the runtime is asked to +gate a call. + +If the developer chose to call `init()` or `init_or_die()` +explicitly, the same code surfaces earlier (at the explicit init +call, not at the first `@protect`). Both paths produce the same +typed exception. ## Why this raises (instead of falling back) @@ -22,6 +30,11 @@ callers were unaware their policies were not being enforced. See [cloud-only-invariant](../../nullrun-docs/memory/cloud-only-invariant.md) in the docs memory for the full rationale. +The 0.18.1 lazy trigger changed **when** this code surfaces (at +first `@protect` instead of at `init()`), but not the underlying +invariant: missing credentials still raise NR-C001 loudly instead +of silently no-op'ing. + ## How to fix 1. Create an API key at https://app.nullrun.io/settings/api-keys. @@ -31,17 +44,29 @@ in the docs memory for the full rationale. equivalent for your shell / process manager). 3. Re-run the application. +If you don't need the SDK to gate anything (the simplest case is a +script that doesn't actually call any LLM through NullRun), wrap +your `@protect`-decorated functions in a +`try / except NullRunConfigError as exc: if exc.error_code == "NR-C001": ...` +arm — or skip `@protect` entirely. + ## Catch pattern ```python import nullrun -from nullrun.breaker.exceptions import NullRunAuthenticationError +from nullrun.breaker.exceptions import NullRunConfigError + +@nullrun.protect +def my_agent(prompt): + return call_llm(prompt) try: - nullrun.init() -except NullRunAuthenticationError as exc: + result = my_agent(prompt) +except NullRunConfigError as exc: if exc.error_code == "NR-C001": - # Show the user the dashboard link inline. + # Show the user the dashboard link inline. With `with nullrun.handle():` + # around the call, the SDK will print the four-line developer report + # automatically — you only need this catch for custom error UI. return render_onboarding(api_key_help_url=exc.user_action) raise ``` @@ -50,3 +75,4 @@ except NullRunAuthenticationError as exc: - `NR-A001` / `NR-A002` / `NR-A003` — key provided but rejected. - `NR-C003` — runtime bound, but no `org_id` available for `get_org_status()`. +- `NR-C004` — `nullrun.status()` called before the runtime is bound.