chore(release): 0.18.1 — lower-friction UX (lazy @protect + dev error report + zero-activity diagnostic + drop dead extras + deprecate langgraph.wrapper) - #110
Merged
Conversation
… report + zero-activity diagnostic + drop dead extras + deprecate langgraph.wrapper)
…report (DEF-DEV-REPORT-EMPTY)
The catch-all exit path previously printed only the catalog user-message
("There's a configuration issue. Please contact support.") to stderr.
That wording is correct for end-users but gave a developer running an
example with a missing NULLRUN_API_KEY zero actionable detail -- they
saw the friendly line, the script exited 1, and they had no way to
tell whether the failure was auth, gate, transport, or config.
New helper nullrun._handle._render_dev_error_report renders a four-line
report answering the four questions a developer actually asks:
1. what -- the stage that failed (auth / gate / track / ...)
2. where -- the wire endpoint + status code + transport source
3. why -- the underlying exception message + machine error_code
4. how to fix -- the user_action from the typed class
Plus an optional fifth docs: line carrying the docs URL.
The catalog format_user_message wording is the headline so end-user-
facing deployments still get a clean single sentence; the structured
detail follows on its own line.
Defensive: handle()/init_or_die() wrap the helper in try/except so a
buggy report builder cannot freeze a script that would otherwise exit.
Falls back to the legacy single-line behaviour.
- src/nullrun/_handle.py:
- _render_dev_error_report(exc, user_message): new -- four-line
structured report. Reads error_code / user_action / retryable /
docs_url / endpoint / status_code / source off the exception.
Caps why: line at 400 chars + ellipsis so a verbose backend
response doesn't blow up the terminal.
- handle(): catches NullRunError, calls _render_dev_error_report,
prints to stderr, sys.exit(exit_code). Defensive fallback to
format_user_message if the helper raises.
- init_or_die(): same path -- a missing API key at startup now
prints the four-line report naming the missing env var, the
dashboard URL to obtain a key, and the docs page, instead of
just the catalog headline.
- tests/test_dev_error_report.py (new, 11 tests):
- test_report_includes_what_where_why_and_fix
- test_report_stage_derived_from_class_name_when_no_endpoint
- test_report_includes_transport_endpoint_and_source
- test_report_truncates_long_underlying_messages
- test_report_omits_fix_line_when_user_action_is_empty
- test_report_includes_docs_url
- test_handle_prints_full_dev_report
- test_guarded_prints_full_dev_report
- test_handle_falls_back_to_legacy_on_helper_bug
- test_handle_report_uses_class_name_for_unknown_endpoint
- test_handle_and_guarded_do_not_require_runtime
11/11 pin tests pass. Wire-format unchanged. SDK_MIN_VERSION unchanged.
…0x with no LLM events (DEF-ZERO-ACTIVITY-DIAG) The user's @Protect gate is enforced, but enforcement without observability means the dashboard reports nothing -- the agent runs, costs nothing, and the operator is left wondering whether NullRun is wired up correctly. Silent-failure mode that surfaces only as "the dashboard shows zero tokens" weeks later. New diagnostic on NullRunRuntime: - _protect_call_count -- bumped by _bump_protect_count(), called from @Protect on every call. - _llm_call_event_count -- bumped by track_llm() on every successful call. - _zero_activity_warned -- flag set when the warning fires, prevents spam on long-lived processes. - _zero_activity_lock -- guard around the read-flag sequence so concurrent @Protect calls cannot produce duplicate warnings. Threshold: 50 @Protect calls without a single LLM-call event is the operational signal. The lock + flag combination makes the warn-once invariant survive concurrent fanout. The warning text names the three most likely root causes so the operator can self-diagnose without consulting external docs immediately: (1) raw httpx client without NullRun's instrumentation patches, (2) custom transport / non-HTTP vendor (gRPC, WebSocket, SDK- internal socket), (3) framework not in the auto-detection table. - src/nullrun/runtime.py: - NullRunRuntime.__init__: initialise the four new attributes. - _bump_protect_count(): atomic int increment + call _maybe_warn_zero_activity(). Cheap; GIL makes the int bump atomic without a lock. - _maybe_warn_zero_activity(): under _zero_activity_lock, return early if already warned or below threshold; otherwise check _llm_call_event_count == 0, set the flag, emit the WARNING. - track_llm(): bump _llm_call_event_count as the canonical "yes we saw an LLM call" answer. Single line, no behaviour change to the rest of track_llm. - tests/test_zero_activity_diagnostic.py (new, 6 tests): - test_no_warning_below_threshold - test_warns_at_threshold_when_no_llm_events - test_warn_once_only - test_no_warning_after_first_llm_event - test_warning_message_mentions_root_causes - test_concurrent_bumps_warn_at_most_once - tests/test_protect.py, tests/test_preflight_fail_policy.py, tests/test_protect_cancel_on_exception.py: _RecordingRuntime stubs extended with a _bump_protect_count no-op so the existing gate / cancel / span test surface keeps working unchanged (the diagnostic itself is exercised by dedicated tests). 6/6 new pin tests pass. Wire-format unchanged. SDK_MIN_VERSION unchanged.
…the canonical path For typical LangGraph usage, nullrun.init_or_die() (or just @Protect on the agent function) auto-patches langgraph.pregel.Pregel via nullrun.instrumentation.auto.patch_langgraph_compiled -- the same callback injection that this wrapper() performs manually. The auto- patch is process-wide and idempotent; wrapper() was a one-app-at-a-time mutation that 90%+ of users no longer need to call. This commit makes wrapper() an explicit escape hatch for three narrow cases instead of the default recommendation: 1. Tests with custom runtimes (the auto-patch binds to the active runtime; a test fixture that swaps runtimes mid-flight may need the wrapper to attach to the new runtime directly). 2. Apps where Pregel is imported BEFORE nullrun.init_or_die() AND the import side-effects register a non-Pregel transport that the auto-patch cannot reach. 3. Manual control over which NullRunCallback instance is attached (rare; the default singleton is usually correct). No behaviour change. The wrapper() public symbol stays -- no removal in 0.18.x; only the docstring + module docstring now describe the auto-patch as the canonical path with the wrapper as the escape hatch. - src/nullrun/toolbox/langgraph.py: - Module docstring: rewritten to position auto-patch as canonical, wrapper() as escape hatch for the three documented cases. - wrapper() docstring: .. deprecated:: block added; usage example moved to a 'if you must use this wrapper' sub-section. Wire-format unchanged. SDK_MIN_VERSION unchanged.
…tc.)
NullRun never imports the vendor SDKs listed in [openai] / [anthropic]
/ [mistral] / [gemini] / [cohere] / [bedrock]. HTTP-level
instrumentation (patch_httpx + 5 URL-keyed extractors) covers OpenAI,
Azure, Anthropic, Mistral, Gemini, Cohere, and Bedrock without those
vendor packages -- all of those vendors route through httpx, and
NullRun parses the response body by URL host. Installing the extras
was dead weight that pulled vendor SDKs into user environments for
no observable benefit.
Kept framework extras ([opentelemetry], [langgraph], [agents],
[langchain], [llama-index], [crewai], [autogen]) -- those still
install a vendor SDK that NullRun subscribes to via an event hook
(langgraph.pregel.Pregel, langchain_core.callbacks.BaseCallbackManager,
agents.Runner, llama_index.core.instrumentation, crewai.Crew,
autogen_agentchat.agents.BaseChatAgent).
Also dropped:
- [fastapi]: the integrations submodule is lazy-imported; users
who don't use FastAPI don't pay its install cost, and
nullrun.integrations.__init__ doesn't change.
- [all]: the convenience meta-extra was a copy of the above; with
provider extras gone, [all] is now identical to the framework-
extras list and was removed.
Result: 'pip install nullrun' alone is now sufficient for the HTTP-
level + @Protect flow. Framework auto-instrumentation activates
transparently when the user installs any one of the framework extras.
- pyproject.toml [project.optional-dependencies]:
- Removed: openai, anthropic, mistral, gemini, cohere, bedrock,
fastapi, all.
- Kept + expanded comment: opentelemetry, langgraph, agents,
langchain, llama-index, crewai, autogen.
Wire-format unchanged. SDK_MIN_VERSION unchanged.
…zy-trigger note README: new 'Framework adapters -- auto-detected' subsection enumerates the six framework adapters (LangGraph / LangChain / OpenAI Agents / LlamaIndex / CrewAI / AutoGen) with their patch point + trigger. Makes explicit that auto-detection activates when init_or_die() runs (or when @Protect first fires) -- the user does NOT need to choose which extra to install. Also clarifies the lazy-trigger contract: 'If you call @Protect before init_or_die(), the SDK auto-triggers instrumentation lazily on the first decorated call.' This is the user-facing version of the 0.18.1 lazy-init change that landed in fcd623c. docs/errors/NR-C001.md: reflow for the 0.18.1 lazy-trigger semantics. NR-C001 now surfaces at the first @Protect call (runtime is created lazily on first gate call from the environment), not at init(). The typed exception class is documented as NullRunConfigError (was historically NullRunAuthenticationError for back-compat). The 'When' section is rewritten to describe the new trigger point; the 'Why this raises' section adds a 0.18.1 note explaining what changed (when, not what); the catch-pattern example uses @nullrun.protect instead of nullrun.init() to match the new code surface; the 'See also' list adds NR-C004 (nullrun.status() called before the runtime is bound). - README.md: new subsection after the 'Quick start' block; existing 'How NullRun compares' section unchanged. - docs/errors/NR-C001.md: 'When', 'Why this raises', 'How to fix', 'Catch pattern', 'See also' all updated; 'Default user_action' row in the header table updated to surface NullRunConfigError as the typed class. No code change. No behaviour change.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Patch release 0.18.1 — lower-friction UX. Closes four silent-failure modes at once:
@protectauto-attaches a default tool-params extractor (fcd623c). The first call from@protectstampsToolParamsExtractor(include_all=True)on the decorated function so the wire payload carriestool_name + paramsfor every protected call — no separate@sensitiveneeded for the common case. Bare@sensitiveis now deprecated (DeprecationWarningin 0.18.x, removed in 0.19.x);@sensitive(impact=...)remains the advanced API for typedBusinessImpact+ SHA-256action_digest.@protectlazy-triggersauto_instrument()on first invocation. The user can write@protectbeforeinit_or_die()(or skipinitentirely ifNULLRUN_API_KEYis set); the runtime is created on first gate call and patcheshttpx+ framework adapters in a single process-wide idempotent step. Closes the "I added@protectbut nothing tracks tokens" silent-failure mode.Zero-activity diagnostic on
NullRunRuntime(DEF-ZERO-ACTIVITY-DIAG). When@protectfires 50+ times without a single LLM event, the runtime emits a one-time WARNING atlogging.WARNINGnaming the three most likely root causes (raw httpx outside the patchable surface, custom transport / gRPC, framework not on the auto-detection table). Warn-once invariant under concurrent@protectcalls is preserved by_zero_activity_lock.Four-line developer error report from
handle()/guarded()/init_or_die()(DEF-DEV-REPORT-EMPTY). The catch-all exit path previously printed only the catalog user-message — end-user wording that gave a developer zero actionable detail. The new_render_dev_error_report()helper emits a structured report answering the four questions a developer actually asks:what(stage that failed) /where(wire endpoint + status + transport source) /why(underlying exception message + machineerror_code) /how to fix(typed classuser_action). Defensivetry/exceptfallback to the legacy single-line behaviour so a buggy helper cannot freeze a script that would otherwise exit.Plus two cleanups:
Removed dead
pipextras ([openai],[anthropic],[mistral],[gemini],[cohere],[bedrock],[all],[fastapi]). NullRun never imported these vendor SDKs — HTTP-level instrumentation (patch_httpx+ 5 URL-keyed extractors) covers OpenAI / Azure / Anthropic / Mistral / Gemini / Cohere / Bedrock without them.pip install nullrunalone is now sufficient for the HTTP-level +@protectflow. Framework extras stay.toolbox.langgraph.wrapper()marked DEPRECATED in its docstring.init_or_die()/@protectauto-patcheslanggraph.pregel.Pregelviapatch_langgraph_compiled— same callback injection the wrapper does manually, but process-wide and idempotent.wrapper()remains as an escape hatch for three narrow cases (tests with custom runtimes, Pregel imported before init, manual callback control). No removal — the public symbol stays.Wire-format unchanged. SDK_MIN_VERSION unchanged.
Added
DEF-ZERO-ACTIVITY-DIAG — zero-activity diagnostic on
NullRunRuntime(src/nullrun/runtime.py,0d6a6b9)._bump_protect_count()is invoked by@protecton every call;_llm_call_event_countis bumped bytrack_llm()on every successful call. When@protectfires 50+ times without a single LLM event, the runtime emits a one-time WARNING atlogging.WARNINGnaming the three most likely root causes. Warn-once invariant under concurrent@protectcalls is preserved by_zero_activity_lock. 6 new tests intests/test_zero_activity_diagnostic.pypin the warn-once / threshold / concurrent-bump / message-content invariants.DEF-DEV-REPORT-EMPTY — four-line developer error report from
handle()/guarded()/init_or_die()(src/nullrun/_handle.py,cc79241). New helper_render_dev_error_report()readserror_code/user_action/retryable/docs_url/endpoint/status_code/sourceoff the exception, caps thewhy:line at 400 chars + ellipsis, and emits a structured report answeringwhat/where/why/how to fix/docs:questions. The catalog headline is preserved as the first line so end-user-facing deployments still get a clean single sentence. Defensivetry/exceptfallback inhandle()/init_or_die()so a buggy helper cannot freeze a script that would otherwise exit. 11 new tests intests/test_dev_error_report.pypin the four-line / what / where / why / how-to-fix / docs-URL invariants.@protectlazy-triggersauto_instrument()on first invocation (src/nullrun/decorators.py,e62718fpart offcd623c). The first decorated function call installs the runtime and patcheshttpx+ framework adapters in a single process-wide idempotent step, behind a lock so concurrent@protectcalls cannot double-fire. Never raises — a vendor SDK breaking change must not block the enforcement gate. Closes the "I added@protectbut nothing tracks tokens" silent-failure mode.@protectauto-attaches a default tool-params extractor (src/nullrun/decorators.py+src/nullrun/extractor.py,e62718fpart offcd623c). The first call from@protectstampsToolParamsExtractor(include_all=True)on the decorated function so the wire payload carriestool_name + paramsfor every protected call. The auto-attached extractor carries_nullrun_auto_attached=Trueso_enforce_sensitive_toolcan distinguish "developer opted into the policy path" from "SDK auto-derived for tooling reasons"; the policy gate short-circuits on auto-attached extractors so bare@protectstays cheap (no extra/executeround-trip). Bounded extraction: oversized string values (>=1024 bytes) get a deterministic...[truncated:N bytes]suffix; circular references return the partial walk instead of raisingRecursionError; dropped values (float, bytes, custom) emit a single aggregate DEBUG log line with count + type names, never one log line per dropped field. 9 new tests intests/test_protect_only_public_api.pypin the auto-attach / chain-walk / bare-@sensitive-deprecation invariants.Changed
Removed dead provider extras from
pyproject.toml(ab8c183):[openai],[anthropic],[mistral],[gemini],[cohere],[bedrock],[all],[fastapi]. NullRun never imported these vendor SDKs — HTTP-level instrumentation (patch_httpx+ 5 URL-keyed extractors) covers OpenAI, Azure, Anthropic, Mistral, Gemini, Cohere, and Bedrock without them. Kept framework extras ([opentelemetry],[langgraph],[agents],[langchain],[llama-index],[crewai],[autogen]) — those still install a vendor SDK that NullRun subscribes to via an event hook.pip install nullrunalone is now sufficient for the HTTP-level +@protectflow.toolbox.langgraph.wrapper()marked DEPRECATED in its docstring (9a0c2de,src/nullrun/toolbox/langgraph.py).init_or_die()/@protectauto-patcheslanggraph.pregel.Pregelviapatch_langgraph_compiled— same callback injection the wrapper does manually, but process-wide and idempotent.wrapper()remains as an escape hatch for three narrow cases (tests with custom runtimes, Pregel imported before init, manual callback control). No removal — the public symbol stays.Bare
@sensitiveemitsDeprecationWarning(e62718fpart offcd623c,src/nullrun/decorators.py).@sensitive(impact=...)remains the advanced API for typedBusinessImpact+ SHA-256action_digest. Bare@sensitiveis removed in0.19.x; emit isDeprecationWarningin0.18.xonly.README — new
Framework adapters — auto-detectedsubsection enumerates the six framework adapters (LangGraph / LangChain / OpenAI Agents / LlamaIndex / CrewAI / AutoGen) with their patch point + trigger (c7029cb). Makes explicit that auto-detection activates wheninit_or_die()runs (or when@protectfirst fires) — the user does NOT need to choose which extra to install. Also clarifies the lazy-trigger contract:If you call @protect before init_or_die(), the SDK auto-triggers instrumentation lazily on the first decorated call.docs/errors/NR-C001.md— reflow for the 0.18.1 lazy-trigger semantics (c7029cb). NR-C001 now surfaces at the first@protectcall (runtime is created lazily on first gate call from the environment), not atinit(). The typed exception class is documented asNullRunConfigError(was historicallyNullRunAuthenticationErrorfor back-compat).Tests
tests/test_zero_activity_diagnostic.py(new, 6 tests) — pins the warn-once / threshold / concurrent-bump / message-content invariants on the zero-activity diagnostic.tests/test_dev_error_report.py(new, 11 tests) — pins the four-line / what / where / why / how-to-fix / docs-URL invariants on the new error report.tests/test_protect_only_public_api.py(new, 9 tests, landed infcd623c) — pins the auto-attach / chain-walk / bare-@sensitive-deprecation invariants on the@protectcontract change.tests/test_protect.py,tests/test_preflight_fail_policy.py,tests/test_protect_cancel_on_exception.py—_RecordingRuntimestubs extended with a_bump_protect_countno-op so the existing gate / cancel / span test surface keeps working unchanged.Verification
ruff check src testsmypy src/nullrunpytest -qfcd623c, 11 fromtest_dev_error_report.py, 6 fromtest_zero_activity_diagnostic.py)nullrun.__version__0.18.1dist_local/, no*.defect*)Commits included