Skip to content

feat(tools): add in-tree local OpenAI-compatible typed-decision provider - #1551

Merged
potiuk merged 5 commits into
apache:mainfrom
liwenjie200543:feat/typed-decision-local
Oct 9, 2026
Merged

potiuk merged 5 commits into
apache:mainfrom
liwenjie200543:feat/typed-decision-local

Conversation

@liwenjie200543

Copy link
Copy Markdown
Contributor

Summary

  • Implements feat(adapter/typed-decision-local): local model backend for contract:typed-decision (llama.cpp / Ollama / vLLM) #1431: a second DecisionProvider backend for contract:typed-decision targeting self-hosted OpenAI-compatible runtimes — Ollama (:11434/v1, the default), llama.cpp server (:8080/v1), and vLLM (:8000/v1) — so structured decisions can run offline with zero third-party egress.
  • New in-tree local provider (providers/local.py, stdlib-only urllib.request), registered alongside jev; the default resolution path is unchanged (unset MAGPIE_TYPED_DECISION_PROVIDER still prefers a configured jev). jev.py itself is untouched.
  • Loopback privacy posture (the boundary raised in the feat(adapter/typed-decision-local): local model backend for contract:typed-decision (llama.cpp / Ollama / vLLM) #1431 discussion): plain HTTP is only allowed for loopback hosts, which the privacy-LLM gate default-approves; environment proxies (HTTP_PROXY / HTTPS_PROXY) are disabled for every request and redirects are rejected, so a loopback URL stays loopback for the whole round trip.
  • Every operation sends a strict response_format: json_schema (the choice label pinned to the candidate options via enum) and re-validates the parsed decision against the contract — label ∈ options, scale bounds, [0.0, 1.0] ranges, booleans and non-finite numbers rejected — failing open with TypedDecisionUnavailable exactly like jev.py.

Type of change

  • Python package (tools/*/ with pyproject.toml)
  • Documentation (docs/, README.md, CONTRIBUTING.md)

Test plan

  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • Other: ruff format --check clean. 33 new cases in tools/typed-decision/tests/test_local_provider.py, including an end-to-end regression with a real local HTTP server: with HTTP_PROXY / HTTPS_PROXY set to a dead port, the request lands on the loopback server directly (proving the proxy bypass) and its 302 to a non-loopback host raises TypedDecisionUnavailable instead of being followed.

RFC-AI-0004 compliance

  • Sandbox — no new unrestricted host access: loopback default, non-loopback HTTPS hosts still need an explicit opt-in entry; env proxies blocked, redirects rejected.
  • Vendor neutrality — a second independent backend for contract:typed-decision; adapters registry and the hand-maintained vendor-neutrality tables updated.
  • Privacy LLM — the full outbound message content (prompt + appended task instruction) is assembled first and passed through enforce_privacy_gate before the request is built.

Linked issues

Closes #1431

Notes for reviewers

  • The generated vendor-neutrality score block is deliberately unchanged: the scorer counts a tool's **Vendor:** line, so contract:typed-decision still reports "needs 1 more" even though a local backend now ships in-tree. Whether self-hosted runtimes count as a second vendor is a policy call I didn't want to make unilaterally — the shipped adapter is recorded in the hand-maintained tables (docs/vendor-neutrality.md, docs/adapters/registry.md) instead.
  • task_instruction (the scale bounds for score, the option set for choice) travels in the message content because a generic chat model has no structured operation field; the strict JSON schema constrains the shape, and the bounds are re-validated client-side after parsing.
  • Structured-output requirement: Ollama, llama.cpp server, and vLLM all support response_format: json_schema; a runtime that does not simply yields TypedDecisionUnavailable (fail-open), never a degraded parse. Content fenced in ```-blocks (some runtimes emit it even under a JSON schema) is unwrapped before parsing.

Gen-AI disclosure

Authored with ZCode (GLM) under
docs/ai-contribution-policy.md;
every line reviewed and the checks above run locally by me before posting.

Implements a second DecisionProvider backend for contract:typed-decision
targeting self-hosted OpenAI-compatible runtimes (Ollama, llama.cpp
server, vLLM), per apache#1431:

- strict response_format json_schema per operation (label enum pinned to
  the candidate options), with contract-level revalidation (label in
  options, scale bounds, [0,1] ranges, booleans and non-finite numbers
  rejected) and fail-open semantics identical to the Jev provider
- loopback privacy posture: plain HTTP only for loopback hosts,
  HTTP(S)_PROXY env proxies disabled, redirects rejected — a loopback URL
  stays loopback for the whole request; pinned by an end-to-end test with
  a real local server, proxies set, and a 302 to a non-loopback host
- optional Bearer key for authenticated local runtimes; model name via
  MAGPIE_TYPED_DECISION_LOCAL_MODEL (required) and endpoint via
  MAGPIE_TYPED_DECISION_LOCAL_ENDPOINT (defaults to Ollama)

The adapter registry and the hand-maintained vendor-neutrality tables now
list the shipped local provider. The generated neutrality score itself is
unchanged (the scorer counts the tool's Vendor line, so crediting local
runtimes as a second backend vendor is a policy question left for
maintainers).
The counts were zeroed by a scorer side effect on a non-POSIX
checkout (skill frontmatter scan found no files); restore the
regenerated values: 79/79, 22/57, ASF = 16, agnostic = 63.

@Kaap10 Kaap10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed and verified locally against tools/typed-decision.

  • Vendor Neutrality: Clean in-tree local backend for contract:typed-decision using standard library urllib.request with zero third-party dependencies.
  • Privacy & Security: Solid loopback privacy posture — disabling environment proxy resolution (HTTP_PROXY/HTTPS_PROXY) and blocking 302 redirects ensures zero egress leakage.
  • Fail-Open Discipline: Strict handling of non-finite numbers, booleans, markdown fences, and schema errors falling back cleanly to TypedDecisionUnavailable.
  • Tests: Ran the full test suite locally — all 34 unit tests in test_local_provider.py, mypy, and ruff pass cleanly.

LGTM!

@onlyarnav onlyarnav left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer Review Summary

Thank you for this high-quality contribution, @liwenjie200543!

The overall architecture and implementation of the local provider are solid:

  • Zero Third-Party Dependencies: Pure standard library urllib.request implementation without adding heavyweight SDKs.
  • Robust Privacy & Loopback Posture: Disabling environment proxies via ProxyHandler({}) and blocking HTTP redirects with NoRedirectHandler ensures loopback traffic strictly remains on loopback throughout the request lifecycle.
  • Fail-Open Discipline: Strict type checking (rejecting booleans as floats), scale validation, and exception handling cleanly map to TypedDecisionUnavailable.
  • Comprehensive Test Suite: Excellent unit test hygiene with the live loopback proxy-bypass regression test.

I have flagged a few minor improvements inline on local.py:

  1. Explicit temperature: 0.0 (L387): Enforce zero temperature in the chat completion payload to ensure deterministic classification, scoring, and probability estimates across local runtimes (which otherwise default to 0.7–1.0).
  2. Reasoning model / preamble fence extraction (L219): Consider stripping <think> tags and matching markdown code fences across the message body rather than requiring text.startswith("```"), to accommodate models that emit reasoning tokens or preambles.
  3. Defensive hostname normalization (L85): Normalize (parsed.hostname or "").lower() defensively when validating loopback endpoints.

Once these minor items are addressed, this is in great shape to merge!

Comment thread tools/typed-decision/src/typed_decision/providers/local.py
Comment thread tools/typed-decision/src/typed_decision/providers/local.py Outdated
Comment thread tools/typed-decision/src/typed_decision/providers/local.py Outdated
potiuk added 3 commits October 7, 2026 15:52
Generated-by: Copilot SDK in VS Code
Generated-by: Copilot SDK in VS Code
The local provider's transport check kept its own copy of the loopback
hosts. Import the privacy checker's set instead, so the transport check
and the gate cannot disagree about which hosts count as local-only.

Generated-by: Claude Opus 5

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. A clean, stdlib-only second backend for contract:typed-decision, with a privacy posture that holds end to end: loopback-only plain HTTP, no environment proxies, no redirects, and the gate sees the full outbound content. CI is green and the branch merges cleanly with current main.

@onlyarnav's three points (temperature 0.0, reasoning tags and preambles before a fenced decision, case-insensitive loopback host) are addressed in c468014 with test coverage, and the threads are resolved. I'm merging on that basis.

On your vendor-neutrality question: yes, an in-tree backend that runs against any self-hosted OpenAI-compatible runtime is independent of TypeSafe and should count as a second vendor. The scorer currently reads one **Vendor:** line per tool README, so crediting a second provider that lives in the same tool needs a scorer change. That will be a follow-up, not something to fix here.

I pushed one small fixup on top (de4bd0a): the provider now imports the privacy checker's loopback host set instead of keeping its own copy, so the transport check and the gate cannot drift apart.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
Contributing guide.

@potiuk
potiuk merged commit 75c057d into apache:main Oct 9, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(adapter/typed-decision-local): local model backend for contract:typed-decision (llama.cpp / Ollama / vLLM)

4 participants