[Nadeesha D Shalom (Nadeesha-D-Shalom)]

add(marketplace): load live local catalogs directly from apm.yml

TL;DR

Allow apm marketplace add ./apm.yml --name alice-dev to register an explicitly selected local authoring catalog without generating JSON first.
Later explicit operations reread that file, while each invocation uses one configuration snapshot.
Selected installs retain the actual package dependency and existing lockfile semantics.

Issue and approved scope

Fixes #2729.

Human approval: #2729 (comment)

Review contact: Daniel Meppiel (@danielmeppiel).

This implements the bounded local-authoring change.
Registry-backed authoring, durable catalog-following intent, and the separate enterprise API conformance scenario remain with their respective issues.

Important

Registration creates a local discovery source. It does not publish packages, install inventory, activate plugins, or run package hooks.
Local paths and hashes remain local dependencies; they do not distribute files to other machines.

Problem (WHY)

• Authoring already lives in YAML, but registration previously attempted to decode the file as JSON. The canonical issue states that "Marketplace metadata and package entries can be declared in apm.yml".
• Requiring an intermediate generated catalog adds a separate step and stale-output risk: the issue describes generated output becoming "stale relative to apm.yml".
• The change is backed by executable scenarios rather than the example command alone, following "Instruct the agent to validate its own work before moving on.".

Approach (WHAT)

• Recognize only an explicitly selected local apm.yml; retain existing source classification for JSON, Git and hosted JSON.
• Reuse canonical authoring validation, ref resolution, catalog mapping and path containment.
• Cache the successful in-memory catalog only in the current Click invocation.
• Expose bounded --offline behavior on marketplace add and marketplace browse.

Implementation (HOW)

Area	Change
src/apm_cli/marketplace/authoring_source.py	Coordinates existing owners, validates author-relative paths, retains the invocation snapshot, and translates failures into actionable marketplace errors.
src/apm_cli/marketplace/client.py	Routes explicitly selected local authoring files to that adapter; preserves the existing catalog cache/fetch paths for other sources.
src/apm_cli/marketplace/builder.py	Composes a discovery view using local metadata and existing mappers; keeps native output configuration intact and avoids credential initialization for local-only resolution.
src/apm_cli/commands/marketplace/__init__.py	Adds examples and bounded offline options; validates offline source eligibility before probing. Existing registration still occurs only after a successful fetch.
.apm/architecture/owners/marketplace-plugins.json and scripts/architecture_linter/checks/marketplace_catalog_and_contract.py	Register and enforce canonical routing and the no-writing boundary.
tests/unit/marketplace/test_authoring_source.py	Covers snapshots, containment, offline resolution, native-output stability, metadata and local-only credential avoidance.
tests/integration/test_marketplace_authoring_lifecycle.py	Exercises the installed CLI and packaged executable using owned local packages and Git origins. Included in the existing lifecycle_smoke selection.
tests/integration/test_architecture_authoring_source.py	Confirms the registered guard rejects an injected packing call.
Marketplace CLI, producer and consumer documentation; packages/apm-guide/.apm/skills/apm-usage/commands.md	Document live-source behavior, offline limits, side effects and portability boundaries.

The dashed nodes identify the new local input and its in-memory discovery view; existing consumer and lockfile owners remain downstream.

flowchart LR
    A[Explicit local apm.yml] --> B[Canonical authoring parser]
    B --> C[Canonical ref resolver]
    C --> D[In-memory discovery composer]
    D --> E[Existing marketplace consumer]
    E --> F[Browse or view]
    E --> G[Install selected actual dependency]
    G --> H[Existing manifest and lockfile]
    I[JSON or Git or HTTPS catalog] --> E
    classDef added stroke-dasharray:5 5
    class A,D added

Loading

The diagram was rendered successfully with mmdc.

Trade-offs

• Discovery does not fetch cosmetic manifests for every remote package. Explicit curator fields and existing local metadata remain available; remote ref resolution is still explicit and can require network access.
• --offline is limited to this local YAML path. Full commit SHAs and local entries work without network; tags and ranges are not persisted for later offline invocations.
• Native pluginRoot layout hints are omitted from the temporary discovery view so authored paths remain relative to the selected file. Native generated catalogs retain their original representation, verified before and after discovery composition.
• Snapshots are invocation-local, with no watcher or persistent generated preview. A later explicit operation validates the current file again and reports edits or deletion.

Benefits

1. Register and browse a valid local authoring file without running apm pack.
2. Install one selected package without installing an unselected package's inventory or executable hooks.
3. Replay a remote dependency from its lock in a fresh configuration scope after removing the author's preview directory.

Validation

Python 3.12.15 and dependencies installed from the unchanged uv.lock with uv sync --frozen --extra dev --extra build.

Marketplace regression suite and new architecture guard:

1785 passed, 5 deselected in 90.29s (0:01:30)

The five deselections are symlink-related cases requiring privileges unavailable to this Windows account. They are not disabled in the committed suite.

PyInstaller 6.17.0 packaged executable, selected through APM_BINARY_PATH:

8 passed in 38.87s

Full Ruff lint and formatting:

All checks passed!
2095 files already formatted

Full duplication check (pylint R0801):

Your code has been rated at 10.00/10

Architecture linter exited 0. Auth and assertion/duplicate ratchets:

[+] auth-signal lint clean
[+] assertion-quality ratchet clean: AQ001=4, AQ002=12
[+] exact test duplicate ratchet clean: 1418 files, 0 allowed duplicate group(s)

Documentation build and generated-page link checks:

20:55:52 [build] 126 page(s) built in 43.50s
[+] Checked 1113 relative link(s) across generated pages. No broken relative links found.

git diff --check passed. These are local Windows results; required GitHub checks and maintainer review still apply.

Scenario Evidence

#	Scenario (user promise)	Principle(s)	Test(s) proving it	Type
1	Register, browse and view the authoring file from another directory without changing author or consumer files.	DevX (pragmatic as npm), Secure by default	tests/integration/test_marketplace_authoring_lifecycle.py::test_live_local_catalog_register_browse_view_install_and_edit	e2e
2	See an edited catalog on the next operation and install only the selected nested local package.	DevX (pragmatic as npm), Portability by manifest	Same installed-CLI lifecycle scenario as row 1.	e2e
3	Reject missing, malformed or unusable YAML without replacing an existing registration; reject later broken live reads.	Secure by default	tests/integration/test_marketplace_authoring_lifecycle.py::test_failed_reregistration_preserves_registered_source	e2e
4	Install the actual remote Git package and replay without the preview directory or its registration.	Vendor-neutral, Governed by policy	tests/integration/test_marketplace_authoring_lifecycle.py::test_remote_entry_records_real_source_and_replays_without_preview	e2e
5	Continue registering existing JSON files and Git catalogs without authoring ref resolution.	Vendor-neutral, DevX (pragmatic as npm)	tests/integration/test_marketplace_authoring_lifecycle.py::test_existing_catalog_sources_still_register_without_authoring_resolution	e2e
6	Keep nested and bare authored package paths anchored to the authoring file (#2729 regression trap).	Secure by default, Portability by manifest	tests/unit/marketplace/test_authoring_source.py::test_plugin_root_never_rebases_authored_package_paths	component
7	Preserve native generated-output semantics after browsing a local authoring preview.	Multi-harness support	tests/unit/marketplace/test_authoring_source.py::test_discovery_composition_does_not_change_native_output	component
8	Avoid remote credential lookup for a local-only catalog and avoid network ref lookup for pinned offline commits.	Secure by default, DevX (pragmatic as npm)	tests/unit/marketplace/test_authoring_source.py::test_local_catalog_does_not_resolve_remote_credentials; ::test_pinned_remote_commit_needs_no_ref_or_metadata_network	component
9	Keep one configuration snapshot within an invocation, then reread next time.	Governed by policy	tests/unit/marketplace/test_authoring_source.py::test_invocation_snapshot_is_shared_but_next_invocation_rereads	component

How to test

• Install the locked development environment: uv sync --frozen --extra dev --extra build.
• Run uv run --frozen pytest tests/unit/marketplace tests/integration/test_architecture_authoring_source.py; expect the marketplace suite to pass on the CI platform.
• Run uv run --frozen pytest tests/integration/test_marketplace_authoring_lifecycle.py; expect all eight scenarios to pass using the installed CLI. For packaged validation, build with build/apm.spec, set APM_BINARY_PATH to the resulting executable, and repeat.
• Register a valid local catalog with apm marketplace add ./apm.yml --name alice-dev, edit it, and browse again. Expect current entries with no generated catalog writes; a malformed edit must fail without replacing registration.
• Run the repository Ruff, pylint duplication, auth/architecture and test-quality guards, then cd docs && npm ci && npm run build; expect clean checks and valid relative links.

Type of change and testing

• New feature
• Documentation
• Tested locally, including the packaged executable
• Added behavioral and architecture regression tests
• Required GitHub checks and maintainer review completed

Spec conformance

• N/A: no normative requirement, public schema or lockfile format changes. This adds a local authoring input to existing registration and package-source contracts.

[microsoft-github-policy-service]

Nadeesha D Shalom (@Nadeesha-D-Shalom) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your
contributions to Microsoft open source projects. This Agreement is effective as of the latest signature
date below.

1. Definitions.
   “Code” means the computer software code, whether in human-readable or machine-executable form,
   that is delivered by You to Microsoft under this Agreement.
   “Project” means any of the projects owned or managed by Microsoft and offered under a license
   approved by the Open Source Initiative (www.opensource.org).
   “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
   Project, including but not limited to communication on electronic mailing lists, source code control
   systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
   discussing and improving that Project, but excluding communication that is conspicuously marked or
   otherwise designated in writing by You as “Not a Submission.”
   “Submission” means the Code and any other copyrightable material Submitted by You, including any
   associated comments and documentation.
2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any
   Project. This Agreement covers any and all Submissions that You, now or in the future (except as
   described in Section 4 below), Submit to any Project.
3. Originality of Work. You represent that each of Your Submissions is entirely Your original work.
   Should You wish to Submit materials that are not Your original work, You may Submit them separately
   to the Project if You (a) retain all copyright and license information that was in the materials as You
   received them, (b) in the description accompanying Your Submission, include the phrase “Submission
   containing materials of a third party:” followed by the names of the third party and any licenses or other
   restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
   guidelines concerning Submissions.
4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else
   for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
   Submission is made in the course of Your work for an employer or Your employer has intellectual
   property rights in Your Submission by contract or applicable law, You must secure permission from Your
   employer to make the Submission before signing this Agreement. In that case, the term “You” in this
   Agreement will refer to You and the employer collectively. If You change employers in the future and
   desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
   and secure permission from the new employer before Submitting those Submissions.
5. Licenses.

• Copyright License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
  Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
  the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
  parties.
• Patent License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
  Your patent claims that are necessarily infringed by the Submission or the combination of the
  Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
  import or otherwise dispose of the Submission alone or with the Project.
• Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
  No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
  granted by implication, exhaustion, estoppel or otherwise.

6. Representations and Warranties. You represent that You are legally entitled to grant the above
   licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
   have disclosed under Section 3). You represent that You have secured permission from Your employer to
   make the Submission in cases where Your Submission is made in the course of Your work for Your
   employer or Your employer has intellectual property rights in Your Submission by contract or applicable
   law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
   have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
   You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
   REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
   EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
   PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
   NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
7. Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
   You later become aware that would make Your representations in this Agreement inaccurate in any
   respect.
8. Information about Submissions. You agree that contributions to Projects and information about
   contributions may be maintained indefinitely and disclosed publicly, including Your name and other
   information that You submit with Your Submission.
9. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
   the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
   Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
   exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
   defenses of lack of personal jurisdiction and forum non-conveniens.
10. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
    supersedes any and all prior agreements, understandings or communications, written or oral, between
    the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

[Copilot]

🟢 Approval recommended

No blocking issues were identified, and focused regression coverage supports the canonical routing, containment, snapshot and compatibility behavior.

No open issues 🎉

What changed in this PR

Implements #2729 by allowing explicitly selected local apm.yml files to serve as live marketplace catalogs, reusing existing authoring and installation machinery without generating JSON.

Changes:

• Adds invocation-local catalog snapshots, canonical validation and path containment, with fresh reads on later operations.
• Adds bounded --offline support for registration and browsing while preserving existing catalog and dependency workflows.
• Adds regression tests, architecture guards and documentation for local previews.

File	Description
tests/​unit/​marketplace/​test_authoring_source.py	Tests snapshots, containment, offline behavior and native-output preservation.
tests/​integration/​test_marketplace_authoring_lifecycle.py	Covers discovery, selected installation, replay and existing-source compatibility.
tests/​integration/​test_architecture_authoring_source.py	Tests rejection of packing calls in the read-only adapter.
src/​apm_cli/​marketplace/​client.py	Routes explicitly selected local YAML catalogs to the adapter.
src/​apm_cli/​marketplace/​builder.py	Composes discovery catalogs and avoids local-only credential initialization.
src/​apm_cli/​marketplace/​authoring_source.py	Coordinates validation, resolution, containment and invocation snapshots.
src/​apm_cli/​commands/​marketplace/​__init__.py	Adds local YAML examples and bounded offline options.
scripts/​architecture_linter/​checks/​marketplace_catalog_and_contract.py	Enforces canonical routing and the adapter's no-writing boundary.
packages/​apm-guide/​.apm/​skills/​apm-usage/​commands.md	Documents live previews and offline limits.
docs/​src/​content/​docs/​reference/​cli/​marketplace.md	Explains command behavior, snapshots and dependency portability.
docs/​src/​content/​docs/​producer/​publish-to-a-marketplace.md	Adds the producer's direct local-preview workflow.
docs/​src/​content/​docs/​consumer/​installing-from-marketplaces.md	Explains discovery and installation from local authoring catalogs.
.apm/​architecture/​owners/​marketplace-plugins.json	Registers ownership and the guard for live authoring catalogs.

_{Review effort: Balanced}