Document conditional dependencies and installable extras

articles/_release-notes/v0.13.x.md

@@ -6,6 +6,74 @@ list_title: Versions 0.13.x
 
 # Release Notes - Versions 0.13.x
 
+## Version 0.13.23
+
+### New Features
+* The same distribution may now be declared more than once under mutually
+  exclusive environment markers, so that `foo==1.0` on one interpreter and
+  `foo==2.0` on another is expressible. Markers are now part of a dependency's
+  identity, and validation accepts a group of same-named declarations as long as
+  no two of them apply at once. Identical conditions on one name remain an error,
+  since no environment can tell them apart.
+* Extras groups declared with `depends_on(..., extra=...)` can now be installed
+  into the VEnvs PyBuilder builds, so that the code an extra guards can actually
+  be tested. The new `install_dependencies_extras` property selects them: a list
+  of names, a single name, or `"*"` for every declared group. It defaults to
+  `[]`, which is the previous behavior. Selecting a group the project does not
+  declare fails the build and lists the ones it does.
+* Dependencies are now selected against the marker environment of the target
+  interpreter rather than the one running the build. `PythonEnv.marker_env`
+  carries the full set of PEP 508 variables, probed from that interpreter with
+  PyBuilder's vendored *packaging*, so a VEnv built from a different Python
+  resolves its own conditional dependencies correctly.
+* `list_dependencies` now prints the declared extras groups, each marked with
+  whether the current selection installs it. `pyb -i` reports the group a
+  dependency belongs to under the new `extra` key.
+
+### Changed Features
+* `project.dependencies` now returns the base runtime dependencies plus whatever
+  `install_dependencies_extras` selects, which is what every install and
+  consumption site wants. The new `project.base_dependencies` returns the base
+  ones alone, and is what the generated `setup.py` uses for `install_requires`
+  and `dependency_links` - a build-time decision to test an extra must never
+  become a mandatory requirement of the published distribution. Plugins that
+  render distribution metadata from `project.dependencies` should move to
+  `project.base_dependencies`.
+
+### Bugs Fixed
+* A dependency declared twice with different markers was silently discarded.
+  Same name and version with different conditions collided on identity, the
+  second declaration was dropped, and nothing was reported - so a Windows-only
+  dependency would disappear because a Linux-only one happened to be declared
+  first.
+* Declaring a conditional pin was a hard build failure. `validate_dependencies`
+  counted by name alone, so the correct way to express a per-interpreter version
+  broke the build.
+* Dependencies whose markers do not apply are no longer installed, and no longer
+  written to the constraints file. They were queued on every build, which
+  defeated the "already up-to-date, skipped" path; and where two entries on one
+  name were both live, pip intersected them into a `ResolutionImpossible` that
+  named neither declaration. PyBuilder now reports such a conflict itself,
+  naming both.
+* `depends_on("foo[security]")` was a silent no-op on incremental builds.
+  Nothing in installed metadata records which extras were requested, so with
+  plain `foo` already installed at a satisfying version the extra was skipped and
+  its requirements never arrived. PyBuilder now resolves what the extra requires
+  from the installed distribution's own metadata and verifies those are present,
+  falling back to handing the dependency to pip when the installed version no
+  longer offers the extra at all.
+* A dependency carrying environment markers generated a syntactically invalid
+  `setup.py`, because the rendered requirement was interpolated into single
+  quotes and markers are conventionally written with single quotes of their own:
+  `'pywin32>=300; sys_platform == 'win32''`. Building the distribution failed
+  with a `SyntaxError` from the backend. Values are now rendered as proper Python
+  literals, which also fixes any other string containing a quote or a backslash.
+* `install_requires` dropped a dependency's own extras, so
+  `depends_on("foo[bar]")` was published as plain `foo`.
+* Extras group names are now normalized, so that `extra="Security"` and
+  `extra="security"` are one group rather than two emitted separately into
+  `extras_require`.
+
 ## Version 0.13.22
 
 ### New Features

documentation/coding-agents.md

@@ -117,8 +117,24 @@ def initialize(project):
     # From a requirements file
     project.depends_on_requirements("requirements.txt")
     project.build_depends_on_requirements("requirements-dev.txt")
+
+    # Conditional dependency. The same distribution may be declared more than once
+    # as long as no two declarations apply in the same environment
+    project.depends_on("numpy", "==1.26.4", markers="python_version < '3.12'")
+    project.depends_on("numpy", "==2.1.0", markers="python_version >= '3.12'")
+
+    # Optional dependency, published under extras_require
+    project.depends_on("cryptography", ">=42", extra="security")
+
+    # Install that group into the build and test venvs so its code path is tested.
+    # Accepts a name, a list of names, or "*" for every declared group
+    project.set_property("install_dependencies_extras", ["security"])
 ```
 
+Extras selected with `install_dependencies_extras` are installed into the venvs but are
+*not* published as mandatory requirements — they stay in `extras_require`. Naming a
+group the project does not declare fails the build.
+
 ### Writing Tests
 
 Unit tests go in `src/unittest/python/` and must match the glob `*_tests.py`:
@@ -252,6 +268,9 @@ If directories are customized, specify the actual paths:
 Edit `build.py` and add to the initializer:
 - Runtime: `project.depends_on("package-name", ">=1.0")`
 - Build/test: `project.build_depends_on("package-name")`
+- Conditional: `project.depends_on("package-name", markers="sys_platform == 'win32'")`
+- Optional: `project.depends_on("package-name", extra="group-name")`, installed into the
+  venvs only when the group is named in `install_dependencies_extras`
 ```
 
 ### Example CLAUDE.md

documentation/manual.md

@@ -359,6 +359,118 @@ fall back to the system Python. This is a legacy mode primarily used for debuggi
 and is not recommended for normal builds, as it can lead to dependency conflicts
 and unreliable coverage results.
 
+## Conditional Dependencies
+
+A dependency may carry PEP 508 environment markers, either as a keyword argument or
+as part of the requirement string:
+
+<pre><code>@init
+def initialize(project):
+    project.depends_on("pywin32", "&gt;=300", markers="sys_platform == 'win32'")
+    project.depends_on("tomli; python_version &lt; '3.11'")
+</code></pre>
+
+Markers are part of a dependency's identity, so the same distribution may be declared
+more than once under mutually exclusive conditions. This is how a version that differs
+per interpreter is expressed:
+
+<pre><code>@init
+def initialize(project):
+    project.depends_on("numpy", "==1.26.4", markers="python_version &lt; '3.12'")
+    project.depends_on("numpy", "==2.1.0", markers="python_version &gt;= '3.12'")
+</code></pre>
+
+Both declarations are published - `install_requires` carries each with its marker, and
+the consumer's installer picks the one that applies. Within the build, only the
+applicable one is installed, and only it reaches the constraints file.
+
+The rule PyBuilder enforces is pip's own: for any one distribution, at most one
+declaration may apply in a given environment. Two declarations that both apply, or two
+that carry identical conditions, fail validation with
+
+<pre><code>Runtime dependency 'numpy' has been defined multiple times.</code></pre>
+
+Markers are evaluated against the marker environment of the *target* interpreter, not
+the one running the build, so a VEnv created from a different Python resolves its own
+conditional dependencies. Project validation runs before any VEnv exists and therefore
+uses the interpreter running the build - which is the one every VEnv is created from.
+
+## Extras in Virtual Environments
+
+Assigning a dependency to an extras group with `extra=` publishes it under
+`extras_require`, for consumers to install with `pip install mypackage[security]`. By
+itself that group is never installed into the build, which means the code it guards
+cannot be tested.
+
+The `install_dependencies_extras` property selects groups for installation:
+
+<table class="table table-striped">
+  <tr>
+    <th>Value</th>
+    <th>Meaning</th>
+  </tr>
+  <tr>
+    <td><code>[]</code> (default) or <code>None</code></td>
+    <td>No extras are installed.</td>
+  </tr>
+  <tr>
+    <td><code>"security"</code></td>
+    <td>That one group.</td>
+  </tr>
+  <tr>
+    <td><code>["security", "speedups"]</code></td>
+    <td>Those groups.</td>
+  </tr>
+  <tr>
+    <td><code>"*"</code></td>
+    <td>Every group the project declares.</td>
+  </tr>
+</table>
+
+<pre><code>@init
+def initialize(project):
+    project.depends_on("cryptography", "&gt;=42", extra="security")
+    project.depends_on("pywin32", "&gt;=300", extra="windows",
+                       markers="sys_platform == 'win32'")
+    project.depends_on("sphinx", "&gt;=7", extra="docs")
+
+    # Install the security extra into the build and test venvs so that the code
+    # path it guards is actually exercised by the tests
+    project.set_property("install_dependencies_extras", ["security"])
+</code></pre>
+
+Naming a group the project does not declare fails the build and lists the declared
+ones. Group names are normalized, so `extra="Security"` and `extra="security"` are the
+same group.
+
+The selection applies to the build and test VEnvs and to the `install_dependencies` and
+`install_runtime_dependencies` tasks alike. Because it is an ordinary property, it can
+be scoped to an environment - for instance to exercise the heavy groups only in CI:
+
+<pre><code>@init(environments="ci")
+def initialize_ci(project):
+    project.set_property("install_dependencies_extras", "*")
+</code></pre>
+
+For a different selection per VEnv, compose `venv_dependencies` directly; an explicit
+entry always wins over the default:
+
+<pre><code>@init
+def initialize(project):
+    project.set_property("venv_dependencies", {
+        "test": project.base_dependencies + project.extras_dependencies["security"],
+    })
+</code></pre>
+
+Selecting an extra is a build-time decision and never changes what is published:
+`install_requires` carries the base dependencies only, and the selected groups remain in
+`extras_require`. `project.dependencies` returns the base dependencies plus the selected
+groups, which is what installation uses; `project.base_dependencies` returns the base
+ones alone, which is what the generated `setup.py` uses.
+
+Run `pyb list_dependencies` to see the declared groups and which of them the current
+selection installs.
+
 ## Unit Testing in Detail
 
 The `python.unittest` plugin executes unit tests using Python's `unittest` module

documentation/plugins.md

@@ -622,14 +622,18 @@ Note that the `*_depends_on` methods accept the following arguments :
   <tr>
     <td>extra</td>
     <td>Optional keyword argument (<code>None</code> default). Only available on <code>depends_on</code>.
-        Assigns the dependency to an extras group (e.g. <code>extra="security"</code>). Users can then install it
-        via <code>pip install mypackage[security]</code>.</td>
+        Assigns the dependency to an extras group (e.g. <code>extra="security"</code>), which is published in
+        <code>extras_require</code> so that users can install it via <code>pip install mypackage[security]</code>.
+        The group may additionally be installed into the VEnvs of this build - so that the code it guards can be
+        tested - by naming it in <code>install_dependencies_extras</code>. Group names are normalized, so
+        <code>"Security"</code> and <code>"security"</code> are the same group.</td>
   </tr>
 
   <tr>
     <td>markers</td>
     <td>Optional keyword argument (<code>None</code> default). PEP 508 environment markers for conditional
-        dependencies (e.g. <code>markers="sys_platform == 'win32'"</code>).</td>
+        dependencies (e.g. <code>markers="sys_platform == 'win32'"</code>). Markers are part of a dependency's
+        identity, so the same distribution may be declared several times under mutually exclusive conditions.</td>
   </tr>
 </table>
 
@@ -689,6 +693,19 @@ The logic of version goes as follows:
     <td><code>[ ]</code></td>
     <td>Tell newer versions of pip that it's OK to install those dependencies insecurely (externally hosted, potentially unverified)</td>
   </tr>
+
+  <tr>
+    <td>install_dependencies_extras</td>
+    <td>List of strings, string, or <code>"*"</code></td>
+    <td><code>[ ]</code></td>
+    <td>Extras groups to install alongside the runtime dependencies, into the build and test VEnvs as well as
+        through the <code>install_dependencies</code> and <code>install_runtime_dependencies</code> tasks.
+        <code>[ ]</code> or <code>None</code> installs none of them, which is the default; a single name such as
+        <code>"security"</code> installs that group; a list installs those groups; <code>"*"</code> installs every
+        group the project declares. Names are normalized, and naming a group the project does not declare fails
+        the build. Extras selected here are <em>not</em> published as mandatory requirements of the distribution -
+        they remain in <code>extras_require</code> only.</td>
+  </tr>
 </table>
 
 ### Creating a source distribution

documentation/project-info.md

@@ -108,7 +108,7 @@ Object with four sub-keys:
 
 | Key | Contents |
 |-----|----------|
-| `runtime` | Dependencies from `depends_on()` |
+| `runtime` | Dependencies from `depends_on()`, plus the extras groups selected by `install_dependencies_extras` |
 | `build` | Dependencies from `build_depends_on()` |
 | `plugin` | Dependencies from `plugin_depends_on()` |
 | `extras` | Object mapping extra name to dependency array |
@@ -121,12 +121,22 @@ Each dependency is an object:
   "version": ">=2.28",
   "url": null,
   "extras": null,
+  "extra": null,
   "markers": "sys_platform == 'linux'",
   "declaration_only": false,
   "type": "dependency"
 }
 ```
 
+`extras` are the extras of the dependency itself, as in `depends_on("requests[socks]")`.
+`extra` is the extras group of *this* project that the dependency was declared under, as
+in `depends_on("requests", extra="http")`, and is `null` for a base dependency. A
+dependency belonging to a selected group appears both in `runtime` and under its group in
+`extras`.
+
+Because markers are part of a dependency's identity, one distribution may appear several
+times in `runtime` with different `markers` and versions.
+
 Requirements files have `"type": "requirements_file"` and only `name`,
 `version` (always null), and `declaration_only` fields.
 

documentation/tutorial.md

@@ -615,6 +615,41 @@ project metadata into CI/CD scripts. See the
 [project-info documentation](/documentation/project-info.html) for the full
 JSON schema and integration examples.
 
+## Conditional and Optional Dependencies
+
+Our project so far declares one build dependency. Two further shapes are worth knowing
+about before you outgrow the basics.
+
+A dependency may carry PEP 508 environment markers, and the same distribution may be
+declared more than once as long as no two declarations apply at the same time:
+
+```python
+@init
+def set_properties(project):
+    project.depends_on("numpy", "==1.26.4", markers="python_version < '3.12'")
+    project.depends_on("numpy", "==2.1.0", markers="python_version >= '3.12'")
+```
+
+Both are published; within the build, only the one that applies is installed.
+
+A dependency may also be optional, assigned to an extras group that consumers opt into
+with `pip install helloworld[security]`:
+
+```python
+@init
+def set_properties(project):
+    project.depends_on("cryptography", ">=42", extra="security")
+
+    # Install that group into the build and test venvs as well, so the code it
+    # guards is exercised by our tests rather than merely shipped
+    project.set_property("install_dependencies_extras", ["security"])
+```
+
+Without that property the group is published but never installed, and the code path it
+guards cannot be tested. Run `pyb list_dependencies` to see the declared groups and which
+of them the current selection installs. See the
+[manual](/documentation/manual.html) for the full behavior.
+
 ## Recap
 
 In this tutorial we saw how PyBuilder can be used to "build" a typical Python project. Building in an interpreted