diff --git a/articles/_release-notes/v0.13.x.md b/articles/_release-notes/v0.13.x.md index 12641d7..4c6c7d8 100644 --- a/articles/_release-notes/v0.13.x.md +++ b/articles/_release-notes/v0.13.x.md @@ -6,6 +6,74 @@ list_title: Versions 0.13.x # Release Notes - Versions 0.13.x +## Version 0.13.23 + +### New Features +* The same distribution may now be declared more than once under mutually + exclusive environment markers, so that `foo==1.0` on one interpreter and + `foo==2.0` on another is expressible. Markers are now part of a dependency's + identity, and validation accepts a group of same-named declarations as long as + no two of them apply at once. Identical conditions on one name remain an error, + since no environment can tell them apart. +* Extras groups declared with `depends_on(..., extra=...)` can now be installed + into the VEnvs PyBuilder builds, so that the code an extra guards can actually + be tested. The new `install_dependencies_extras` property selects them: a list + of names, a single name, or `"*"` for every declared group. It defaults to + `[]`, which is the previous behavior. Selecting a group the project does not + declare fails the build and lists the ones it does. +* Dependencies are now selected against the marker environment of the target + interpreter rather than the one running the build. `PythonEnv.marker_env` + carries the full set of PEP 508 variables, probed from that interpreter with + PyBuilder's vendored *packaging*, so a VEnv built from a different Python + resolves its own conditional dependencies correctly. +* `list_dependencies` now prints the declared extras groups, each marked with + whether the current selection installs it. `pyb -i` reports the group a + dependency belongs to under the new `extra` key. + +### Changed Features +* `project.dependencies` now returns the base runtime dependencies plus whatever + `install_dependencies_extras` selects, which is what every install and + consumption site wants. The new `project.base_dependencies` returns the base + ones alone, and is what the generated `setup.py` uses for `install_requires` + and `dependency_links` - a build-time decision to test an extra must never + become a mandatory requirement of the published distribution. Plugins that + render distribution metadata from `project.dependencies` should move to + `project.base_dependencies`. + +### Bugs Fixed +* A dependency declared twice with different markers was silently discarded. + Same name and version with different conditions collided on identity, the + second declaration was dropped, and nothing was reported - so a Windows-only + dependency would disappear because a Linux-only one happened to be declared + first. +* Declaring a conditional pin was a hard build failure. `validate_dependencies` + counted by name alone, so the correct way to express a per-interpreter version + broke the build. +* Dependencies whose markers do not apply are no longer installed, and no longer + written to the constraints file. They were queued on every build, which + defeated the "already up-to-date, skipped" path; and where two entries on one + name were both live, pip intersected them into a `ResolutionImpossible` that + named neither declaration. PyBuilder now reports such a conflict itself, + naming both. +* `depends_on("foo[security]")` was a silent no-op on incremental builds. + Nothing in installed metadata records which extras were requested, so with + plain `foo` already installed at a satisfying version the extra was skipped and + its requirements never arrived. PyBuilder now resolves what the extra requires + from the installed distribution's own metadata and verifies those are present, + falling back to handing the dependency to pip when the installed version no + longer offers the extra at all. +* A dependency carrying environment markers generated a syntactically invalid + `setup.py`, because the rendered requirement was interpolated into single + quotes and markers are conventionally written with single quotes of their own: + `'pywin32>=300; sys_platform == 'win32''`. Building the distribution failed + with a `SyntaxError` from the backend. Values are now rendered as proper Python + literals, which also fixes any other string containing a quote or a backslash. +* `install_requires` dropped a dependency's own extras, so + `depends_on("foo[bar]")` was published as plain `foo`. +* Extras group names are now normalized, so that `extra="Security"` and + `extra="security"` are one group rather than two emitted separately into + `extras_require`. + ## Version 0.13.22 ### New Features diff --git a/documentation/coding-agents.md b/documentation/coding-agents.md index 7f86b99..9bb34f6 100644 --- a/documentation/coding-agents.md +++ b/documentation/coding-agents.md @@ -117,8 +117,24 @@ def initialize(project): # From a requirements file project.depends_on_requirements("requirements.txt") project.build_depends_on_requirements("requirements-dev.txt") + + # Conditional dependency. The same distribution may be declared more than once + # as long as no two declarations apply in the same environment + project.depends_on("numpy", "==1.26.4", markers="python_version < '3.12'") + project.depends_on("numpy", "==2.1.0", markers="python_version >= '3.12'") + + # Optional dependency, published under extras_require + project.depends_on("cryptography", ">=42", extra="security") + + # Install that group into the build and test venvs so its code path is tested. + # Accepts a name, a list of names, or "*" for every declared group + project.set_property("install_dependencies_extras", ["security"]) ``` +Extras selected with `install_dependencies_extras` are installed into the venvs but are +*not* published as mandatory requirements — they stay in `extras_require`. Naming a +group the project does not declare fails the build. + ### Writing Tests Unit tests go in `src/unittest/python/` and must match the glob `*_tests.py`: @@ -252,6 +268,9 @@ If directories are customized, specify the actual paths: Edit `build.py` and add to the initializer: - Runtime: `project.depends_on("package-name", ">=1.0")` - Build/test: `project.build_depends_on("package-name")` +- Conditional: `project.depends_on("package-name", markers="sys_platform == 'win32'")` +- Optional: `project.depends_on("package-name", extra="group-name")`, installed into the + venvs only when the group is named in `install_dependencies_extras` ``` ### Example CLAUDE.md diff --git a/documentation/manual.md b/documentation/manual.md index 16b5c93..b16f66e 100644 --- a/documentation/manual.md +++ b/documentation/manual.md @@ -359,6 +359,118 @@ fall back to the system Python. This is a legacy mode primarily used for debuggi and is not recommended for normal builds, as it can lead to dependency conflicts and unreliable coverage results. +## Conditional Dependencies + +A dependency may carry PEP 508 environment markers, either as a keyword argument or +as part of the requirement string: + +
@init
+def initialize(project):
+    project.depends_on("pywin32", ">=300", markers="sys_platform == 'win32'")
+    project.depends_on("tomli; python_version < '3.11'")
+
+ +Markers are part of a dependency's identity, so the same distribution may be declared +more than once under mutually exclusive conditions. This is how a version that differs +per interpreter is expressed: + +
@init
+def initialize(project):
+    project.depends_on("numpy", "==1.26.4", markers="python_version < '3.12'")
+    project.depends_on("numpy", "==2.1.0", markers="python_version >= '3.12'")
+
+ +Both declarations are published - `install_requires` carries each with its marker, and +the consumer's installer picks the one that applies. Within the build, only the +applicable one is installed, and only it reaches the constraints file. + +The rule PyBuilder enforces is pip's own: for any one distribution, at most one +declaration may apply in a given environment. Two declarations that both apply, or two +that carry identical conditions, fail validation with + +
Runtime dependency 'numpy' has been defined multiple times.
+ +Markers are evaluated against the marker environment of the *target* interpreter, not +the one running the build, so a VEnv created from a different Python resolves its own +conditional dependencies. Project validation runs before any VEnv exists and therefore +uses the interpreter running the build - which is the one every VEnv is created from. + +## Extras in Virtual Environments + +Assigning a dependency to an extras group with `extra=` publishes it under +`extras_require`, for consumers to install with `pip install mypackage[security]`. By +itself that group is never installed into the build, which means the code it guards +cannot be tested. + +The `install_dependencies_extras` property selects groups for installation: + + + + + + + + + + + + + + + + + + + + + + +
ValueMeaning
[] (default) or NoneNo extras are installed.
"security"That one group.
["security", "speedups"]Those groups.
"*"Every group the project declares.
+ +
@init
+def initialize(project):
+    project.depends_on("cryptography", ">=42", extra="security")
+    project.depends_on("pywin32", ">=300", extra="windows",
+                       markers="sys_platform == 'win32'")
+    project.depends_on("sphinx", ">=7", extra="docs")
+
+    # Install the security extra into the build and test venvs so that the code
+    # path it guards is actually exercised by the tests
+    project.set_property("install_dependencies_extras", ["security"])
+
+ +Naming a group the project does not declare fails the build and lists the declared +ones. Group names are normalized, so `extra="Security"` and `extra="security"` are the +same group. + +The selection applies to the build and test VEnvs and to the `install_dependencies` and +`install_runtime_dependencies` tasks alike. Because it is an ordinary property, it can +be scoped to an environment - for instance to exercise the heavy groups only in CI: + +
@init(environments="ci")
+def initialize_ci(project):
+    project.set_property("install_dependencies_extras", "*")
+
+ +For a different selection per VEnv, compose `venv_dependencies` directly; an explicit +entry always wins over the default: + +
@init
+def initialize(project):
+    project.set_property("venv_dependencies", {
+        "test": project.base_dependencies + project.extras_dependencies["security"],
+    })
+
+ +Selecting an extra is a build-time decision and never changes what is published: +`install_requires` carries the base dependencies only, and the selected groups remain in +`extras_require`. `project.dependencies` returns the base dependencies plus the selected +groups, which is what installation uses; `project.base_dependencies` returns the base +ones alone, which is what the generated `setup.py` uses. + +Run `pyb list_dependencies` to see the declared groups and which of them the current +selection installs. + ## Unit Testing in Detail The `python.unittest` plugin executes unit tests using Python's `unittest` module diff --git a/documentation/plugins.md b/documentation/plugins.md index de9898d..a31ab51 100644 --- a/documentation/plugins.md +++ b/documentation/plugins.md @@ -622,14 +622,18 @@ Note that the `*_depends_on` methods accept the following arguments : extra Optional keyword argument (None default). Only available on depends_on. - Assigns the dependency to an extras group (e.g. extra="security"). Users can then install it - via pip install mypackage[security]. + Assigns the dependency to an extras group (e.g. extra="security"), which is published in + extras_require so that users can install it via pip install mypackage[security]. + The group may additionally be installed into the VEnvs of this build - so that the code it guards can be + tested - by naming it in install_dependencies_extras. Group names are normalized, so + "Security" and "security" are the same group. markers Optional keyword argument (None default). PEP 508 environment markers for conditional - dependencies (e.g. markers="sys_platform == 'win32'"). + dependencies (e.g. markers="sys_platform == 'win32'"). Markers are part of a dependency's + identity, so the same distribution may be declared several times under mutually exclusive conditions. @@ -689,6 +693,19 @@ The logic of version goes as follows: [ ] Tell newer versions of pip that it's OK to install those dependencies insecurely (externally hosted, potentially unverified) + + + install_dependencies_extras + List of strings, string, or "*" + [ ] + Extras groups to install alongside the runtime dependencies, into the build and test VEnvs as well as + through the install_dependencies and install_runtime_dependencies tasks. + [ ] or None installs none of them, which is the default; a single name such as + "security" installs that group; a list installs those groups; "*" installs every + group the project declares. Names are normalized, and naming a group the project does not declare fails + the build. Extras selected here are not published as mandatory requirements of the distribution - + they remain in extras_require only. + ### Creating a source distribution diff --git a/documentation/project-info.md b/documentation/project-info.md index 865ecfc..80e282f 100644 --- a/documentation/project-info.md +++ b/documentation/project-info.md @@ -108,7 +108,7 @@ Object with four sub-keys: | Key | Contents | |-----|----------| -| `runtime` | Dependencies from `depends_on()` | +| `runtime` | Dependencies from `depends_on()`, plus the extras groups selected by `install_dependencies_extras` | | `build` | Dependencies from `build_depends_on()` | | `plugin` | Dependencies from `plugin_depends_on()` | | `extras` | Object mapping extra name to dependency array | @@ -121,12 +121,22 @@ Each dependency is an object: "version": ">=2.28", "url": null, "extras": null, + "extra": null, "markers": "sys_platform == 'linux'", "declaration_only": false, "type": "dependency" } ``` +`extras` are the extras of the dependency itself, as in `depends_on("requests[socks]")`. +`extra` is the extras group of *this* project that the dependency was declared under, as +in `depends_on("requests", extra="http")`, and is `null` for a base dependency. A +dependency belonging to a selected group appears both in `runtime` and under its group in +`extras`. + +Because markers are part of a dependency's identity, one distribution may appear several +times in `runtime` with different `markers` and versions. + Requirements files have `"type": "requirements_file"` and only `name`, `version` (always null), and `declaration_only` fields. diff --git a/documentation/tutorial.md b/documentation/tutorial.md index 345c99d..6566abe 100644 --- a/documentation/tutorial.md +++ b/documentation/tutorial.md @@ -615,6 +615,41 @@ project metadata into CI/CD scripts. See the [project-info documentation](/documentation/project-info.html) for the full JSON schema and integration examples. +## Conditional and Optional Dependencies + +Our project so far declares one build dependency. Two further shapes are worth knowing +about before you outgrow the basics. + +A dependency may carry PEP 508 environment markers, and the same distribution may be +declared more than once as long as no two declarations apply at the same time: + +```python +@init +def set_properties(project): + project.depends_on("numpy", "==1.26.4", markers="python_version < '3.12'") + project.depends_on("numpy", "==2.1.0", markers="python_version >= '3.12'") +``` + +Both are published; within the build, only the one that applies is installed. + +A dependency may also be optional, assigned to an extras group that consumers opt into +with `pip install helloworld[security]`: + +```python +@init +def set_properties(project): + project.depends_on("cryptography", ">=42", extra="security") + + # Install that group into the build and test venvs as well, so the code it + # guards is exercised by our tests rather than merely shipped + project.set_property("install_dependencies_extras", ["security"]) +``` + +Without that property the group is published but never installed, and the code path it +guards cannot be tested. Run `pyb list_dependencies` to see the declared groups and which +of them the current selection installs. See the +[manual](/documentation/manual.html) for the full behavior. + ## Recap In this tutorial we saw how PyBuilder can be used to "build" a typical Python project. Building in an interpreted