feat(toolchain): package standard library into zip file for hermetic runtimes - #4146
Draft
rickeylev wants to merge 17 commits into
Draft
feat(toolchain): package standard library into zip file for hermetic runtimes#4146rickeylev wants to merge 17 commits into
rickeylev wants to merge 17 commits into
Conversation
…runtimes Package the Python standard library into a platform-appropriate compressed zip archive and wire it into hermetic toolchain py_runtime definitions so standard library modules can be imported directly from the zip. Add a private zip_stdlib rule wrapping @bazel_tools//tools/zip:zipper to construct the standard library archive with prefix stripping and a mandatory output location. Update hermetic_runtime_repo_setup with a _define_zip_stdlib helper to create zip_stdlib targets targeting platform-specific standard library zip paths (lib/pythonXY[t].zip on POSIX, pythonXY[t].zip on Windows) and include the resulting zip target in py3_runtime's files attribute.
Add a pytest_test target in tests/zip_stdlib to assert pure Python standard library modules are loaded via zipimport from the hermetic runtime zip.
…in test Exclude on-disk standard library files from the hermetic runtime files filegroup so only the zipped archive is provided at runtime. Add test assertions in zip_stdlib_test to verify unzipped standard library files are not present on disk in runfiles.
…ry zipping Allow users to configure whether the standard library is packaged into a zip file using the zip_stdlib string flag (values: 'yes', 'no', default: 'yes'). - Define ZipStdlibFlag in python/private/flags.bzl and declare string_flag zip_stdlib in python/config_settings/BUILD.bazel. - Define _is_zip_stdlib_yes and _is_zip_stdlib_no config settings in construct_config_settings using labels.ZIP_STDLIB. - Use _IS_ZIP_STDLIB_YES and _IS_ZIP_STDLIB_NO in hermetic runtime repo setup to conditionally package and include the stdlib zip or on-disk files. - Add analysis tests covering flag disabled behavior.
Document the zip_stdlib flag in the configuration settings API documentation so users understand the option to package or retain standard library files on disk.
…e wrapping Add manual tags to intermediate filegroups and shorten test implementation function name to comply with line length limits.
… tag handling Wildcard builds failed because helper targets in hermetic_runtime_repo_setup were missing the manual tag, and zip_stdlib referenced ctx.attr.strip_prefix instead of the local variable. Mark helper targets as manual, fix strip_prefix reference, and move list_add_unique to util.bzl to merge tags cleanly.
…reters When standard library zipping is enabled, CPython requires landmark files (like os.py) to resolve its prefix, Windows venvs need pythonXY.zip in Scripts/, and exec actions running bare interpreters need the zip in PYTHONPATH. Expose zip_stdlib in PyRuntimeInfo and py_runtime, retain landmark files in files_base, include the zip in Windows venv_bin_files, and add it to action_env in actions_run.
…ackages CPython on Windows requires DLLs/ in sys.path and registered with os.add_dll_directory to load C-extensions such as _socket and _ctypes. Symlink DLLs/ when creating Windows venvs and add it in stage 2 bootstrap. Additionally, python-build-standalone bundles _distutils_hack in site-packages which caused ModuleNotFoundError when the bare interpreter invoked distutils-precedence.pth. Exclude bundled site-packages from files_base while preserving site-packages landmark files.
On Windows, UNC prefixes injected into target_root corrupt sys.path, and sys.path.insert for DLLs alters sys.path ordering; use abspath and os.add_dll_directory instead. Symlink python*.zip into Windows venvs. Actions running bare interpreters like py_console_script_gen need access to the stdlib zip, so execute via py_interpreter_program and actions_run. Finally, keep encodings/*.py in landmark files on disk so FrozenImporter and early site-packages initialization can resolve utf-8-sig on Python 3.15 before _bazel_site_init runs.
On Windows, runtime virtual environments execute python from the venv Scripts directory, where CPython's getpath expects the stdlib zip to reside. When missing, early startup fails to locate the stdlib zip. Additionally, native C extensions located in DLLs/ were omitted from sys.path, and premature sys.base_prefix entries violated sys.path ordering invariants. Symlink the standard library zip into venv bin and recreate encodings in venv Lib. Add runtime DLLs to sys.path after stdlib archives and omit bare prefix roots from early sys.path positions.
Python on Windows looks for the stdlib zip in the virtual environment root (PREFIX) and BASE_PREFIX during startup. Placing it only in the venv Scripts directory prevents CPython (notably 3.15) from finding standard library encodings when importing site. Symlink stdlib zip files into both the venv root and venv bin directory at build time in py_executable.bzl and at runtime in the bootstrap template. Also scan the actual interpreter directory in the runfiles tree at bootstrap runtime.
In Python 3.15, site.py decodes .pth files using utf-8-sig. When running in a venv on Windows, getpath sets sys._stdlib_dir to the venv Lib dir because site-packages exists within it. FrozenImporter resolves submodules of encodings strictly via encodings.__path__ (sys._stdlib_dir/encodings). If loose codec files are missing from that directory, imports fail with LookupError. Extract standard library encodings into the venv Lib/encodings directory at bootstrap on Windows if not already present. Also include all encodings files in files_landmark.
…aths Subprocess output on Windows emits carriage returns (\r\n) which leaves trailing \r on resolved sys.executable and sys.base_prefix paths when split on newline. This causes subsequent path lookups, directory listing, and pyvenv.cfg generation to point to non-existent paths. Use splitlines() and strip() when parsing subprocess output, interpreter arguments, and runtime venv symlinks. Ensure standard library encodings are extracted to both python_home and venv Lib directories on Windows when required by Python 3.15.
…n 3.15 stdlib zip In Python 3.15, site.py decodes .pth files using utf-8-sig. When frozen modules are enabled (the default), encodings is loaded as a frozen package whose submodules are only searched within sys._stdlib_dir on disk. With stdlib zip packaging, loose codec files are not present on disk, causing LookupError: unknown encoding: utf-8-sig. Pass -Xfrozen_modules=off when probing and executing Python subprocesses on Windows so encodings and its submodules are imported via zipimport directly from the stdlib zip archive. Also restrict extraction targets to the writable venv directory to avoid PermissionError on read-only runfiles trees.
Wrap lines to 80 columns across Starlark and runtime bootstrap templates, normalize tag arguments in toolchain macros, and canonicalize the Windows platform label reference.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hermetic Python runtimes include thousands of individual standard
library files. Staging each file in runfiles trees consumes inodes,
increases manifest overhead, and degrades test invocation latency.
Python natively supports importing modules from a zip archive via
zipimport. Packaging pure Python standard library modules into a single
compressed archive substantially shrinks runfiles trees and accelerates
runfiles creation.
Update hermetic toolchain runtimes to package standard library modules
into a zip archive and omit loose files from runfiles trees. Add a
//python/config_settings:zip_stdlib string flag ('yes'/'no', defaulting
to 'yes') so users can opt out and retain on-disk files when necessary.
Work towards #1653.