Repository navigation
Decide on the first module to port #30
Description
We need to choose what the first module we migrate to Rust will be. This will be the only module we port in 3.16. I see three leading options (but please suggest other options if they seem compelling!):
base64
Base64 is a very commonly used algorithm that is also relatively simple. We could use https://docs.rs/base64/latest/base64/ which supports multiple alphabets and padding behaviors which would make implementation much easier, and performs significantly better than the _binascii C implementation. We could also use https://docs.rs/base64-simd/latest/base64_simd/ which would be even faster.
zlib
zlib is used in web compression, zip files (notably including wheels!), gzip files, and many other places. We could use https://github.com/trifectatechfoundation/zlib-rs which is faster than zlib, and about as fast as zlib-ng. The output might differ compared to zlib as the implementation is based on zlib-ng.
bz2
The bz2 module is not used nearly as much as the other two, but that can be a blessing if we want to reduce risks. We could use https://github.com/trifectatechfoundation/libbzip2-rs, which is a safe bzip2 implementation which is also 5-10% faster. I think the benefit here would be a performance and safety gain. The output should also be identical to the existing ubiquitous C library which may not be the case for zlib.
Activity
Qelxiros commentedon Apr 11, 2026
Hi! I'm new here, but not to Rust or Python. I think base64 might be the best option to start with for a couple reasons. First, it's widely used, so lots of people have experience using it and can easily test the rust implementation. Second, the base64_simd crate looks super straightforward to use, so it's a low(zero?)-effort performance benefit that'll be immediately apparent in profiling.
hugovk commentedon Apr 11, 2026
Anything on pip scale will have a huge impact!
emmatyping commentedon Apr 11, 2026
@Qelxiros welcome!
One thing to keep in mind for the base64 module is that while the vast majority of use cases are standard base64 computation, it also supports base16, base32, and base85. We could skip those or implement those ourselves, but I think they should probably be considered when making this decision.
A blessing and a curse :)
I believe deflate decompression is a bottleneck, or at least a large portion of the time, in pip installation, so moving everyone to zlib-rs would make wheel installs with pip significantly faster if I'm not mistaken.
Qelxiros commentedon Apr 11, 2026
Thanks for the extra context! It looks like https://github.com/Nugine/simd includes both base32 and hex with simd support, but I'm not sure about base85. Is it possible to have a split of some sort (either via Rust FFI or CPython's build system) where the common cases use Rust for SIMD and base85 keeps its C implementation?
egorsmkv commentedon Apr 11, 2026
I would suggest json module because it's more popular nowadays in many areas of applied programming by everyday user
dimaqq commentedon Apr 16, 2026
My 2c: zlib.
Go little big or go home. Depending on an external (?) crate will be and interesting human problem to solve.
emmatyping commentedon Apr 17, 2026
This could make sense, but I expect people will want a library that is about as fast as some of the faster options, such as orjson. It might be challenging to find a Rust crate that competes with these libraries.
Makes sense! I think there are definite ecosystem arguments to be made here!
emmatyping commentedon Apr 20, 2026
After the meeting last week, I think the concensus among attendees was the
zlibmodule had a great balance of simplicity in implementation and significant impact. If anyone is interested in working on implementations of the other modules targetting 3.17, I'd be happy to see those! For 3.16, let's move forward withzlib.