[jonasdedden]

Part of #6399 (protocols).

Adds pyo3::pyprotocol! (with experimental-inspect), which declares a typing.Protocol that only exists in the generated stubs:

pyo3::pyprotocol! {
    /// A binary stream that can be read from and written to
    pub SupportsReadWrite {
        #[pyo3(signature = (n = -1, /))]
        fn read(&self, n: isize) -> Py<PyBytes>;
        fn write(&self, data: &[u8]) -> usize;
    }
}

impl<'py> FromPyObject<'_, 'py> for ReadWriteFile<'py> {
    const INPUT_TYPE: PyStaticExpr = SupportsReadWrite::TYPE_HINT;
    // ...
}

By default the protocol is private, next to where it's used:

@type_check_only
class _SupportsReadWrite(Protocol):
    """
    A binary stream that can be read from and written to
    """
    def read(self, n: int = ..., /) -> bytes: ...
    def write(self, data: bytes, /) -> SupportsIndex: ...

def write_to(f: _SupportsReadWrite, data: bytes) -> int: ...

A module can make it public by exporting it, like a class. It's then declared there as SupportsReadWrite, even if nothing uses it, and can be imported in an if TYPE_CHECKING: block:

#[pymodule]
mod my_module {
    #[pymodule_export]
    use super::SupportsReadWrite;
}

• Works from library crates. The protocol is a protocol chunk with its methods inline. pyo3-introspection puts a private one as _Name in the deepest module containing all modules that use it, and leaves out unused ones. An exported one goes in the exporting module under its own name. The extension owning the Python API decides what's public, so a library like pyo3-file doesn't have to know its users' modules, and it can bundle its protocols in a #[pymodule] that an extension exports in one line.
• A name clash with another member of the module is an error, no automatic renaming; #[pyo3(name = "...")] sets another name. A protocol can only be exported by one module.
• It's decorated @type_check_only and not listed in __all__, so the stubs pass stubtest.
• The id is module_path!() plus the name, so there is one chunk per protocol. A static can't depend on a generic parameter, so protocols that depend on T can't be expressed.
• Arguments use IntoPyObject::OUTPUT_TYPE and return values FromPyObject::INPUT_TYPE, since Rust is the caller. Parameters are positional-only unless a signature is given, and annotations from the signature are kept. Methods must take &self.
• Protocols and their methods have their own #[pyo3] options: name and crate for protocols, name and signature for methods.
• Only methods are supported for now: no attributes, properties or generic protocols (documented in the guide).

[jonasdedden]

@Tpt did you have a chance to look at this PR yet, and if yes, what do you generally think? It's a bit clunky, but hopefully a first step in a direction of freely defining Protocols per function, right? It would actually unblock some use case of a PyO3 IO library I want to write.

[Tpt]

@jonasdedden Sorry for the slow reaction.

Thank you for pushing this.

A possible downside I see: we will need to be very careful to not generate conflicting "introspection ids" to properly map the generated class chunk with its usage. If we have stuff like impl<T> FromPyObject for Container<T> and generate the class chunk inside of the impl block we might end up with a class chunk for each T used.

A way to circumvent that and avoid the need for "attach_to_root": true part is to extend our type expression introspection AST to add a magic "protocol" variant that is an inline class definition.
Then, at stub generation time we create a class not part of __all__ in the current module and substitute the inline class definition with just the class name while handling carefully deduplication. We can handle deduplication and conflict detection on class names fairly easily at this stage.

The downside I see is that we move our type expression introspection AST beyond what Python supports. WDYT?

[jonasdedden]

@Tpt thanks for your remarks! I reworked this a bit (which comes with a much larger PR, sorry 😢).

Library crates now use a pyprotocol! macro instead of writing JSON by hand. It emits the class and method chunks with a fixed id built from module_path!(), so there's exactly one chunk per protocol. A static can't depend on a generic parameter anyway, so the per-T duplication can't happen. This comes with a downside that a protocol whose shape depends on T can't be expressed, which I guess can be okay for now? The expression AST stays plain Python.

The generated protocols are marked @typing.type_check_only, since they only exist in the stubs and aren't runtime-importable (which is exactly what this decorator was made for). I checked which type checker versions accept it, and it's basically every one:

Type checker	Supported since
mypy	0.710 (June 2019)
pyright	1.0.65 (September 2019)
pyrefly	first working release (0.2.0)
ty	first working release

With that actually, everything from the original proposal that I could think of for Protocol definitions would be supported 🤔

[jonasdedden]

@Tpt any thoughts on this? :) (sorry for pinging 😅 )

[Tpt]

@jonasdedden Sorry for the answer lag.

Overall, I would love to get a better overview of where we want to go in #6399 to make sure we heading in the right direction to support what we need to support, both in term of UX and implementation. But it seems it's heading in the right direction.

Two (large) open questions:

1. what about using protocols defined this way in custom type annotations? How would it work? A common use case I see is stating that an argument must comply with a protocol.

2. [crazy idea] do we want protocols to go further than stubs? For example I think that from:

pyo3::pyprotocol! {
    /// A binary stream that can be read from and written to
    pub SupportsRead {
        #[pyo3(signature = (n = -1, /))]
        fn read(&self, n: isize) -> PyResult<Py<PyBytes>>;
    }
}

we might actually codegen something that would allow to use the type in the code like:

#[pyfunction]
fn do_io(io_object: Bound<'_, SupportsRead>) -> PyResult<()> {
   let read = io_object.read(10)?;
   Ok(())
}

It might look something like:

#[repr(transparent)]
pub struct SupportsRead(PyAny);

impl PyTypeInfo for SupportsRead { ... }

pub trait SupportsReadMethods {
    fn read(&self, n: isize) -> PyResult<Py<PyBytes>>;
}

impl SupportsReadMethods for Bound<'_, SupportsRead> {
    fn read(&self, n: isize) -> PyResult<Py<PyBytes>> { ... }
}

This would go in the same line of what other tools like wasm_bindgen do.
Some stuff like default arguments and named arguments might be "fun" to implement though...

[jonasdedden]

1. what about using protocols defined this way in custom type annotations? How would it work? A common use case I see is stating that an argument must comply with a protocol.

Right now a string annotation like "SupportsRead" doesn't work: it's opaque, so the protocol is never added and the name dangles (mypy/stubtest fail). But Bound<'_, SupportsRead> already gives the right annotation and import if the type implements PyTypeCheck, so Q2 would cover "argument must follow this protocol". For things like list[SupportsRead] in a signature, we could later allow a Rust const there, e.g. (a: {SupportsRead::TYPE_HINT}).

[jonasdedden]

[crazy idea] do we want protocols to go further than stubs? For example I think that from:

I briefly explored your ideas, but I think it has some rough edges. I haven't had time to dig into details yet, here is a Claude exploration of why things aren't ideal, with at least what I tried out:

I hand-wrote your sketch and it works: Bound<'_, SupportsRead> gets the right stub annotation, and calls go through typed methods instead of call_method strings. Compared to what this PR does today, though, generating the calls brings problems we currently don't have:

• Runtime class: today nothing needs one. A generated type has to implement PyTypeCheck, and classinfo_object has nothing real to return, since the protocol doesn't exist at runtime. Returning the name as a string breaks its contract.
• Type checks: today the user's FromPyObject decides what to check, if anything. A generated type has to pick one behaviour for everyone. hasattr checks miss wrong signatures and run user code; skipping checks moves errors to call time.
• Defaults: today a default is only a stub declaration and Rust passes whatever it wants. Generated methods have to map defaults to Rust somehow, e.g. Option<T>. Python has no "absent" value like JS undefined, so that needs extra rules.
• Return types: today -> Py<PyBytes> is enough because the methods are never called. Generated calls need PyResult<T>, so the syntax would change.
• Dunder methods: today Rust code uses the right API, e.g. PyAnyMethods::len() (PyObject_Size). Generated call_method("__len__") looks the method up on the instance instead of the type, so with o.__len__ = lambda: 99, it returns 99 while len(o) is 3. Dunders would need a mapping to the C API.
• Name clashes: today nothing is added to Bound<…>. A generated method named len silently shadows PyAnyMethods::len: I got 7 from p.len() and 3 from p.as_any().len().
• Overloads and async: today call_method can call an overloaded or async method however needed. Generated methods would need overload support, which PyO3 doesn't have, and a way to drive awaitables.
• External protocols: today typeshed's protocols work directly as INPUT_TYPE (_typeshed.SupportsRead[bytes]). Generated types only exist for protocols we declare, unless we add a mode to bind existing ones.
• Stability: today it's stub-only and behind experimental-inspect. Generated types would be runtime API, so the gate would go again, and every protocol brings a …Methods trait to import.

I think it's still worth it for the simple cases, but as a follow-up in #6399, not in this PR.

[Tpt]

Thank you for the iteration. It's looking much better

[codspeed]

Merging this PR will not alter performance

✅ 143 untouched benchmarks
⏩ 6 skipped benchmarks¹

---

_{Comparing jonasdedden:introspection-attach-to-root (6e38be2) with main (f41d7e8)}

Footnotes

1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[Tpt]

Thank you for the iteration and the improvements!

I still need to review the pyo3-introspection code carefully and I think this PR is good to go in my point of view

[Tpt]

Thanks! It looks good to me. Just to last nit/open questions

[jorenham]

Do I understand correctly that only Protocol methods, and not attributes or properties, are not supported? And generic Protocols also not?

[Tpt]

Do I understand correctly that only Protocol methods, and not attributes or properties, are not supported? And generic Protocols also not?

Yes! This is definitely something we can add later (generic is something not tackled yet at all in our stub generation).

[jorenham]

Since it's @type_check_only, it might be a bit cleaner to prefix this with _, otherwise someone, or some tool, might misinterpret this as part of the public API.

[jonasdedden]

This was already part of another discussion, have a look here: #6456 (comment) I gave an example of a typical use case I often have, where one would actually import Protocol from an external package. This is nice, as this negates the need to somehow re-wrap or re-create a Protocol internally (introducing the possibility of getting the Protocols out of sync in the worst and introducing code duplication in the best case).

So I'd say, at least my primary intention behind bringing Protocol support actually IS to make it importable by external packages, i.e. to make it part of a (type-checking-only) public API. The decorator @type_check_only fits this use case perfectly IMHO and as far as I understand it's specifically made for this. That means, at least from the Python docs, I wouldn't consider this to be an anti-pattern.

[jorenham]

Seems a bit risky though, given

• A protocol only appears in the stubs if a type hint refers to it.
  It is declared in the root module, so that using it in more modules doesn't move it.

So this could easily lead to accidental breakage if internally that protocol isn't used anymore, but downstream libraries still use it.

[jonasdedden]

Happy to remove the feature that drops a Protocol if it's unused, if people are generally fine with the type-check-only Protocols.

I believe exporting the Protocols in general (make them runtime-importable) is a bit tricky, as much of this is behind the experimental-inspect feature, which you wouldn't have enabled generally for building the release wheels. Correct me if I'm wrong here.

[jorenham]

I think that keeping protocols private and type-check-only by default makes sense. We also have a lot of such protocols in the NumPy stubs, for example, and would like to keep it that way, so that we don't have to worry about year-long deprecation periods when we remove or change something about them. For those that explicitly make their protocols part of their public API, of which optype is a rather extreme example (it has 337 runtime-accessible protocols in its public API), then they'll likely won't use pyo3 for that, since there's no real advantage in doing so, and just define them in a .py directly.

[jonasdedden]

For those that explicitly make their protocols part of their public API, [...], then they'll likely won't use pyo3 for that, since there's no real advantage in doing so, and just define them in a .py directly.

A very concrete example would be pyo3-file; here, a PyO3 file object type can generic over it being readable or not, writeable or not, seekable or not, with any combination of that being possible; In theory, one could manually write a Python package and expose it to PyPI where every possible Protocol is defined that exactly matches the specific type requirements of that crate, and ensure the exact type definitions somehow are kept in sync; also, the built PyO3 package then depends on this Python packge, or this is somehow automatically re-exported into every PyO3 package that uses pyo3-file internally as part of the wheel build step. I believe there is value in doing this entirely through PyO3 instead, as AFAIS this is removing a whole lot of failure modes, and instead we have one source of truth for everything.

We also have a lot of such protocols in the NumPy stubs, for example, and would like to keep it that way, so that we don't have to worry about year-long deprecation periods when we remove or change something about them.

As you and also @Tpt raised this issue though, I agree that above use case may be a very niche requirement, and it may be nicer to let the default be way more defensive.

I'd propose the following:

• Let Protocols ALWAYS be type-check-only (as this is a reflection of what's actually happening)
• Let Protocols be private (i.e. _-prefixed) by default
  • If they are private, they actually could be placed as close to the actual usage as possible again, i.e. not be placed into the root __init__.pyi by default
• Provide an opt-in way to expose a Protocol as part of a (type-check-only) public API if the user chooses so
  • In this case, the user has to provide a name explicitly (as otherwise, name changes are a breaking change for users); I guess it would be nice to enforce exposing it to __init__.pyi then (as otherwise, moving defintion into other modules also is a breaking change).

[jorenham]

Sounds great!

[Tpt]

I really like this proposal too! Thank you for iterating on it!

An API proposal: if the module = "foo.bar" macro parameter is set then the protocol is exposed publicly in the corresponding module. If not it is private. This way the place where the protocol is exposed is an explicit decision

[jonasdedden]

Thanks! I went with the same idea (private by default, explicit opt-in), but made the opt-in an export instead of a module option: #[pymodule_export] use some_crate::SupportsRead; makes it public in that module.

The reason is library crates: something like pyo3-file declares the protocols but can't know the module names of the extensions using it, so module = "..." on the declaration wouldn't work there. With the export, the extension that owns the Python API decides where (and whether) it's public, the same way as for classes. A library can also bundle its protocols in a #[pymodule], so an extension can export them all with one line.

Private protocols go as _Name into the deepest module containing all their users. I also dropped the automatic renaming: any name clash is now an error, and #[pyo3(name = "...")] resolves it.

I also added the limitations jorenham mentioned (methods only, no attributes, properties or generics) to the guide.

[jorenham]

Do I understand correctly that only Protocol methods, and not attributes or properties, are not supported? And generic Protocols also not?

Yes! This is definitely something we can add later (generic is something not tackled yet at all in our stub generation).

Maybe it's an idea to document this as "Limitations" or something then?