feat(ffi): generate Python bindings as a CPython extension

.github/workflows/nightly.yml

@@ -181,8 +181,9 @@ jobs:
       - name: Clang
         run: nix develop .#clang --command bash -c 'export CC=gcc; just ffi test cpp && just ffi moq-cpp'
 
-  # The C++ header with the compilers the job above lacks, through the same checks, and on
-  # macOS the Swift with Apple's toolchain.
+  # The C++ header with the compilers the job above lacks, through the same checks, the
+  # Python on the platforms the job above lacks, and on macOS the Swift with Apple's
+  # toolchain.
   ffi-cpp:
     name: Export moq-net to C++ (${{ matrix.name }})
     runs-on: ${{ matrix.os }}
@@ -232,6 +233,13 @@ jobs:
       - name: Test
         run: just ffi test cpp
 
+      # The Python, on the runner's CPython, which Windows names `python`: its proactor
+      # loop reads wakes through a task rather than a reader callback.
+      - name: Python
+        run: just ffi test python
+        env:
+          PYTHON: ${{ runner.os == 'Windows' && 'python' || 'python3' }}
+
       - name: Export
         run: just ffi moq-cpp
 

ffi/MIGRATE.md

@@ -2,6 +2,11 @@
 
 ## Unreleased
 
+- Every export writes `python/<prefix>.py`, so an export fails when two items take the same
+  Python name, such as a type `Foo` beside an error type's `FooException`, or when one
+  takes a name the Python's runtime uses unqualified, each starting with `_`: give it
+  another name with a `rename`. The crate it writes has a build script, and a `python`
+  feature that builds the CPython extension the Python loads.
 - Every export writes `swift/<prefix>.swift`, whose runtime claims the C names
   `<prefix>_atomic_load`, `<prefix>_atomic_store`, `<prefix>_atomic_add`,
   `<prefix>_atomic_exchange`, `<prefix>_atomic_cas`, and `<PREFIX>_ATOMICS_H`. An item that

ffi/README.md

@@ -27,8 +27,9 @@ A `[replace]` from the source it replaces, which no `[patch]` can carry, is an e
 Beside the C header, `include/<prefix>.hpp` holds the same API in C++, and
 `kotlin/<prefix>.kt` holds it in Kotlin, over JNI functions the crate builds with
 `--features kotlin`, and `kotlin/<prefix>.pro` the R8 and ProGuard rules keeping what
-those functions look up by name, and `swift/<prefix>.swift` holds it in Swift, over the C
-header as a clang module.
+those functions look up by name. `python/<prefix>.py` holds it in Python, over a CPython
+extension the crate builds with `--features python`, and `swift/<prefix>.swift` holds it in
+Swift, over the C header as a clang module.
 
 ```toml
 # ffi.toml, beside the package's Cargo.toml unless --manifest-path says otherwise.
@@ -273,6 +274,59 @@ in as a `ByteArray`, bytes returned as their owner, and anything else encoded in
   Android library ships it as a consumer rule (`consumerProguardFiles`), which its AAR
   carries to each app that shrinks it.
 
+## Python
+
+`<prefix>.py` is Python over a CPython extension that `--features python` builds into the
+same library on the stable ABI, so one build loads in CPython 3.11 and later. The module
+loads the library from beside itself: `lib<prefix>.so`, `lib<prefix>.dylib`, or
+`<prefix>.dll`. Its glue converts each Python value to the C value and calls the C ABI
+function in-process, so a call is one crossing, which releases the GIL while Rust runs.
+
+- **Names** are in the module: types in PascalCase without the prefix, members in
+  snake_case without their type's name (`moq.TrackProducer.append_group`). A keyword, a
+  member Python gives every such class, or a method named as a field takes a trailing `_`.
+  Constructors such as `new` stay static methods: `Counter.new(1)`.
+- **Objects** are classes owning their handle, freed once their last reference goes, or
+  sooner by `close()` or a `with` block once no call holds it; a call after it raises
+  `ValueError`. A `self` method, or passing the object by value, closes it once the call
+  returns. `Clone` is `clone()`, and `PartialEq` is `==`, unhashable. A field is a
+  property, and its setter a method, `set_first(value)`, since setting an object moves it
+  in and closes it, which an assignment would hide.
+- **Records** are slotted dataclasses, compared by their fields, and **enums** an `Enum`
+  of upper-case members, or for one with data a class of a dataclass per variant,
+  `Event.Moved(p)`. `Display` is `str()`, `Debug` `repr()`, and `Default` a static
+  `default()`. Constants are module or class attributes, made anew on each read unless a
+  literal.
+- **Bytes** passed in are any buffer: `bytes` is read in place, and others, which Python
+  may change while Rust reads, are copied. Bytes Rust returns, or a record holds, are a
+  read-only `memoryview` lending Rust's buffer until it and its views are released. Those
+  Rust lends a closure or trait method are copied into `bytes`. Strings are `str`,
+  `Option<T>` `T | None`, a tuple a `tuple`, `Duration` `datetime.timedelta`, truncated to
+  the microsecond, with `timedelta.max` for any longer and for `Duration::MAX`, a `Range` a
+  `range`, and an `Err` a raised `<E>Exception` holding it in `error`, whose `str` is its
+  `Display`.
+- A **path** is a `str`, or any path passed in, which crosses as `os.fsencode` encodes it, so
+  one that is not UTF-8 keeps its bytes. A `SocketAddr` is the `socket` module's tuple,
+  `(host, port)` or `(host, port, flowinfo, scope_id)`, of that family for `V4` or `V6`.
+  A `Vec<T>` or `&[T]` is a `list`, and passing a `Vec` by value closes each object in it.
+  An integer out of its Rust type's range raises `OverflowError`.
+- **Traits** are abstract classes an implementation subclasses; once Rust drops one, its
+  `close()` is called if it has one.
+- **Closures** lent for a call are any callable, which Rust calls only during the call,
+  from any thread. An object it lends is closed once the call returns. An exception a
+  closure or trait method raises prints its traceback and aborts, since Rust has nowhere
+  to unwind it to.
+- **Futures** are `async def`s, which run nothing until awaited, or for a function that is
+  not `async`, an awaitable the call returns. They resume on the running event loop and
+  never inside a wake: a wake from any thread queues the future's token and writes a byte
+  to a socket the loop reads, without the GIL. Cancelling the task frees the future and
+  loses nothing. The call polls it once before suspending, and a future is awaited once,
+  as a coroutine is.
+- A **waker** is a `Waker(callback)`, which Rust may wake from any thread, calling
+  `callback` on its event loop.
+- **The contract** is checked when the module imports, before any other call: a Python
+  file and a library from different builds raise `ImportError`.
+
 ## Swift
 
 `swift/<prefix>.swift` is Swift over the C ABI, for Apple's platforms and Linux. It imports
@@ -327,12 +381,14 @@ its Rust path, its span, and its docs. A type is in the exact form Rust takes or
 (`&str` apart from `String`, `Box<dyn Trait>` apart from `impl Trait`), so a generator can
 borrow where Rust borrows. [`src/abi.rs`](src/abi.rs) decides the C ABI from it once,
 names and signatures, so [`src/c.rs`](src/c.rs), [`src/cpp.rs`](src/cpp.rs),
-[`src/kotlin.rs`](src/kotlin.rs), [`src/swift.rs`](src/swift.rs), and
-[`src/glue.rs`](src/glue.rs) agree; the runtime the glue includes is
-[`runtime/rt.rs`](runtime/rt.rs), the one the C++ header includes
+[`src/kotlin.rs`](src/kotlin.rs), [`src/python.rs`](src/python.rs),
+[`src/swift.rs`](src/swift.rs), and [`src/glue.rs`](src/glue.rs) agree; the runtime the
+glue includes is [`runtime/rt.rs`](runtime/rt.rs), the one the C++ header includes
 [`runtime/rt.hpp`](runtime/rt.hpp), the ones Kotlin and its JNI glue include
-[`runtime/rt.kt`](runtime/rt.kt) and [`runtime/jni.rs`](runtime/jni.rs), and the one the
-Swift includes [`runtime/rt.swift`](runtime/rt.swift).
+[`runtime/rt.kt`](runtime/rt.kt) and [`runtime/jni.rs`](runtime/jni.rs), the ones Python
+and its extension include [`runtime/rt.py`](runtime/rt.py) and
+[`runtime/python.rs`](runtime/python.rs), and the one the Swift includes
+[`runtime/rt.swift`](runtime/rt.swift).
 
 ## Census
 
@@ -345,13 +401,13 @@ refusal, which is a bug in the exporter's inference rather than in the crate.
 ## Tests
 
 ```bash
-just ffi test     # export the fixtures, build them, and drive api from C, C++, Kotlin, and Swift
-just ffi moq      # export moq's crates at the pinned revision, build them, and run moq/moq.c, moq/moq.cpp, moq/moq.kt, and moq/moq.swift
+just ffi test     # export the fixtures, build them, and drive api from C, C++, Kotlin, Python, and Swift
+just ffi moq      # export moq's crates at the pinned revision, build them, and run moq/moq.c, moq/moq.cpp, moq/moq.kt, moq/moq.py, and moq/moq.swift
 just ffi moq-cpp  # the same export, running only moq/moq.cpp
 just ffi moq-swift  # the same export, running only moq/moq.swift
 just ffi census   # census each into target/census
 nix develop .#android --command just ffi android  # the Kotlin test, shrunk by R8, on the Android device adb reaches
-nix develop .#bench --command just ffi bench  # frame writes and reads, through the export from C, C++, Kotlin, and Swift, and moq's rs/moq-c
+nix develop .#bench --command just ffi bench  # frame writes and reads, through the export from C, C++, Kotlin, Python, and Swift, and moq's rs/moq-c
 ```
 
 The `api` fixture covers each rule once, and names the `dep` fixture's types, which
@@ -367,6 +423,10 @@ same on ART: the Kotlin and `tests/api.kt`, shrunk by R8 into an app's dex with
 generated rules keeping what JNI looks up, against the library built by the NDK for the
 device's ABI, which ANDROID_HOME and ANDROID_NDK_HOME locate. Nightly runs it on an
 x86_64 emulator at API 24, the oldest moq's Android library supports, and at API 35.
+`tests/api.py` runs the script through the Python, in CPython's development mode with
+warnings as errors, checks its futures the same way, what frees an object or bytes nobody
+closed, and the values Python's types bound, and breaks each runtime check, which must raise or
+abort with its message; the Python with another contract must refuse the library.
 `tests/api.swift` runs the script through the Swift, built as a module in Swift 6's
 language mode and type-checked in Swift 5's, checks its futures the same way and that
 releasing frees, and breaks each runtime check, which must trap or abort with its message.
@@ -383,7 +443,9 @@ QUIC, both on moq-tokio's fallback runtime, whose threads wake C's. `moq/moq.cpp
 same through the C++ header, short of the catalog's field, at C++17 without exceptions or
 RTTI, as OBS builds it, serving on an executor's thread and blocking on each read, and
 `moq/moq.kt` through the Kotlin, serving in a coroutine and suspending on each read, and
-`moq/moq.swift` through the Swift, serving in a task and awaiting each read.
+`moq/moq.py` through the Python, running the origin's driver and the server as asyncio
+tasks and awaiting each read, and `moq/moq.swift` through the Swift, serving in a task and
+awaiting each read.
 
 The C++ builds take `CXX`, or `c++`, with every warning an error and `-Wshadow`, which
 stands in for MSVC's warnings that a name hides another. On Windows they take MSVC's `cl`

ffi/fixtures/api/src/lib.rs

@@ -195,6 +195,20 @@ pub fn divide(a: u64, b: u64) -> Result<(u64, u64), Error> {
 	}
 }
 
+/// The unit corner, borrowed for good, failing unless `unit`: a borrowed tuple returns as
+/// one value, not an out per element.
+pub fn corner(unit: bool) -> Result<&'static (u32, u32), Error> {
+	match unit {
+		true => Ok(&(1, 1)),
+		false => Err(Error::Overflow),
+	}
+}
+
+/// [`corner`]'s, from a future.
+pub async fn corner_later() -> &'static (u32, u32) {
+	&(1, 1)
+}
+
 /// `n` narrowed as [`widths`] does, unless 0: a tuple inside an `Option` is a struct.
 pub fn spread(n: u64) -> Option<(u8, i16, u32, u64)> {
 	(n != 0).then(|| widths(n))
@@ -581,6 +595,12 @@ pub fn filter(lines: Vec<u8>, mut keep: impl FnMut(u8, &str) -> bool + Send) ->
 	})
 }
 
+/// Logs a line once polled, returning whether it was kept: the future holds `log` until it
+/// is dropped.
+pub fn log_later(log: Box<dyn Log>) -> impl Future<Output = bool> + use<> {
+	async move { log.log(0, "later") }
+}
+
 /// Holds a foreign implementation as a trait object, dropping it with itself.
 pub struct Logger {
 	sink: Box<dyn Log>,
@@ -1313,6 +1333,11 @@ pub fn ports(addrs: Vec<std::net::SocketAddr>) -> Vec<u16> {
 	addrs.iter().map(std::net::SocketAddr::port).collect()
 }
 
+/// `ttl` doubled, saturating, so `Duration::MAX` stays forever.
+pub fn doubled_ttl(ttl: Duration) -> Duration {
+	ttl.saturating_mul(2)
+}
+
 /// `name` in `dir`.
 pub fn file(dir: &std::path::Path, name: &str) -> std::path::PathBuf {
 	dir.join(name)

ffi/justfile

@@ -46,7 +46,7 @@ run *args:
 census checkout="": (standin checkout)
     cargo run --locked --quiet -- --config moq/ffi.toml --manifest-path target/moq-ffi/Cargo.toml --census target/census
 
-# Time frame writes through the export, from C, C++, Kotlin, and Swift, and through moq's hand-written rs/moq-c, in the bench shell: nix develop .#bench.
+# Time frame writes through the export, from C, C++, Kotlin, Python, and Swift, and through moq's hand-written rs/moq-c, in the bench shell: nix develop .#bench.
 bench checkout="": (moq checkout)
     #!/usr/bin/env bash
     set -euo pipefail
@@ -75,10 +75,11 @@ bench checkout="": (moq checkout)
         target/bench-export
         target/bench-export-cpp
         kotlin -Djava.library.path="$out/target/release" -cp "target/bench-kt:$KOTLINX_COROUTINES" ExportKt
+        PYTHONPATH="$out/python" python3 -P moq/bench/export.py
         if $swift; then target/bench-export-swift; fi
     done
 
-# Export moq's crates' C ABI, build it, and run moq/moq.c, moq/moq.cpp, moq/moq.kt, and moq/moq.swift against it, from the pinned moq revision or a checkout.
+# Export moq's crates' C ABI, build it, and run moq/moq.c, moq/moq.cpp, moq/moq.kt, moq/moq.py, and moq/moq.swift against it, from the pinned moq revision or a checkout.
 moq checkout="": (moq-cpp checkout)
     #!/usr/bin/env bash
     set -euo pipefail
@@ -90,6 +91,8 @@ moq checkout="": (moq-cpp checkout)
     rm -rf target/moq-kt
     kotlinc -Werror -cp "$KOTLINX_COROUTINES" "$out/kotlin/moq.kt" moq/moq.kt -d target/moq-kt
     kotlin -Djava.library.path="$out/target/release" -cp "target/moq-kt:$KOTLINX_COROUTINES" MoqKt
+    # The Python, through the CPython extension the python feature builds.
+    PYTHONPATH="$out/python" python3 -P -X dev -W error moq/moq.py
     if [ "{{ os() }}-{{ arch() }}" = linux-aarch64 ]; then
         echo "skipping moq/moq.swift: nixpkgs builds no Swift for aarch64-linux (quest/m2/uniffi-swift-arm.md)" >&2
     else
@@ -153,7 +156,17 @@ moq-export checkout="": (standin checkout)
     cp target/moq-ffi/Cargo.lock "$out/Cargo.lock"
     # A later nightly makes this future-compat lint a hard error; fail on it in moq's crates
     # and the glue now, while the pin can still move. rustc prints the native libraries.
-    RUSTFLAGS="-D recursion_depth_exceeding_limit --print=native-static-libs" cargo build --quiet --release --manifest-path "$out/Cargo.toml" --features kotlin 2>&1 | tee target/moq-build.log
+    export RUSTFLAGS="-D recursion_depth_exceeding_limit --print=native-static-libs"
+    # The CPython extension's symbols would leave the C programs' links undefined, so it
+    # builds apart, before the library they link, and moves beside the Python.
+    case "{{ os() }}" in
+        windows) lib=moq.dll ;;
+        macos) lib=libmoq.dylib ;;
+        *) lib=libmoq.so ;;
+    esac
+    cargo build --quiet --release --manifest-path "$out/Cargo.toml" --features python
+    mv "$out/target/release/$lib" "$out/python/$lib"
+    cargo build --quiet --release --manifest-path "$out/Cargo.toml" --features kotlin 2>&1 | tee target/moq-build.log
     sed -n 's/^note: native-static-libs: //p' target/moq-build.log > target/moq-native
 
 # Write target/moq-ffi, an empty library on every crate moq/ffi.toml exports, from the pinned moq revision or a checkout: the check builds with all of them, as moq-ffi's remainder will.

ffi/moq/bench/export.py

@@ -0,0 +1,126 @@
+# bench/export.kt's frame writes and reads through the generated Python, so the two differ
+# by what CPython and the Python wrapper add, with the calls bench/ffi/README.md compares
+# BoltFFI's Python on. Each round runs twice, the first to warm CPython's caches.
+
+import asyncio
+import time
+
+import moq
+
+GROUPS = 1000
+FRAMES = 100
+
+
+def per_frame(start):
+	"""Nanoseconds per frame since `start`."""
+	return (time.perf_counter_ns() - start) / (GROUPS * FRAMES)
+
+
+def round(loop):
+	broadcast = moq.BroadcastInfo.new().produce()
+	with moq.Timescale.MICRO as micro:
+		track = broadcast.create_track("bench", moq.TrackInfo.default().with_timescale(micro))
+
+	# A frame of 100 zero bytes, as bench/export.c writes.
+	frame = bytes(100)
+	micros = 0
+
+	# Each frame stamped through a fresh timestamp object, as a caller holding an int must.
+	start = time.perf_counter_ns()
+	for _ in range(GROUPS):
+		with track.append_group() as group:
+			for _ in range(FRAMES):
+				with moq.Timestamp.from_micros(micros) as at:
+					group.write_frame(at, frame)
+				micros += 1
+			group.finish()
+	stamped = per_frame(start)
+
+	# The same, reusing one timestamp object: what the object costs.
+	at = moq.Timestamp.from_micros(micros)
+	start = time.perf_counter_ns()
+	for _ in range(GROUPS):
+		with track.append_group() as group:
+			for _ in range(FRAMES):
+				group.write_frame(at, frame)
+			group.finish()
+	reused = per_frame(start)
+
+	# The floor: a hold of a handle and a bool back.
+	start = time.perf_counter_ns()
+	closed = 0
+	for _ in range(GROUPS * FRAMES):
+		if track.is_closed():
+			closed += 1
+	call = per_frame(start)
+
+	# A record returned: the track's info, its fields made as Python objects.
+	start = time.perf_counter_ns()
+	for _ in range(GROUPS * FRAMES):
+		track.info()
+	info = per_frame(start)
+
+	# Reads: each frame written, then read through an awaitable ready at its first poll,
+	# so it never suspends.
+	async def ready():
+		with track.consume() as consumer:
+			subscriber = await consumer.subscribe(None)
+		last = None
+		start = time.perf_counter_ns()
+		for g in range(GROUPS):
+			with track.append_group() as group, await subscriber.recv_group() as reader:
+				for f in range(FRAMES):
+					group.write_frame(at, frame)
+					read = await reader.read_frame()
+					if g == GROUPS - 1 and f == FRAMES - 1:
+						last = read
+					else:
+						read.close()
+				group.finish()
+		subscriber.close()
+		return per_frame(start), last
+
+	read, last = loop.run_until_complete(ready())
+
+	# Woken reads: each read awaited before its frame is written, so it pends, and the
+	# write's wake resumes it through the loop.
+	async def woken():
+		with track.consume() as consumer:
+			subscriber = await consumer.subscribe(None)
+		start = time.perf_counter_ns()
+		for _ in range(GROUPS // 10):
+			with track.append_group() as group, await subscriber.recv_group() as reader:
+				for _ in range(FRAMES):
+					pending = asyncio.ensure_future(reader.read_frame())
+					await asyncio.sleep(0)
+					group.write_frame(at, frame)
+					(await pending).close()
+				group.finish()
+		subscriber.close()
+		return per_frame(start) * 10
+
+	wake = loop.run_until_complete(woken())
+
+	# Payload reads: the last frame's 100 bytes, lent without a copy, then released.
+	start = time.perf_counter_ns()
+	with last:
+		for _ in range(GROUPS * FRAMES):
+			with last.payload as payload:
+				closed += payload[0]
+	lent = per_frame(start)
+	at.close()
+	track.close()
+	broadcast.close()
+
+	bang = "!" if closed != 0 else ""
+	return (
+		f"export (Python): {stamped:.0f} ns per stamped 100 B frame, {reused:.0f} reusing the timestamp, "
+		f"{call:.1f} ns per handle call, {info:.0f} ns per record return, {read:.0f} ns per frame written and read, "
+		f"{wake:.0f} ns per frame written and read after a wake, {lent:.0f} ns per payload read{bang}"
+	)
+
+
+loop = asyncio.new_event_loop()
+round(loop)
+print(round(loop))
+loop.close()