[kixelated]

Finishes quest/m1/ffi-export/python.md.

Problem

moq ships Python bindings through the UniFFI fork's ctypes backend, measured at 834 ns per noop(). The exporter generates C, C++, and Kotlin, but no Python, so moq's Python can't move off the fork.

Approach

Every export now writes python/<prefix>.py, plus a CPython extension that the crate builds with --features python on pyo3-ffi 0.30's stable ABI (abi3-py311). One build loads in every CPython from 3.11. As with Kotlin's JNI, the glue skips the C ABI's boundary. It converts each Python value to the C value itself and calls the C ABI function in-process, so a call is one crossing.

• Generator (src/python.rs): a thin Python layer naming each function, plus one Rust extension function per export. Records cross field by field, read with getattr after a type check and made with tp_alloc plus PyObject_GenericSetAttr. No byte encoding.
• Runtime (runtime/python.rs, runtime/rt.py): conversions, object handles, and asyncio wakes.
  • Each call into Rust releases the GIL, so Rust may call Python back from any thread.
  • Callbacks and closures retake the GIL with PyGILState_Ensure.
  • A wake never takes the GIL. It pushes a token and writes one byte to a per-loop socket pair that the loop reads with add_reader, or on a proactor loop with a sock_recv task.
• Futures:
  • An async fn is an async def that runs nothing until awaited. It polls once and returns its value at once if ready.
  • A pending future parks on an asyncio future keyed by its token. A wake that lands during the poll is latched (IDLE/POLLING/PARKED/WOKEN), and the poll reruns.
  • Cancelling the task frees the C future, which loses nothing.
  • A function that returns a future without being async runs at the call and returns the awaitable.
• Drivers: moq/moq.py runs the origin's driver as an asyncio task. It holds one Waiter made from a Waker(event.set), polls on each wake and at each deadline, and stops when the driver finishes.
• Contract: the module checks _n.contract() at import, before any other call. It hashes the header's contract with the generated Python, the glue, and both runtimes.
• Tests:
  • api_python builds the fixture with --features python before the Kotlin build that the C programs link. The extension's Python symbols would otherwise leave their links undefined.
  • It runs tests/api.py under -X dev -W error. That checks the transcript line for line, then futures, objects and bytes nobody closed, value bounds, six broken contracts, and a mismatched contract.
  • The nightly runs it on Windows and macOS too.
  • just ffi moq runs moq/moq.py, and just ffi bench runs moq/bench/export.py.

Choices made without asking, and where Python differs from Kotlin

• Names: types are PascalCase, and members and functions snake_case. A trailing _ goes on keywords, on close, on an Enum's name/value, and on a method that shares a field's name (moq's Position.group). Python has no companion namespace to keep those apart.
• Objects subclass the native _n.Object, and are freed when their last reference goes, or sooner by close() or with. A call after close raises ValueError("Counter is closed"). A call holds the handle, so close() during an await frees it once the call ends.
• Records are @dataclass(slots=True), mutable and compared by fields. Plain enums are enum.Enum with UPPER_SNAKE members. Data enums are a base class with one dataclass per variant, as in Event.Moved(p) and Event.Done(). Debug is repr(), and Display is str().
• Bytes:
  • Passed in, bytes is read in place, and any other buffer is copied, as Python may change it while Rust reads without the GIL.
  • Returned, Rust's buffer is lent as a read-only memoryview with no 1 KiB copy threshold, and release() or with frees it.
  • Bytes lent to a callback are copied to bytes.
• Paths cross as os.fsencode encodes them. A Unix path that isn't UTF-8 round-trips through surrogateescape, where Kotlin throws.
• Durations are datetime.timedelta, truncated to the microsecond. Anything past timedelta.max returns as timedelta.max, which passes back as Duration::MAX.
• Socket addresses are socket tuples. (host, port) is IPv4, and (host, port, flowinfo, scope_id) is IPv6.
• Integers are range-checked and raise OverflowError. bool is strict.
• Errors raise <E>Exception(error), with str(e) being the error's Display. A String error is StringException.
• Callbacks must subclass the generated ABC, and closures must be callable. Both are checked before the call.
• Exceptions in callbacks: an exception raised in a callback or closure prints its traceback and aborts, as Kotlin aborts on a throw.
• Awaiting: a Rust future is awaited once, as a coroutine is, and a second await raises RuntimeError. That is the concurrent-poller semantic.
• Constants: non-literal constants are made anew on each read, through module __getattr__ or a class descriptor.
• Build script: the generated crate gains a build.rs calling pyo3_build_config::add_extension_module_link_args(), the upstream way to link an extension on macOS.

Measured

Under heavy host load (load average 30 to 50 on 24 threads), so these are indicative.

Call overhead:

• The GIL release costs about 25 ns per call: dropped() takes 122 ns released and 97 ns held.
• Skipping the release is only safe for a call that can never block on a lock that an upcall holds, which the glue can't prove, so every call releases it.
• abi3 measures the same as the version-specific API on plain and conversion-heavy calls, within noise (point.offset 316 to 362 ns against 311 to 319 ns; rows with 8 records 2.37 to 2.41 µs against 2.40 to 2.44 µs). abi3 stays.

just ffi bench, three runs of moq/bench/export.py against the same build's C, C++, and Kotlin, with BoltFFI's Python from bench/ffi/README.md for comparison (its numbers come from a different slice and host):

per frame or call	Python	Kotlin	C	BoltFFI Python
handle call (track.is_closed())	122 to 127 ns	12.5 to 13.2 ns	7.1 to 7.3 ns	63 ns (group.sequence())
record return (track.info())	221 to 263 ns			1,120 ns
100 B write, reused timestamp	385 to 549 ns	311 to 344 ns	293 to 308 ns	667 ns
100 B write, stamped	1,028 to 1,531 ns	419 to 617 ns	305 to 330 ns
write and ready read	1,348 to 1,473 ns	866 to 1,203 ns	551 to 560 ns
write and woken read	20.6 to 21.6 µs			10.0 µs (await yield_once())
payload read, zero-copy	379 to 398 ns	84 to 104 ns	16 to 17 ns

The woken read is an ensure_future, a sleep(0), a write, and the wake through the loop. asyncio alone spends about 9 µs of it, and the socket pair about 5 µs.

Generated size for moq:

	raw	gzipped
moq.py	1,034 KiB	198 KiB
moq.py without docstrings	363 KiB	44 KiB
the fork's Python for moq-ffi	653 KiB	66 KiB
stripped extension, over the Kotlin feature's library	+1,993 KiB	+312 KiB

The glue's Rust source is 1,815 KiB and ships compiled.

The export covers moq's crates whole, with every doc comment, rather than moq-ffi's hand-written surface. import moq takes about 100 ms with cached bytecode, half of it asyncio's own import.

Impact

• New: every export writes python/<prefix>.py. The export crate gains a python feature (pyo3-ffi, plus pyo3-build-config for its build script) and a build.rs.
• Breaking: an item whose Python name clashes with another's, or with the runtime's _-prefixed names, is an error that takes a rename (MIGRATE.md).
• api-fixture gains doubled_ttl, corner, corner_later, and log_later, so the Python can check durations at their bounds, borrowed tuples, and a future polled during its free.
• The default dev shell gains python3, and the uniffi shell inherits it.

Alternatives

• ctypes or cffi over the C ABI: ruled out by the Plan, at 834 ns per noop().
• Encoding values into bytes, as Kotlin's JNI glue does: CPython's API reads attributes and builds objects directly, so an encoding would only add a copy.
• Waking through call_soon_threadsafe: Rust's wake would take the GIL, from any thread and often under Rust's own locks. Rust code run holding the GIL, such as a drop in a dealloc, could then deadlock against it, and every wake would contend for the GIL.
• Releasing the GIL only for calls taking a closure or callback: about 25 ns faster, but it could deadlock a call that blocks on a lock an upcall holds.

Follow-ups

• A wake on the loop's own thread could resume the future directly, skipping the socket pair's write and read: about 5 of the woken read's 21 µs.
• just ffi standin and just ts share ts/target/moq, and pin it to different moq revisions. Running both at once in one worktree (just ci beside just ffi bench) fails the export with a missing crate.

Review

Reviewer: Codex, gpt-6.1-sol at medium effort, through quest review.

• Round 4, head cea21c2b, the merge of main after Swift (feat(ffi): generate Swift over the C ABI, as a clang module #406) landed, with conflicts in the ffi README, MIGRATE.md, justfile, src/glue.rs, src/main.rs, tests/fixtures.rs, and the nightly workflow, resolved so both generators coexist: clean.
• Earlier rounds, all fixed:
  • Round 1, head 24e4ddf8: buffers other than bytes read without the GIL (now copied); a close() during a poll freed the future under it (now deferred until the poll returns); a borrowed tuple's return split into one out per element (now whole, with corner and corner_later).
  • Round 2, head b17d4532: Python run during a future's free could poll it with a null pointer (reproduced as an abort; the future now ends before the free releases the GIL, with log_later).
  • Round 3, head fb66b632: clean.

just ci passed on cea21c2b, with api_python and api_swift, and just ffi moq ran all five sessions.

🤖 Generated with Claude Code

[coderabbitai]

Important

• Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 7c9648c7-5cba-435d-af6e-c152ca0d043c

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[moq-bot]

codex / Codex land fixer note:

The merge failure was reproduced with git merge --no-commit --no-ff base: CONFLICT (content): Merge conflict in .github/workflows/nightly.yml. The PR adds Python workflow coverage where base adds Swift coverage. Resolving conflicts under .github/ is prohibited for this fixer, so the merge was aborted and the PR branch remains unchanged. Builds and tests were not run because the protected merge conflict prevents a permitted fix.

Failure log line 1:

Conflicts with the base: .github/workflows/nightly.yml, ffi/MIGRATE.md, ffi/README.md, ffi/justfile, ffi/src/glue.rs, ffi/src/main.rs, ffi/tests/fixtures.rs

(written by codex / Codex)

[moq-bot]

codex / Codex land fixer: git merge --no-commit --no-ff base failed: CONFLICT (content): Merge conflict in .github/workflows/nightly.yml; resolving this protected path is prohibited.. Recommendation: Have the maintainer merge base into the PR, preserving the Python and Swift workflow changes and resolving the FFI conflicts, then rerun the FFI build and tests.

Conflicts with the base: .github/workflows/nightly.yml, ffi/MIGRATE.md, ffi/README.md, ffi/justfile, ffi/src/glue.rs, ffi/src/main.rs, ffi/tests/fixtures.rs

(written by codex / Codex)