feat: implement virtio based host-guest communication

[andreiltd]

See rendered design doc. For broader context see HIP PR: #1112

This patch replaces stack based communication with function calls transported over virtual queues. The documentation covers two current limitations: snapshots do not support retained buffers, and the host owns the transport arena. Addressing both is deferred to future work.

[Copilot]

Note

Copilot was unable to run its full agentic suite in this review.

Pull request overview

This PR migrates host/guest communication fully to the virtqueue transport (removing legacy stack I/O), updates guest dispatch to return typed ReturnValue, and extends snapshot persistence to include canonical virtqueue state.

Changes:

• Replace guest dispatch/host-call plumbing from Vec<u8> stack buffers to typed ReturnValue over virtqueues (including C guest C-API).
• Add a versioned OCI “transport” layer to snapshots to persist virtqueue ring images and validate them on load.
• Remove embedded FlatBuffer byte tables/legacy OUTB actions and introduce ExternalValueSource support for virtqueue-backed decoding.

Reviewed changes

Copilot reviewed 80 out of 88 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/tests/rust_guests/witguest/src/main.rs	Update guest dispatch return type to ReturnValue.
src/tests/rust_guests/simpleguest/src/main.rs	Add byte-chunk guest/host functions and migrate dispatch/returns to ReturnValue.
src/tests/c_guests/c_simpleguest/main.c	Add ByteChunks round-trip and error-return behaviors for C guest testing.
src/schema/function_types.fbs	Remove embedded byte table variants; assign explicit union tags/reservations.
src/hyperlight_host/tests/snapshot_goldens/fixtures.rs	Generate goldens via UninitializedSandbox + config.
src/hyperlight_host/tests/integration_test.rs	Update tests for virtqueue/log delivery and removed legacy stack behavior.
src/hyperlight_host/tests/common/mod.rs	Add helpers for UninitializedSandbox with custom config.
src/hyperlight_host/src/sandbox/uninitialized_evolve.rs	Remove conditional virtq attachment path in evolve.
src/hyperlight_host/src/sandbox/uninitialized.rs	Update snapshot-related tests for new config/layout expectations.
src/hyperlight_host/src/sandbox/snapshot/tripwires.rs	Add transport media-type tripwire and update OUTB action assertions.
src/hyperlight_host/src/sandbox/snapshot/mod.rs	Update snapshot virtq documentation to reflect validation/immutability.
src/hyperlight_host/src/sandbox/snapshot/file/transport.rs	New transport blob codec for persisting virtqueue ring images (with tests).
src/hyperlight_host/src/sandbox/snapshot/file/mod.rs	Persist/load transport layer and validate it against layout.
src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs	Add transport layer media type constants.
src/hyperlight_host/src/sandbox/snapshot/file/digest.rs	Generalize blob verification comment (now used for more blobs).
src/hyperlight_host/src/sandbox/snapshot/file/config.rs	Remove input/output buffer sizes from snapshot config schema.
src/hyperlight_host/src/sandbox/config.rs	Remove input/output region sizing; adjust transport defaults and docs.
src/hyperlight_host/src/sandbox/builder.rs	Remove input/output sizing APIs; add virtqueue transport tuning APIs + tests.
src/hyperlight_host/src/mem/virtq_mem.rs	Adjust imports and offset calculation; extend tests.
src/hyperlight_host/src/mem/virtq/tests.rs	New virtqueue validation/snapshot/restore tests.
src/hyperlight_host/src/mem/virtq/codec.rs	New host-side virtqueue codec for function calls/logs (with tests).
src/hyperlight_host/src/mem/shared_mem.rs	Remove legacy stack I/O helpers and related tests/errors.
src/hyperlight_host/src/mem/mod.rs	Stop exporting legacy virtq_mem module publicly.
src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs	Refactor test layout creation and update FXSAVE test to use scratch.
src/hyperlight_host/src/error.rs	Add poisoning TransportError and VirtqError conversion.
src/hyperlight_host/benches/benchmarks.rs	Update benches to new transport/config + add codec benchmarks for externals.
src/hyperlight_guest_tracing/src/lib.rs	Avoid deadlock by returning None when trace state lock is held; add test.
src/hyperlight_guest_macro/src/lib.rs	Update dispatch macro/docs to return ReturnValue vs Vec<u8>.
src/hyperlight_guest_capi/src/error.rs	Rework C-API error setting to stash HyperlightGuestError instead of stack writing.
src/hyperlight_guest_capi/src/dispatch.rs	Convert C-API dispatch and host calls to typed ReturnValue over transport.
src/hyperlight_guest_bin/src/transport.rs	Pass mailbox address to guest transport initialization.
src/hyperlight_guest_bin/src/lib.rs	Initialize transport earlier (before logging/guest init); export ReturnValue.
src/hyperlight_guest_bin/src/host_comm.rs	Route host calls through transport and return typed ReturnValue.
src/hyperlight_guest_bin/src/guest_logger.rs	Emit logs via virtqueue transport (and tracing path integration).
src/hyperlight_guest_bin/src/guest_function/definition.rs	Keep return values typed (ReturnValue) until transport encoding.
src/hyperlight_guest_bin/src/guest_function/call.rs	Dispatch via virtqueue, send results, and support snapshot checkpoints.
src/hyperlight_guest/src/transport/mod.rs	Add guest-side codec module; rename with_context → with_ctx.
src/hyperlight_guest/src/transport/mem.rs	Add BufferMap-backed Bytes owner retention via GuestMapping.
src/hyperlight_guest/src/transport/codec.rs	New guest-side message decoding for requests/responses (with tests).
src/hyperlight_guest/src/lib.rs	Remove legacy guest_handle modules that depended on stack I/O.
src/hyperlight_guest/src/guest_handle/io.rs	Delete legacy stack-based input/output buffer I/O.
src/hyperlight_guest/src/guest_handle/host_comm.rs	Delete legacy host-call/logging implementation over OUTB + stack I/O.
src/hyperlight_guest/src/guest_handle/handle.rs	Move read_n_bytes_from_user_memory into GuestHandle.
src/hyperlight_guest/src/error.rs	Add conversion from GuestError and tweak bail! formatting.
src/hyperlight_component_util/src/guest.rs	Emit ReturnValue::VecBytes directly instead of size-prefixed flatbuffer helper.
src/hyperlight_common/src/virtq/ring.rs	Improve chain element storage handling; add bookkeeping error; add poll rollback support.
src/hyperlight_common/src/virtq/mod.rs	Normalize bookkeeping error; add zeroed_vec helper and test.
src/hyperlight_common/src/virtq/buffer.rs	Add Segments::append and optimize segment collection; add tests.
src/hyperlight_common/src/transport.rs	Add ExternalValueSource impls for RecvChain/Segments; harden header parsing.
src/hyperlight_common/src/outb.rs	Remove legacy stack actions and add VirtqNotify.
src/hyperlight_common/src/mem.rs	Remove PEB input/output stack regions.
src/hyperlight_common/src/layout.rs	Remove IO-buffer sizing from min scratch; add mailbox in transport arena layout.
src/hyperlight_common/src/flatbuffers/mod.rs	Stop including generated embedded byte payload tables.
src/hyperlight_common/src/flatbuffers/hyperlight/generated/return_value_generated.rs	Update generated union max/values for removed variants.
src/hyperlight_common/src/flatbuffers/hyperlight/generated/return_value_box_generated.rs	Remove accessors/verification/debug for removed return variants.
src/hyperlight_common/src/flatbuffers/hyperlight/generated/parameter_value_generated.rs	Update generated union max/values for removed variants.
src/hyperlight_common/src/flatbuffers/hyperlight/generated/parameter_generated.rs	Remove accessors/verification/debug for removed parameter variants.
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlvecbytes_generated.rs	Remove generated embedded byte-vector table.
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlsizeprefixedbytechunks_generated.rs	Remove generated embedded size-prefixed bytechunks table.
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlsizeprefixedbuffer_generated.rs	Remove generated embedded size-prefixed buffer table.
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlbytechunks_generated.rs	Remove generated embedded bytechunks table.
src/hyperlight_common/src/flatbuffer_wrappers/util.rs	Remove legacy get_flatbuffer_result; keep capacity estimator aligned to external bytes.
src/hyperlight_common/src/flatbuffer_wrappers/guest_log_level.rs	Update comment referencing renamed log test.
src/hyperlight_common/src/flatbuffer_wrappers/codec.rs	Document ExternalValueSource implementations and expectations.
src/hyperlight_common/src/arch/amd64/layout.rs	Remove IO-buffer sizing from architecture minimum scratch computation.
src/hyperlight_common/src/arch/aarch64/layout.rs	Remove IO-buffer sizing from architecture minimum scratch computation.
fuzz/fuzz_targets/host_call.rs	Update fuzz harness to new sandbox initialization/config approach and new error cases.
fuzz/fuzz_targets/fuzz_push_pop_buffer.rs	Remove fuzz target for deleted stack push/pop I/O.
fuzz/Cargo.toml	Remove fuzz binary entry for deleted target.
docs/snapshot-versioning.md	Document added transport version marker and updated ABI scope.
docs/snapshot-oci-format.md	Document new OCI transport layer and its framing.
docs/paging-development-notes.md	Update documentation to reflect virtqueue transport arena usage.
docs/README.md	Link new virtqueue host/guest communication doc.
Justfile	Remove deleted fuzz target; update isolated/coverage test names.
CHANGELOG.md	Document virtqueue-only transport, snapshot transport layer, removed legacy stack I/O, etc.

Suppressed comments (3)

src/hyperlight_common/src/transport.rs:1

• The header now derives CheckedBitPattern/NoUninit (to validate enum discriminants), but as_bytes() still uses bytemuck::bytes_of(self). Depending on the bytemuck version, bytes_of may require Pod, which MsgHeader cannot safely implement because it contains MsgKind (not all u8 bit-patterns are valid). To avoid potential compilation failures or accidental UB, consider making as_bytes() produce a [u8; MsgHeader::SIZE] via explicit little-endian encoding, and adjust callers/tests accordingly (or provide a write_to(&mut [u8; SIZE]) helper).

// SPDX-License-Identifier: Apache-2.0

src/hyperlight_common/src/transport.rs:88

• The header now derives CheckedBitPattern/NoUninit (to validate enum discriminants), but as_bytes() still uses bytemuck::bytes_of(self). Depending on the bytemuck version, bytes_of may require Pod, which MsgHeader cannot safely implement because it contains MsgKind (not all u8 bit-patterns are valid). To avoid potential compilation failures or accidental UB, consider making as_bytes() produce a [u8; MsgHeader::SIZE] via explicit little-endian encoding, and adjust callers/tests accordingly (or provide a write_to(&mut [u8; SIZE]) helper).

    pub fn as_bytes(&self) -> &[u8] {
        bytemuck::bytes_of(self)
    }

src/hyperlight_guest_bin/src/host_comm.rs:1

• The new type-mismatch error is quite minimal, which can make debugging host/guest ABI issues harder (especially now that all calls go through virtqueues). Consider including at least the host function name and the expected return type (and optionally the actual ReturnValue variant) in the error message so failures can be diagnosed without adding extra tracing.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ludfjig]

I am curious if you have run any of the benchamrks compared to main and could share?

Question about ergonomics orthogonal to this pr: Could the call on the host accept slice like &[u8] (or &[&[u8]] and &str instead of the owned counterparts? At least 1 copy is necessary on the host I believe, so looks doable on the surface maybe? I think it could get rid of some clones if host only has access to a slice and then needs to clone it unnecessarily.

Guest functions which take a String have the bytes copied from the flatbuffer into a new String if I understand correctly. Could we consider supporting guest functions which take something similar to a &str pointing directly into the flatbuffer to get rid of copy? Flatbuffer already supports it I think. An alternative could perhaps be to document that if perf is important, use ByteChunks instead of String.

Will get back reviewing this in more detail shortly!