[Roadmap] TransferQueue Q2 roadmap #49

Performance

MooncakeStore [Call for Contribution]

• Use MooncakeStore put_from and get_into + serial_utils.py of TQ to eliminate the complex conditional branch for tensor and non-tensor data, and use zero-copy to improve performance. @0oshowero0

  TransferQueue/transfer_queue/storage/clients/mooncake_client.py

  Lines 112 to 133 in 4e9ae22

  	tensor_keys = []
  	tensor_values = []
  	non_tensor_keys = []
  	non_tensor_values = []
  	for key, value in zip(keys, values, strict=True):
  	if isinstance(value, torch.Tensor):
  	tensor = value.contiguous()
  	# TODO: use gpu direct rdma instead
  	if tensor.device.type == "cuda":
  	tensor = tensor.cpu()
  	tensor_keys.append(key)
  	tensor_values.append(tensor)
  	else:
  	non_tensor_keys.append(key)
  	non_tensor_values.append(pickle.dumps(value))
  	if tensor_keys:
  	self._batch_put_tensors(tensor_keys, tensor_values)
  	if non_tensor_keys:
  	self._batch_put_bytes(non_tensor_keys, non_tensor_values)

  [perf] Refactor MooncakeStore backend with zero-copy upsert API #77
• Support GPU tensor transfer using GDR
• Use MooncakeStore provided buffer pool implementation [Store] Introduce buffer pool for zero copy interfaces kvcache-ai/Mooncake#2095

Optimization

MooncakeStore [Call for Contribution]

• Use hard-pin and upsert mechanism to enable manual data lifetime control [RFC]: Hard Pin and Upsert for Mooncake Store kvcache-ai/Mooncake#1645 [Store] Add hard pin mechanism for eviction-protected objects kvcache-ai/Mooncake#1728 [Store] Add Upsert API for in-place object updates kvcache-ai/Mooncake#1662 @0oshowero0 [perf] Refactor MooncakeStore backend with zero-copy upsert API #77
• Update MooncakeStore version to prevent handshake issue [TE] Fix simultaneous open handshake in RdmaEndpoint kvcache-ai/Mooncake#1733
• Use batch_remove (when ready) to replace remove_by_regex @0oshowero0 [perf] Refactor MooncakeStore backend with zero-copy upsert API #77

Yuanrong

• Improve usability by eliminate etcd @dpj135 @tianyi-ge [feat] Support metastore mode for Yuanrong backend init #74
• Support one-click auto init for Yuanrong backend Add automatic Yuanrong startup to interface.py #60 @dpj135 Add automatic Yuanrong startup to interface.py #60

General

• Refactor BatchMeta as ordinary class to avoid extra transformation before sending @0oshowero0
• Extract common optimizations from Yuanrong backend to upper level

Stress Test

• Large-scale stress test TransferQueue Stress Testing - 0.1.6 @0oshowero0

Integrations

verl

• Follow up upstream requirements

ROLL

• Support ROLL integration
• Implement LazyDataProto that prevent massive modification for single-controller [design doc]https://www.yuque.com/haomingzi-lfse7/lhp4el/ffxkyne0piygsp8d?singleDoc# 《LazyDataProto》) @0oshowero0

slime

• Discuss with the community for TQ integration

Documentation & Tutorial

• Website: Build a dedicated documentation website.
• API Reference: Generate a comprehensive API list.
• Onboarding: Provide guiding docs to simplify the user onboarding process.
• Tutorial: Add pluggable backend tutorial, demonstrate how to switch between supported backends and illustrate the custom_backend_meta usage. @tianyi-ge

CI/CD

• Unified Workflow: Unify CI experience across GitCode and GitHub (addressing missing tests on GitCode).
• Performance Checkpoint: Add periodic performance testing to detect regressions in new PRs.
• Refactoring: Refactor unit test logic for better maintainability.