SpatialCrafter: Single Image World Modeling with Generative 3D Proxies

Chuan Fang, Lingteng Qiu, Yixun Liang, Rui Chen, Kunming Luo, Zhaohua Zheng, Tongyuan Bai, Feipeng Tian, Zilong Dong, Zihan Zhou, Ping Tan

HKUST Spatial Artificial Intelligence Lab; Manycore Tech Inc; Alibaba Group; Jilin University;

TL;DR: Given a single image and a consecutive camera trajectory, SpatialCrafter builds generative 3D proxies from a single image to enable consistent, controllable video generation.

✨ News

• [Oct, 2026] Initial release of SpatialCrafter!
• [Aug, 2026] Paper is released on arXiv!
• [Jul, 2026] SpatialCrafter is accepted to SIGGRAPH-Asia 2026!

📋 Release Plan

• Inference code && Pre-trained model weights
• Data-Engine code
• Training code

SpatialCrafter Models

Model	Download
SpatialCrafter-1.0	🤗 HuggingFace

Usage

🔧 Installation

Tested with the following environment:

• Python 3.10
• PyTorch 2.5.1
• CUDA Version 12.4
• NVIDIA A100 and H20

# clone the repository
git clone https://github.com/fangchuan/SpatialCrafter.git
cd SpatialCrafter

python -m venv .venv
source .venv/bin/activate

pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu124
pip install --no-build-isolation -e .

High quality 3D Scene Data Engine

Please refer to the Data Engine documentation for more details on how to produce the high quality 3D scene data.

Inference

# download the weights of Wan2.1, DepthAnything V3
huggingface-cli download alibaba-pai/Wan2.1-Fun-V1.1-14B-Control --local-dir ./checkpoints/PAI/Wan2.1-Fun-V1.1-14B-Control
huggingface-cli download depth-anything/DA3NESTED-GIANT-LARGE-1.1 --local-dir ./checkpoints/DA3NESTED-GIANT-LARGE-1.1
export DIFFSYNTH_MODEL_BASE_PATH=$PWD/checkpoints
export DA3NESTED_MODEL_BASE_PATH=$PWD/checkpoints/DA3NESTED-GIANT-LARGE-1.1
# download the weights of TRELLIS-Scene model and SpatialCrafter
huggingface-cli download manycore-research/SpatialCrafter --local-dir ./checkpoints/SpatialCrafter

# TODO: Single image-to-RGBD Video generation
python scripts/infer_rgbd_video.py \
       --test_split_file test_data/tests.csv    \
       --ss_vae_config_file ./checkpoints/SpatialCrafter/TRELLIS_Scene/sft_ss_vae_64/config.json \
       --ss_dit_config_file ./checkpoints/SpatialCrafter/TRELLIS_Scene/sft_ss_dit_64/config.json \
       --slat_vae_config_file ./checkpoints/SpatialCrafter/TRELLIS_Scene/sft_slat_vae_64/config.json \
       --slat_dit_config_file ./checkpoints/SpatialCrafter/TRELLIS_Scene/sft_slat_dit_64/config.json \
       --wanvideo_ckpt_path ./checkpoints/SpatialCrafter/deferred_geo_refiner_rgbd.safetensors  \
       --depth_concat width_wise     \
       --video_size 512  \
       --output_path ./outputs_rgbd

# TODO: Single image-to-RGB-SCM (scene corrdinate map) Video generation
python scripts/infer_rgb_scm_video.py \
       --test_split_file test_data/tests.csv    \
       --ss_vae_config_file ./checkpoints/SpatialCrafter/TRELLIS_Scene/sft_ss_vae_64/config.json \
       --ss_dit_config_file ./checkpoints/SpatialCrafter/TRELLIS_Scene/sft_ss_dit_64/config.json \
       --slat_vae_config_file ./checkpoints/SpatialCrafter/TRELLIS_Scene/sft_slat_vae_64/config.json \
       --slat_dit_config_file ./checkpoints/SpatialCrafter/TRELLIS_Scene/sft_slat_dit_64/config.json \
       --wanvideo_ckpt_path ./checkpoints/SpatialCrafter/deferred_geo_refiner_rgbscm.safetensors  \
       --depth_concat width_wise     \
       --video_size 512  \
       --output_path ./outputs_rgb_scm

License

This project is released under the Apache License 2.0.

Citation

@article{fang2026spatialcrafter,
  title={SpatialCrafter: Single Image World Modeling with Generative 3D Proxies},
  author={Fang, Chuan and Qiu, Lingteng and Liang, Yixun and Chen, Rui and Luo, Kunming and Zheng, Zhaohua and Bai, Tongyuan and Tian, Feipeng and Dong, Zilong and Zhou, Zihan and Tan, Ping},
  journal={arXiv preprint arXiv:2608.27073},
  year={2026}
}

Announcement on Revision of TRELLIS and DiffSynth Codebase

The TRELLIS codebase used in this project have been revised from their original versions to better suit the 3D scene dataset, and DiffSynth-Studio has been revised for single image-to-RGBD and single image-to-RGB-SCM video generation tasks. These revisions include modifications to the model configurations and training codes to ensure compatibility and improved performance within the SpatialCrafter framework.

Please note that these revised models are intended for research purposes within the context of the SpatialCrafter project and may not reflect the original performance characteristics of the unmodified TRELLIS and WanVideo models.

Acknowledgements

We thank the teams behind TRELLIS and DiffSynth-Studio for open-sourcing their models, and the authors of SpatialGen for the website template.