candle

This is the visioncortex fork — upstream candle at 0.11.0 (2cf0e66d) plus the fixes and the ROCm and Vulkan backends listed below, published as 0.11.1 on branch main. The public API is upstream's; every change is a fix, a backend or a performance path behind it. The list is kept in order of the layers as they sit on upstream.

Notable fixes in this fork

CPU

• Large copies run on all cores. copy2d (cat, narrow, pad), strided-source copies (transpose, pixel shuffle, contiguous() of a view), nearest upsample and the NCHW→NHWC transpose inside the tiled conv2d were serial loops; whole-image CNN inference on a 4k-edge raster ran on one core with the rayon pool idle (16 threads bought 2.2× over one). They fan out over rayon above 256k elements with byte-identical results; the 1×1 conv's input reshape likewise.
• The backward pass runs on all cores too. A sum over some dims (every conv bias gradient) walked one element at a time; it now splits over output elements, each still adding its sources in increasing order, so the sums are bit-identical. Large affine, the tiled conv2d's kernel flatten (activation-sized when the kernel is an output gradient) and the transpose conv's input packing are parallel as well.
• A tensor's first gradient is stored as it comes rather than added to a fresh zeros_like, which cost a zero buffer and a full add per gradient (CPU and GPU alike). Values are unchanged. With all of this a small conv net's CPU training step went from 9.2 s to 1.9 s on 32 cores, with byte-identical weights.
• conv1d with a strided kernel view matmul'd the original buffer at its old offset (the cpu conv2d fix of #3736 had not reached conv1d).
• The thread count is computed once. get_num_threads re-parsed /proc/cpuinfo (num_cpus::get_physical) on every matmul and every fanned-out copy: a 512×512 x2 pass opened it 20k times and a whole-image x8 chain spent 30% of its CPU time in the kernel (x8 26.6 s → 18.4 s on 16 cores, system time 207 s → 22 s).

Metal (Apple Silicon)

• MetalDevice::new panicked on M-series: the system default device is used.
• The conv weight-gradient was wrong through a non-contiguous im2col and training diverged; the gradient's inputs are made contiguous.
• Errored in-flight command buffers are surfaced from flush_and_wait_current instead of their output being read as garbage.
• Wired-memory growth from the buffer pool is bounded, the pool sweep is non-blocking (it could self-deadlock), and the residency set is skipped where the class is missing.
• conv1d/conv2d with a non-contiguous kernel view used the original buffer at its old start offset; index_select bailed on strided ids where CPU/CUDA accept them; a zero-length buffer came back nil from newBufferWithBytes.

CUDA

• cudarc dynamic loading probes the CUDA DLLs at runtime: a plain-cuda binary degrades to CPU instead of failing to start on a machine without the driver.

ROCm — a new backend, --features rocm

• candle-core/src/rocm_backend and the candle-rocm-kernels crate: the HIP backend begun in upstream #3424, continued in cryptomilk/candle-rocm and rebased here (lineage in candle-rocm-kernels/README.md). The shared candle-kernels CUDA sources are compiled under HIP through a header shim, so nothing drifts from the CUDA kernels.
• Convolutions through MIOpen, pooling, upsample, gather/scatter/index_add, cmp/where, arg_sort, RNG, rotary embeddings, softmax/norms; F8E4M3 and bf16 arithmetic; quantized GGUF tensors with AMD WMMA MMQ/MMVQ matmul paths retuned for RDNA geometry, imatrix quantization and the quantized MoE forward.
• A VMM pool for address-stable allocations under HIP graph capture/replay, host↔device copies off the legacy stream, async host-to-device, a bounded allocator cache so shape-drifting decode cannot hoard VRAM (CANDLE_ROCM_CACHE_LIMIT_MB), the materialized im2col buffer capped at 256 MB through chunked kernels.
• Tests needing hipMemCreate skip on APUs that lack it (gfx1151, Strix Halo), so the suite runs on integrated parts as well as discrete cards.

Vulkan — a new backend, --features vulkan (inference)

• candle-core/src/vulkan_backend: the Vulkan backend of upstream #4096 (nglmercer's port of rexlunae/candle-core joshua-vulkan), applied here as its Vulkan-only diff. ash loads libvulkan at runtime and naga compiles the GLSL kernels to SPIR-V on first use, so neither a Vulkan SDK nor glslc is needed to build or run; any Vulkan 1.1 GPU with a conformant driver (RADV, NVIDIA, Intel ANV, Mesa on Android) is a target. f32 kernels for elementwise ops, strided copies, casts, reductions, index ops, GEMM/GEMV, softmax/RMSNorm/RoPE and the quantized GGUF matmuls; software devices (lavapipe) are refused unless CANDLE_VULKAN_ALLOW_CPU=1. Ops without a kernel take a counted CPU round-trip (vulkan_backend::stats(), CANDLE_VULKAN_TRACE=1).
• Added here on top of the PR: conv2d (im2col in chunks of whole output rows, capped at 256 MB, then GEMM straight into the output planes), conv_transpose2d, avg/max pooling and nearest upsampling as native kernels — the ops a convolutional net spends its time in, which the PR still ran on the CPU. Ported from candle-kernels/src/conv.cu, so the semantics (including strided inputs and kernels) match the CUDA and ROCm paths exactly; vulkan_conv_pool_upsample_match_cpu checks them against the CPU backend.
• Tuned for convolutional nets: freed blocks stay mapped for reuse (CANDLE_VULKAN_CACHE_LIMIT_MB, default 1024), conv2d reuses a device-owned im2col scratch, and a reduction of few long rows (group norm) is split across ~1k work-groups. On a Strix Halo iGPU (RADV) a 4.5M-parameter conv U-Net's x8 inference runs in 11.4 s against 12.1 s through ROCm and 26 s on 16 CPU cores, with the output within 1 LSB of the CPU's. cargo run --release -p candle-core --features vulkan --example conv_bench -- vulkan times the building blocks per device.
• Discrete cards with Resizable BAR, and APUs whose unified memory is not host-cached: host reads of the tensor mapping run at ~0.1 GB/s, so downloads are copied by the GPU into a host-cached staging buffer first (7× faster on Strix Halo; a host-cached unified type is preferred where a driver offers one, as Intel's does). vulkan_backend::best_adapter() ranks adapters for a caller without a preference: a Resizable-BAR card, then an integrated GPU, never a software device; vk_memtypes prints the heaps and types behind the choice.
• A discrete card without Resizable BAR exposes only its 256 MB BAR window as host-visible device memory; since every tensor is mapped, the device is refused at bind (a caller's Auto falls back to the CPU) unless CANDLE_VULKAN_SYSTEM_MEMORY=1 asks to run from system memory over PCIe (the conv im2col scratch then lives in real VRAM; tensors do not). Measured on a GeForce MX150 (PCIe 3 x4): two passes take 45 s from system memory against 18.7 s on its CPU, so the refusal stays the default. Weak iGPUs (Intel UHD 620) run correctly but no faster than their CPU; pick the device per machine.

Candle is a minimalist ML framework for Rust with a focus on performance (including GPU support) and ease of use. Try our online demos: whisper, LLaMA2, T5, yolo, Segment Anything.

Get started

Make sure that you have candle-core correctly installed as described in Installation.

Let's see how to run a simple matrix multiplication. Write the following to your myapp/src/main.rs file:

use candle_core::{Device, Tensor};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let device = Device::Cpu;

    let a = Tensor::randn(0f32, 1., (2, 3), &device)?;
    let b = Tensor::randn(0f32, 1., (3, 4), &device)?;

    let c = a.matmul(&b)?;
    println!("{c}");
    Ok(())
}

cargo run should display a tensor of shape Tensor[[2, 4], f32].

Having installed candle with Cuda support, simply define the device to be on GPU:

- let device = Device::Cpu;
+ let device = Device::new_cuda(0)?;

On an AMD GPU, build with the rocm feature and use:

- let device = Device::Cpu;
+ let device = Device::new_rocm(0)?;

On any GPU with a Vulkan driver, build with the vulkan feature and use:

- let device = Device::Cpu;
+ let device = Device::new_vulkan(0)?;

For more advanced examples, please have a look at the following section.

Check out our examples

These online demos run entirely in your browser:

• yolo: pose estimation and object recognition.
• whisper: speech recognition.
• LLaMA2: text generation.
• T5: text generation.
• Phi-1.5, and Phi-2: text generation.
• Segment Anything Model: Image segmentation.
• BLIP: image captioning.

We also provide some command line based examples using state of the art models:

• LLaMA v1, v2, and v3: general LLM, includes the SOLAR-10.7B variant.
• Falcon: general LLM.
• Codegeex4: Code completion, code interpreter, web search, function calling, repository-level
• GLM4: Open Multilingual Multimodal Chat LMs by THUDM
• Gemma v1 and v2: 2b and 7b+/9b general LLMs from Google Deepmind.
• RecurrentGemma: 2b and 7b Griffin based models from Google that mix attention with a RNN like state.
• Phi-1, Phi-1.5, Phi-2, and Phi-3: 1.3b, 2.7b, and 3.8b general LLMs with performance on par with 7b models.
• StableLM-3B-4E1T: a 3b general LLM pre-trained on 1T tokens of English and code datasets. Also supports StableLM-2, a 1.6b LLM trained on 2T tokens, as well as the code variants.
• Mamba: an inference only implementation of the Mamba state space model.
• Mistral7b-v0.1: a 7b general LLM with better performance than all publicly available 13b models as of 2023-09-28.
• Mixtral8x7b-v0.1: a sparse mixture of experts 8x7b general LLM with better performance than a Llama 2 70B model with much faster inference.
• StarCoder and StarCoder2: LLM specialized to code generation.
• Qwen1.5: Bilingual (English/Chinese) LLMs.
• RWKV v5 and v6: An RNN with transformer level LLM performance.
• Replit-code-v1.5: a 3.3b LLM specialized for code completion.
• Yi-6B / Yi-34B: two bilingual (English/Chinese) general LLMs with 6b and 34b parameters.
• Quantized LLaMA: quantized version of the LLaMA model using the same quantization techniques as llama.cpp.
• Quantized Qwen3 MoE: support gguf quantized models of Qwen3 MoE models.

• Stable Diffusion: text to image generative model, support for the 1.5, 2.1, SDXL 1.0 and Turbo versions.

• Wuerstchen: another text to image generative model.

• yolo-v3 and yolo-v8: object detection and pose estimation models.

• segment-anything: image segmentation model with prompt.

• SegFormer: transformer based semantic segmentation model.
• Whisper: speech recognition model.
• EnCodec: high-quality audio compression model using residual vector quantization.
• MetaVoice: foundational model for text-to-speech.
• Parler-TTS: large text-to-speech model.
• T5, Bert, JinaBert : useful for sentence embeddings.
• DINOv2: computer vision model trained using self-supervision (can be used for imagenet classification, depth evaluation, segmentation).
• VGG, RepVGG: computer vision models.
• BLIP: image to text model, can be used to generate captions for an image.
• CLIP: multi-model vision and language model.
• TrOCR: a transformer OCR model, with dedicated submodels for hand-writing and printed recognition.
• Marian-MT: neural machine translation model, generates the translated text from the input text.
• Moondream: tiny computer-vision model that can answer real-world questions about images.

Run them using commands like:

cargo run --example quantized --release

In order to use CUDA add --features cuda to the example command line. If you have cuDNN installed, use --features cudnn for even more speedups. The opt-in cutile feature supports writing JIT-compiled CUDA kernels in Rust; see the cuTile guide for setup and interop details.

For an AMD GPU add --features rocm instead. This needs a ROCm 6.2+ installation with hipcc on PATH and clang-offload-bundler under $ROCM_PATH (default /opt/rocm). The kernels are the same candle-kernels sources the CUDA backend uses; they are compiled by hipcc at runtime on first use and cached under ~/.cache/candle-rocm, so the first run of a given architecture pays a one-off compile (a few seconds per module, about 70 s for the quantized module) and later runs load from the cache. If MIOpen is installed, --features miopen swaps convolutions onto it, the way cudnn layers over cuda. The ROCm backend started from @airpods69's #3424; see candle-rocm-kernels.

For any GPU with a Vulkan driver add --features vulkan. Nothing is needed at build time: the Vulkan loader is opened at runtime and the GLSL kernels are compiled to SPIR-V in-process by naga on first use. The backend is inference-oriented (no RNG or backward-specific kernels; those ops fall back to the CPU and are counted in vulkan_backend::stats()).

There are also some wasm examples for whisper and llama2.c. You can either build them with trunk or try them online: whisper, llama2, T5, Phi-1.5, and Phi-2, Segment Anything Model.

For LLaMA2, run the following command to retrieve the weight files and start a test server:

# install target platform 'wasm32-unknown-unknown'
rustup target add wasm32-unknown-unknown

cd candle-wasm-examples/llama2-c
wget https://huggingface.co/spaces/lmz/candle-llama2/resolve/main/model.bin
wget https://huggingface.co/spaces/lmz/candle-llama2/resolve/main/tokenizer.json
trunk serve --release --port 8081

And then head over to http://localhost:8081/.

Useful External Resources

• candle-tutorial: A very detailed tutorial showing how to convert a PyTorch model to Candle.
• candle-lora: Efficient and ergonomic LoRA implementation for Candle. candle-lora has
  out-of-the-box LoRA support for many models from Candle, which can be found here.
• candle-video: Rust library for text-to-video generation (LTX-Video and related models) built on Candle, focused on fast, Python-free inference.
• optimisers: A collection of optimisers including SGD with momentum, AdaGrad, AdaDelta, AdaMax, NAdam, RAdam, and RMSprop.
• candle-vllm: Efficient platform for inference and serving local LLMs including an OpenAI compatible API server.
• candle-ext: An extension library to Candle that provides PyTorch functions not currently available in Candle.
• candle-coursera-ml: Implementation of ML algorithms from Coursera's Machine Learning Specialization course.
• kalosm: A multi-modal meta-framework in Rust for interfacing with local pre-trained models with support for controlled generation, custom samplers, in-memory vector databases, audio transcription, and more.
• candle-sampling: Sampling techniques for Candle.
• gpt-from-scratch-rs: A port of Andrej Karpathy's Let's build GPT tutorial on YouTube showcasing the Candle API on a toy problem.
• candle-einops: A pure rust implementation of the python einops library.
• atoma-infer: A Rust library for fast inference at scale, leveraging FlashAttention2 for efficient attention computation, PagedAttention for efficient KV-cache memory management, and multi-GPU support. It is OpenAI api compatible.
• llms-from-scratch-rs: A comprehensive Rust translation of the code from Sebastian Raschka's Build an LLM from Scratch book.
• vllm.rs: A minimalist vLLM implementation in Rust based on Candle.

If you have an addition to this list, please submit a pull request.

Features

• Simple syntax, looks and feels like PyTorch.
  • Model training.
  • Embed user-defined ops/kernels, such as flash-attention v2.
• Backends.
  • Optimized CPU backend with optional MKL support for x86 and Accelerate for macs.
  • CUDA backend for efficiently running on GPUs, multiple GPU distribution via NCCL.
  • ROCm/HIP backend for AMD GPUs, sharing the CUDA kernel sources, with optional MIOpen convolutions.
  • WASM support, run your models in a browser.
• Included models.
  • Language Models.
    • LLaMA v1, v2, and v3 with variants such as SOLAR-10.7B.
    • Falcon.
    • StarCoder, StarCoder2.
    • Phi 1, 1.5, 2, and 3.
    • Mamba, Minimal Mamba
    • Gemma v1 2b and 7b+, v2 2b and 9b.
    • Mistral 7b v0.1.
    • Mixtral 8x7b v0.1.
    • StableLM-3B-4E1T, StableLM-2-1.6B, Stable-Code-3B.
    • Replit-code-v1.5-3B.
    • Bert.
    • Yi-6B and Yi-34B.
    • Qwen1.5, Qwen1.5 MoE, Qwen3 MoE.
    • RWKV v5 and v6.
  • Quantized LLMs.
    • Llama 7b, 13b, 70b, as well as the chat and code variants.
    • Mistral 7b, and 7b instruct.
    • Mixtral 8x7b.
    • Zephyr 7b a and b (Mistral-7b based).
    • OpenChat 3.5 (Mistral-7b based).
    • Qwen3 MoE (16B-A3B, 32B-A3B)
  • Text to text.
    • T5 and its variants: FlanT5, UL2, MADLAD400 (translation), CoEdit (Grammar correction).
    • Marian MT (Machine Translation).
  • Text to image.
    • Stable Diffusion v1.5, v2.1, XL v1.0.
    • Wurstchen v2.
  • Image to text.
    • BLIP.
    • TrOCR.
  • Audio.
    • Whisper, multi-lingual speech-to-text.
    • EnCodec, audio compression model.
    • MetaVoice-1B, text-to-speech model.
    • Parler-TTS, text-to-speech model.
  • Computer Vision Models.
    • DINOv2, ConvMixer, EfficientNet, ResNet, ViT, VGG, RepVGG, ConvNeXT, ConvNeXTv2, MobileOne, EfficientVit (MSRA), MobileNetv4, Hiera, FastViT.
    • yolo-v3, yolo-v8.
    • Segment-Anything Model (SAM).
    • SegFormer.
• File formats: load models from safetensors, npz, ggml, or PyTorch files.
• Serverless (on CPU), small and fast deployments.
• Quantization support using the llama.cpp quantized types.

How to use

Cheatsheet:

	Using PyTorch	Using Candle
Creation	torch.Tensor([[1, 2], [3, 4]])	Tensor::new(&[[1f32, 2.], [3., 4.]], &Device::Cpu)?
Creation	torch.zeros((2, 2))	Tensor::zeros((2, 2), DType::F32, &Device::Cpu)?
Indexing	tensor[:, :4]	tensor.i((.., ..4))?
Operations	tensor.view((2, 2))	tensor.reshape((2, 2))?
Operations	a.matmul(b)	a.matmul(&b)?
Arithmetic	a + b	&a + &b
Device	tensor.to(device="cuda")	tensor.to_device(&Device::new_cuda(0)?)?
Dtype	tensor.to(dtype=torch.float16)	tensor.to_dtype(&DType::F16)?
Saving	torch.save({"A": A}, "model.bin")	candle::safetensors::save(&HashMap::from([("A", A)]), "model.safetensors")?
Loading	weights = torch.load("model.bin")	candle::safetensors::load("model.safetensors", &device)

Structure

• candle-core: Core ops, devices, and Tensor struct definition
• candle-nn: Tools to build real models
• candle-examples: Examples of using the library in realistic settings
• candle-kernels: CUDA custom kernels
• candle-rocm-kernels: ROCm/HIP support, compiling the candle-kernels sources with hipcc
• candle-datasets: Datasets and data loaders.
• candle-transformers: transformers-related utilities.
• candle-flash-attn: Flash attention v2 layer.
• candle-onnx: ONNX model evaluation.

FAQ

Why should I use Candle?

Candle's core goal is to make serverless inference possible. Full machine learning frameworks like PyTorch are very large, which makes creating instances on a cluster slow. Candle allows deployment of lightweight binaries.

Secondly, Candle lets you remove Python from production workloads. Python overhead can seriously hurt performance, and the GIL is a notorious source of headaches.

Finally, Rust is cool! A lot of the HF ecosystem already has Rust crates, like safetensors and tokenizers.

Other ML frameworks

• dfdx is a formidable crate, with shapes being included in types. This prevents a lot of headaches by getting the compiler to complain about shape mismatches right off the bat. However, we found that some features still require nightly, and writing code can be a bit daunting for non rust experts.

  We're leveraging and contributing to other core crates for the runtime so hopefully both crates can benefit from each other.

• burn is a general crate that can leverage multiple backends so you can choose the best engine for your workload.

• tch-rs Bindings to the torch library in Rust. Extremely versatile, but they bring in the entire torch library into the runtime. The main contributor of tch-rs is also involved in the development of candle.

Common Errors

Missing symbols when compiling with the mkl feature.

If you get some missing symbols when compiling binaries/tests using the mkl or accelerate features, e.g. for mkl you get:

  = note: /usr/bin/ld: (....o): in function `blas::sgemm':
          .../blas-0.22.0/src/lib.rs:1944: undefined reference to `sgemm_' collect2: error: ld returned 1 exit status

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo

or for accelerate:

Undefined symbols for architecture arm64:
            "_dgemm_", referenced from:
                candle_core::accelerate::dgemm::h1b71a038552bcabe in libcandle_core...
            "_sgemm_", referenced from:
                candle_core::accelerate::sgemm::h2cf21c592cba3c47 in libcandle_core...
          ld: symbol(s) not found for architecture arm64

This is likely due to a missing linker flag that was needed to enable the mkl library. You can try adding the following for mkl at the top of your binary:

extern crate intel_mkl_src;

or for accelerate:

extern crate accelerate_src;

Cannot run the LLaMA examples: access to source requires login credentials

Error: request error: https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/tokenizer.json: status code 401

This is likely because you're not permissioned for the LLaMA-v2 model. To fix this, you have to register on the huggingface-hub, accept the LLaMA-v2 model conditions, and set up your authentication token. See issue #350 for more details.

Docker build

When building CUDA kernels inside a Dockerfile, nvidia-smi cannot be used to auto-detect compute capability.

You must explicitly set CUDA_COMPUTE_CAP, for example:

FROM nvidia/cuda:12.9.0-devel-ubuntu22.04

# Install git and curl
RUN set -eux; \
  apt-get update; \
  apt-get install -y curl git ca-certificates;

# Install Rust
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

# Clone candle repo
RUN git clone https://github.com/huggingface/candle.git

# Set compute capability for the build
ARG CUDA_COMPUTE_CAP=90
ENV CUDA_COMPUTE_CAP=${CUDA_COMPUTE_CAP}

# Build with explicit compute cap
WORKDIR /app
COPY . .
RUN cargo build --release features cuda

Compiling with flash-attention fails

/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:

This is a bug in gcc-11 triggered by the Cuda compiler. To fix this, install a different, supported gcc version - for example gcc-10, and specify the path to the compiler in the NVCC_CCBIN environment variable.

env NVCC_CCBIN=/usr/lib/gcc/x86_64-linux-gnu/10 cargo ...

Linking error on windows when running rustdoc or mdbook tests

Couldn't compile the test.
---- .\candle-book\src\inference\hub.md - Using_the_hub::Using_in_a_real_model_ (line 50) stdout ----
error: linking with `link.exe` failed: exit code: 1181
//very long chain of linking
 = note: LINK : fatal error LNK1181: cannot open input file 'windows.0.48.5.lib'

Make sure you link all native libraries that might be located outside a project target, e.g., to run mdbook tests, you should run:

mdbook test candle-book -L .\target\debug\deps\ `
-L native=$env:USERPROFILE\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.42.2\lib `
-L native=$env:USERPROFILE\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.48.5\lib

ROCm errors

Kernel compilation failed: could not run hipcc: No such file or directory (os error 2). Is ROCm installed?

The ROCm backend compiles kernels at runtime, so hipcc must be on PATH at run time, not just at build time — add /opt/rocm/bin to PATH. The matching clang-offload-bundler is taken from $ROCM_PATH (default /opt/rocm), so point ROCM_PATH at your install if it lives elsewhere.

ROCm error: hipGetDeviceProperties failed for device 0 ... set CANDLE_ROCM_ARCH to build kernels anyway

or a launch failing with an "invalid device function"

The target architecture is read from the device the Device::new_rocm(ordinal) call opened, via hipGetDeviceProperties. Override it if that is unavailable or reports the wrong target:

CANDLE_ROCM_ARCH=gfx1101 cargo run --release --example bert --features rocm

The cache key covers the kernel sources, the HIP shim headers, the compile flags and the toolchain version, so editing a kernel or upgrading ROCm invalidates the affected entries on its own. To force a recompile anyway — say a crash left a truncated code object behind:

CANDLE_ROCM_FORCE_RECOMPILE=1 cargo run --release --example bert --features rocm
make rocm-cache-clean   # or: rm -rf ~/.cache/candle-rocm

Set CANDLE_ROCM_CACHE_DIR to move the cache elsewhere. Without a writable $HOME — containers, CI, service accounts — it falls back to a per-uid directory under the system temp dir.

Extremely slow model load time with WSL

This may be caused by the models being loaded from /mnt/c, more details on stackoverflow.

Tracking down errors

You can set RUST_BACKTRACE=1 to be provided with backtraces when a candle error is generated.

CudaRC error

If you encounter an error like this one called Result::unwrap()on anErr value: LoadLibraryExW { source: Os { code: 126, kind: Uncategorized, message: "The specified module could not be found." } } on windows. To fix copy and rename these 3 files (make sure they are in path). The paths depend on your cuda version. c:\Windows\System32\nvcuda.dll -> cuda.dll c:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\cublas64_12.dll -> cublas.dll c:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\curand64_10.dll -> curand.dll