Implement AVX-512 intrinsics

General instructions for this can be found at #40, but the list of AVX-512 intrinsics is quite large! This is intended to help track progress but you'll likely want to talk to us out of band to ensure that everything is coordinated.

Intrinsic lists: https://gist.github.com/alexcrichton/3281adb58af7f465cebee49759ae3164

I think the best instruction set to get started with is probably avx512f as it has the constructors for types that we can use for all the other sets:

["AVX512F"]

Dissecting one of the interesting intrinsics here:

/// Compute the absolute value of packed 8-bit integers in a, 
/// and store the unsigned results in dst using writemask k (elements 
/// are copied from src when the corresponding mask bit is not set).
__m512i _mm512_mask_abs_epi8 (__m512i src, __mmask64 k, __m512i a);

the __mmask64 type appears, which is a 64-bit mask where LLVM requires us to implement it as a <64 x i1> vector that must be allocated to a k64 registers. In AVX-512 k registers are mask registers, and what seems to be more interesting is that AVX-512 does seem to support i1 as a type that is legal to lower to a cleared k register with the first bit either set or unset...

So it would be nice to know how does exactly all of this works in LLVM because i1 types are illegal in all other x86 "targets" (e.g. AVX2). Does anybody know?

Another difference with AVX2 is that if we want to use a mask in AVX2 to select values from two u8x32, the mask is an i8x32 with each byte either set or unset but IIUC AVX-512 __mmask32 is also usable for this, but it requires 32bits instead. It would be nice to know if these two (i8x32 as a mask and __mmask32) can interact, and if so, how. going from __mmask32 to i8x32 can probably be done in LLVM as sext <32 x i1> to <32 x i8> and the opposite with a trunc <32 x i8> to <32 x i1> but maybe there is a different way in which these things must be done.

This affects boolean vectors / masks, because bool8x32 would need to be casteable to bool1x32 and vice-versa.

So it would be nice to know how does exactly all of this works in LLVM because i1 types are illegal in all other x86 "targets" (e.g. AVX2). Does anybody know?

I don't understand how it works, but there are possibly relevant slides from the 2017 LLVM meeting, maybe they are useful: https://llvm.org/devmtg/2017-03//assets/slides/avx512_mask_registers_code_generation_challenges_in_llvm.pdf

Another possibly relevant point is that AVX512VL extends the mask registers (and the corresponding intrinsics) to 128- and 256-bit vectors. But, at least when using these from C, LLVM will currently just use blend instructions instead of masks: https://godbolt.org/g/FjU1Xn

But, at least when using these from C, LLVM will currently just use blend instructions instead of masks: https://godbolt.org/g/FjU1Xn

That's a really nice test. Do you know if there is an LLVM bug open for it? I haven't been able to find any.

Hi, has there been any new developments since this was last active? I would like to contribute AVX-512 intrinsics, but I'm not sure what (if anything) is blocking it, so if anyone has any pointers I'd be happy to help!

You can add any intrinsic that does not use __mmask.. types without issues.

If you want to add an intrinsic that uses __mmask..., you would need to add the mask types first. It is unclear what that would take. A #[repr(simd)] struct __mmask64(i64); might just work, or it might fail spectacularly. AFAIK nobody has tried yet.

Clang defines masks like __mmask16 as just (https://github.com/llvm-mirror/clang/blob/master/lib/Headers/avx512fintrin.h#L48):

typedef unsigned char __mmask8;
typedef unsigned short __mmask16;
typedef unsigned int __mmask32;
typedef unsigned long long __mmask64;

So maybe just a wrapper struct without #[repr(simd)] would be enough:

pub struct __mmask8(u8);
pub struct __mmask16(u16);
pub struct __mmask32(u32);
pub struct __mmask64(u64);

Cool! I'll give it a try some time this week, RustConf permitting.

Should AVX-512 intrinsics be split into modules corresponding to their feature flag?

This seems sensible except that I'm not sure how it should interact with the AVX512VL extension, since it seems weird to have the 512/256/128-bit versions of the same intrinsic in different places.

@hdevalence

@hdevalence we currently split the functionality in modules corresponding to their target-feature flag and/or cpuid flag. I expect avx512f, avx512vl, etc. to be their own modules like they are in clang.

This stuff is decided on a 1:1 basis though, whoever sets the PR can get the conversation started. Are there any technical reasons to split it in any other way?

Hmm, but the VL flag is orthogonal to the other flags, so for instance the _mm256_madd52hi_epu64 intrinsic requires IFMA and VL. Where should it live?

@hdevalence

@hdevalence in clang they live in an avx512ifmavl header... avx-512 is complicated :/ many intrinsics require two features...

EDIT: typically the ones that require avx512f + avx512{something_else} live in the {something_else} module though.

avx-512 is complicated :/

no kidding... looking at the AVX-512 Venn diagram:

it seems like the only CPUs that don't have VL extensions for all of their supported AVX-512 instructions are the Xeon Phi cores, which I think are all cancelled now, so it seems like the common case will be that if a CPU supports an instruction it will almost certainly support the VL extensions for it.

In that case, maybe it makes sense to split the intrinsics into modules avx512f, avx512ifma, etc., and then within those modules separately gate the VL variants on the avx512vl flag. This is still correct in the edge case that VL is not present, but seems like a more logical grouping... I think clang maybe can't really do this because C doesn't have a module system.

Does this seem like a sensible arrangement?

it seems like the only CPUs that don't have VL extensions for all of their supported AVX-512 instructions are the Xeon Phi cores, which I think are all cancelled now,

I don't think we should worry about these. Some of these did not support SSE4.2 and IIRC AVX2 either (only AVX-512), and we can't target them with LLVM IIRC.

Does this seem like a sensible arrangement?

Sure. If once we start this way we discover that putting these into their own modules makes things clearer, we can always do that later.

@minybot

I had a look in the compiler and it seems that this is a bug in the implementation of simd_select_bitmask: it should accept u8 inputs when the number of lanes is less than 8. simd_bitmask already supports this by returning u8 when the number of lanes is less than 8.

@minybot @bjorn3 Would one of you be willing to make a PR to fix this in rustc? The relevant code is here: https://github.com/rust-lang/rust/blob/f3c923a13a458c35ee26b3513533fce8a15c9c05/compiler/rustc_codegen_llvm/src/intrinsic.rs#L1272

There is another solution without touching simd_select_bitmask.
Use cast. Take _mm512_mask_extractf32x4_ps (__m128 src, __mmask8 k, __m512 a, int imm8) as an example.
a->(32x4); Cast to (32x16); Cast to (32x8); do bitmask; Cast to (32x4).
There is no cast128_to_256 directly. only 128_to_512, 512_to_256. 512_to_128.

I just went ahead and fixed the issue in rust-lang/rust#77504.

I just went ahead and fixed the issue in rust-lang/rust#77504.

I test it, and it works when the mask size is 4.

For Mask operation in avx512 such as _kadd_mask32, it adds two masks.
According to https://travisdowns.github.io/blog/2019/12/05/kreg-facts.html, the Mask has its own hardware register.
Is there anyway to make sure _kadd_mask32 will generate "kaddd" instruction?

No, but it's fine since we don't guarantee a particular instruction is used for an intrinsic: we leave it to LLVM to decide whether it is better to use a kadd instruction or a normal add instruction.

While working on a private project, I needed masked loading, so I wanted to prepare a PR with implementations for _mm512_mask_load_epi32 and the like. Reading https://github.com/rust-lang/stdarch/blob/master/crates/core_arch/avx512f.md, I found the following:

_mm512_mask_load_epi32 //need i1

_mm512_maskz_load_epi32 //need i1

What is the "need i1" part? I have not found any explanation there.

Currently, I am tempted to implement masked loading like in (as an example)

/// Load packed 32-bit integers from memory into dst using writemask k (elements are copied from src when the corresponding mask bit is not set). mem_addr must be aligned on a 64-byte boundary or a general-protection exception may be generated.
///
/// [Intel's documentation](https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm512_mask_load_epi32&expand=3305)
#[inline]
#[target_feature(enable = "avx512f")]
#[cfg_attr(test, assert_instr(vmovdqa32))]
pub unsafe fn _mm512_mask_load_epi32(src: __m512i, k: __mmask16, mem_addr: *const i32) -> __m512i {
    let loaded = ptr::read(mem_addr as *const __m512i).as_i32x16();
    let src = src.as_i32x16();
    transmute(simd_select_bitmask(k, loaded, src))
}

which follows how _mm512_maskz_mov_epi32 and _mm512_load_epi32 are implemented. If this sounds correct, I might make a PR in the next days.

This is incorrect since _mm512_mask_load_epi32 must not cause page faults on the parts of the vector that are masked off. Your version will still cause these page faults.

To support this properly we need to call an LLVM intrinsic directly. However this intrinsic uses a vector of i1 as argument, which we cannot represent with Rust types. We need additional support in the compiler to call LLVM intrinsics that take a vector of i1 as a parameter.

Makes sense. Many thanks for the explanation.

Another possible implementation for _mm512_mask_load_epi32 would using the asm feature. I have successfully used the following implementation:

#[inline]
pub unsafe fn _mm512_mask_loadu_epi32(src: __m512i, mask: __mmask16, ptr: *const i32) -> __m512i {
    let mut result: __m512i = src;

    asm!(
    "vmovdqu32 {io}{{{k}}}, [{p}]",
    p = in(reg) ptr,
    k = in(kreg) mask,
    io = inout(zmm_reg) result,
    options(nostack), options(pure), options(readonly)
    );

    result
}

If such an implementation would be ok maintenance wise I could try preparing a PR that adds the missing avx512f this way.

If such an implementation would be ok maintenance wise I could try preparing a PR that adds the missing avx512f this way.

Sounds good!

@Amanieu

Just coming from the discussion: rust-lang/portable-simd#28.

Regarding the separation of avx512f intrinsics and and target_feature=avx512f, now, I have enough interest and time to investigate it.
My particular case of interest is using zmm_reg for inline assembly (so need for avx512f intrinsics), but target_feature=avx512f is not stable yet. If it helps the stabilisation of target_feature, I am willing to work on it under some guidance.
@Amanieu What do you think?

I expect that we will be stabilizing AVX-512 soon, thanks to the hard work of many people in implementing the full set of AVX-512 intrinsics in stdarch.

I believe this is a nice time to bring up the topic of avx512vp2intersect 😅 - it is stuck due to no i1 support in rustc. Is there any other way we can implement them? That would truly complete the avx512 set

With #2081 the AVX512 set is finally complete ❤️

Navigation Menu

Repository navigation

Implement AVX-512 intrinsics #310

73 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Implement AVX-512 intrinsics #310

Description

Activity

alexcrichton commented on Jan 29, 2018

gnzlbg commented on Mar 16, 2018

hdevalence commented on Apr 4, 2018

gnzlbg commented on Apr 6, 2018

hdevalence commented on Aug 15, 2018

gnzlbg commented on Aug 15, 2018

gnzlbg commented on Aug 15, 2018

hdevalence commented on Aug 15, 2018

hdevalence commented on Sep 11, 2018

gnzlbg commented on Sep 11, 2018

hdevalence commented on Sep 11, 2018

gnzlbg commented on Sep 11, 2018

hdevalence commented on Sep 11, 2018

gnzlbg commented on Sep 11, 2018

73 remaining items

minybot commented on Oct 1, 2020

Amanieu commented on Oct 3, 2020

minybot commented on Oct 6, 2020

minybot commented on Dec 21, 2020

Amanieu commented on Dec 21, 2020

stopbystudent commented on Apr 7, 2021

Amanieu commented on Apr 7, 2021

stopbystudent commented on Apr 7, 2021

jhorstmann commented on Nov 8, 2021

Amanieu commented on Nov 8, 2021

mert-kurttutan commented on Jul 16, 2024

Amanieu commented on Jul 25, 2024

sayantn commented on Apr 17, 2025

sayantn commented on May 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions