BittyCrab - Re-optimized Bitnet.cpp, less memory and faster with a few tricks.

[MrOldenburg]

I optimized a few things, created a Mojo version against 15 prompts but it bottomed out at 40 tokens a second on my CPU with 24 cores. Bitnet.cpp was about 30-40 tokens per second. Switched to rust and kept optimizing, rust matched the 8 core performance max (P cores) while Mojo got faster slowly while wanted all the cores but never coming close.

Couple tricks, one was to offload the fp16 and use a int8 table, ternary values come out int8 then a quick k lookup which is almost free because the range so small,BitNet's numbers are so tightly compacted like in the paper this gave 100% accuracy at k4. Got up to 70 tokens with a tiny decoder.

The int8 was probably the real win cheap, easy and fp16 was easily offloaded using 900MB with a lot more context which is below the original implementation. The int8 saved about 300mb of ram.

Keeping it small would be good to load onto cellphones or older devices.

Codes here if interested, and yes I used AI, it was a fun project to learn BitNet and how it works inside out.

BittyCrab