Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

~38 TOPS at fp16 is amazing, if the quoted number if fp16 (ANE is fp16 according to this [1] but that honestly seems like a bad choice when people are going smaller and smaller even at the higher level datacenter cards so not sure why apple would use it instead of fp8 natively)

[1]: https://github.com/hollance/neural-engine/blob/master/docs/1...



For reference. The llama.cpp people are not going smaller. Most of those models run on 32 bit floats with the dequantization happening on the fly.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: