Running FP8 with vLLM on RDNA4?

Posted by Mr_Moonsilver@reddit | LocalLLaMA | View on Reddit | 1 comments

I'm having a hard time figuring out if this is possible and am looking for help if someone can point me in the right direction. Also how to find out myself is fine, i.e. which documentation would answer this.

Reply to Post

1 Comments

[-]

AustinM731@reddit

For anyone else who comes across this post looking for if RDNA4 supports FP8 in vLLM. Yes, it will run the model with the Triton kernel. Source: I own 4x R9700s and I run Devstral small 2 @ FP8 with the full 256k context window.