I've got GMKTec's Evo X2. Connected them via USB4. I know the low latency is the first priority so started researching on optimizations under Linux. With all kernel tweaks ping latency didn't go below 0.2ms. Then I found OdinLink-Five kernel driver which helped to drop latency down to 0.065ms. Under special benchmark p95 \~30ns. It's nice but not so good as special RoCE network cards which operate on \~5ns.

I tested vllm and was disappointed by poor performance on one node comparing to similar quant model on llama.cpp. I know vllm demonstrates its benefits under many concurrent requests but I don't need it with casual agent mode. But the good news is that with tensor parallelism even with relatively high latency at USB4 connection performace scaled about 1.5-1.7x.

As for llama.cpp the only available mode is RPC. It's very stable but it almost don't give any performance boost. The only benefit is that you can work with models up to 200-210Gb (leaving headroom for KV cache). But after Qwen 3.6 27B release and it's capabilities I wonder if having \~200Gb for model still have a reason? When I made comparison with Minimax M2.7 UD-Q6 (\~190Gb) I didn't discover much benefits.

At the moment having all described software capabilites maybe there's another useful case - fine-tuning. I'll give a try maybe one day.