Get you some GPUs, it's not worth the hacks around lack of RAM

Couple of things come things come to mind that would have an effect... One is that your GPU P2P is not working and everything is going through host memory when your GPUs want to communicate. Another is the lack of PCIe lanes, having one GPU getting the full x16 from the CPU and the other the x4 from the chipset is the worst possible configuration, since the x4 limits the bandwidth AND all the traffic has to go through system memory thus tanking the performance when you have a lot of context. PCIe bandwidth wasn't a big deal 1-2 years ago when people were just chatting (or just running the llama-bench with 512 prompts and 128 tg), but now with people running huge context lengths it has started mattering a lot. Optimally on a consumer board you'd want to be running the GPUs directly from the CPU lanes on x8 / x8 (bifurcating the x16 between the cards) with P2P working (the GPUs can communicate directly without going through system memory). Adding on the NVLink bridge on top would be optimal. A quick way to test is to run llama.cpp with --split-mode tensor, if running that tanks your performance completely then your GPUs are most likely communicating via system memory and not P2P (assuming your cards have a reasonable amount of PCIe lanes).