I haven't played with all permutations of this, and none recently so things could be different than my general impressions.

I've run linux / windows desktop OS VMs with NO GPU HW exposed to either just some emulated basic (fully virtual, zero involvement with any actual GPU) "SVGA card" if/when desired. And that gets you a GUI X / whatever console on LINUX and the ability to run desktop UI. On older versions of windows anyway (XP, 7, 10 AFAIK) you could also give it as little as a "VGA/SVGA card" which could be simulated virtual when msw runs as a guest and you'd get a very basic msw GUI desktop without much or any 2D/3D acceleration, media / compute acceleration, etc. etc. Maybe newer w11 or w10 even makes a DX10 or better (or whatever, IDK) full GPU mandatory to even install / run a basic msw GUI desktop.

I've also "passed through" full control of a DGPU via VFIO to a single guest OS whether linux or msw so that guest has normal GPU drivers and OS takes full control of the card and there is NOTHING LEFT for ANY GPU directly associated functions of any kind for the host OS or any secondary, tertiary, etc. guest OS. Consumer DGPUs just haven't IME "shared well", like not at all. One OS gets it all.

Some particular / newer hypervisors MAY have some improved "share the GPU acceleration / GUI" functions available if you run a guest OS that itself has the ability to load the right paravirtualized / virtualized "fake GPU" driver which keeps the host OS in actual full control of the DGPU but the guest OS enhanced by the hypervisor and fake accelerated GPU guest drivers does get SOME level of accelerated media / GUI / compute type capabilities. So it looks like maybe a generic not fully featured GPU to the guest but it's way better than nothing for 2D/3D/media if any/all of those work out in the case. IIRC there's been some work on exposing vulkan or such functionality at that level to guest OSs but not like full ROCM / native DX-whatever etc. But the host (linux) OS gets basically almost full GPU capability while the guest runs.

Consumer DGPUs and their drivers really suck in this way since any "normal" computing device like CPU, RAM, network adapter, storage / file system interfaces and layers, audio I/O virtualizes / paravirtualizes pretty well in many cases and you can "share" real or virtual fractional capability of your system HW to N different guests plus have the host OS hammering away doing native stuff and it all just works. Consumer DGPUs, nope, sorry, back to 1990 level "you can't do that".

Enterprise "server" GPUs support SR-IOV based virtualization with some extra driver / management support SW that lets you fractionally divide a compatible server DGPU's capabilities like compute / VRAM / whatever between N guests and it "just works" and then can all run CUDA / whatever apps with their own piece of the pie though maybe not granular and dynamic capability sharing in some ways.

Intel has "promised" to have SR-IOV support in Q4 2025 / Q1 2026 or whatever when they announced the Pro B50 / B60 GPUs. I'll believe it when I see it and may need CPR if they finally un-fsck their "consumer DGPU" driver / support so virtualization "just works" on any of their ARC DGPUs. Nvidia / Amd (AFAICT) have not give any such SR-IOV / virtualization / VDI capability without $$$$ SW licenses and special super expensive server enterprise GPU models etc. etc.

I'd strongly think about buying one or two more "basic" DGPUs even like a $50-$100 generations old model that's "just enough" and do VFIO pass through to your guest(s) so any given guest that needs a non-virtual basic emulated video card can be given full control of at least a BASIC DGPU so "it just works". And then you still have your main good DGPU which you can selectively pass-through VFIO to ONE guest os or use it on the host OS and therefore give the LLM / gaming / etc. capability to at least one OS.

If you run LINUX (or msw to a lesser extent) of course you can run llama.cpp or vllm or whatever LLM inference SW you want from any host or guest OS that works for you for inference. Then just expose an openai inference service API on some virtual / physical network address that can be shared as needed locally / remotely / virtually and any v / p machine you want can get full inference capability / performance as a client using the IP based inference service API. So that's the only good way sharing LLM inference works -- you do it at the API service layer and not try to partition the DGPU itself between multiple OSs unless that's supported for your particular OS / DGPU / driver / GPU SW stack (sharing opencl, vulkan, DX, VGPU, whatever).

GPU acceleration helps msw and to an extent sometimes linux by offering the console GUI 2D/3D/media acceleration as well as video encoding/decoding accel and options like moonlight etc. so even when accessing remotely by even minimally RDP you can get use of GPU accel but usually the guest OS being the remote access server needs a "private" (in its viewpoint) DGPU allocated for that to work.