Hi r/LocalLLaMA 🚀 ,

I am currently struggling with my medical LLM research (language models only, no images/video) on my existing RTX 3060 6GB laptop GPU. As you can imagine, this is a major bottleneck—even simple LoRA experiments on small models are cumbersome due to the severe lack of VRAM. It's time to scale up.

Planned operations include: Intensive fine-tuning (LoRA/QLoRA), distillation, and pruning/quantization of large models (targeting 7B to 70B+) for clinical applications.

I am mainly considering two directions for a new setup:

1. NVIDIA DGX Spark: Full power, maximum VRAM, and complete compatibility with the CUDA ecosystem. This is the ideal solution to ensure research freedom when loading and optimizing large LLMs.
2. AMD-based Alternatives (e.g., future Strix Halo/similar): This option is theoretically cheaper, but I honestly dread the potential extra effort and debugging associated with ROCm and the general lack of ecosystem maturity compared to CUDA, especially for specialized LLM tasks (LoRA, QLoRA, distillation, etc.). I need to focus on research, not fighting drivers.

My questions to the community:

• For someone focused purely on research fine-tuning and optimization of LLMs (LoRA/Distillation), and who wants to avoid software friction—is the DGX Spark (or an equivalent H100 cluster) the only viable path?
• Are experiments like LoRA on 70B+ models even feasible when attempting to use non-NVIDIA/non-high-VRAM alternatives?
• Has anyone here successfully used AMD (Strix Halo or MI300 series) for advanced LLM research involving LoRA and distillation? How painful is it compared to CUDA?

Any perspective from an LLM researcher is greatly appreciated. Thank you!