# Strix Halo AI Inference Node Setup Guide
## MiniMax M2.7 (230B MoE) with TurboQuant on AMD Ryzen AI Max+ 395


**Hardware:**
 AMD Ryzen AI Max+ 395, 128GB unified RAM, 1TB NVMe  
**OS:**
 Ubuntu 24.04 LTS Server  
**Model:**
 MiniMax M2.7 UD-IQ4_XS (108GB, 230B total / 10B active params)  
**Backend:**
 llama.cpp TurboQuant HIP fork, ROCm 7.2  
**Context:**
 131K tokens (via TurboQuant KV cache compression)  
**Benchmark:**
 65 tok/s prefill (pp512), 23.3 tok/s generation (tg128)


---


## Performance


| Workload | Performance | Notes |
|---|---|---|
| Short prompts (<2K tokens) | Prefill in seconds, 23 tok/s gen | Great for one-shot coding tasks |
| Medium prompts (2-10K) | Prefill 30-60 seconds | Good for focused conversations |
| Long conversations (10K+) | Prefill slows linearly | Context accumulation gets painful |
| Massive ingestion (100K+) | ~17 tok/s prefill, ~2 hours | Use RAG instead |


---


## Step 1: BIOS Configuration


1. 
**UMA Frame Buffer Size → 512MB**
 (minimum). The GPU uses GTT for compute, not the BIOS carveout.
2. 
**IOMMU → Disabled**
 (~6% better memory bandwidth).
3. 
**TDP → 85W**
 if configurable.


---


## Step 2: Install Ubuntu 24.04 LTS


Minimal server install, no desktop environment.


```bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential cmake git curl wget \
  pkg-config libssl-dev python3-pip linux-headers-$(uname -r)
```


Verify kernel is 
**6.16.9+**
:


```bash
uname -r
```


If too old:


```bash
sudo add-apt-repository ppa:cappelikan/ppa -y
sudo apt update
sudo apt install mainline -y
sudo mainline --install 6.19.4
sudo reboot
```


---


## Step 3: Configure TTM Kernel Parameters


```bash
sudo nano /etc/default/grub
```


Set:


```
GRUB_CMDLINE_LINUX_DEFAULT="ttm.pages_limit=30720000 amd_iommu=off"
```


```bash
sudo update-grub
sudo reboot
```


Verify:


```bash
sudo dmesg | grep "amdgpu.*memory"
# Expected:
# [drm] amdgpu: 512M of VRAM memory ready
# [drm] amdgpu: 120000M of GTT memory ready
```


---


## Step 4: Install ROCm 7.2


```bash
wget https://repo.radeon.com/amdgpu-install/7.2/ubuntu/noble/amdgpu-install_7.2.70200-1_all.deb -O /tmp/amdgpu.deb
sudo dpkg -i /tmp/amdgpu.deb
sudo apt update
sudo amdgpu-install -y --usecase=rocm,hip --no-dkms
sudo usermod -aG render,video $USER
sudo reboot
```


The `amdgpu-dkms` error during install is safe to ignore — the in-kernel driver works.


### Set HSA Override


ROCm 7.2 doesn't recognize gfx1151 natively. Use gfx1100 override (compatible):


```bash
echo 'export HSA_OVERRIDE_GFX_VERSION=11.0.0' >> ~/.bashrc
echo 'export HSA_ENABLE_SDMA=0' >> ~/.bashrc
echo 'export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1' >> ~/.bashrc
source ~/.bashrc
```


Verify:


```bash
rocminfo | grep "Name:" | head -6
# Should show gfx1100 and AMD Radeon Graphics, no warnings
```


---


## Step 5: Build llama.cpp (TurboQuant HIP Fork)


```bash
cd ~
git clone https://github.com/domvox/llama.cpp-turboquant-hip.git
cd llama.cpp-turboquant-hip
git checkout feature/turboquant-hip-port-clean


HIPCXX=/opt/rocm-7.2.0/lib/llvm/bin/clang++ HIP_PATH=/opt/rocm-7.2.0 \
cmake -S . -B build \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS="gfx1100" \
  -DGGML_HIP_ROCWMMA_FATTN=ON \
  -DCMAKE_BUILD_TYPE=Release \
&& cmake --build build --config Release -- -j $(nproc)
```


Many `-Wunused-value` warnings are normal. Wait for `[100%] Built target llama-server`.


Verify:


```bash
./build/bin/llama-cli --list-devices
# Expected:
# Device 0: AMD Radeon Graphics, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 120000 MiB
```


---


## Step 6: Download MiniMax M2.7


```bash
pip install --break-system-packages huggingface_hub hf_transfer


export HF_HUB_ENABLE_HF_TRANSFER=1
hf download unsloth/MiniMax-M2.7-GGUF \
  --include "*UD-IQ4_XS*" \
  --local-dir ~/models/minimax-m2.7
```


~108GB download. If `hf_transfer` loses progress on resume, use `wget -c` instead:


```bash
mkdir -p ~/models/minimax-m2.7/UD-IQ4_XS
cd ~/models/minimax-m2.7/UD-IQ4_XS
wget -c https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00001-of-00004.gguf
wget -c https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00002-of-00004.gguf
wget -c https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00003-of-00004.gguf
wget -c https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00004-of-00004.gguf
```


---


## Step 7: Create systemd Service


```bash
sudo tee /etc/systemd/system/llama-server.service << 'EOF'
[Unit]
Description=llama.cpp TurboQuant MiniMax M2.7 Inference Server
After=network.target


[Service]
Type=simple
User=luadeveloped
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"
Environment="HSA_ENABLE_SDMA=0"
Environment="GGML_CUDA_ENABLE_UNIFIED_MEMORY=1"
ExecStart=/home/luadeveloped/llama.cpp-turboquant-hip/build/bin/llama-server \
  --model /home/luadeveloped/models/minimax-m2.7/UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00001-of-00004.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --ctx-size 131072 \
  --threads 16 \
  --n-gpu-layers 99 \
  --flash-attn on \
  --no-mmap \
  --jinja \
  --chat-template-kwargs '{"enable_thinking":true}' \
  --metrics \
  --temp 1.0 \
  --top-k 40 \
  --cache-type-k turbo3 \
  --cache-type-v turbo4
Restart=always
RestartSec=10
LimitMEMLOCK=infinity


[Install]
WantedBy=multi-user.target
EOF


sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
```


Change `User=luadeveloped` and all `/home/luadeveloped/` paths to your own username.


---


## Step 8: Verify


```bash
sudo systemctl status llama-server
curl -s http://localhost:8080/v1/models
curl -s http://localhost:8080/health
```


### Benchmark (stop server first):


```bash
sudo systemctl stop llama-server
cd ~/llama.cpp-turboquant-hip
./build/bin/llama-bench \
  -m ~/models/minimax-m2.7/UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00001-of-00004.gguf \
  -ngl 99 -fa 1 -p 512 -n 128
sudo systemctl start llama-server
```


---


## Connecting Tools


The server exposes an OpenAI-compatible API at `http://<node-ip>:8080/v1`.


---


## Memory Layout


```
128GB Total System RAM
├── ~8GB   — OS, kernel, services
├── ~103GB — Model weights (ROCm GPU buffer)
├── ~7.3GB — KV cache (TurboQuant, 131K tokens)
├── ~0.7GB — Compute buffers
└── ~9GB   — Free headroom
```


Without TurboQuant, KV cache at FP16 would be ~32GB — only ~43K tokens of context. TurboQuant compresses it 4.4× to 7.3GB, enabling 131K tokens.


---


## Useful Commands


```bash
# GPU memory
cat /sys/class/drm/card*/device/mem_info_gtt_total | awk '{printf "GTT: %.1f GB\n", $1/1024^3}'


# TTM config
cat /sys/module/ttm/parameters/pages_limit | awk '{printf "TTM: %.1f GB\n", $1*4/1024/1024}'


# Server logs
sudo journalctl -u llama-server -f


# Server health
curl -s http://localhost:8080/health


# ROCm GPU status
rocm-smi
```


---


## Known Issues


- 
**Thinking tags:**
 M2.7 sometimes drops the opening `<think>` tag. The closing `</think>` shows as raw text in Open WebUI. This is a rendering issue, not a model issue.
- 
**amdgpu-dkms error during ROCm install:**
 Safe to ignore. In-kernel driver works.
- 
**HSA_OVERRIDE_GFX_VERSION=11.5.1 does NOT work**
 with ROCm 7.2. Use 
**11.0.0**
.
- 
**Long conversation slowdown:**
 Prefill time scales linearly with accumulated context. Use RAG for large documents instead of stuffing them in the prompt.