Dual A100X local workflow

Posted by vitamins1000@reddit | LocalLLaMA | View on Reddit | 15 comments

Came across these A100X's at work and decided to keep them for internal use. We were not sure what to use them for but I came up with a work flow to use RAG to allow a local model to access our inventory database and have users interact with the model through open web ui. This was all new to me and a great learning experience (claude did most of the heavy lifting). Thought it was cool and wanted to share!

[-]

a_beautiful_rhind@reddit

Damn.. that idle.. They don't have p-states?

[-]

FullstackSensei@reddit

If you're buying a 40k GPU, doubt you're worried about 70Wh idle power.

Those DC cards don't do P-states. I think even the PCIe P100 had something like 25W idle, and that was a 250W card.

[-]

a_beautiful_rhind@reddit

Why not? It's all money. P100 had that problem but P40 didn't.

[-]

FullstackSensei@reddit

P40 is a 1080Ti with different BIOS. P100 is an entirely different chip

Worrying about the power consumption of a data center card is like worrying about the fuel economy of a Ferrari. You can also get older Ferraris for relatively cheap, doesn't change the fact that if fuel economy an issue for you, you're not the target audience for the thing..

[-]

a_beautiful_rhind@reddit

If I was buying several hundred of them I would worry. It's not power consumption under load but at idle.

[-]

bjodah@reddit

Yeah, my CMP 90HX also idles at 70W, and does not seem to support setting other P-states.

[-]

vitamins1000@reddit (OP)

There might be a BIOS setting (APSM) I should enable to lower those but I'm also not sure how much of that is the built in DPU on these cards.

[-]

a_beautiful_rhind@reddit

ASPM won't help. Its only for the link when they're idle.

[-]

jacek2023@reddit

could you show some llama-bench results?

[-]

vitamins1000@reddit (OP)

  Device 0: NVIDIA A100X, compute capability 8.0, VMM: yes, VRAM: 81050 MiB
  Device 1: NVIDIA A100X, compute capability 8.0, VMM: yes, VRAM: 81050 MiB
| model                          |       size |     params | backend    | ngl | threads | ts           |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ------------ | --------------: | -------------------: |
| gpt-oss 120B F16               |  60.87 GiB |   116.83 B | CUDA       | 999 |       8 | 1.00         |           pp512 |      1410.25 ± 19.17 |
| gpt-oss 120B F16               |  60.87 GiB |   116.83 B | CUDA       | 999 |       8 | 1.00         |           tg128 |        123.78 ± 0.52 |
| gpt-oss 120B F16               |  60.87 GiB |   116.83 B | CUDA       | 999 |       8 | 1.00         |           pp512 |      1383.17 ± 11.05 |
| gpt-oss 120B F16               |  60.87 GiB |   116.83 B | CUDA       | 999 |       8 | 1.00         |           tg128 |        123.97 ± 0.20 |

[-]

vitamins1000@reddit (OP)

What's wrong with that? I have a strong love for debian 12.

[-]

matt-k-wong@reddit

that is awesome congratulations. You can run some decent models on those.