Thinking about running dual 4060TIs 16gb. But is there a way to limit power on linux? Am I going to sweat myself to death in the summer?

Posted by LanceThunder@reddit | LocalLLaMA | View on Reddit | 17 comments

Like the title says, i am running linux mint and thinking about upgrading to dual 4070s. it should be a huge upgrade for me. but i would like to be able to limit how much power they draw at least some of the time. even shutting one of them right off when i am not working on LLMs might be good. is this possible and practical? are there any other problems i am not thinking about?

[-]

AppearanceHeavy6724@reddit

4060s are very very slow.

[-]

LanceThunder@reddit (OP)

compared to what? according to my reading its a decent value for what they are selling for.

[-]

AppearanceHeavy6724@reddit

compared to 3060. 2x3060 twice as fast, cheaper if bought used and have 24gb.

[-]

LanceThunder@reddit (OP)

But I am talking about a 4060TI with 16gb. That will be a lot faster than a 3060 with 12gb and the price will still be affordable.

[-]

AppearanceHeavy6724@reddit

Okay, you do not want to listen to good advice, it is up to you.

No, 4060ti for LLM use in paricular is 25% slower (290 Gb/sec bandwidth) than 3060 (360Gb/sec), and the price is $450. 3060 is $220 used, 2x3060 = 440, and with proper software (vllm) you can parallelize them and get twice the performance of 4060. The only negative is much higher power consumption.

[-]

LanceThunder@reddit (OP)

Ok, I think I see what you are saying. You are comparing TWO 3060s to just one 4060ti. I am comparing two of each. You are right that the 3060 wins in a 2 v 1 but I am right if we are talking about a 2 v 2. But if I have the money to spare why would I consider two 3060 over two 4060ti 16gb?

[-]

AppearanceHeavy6724@reddit

Because SINGLE 4060ti is 20% slower than already slow SINGLE 3060. You'd be better of buying 3 3060, as it will have more memory than 4060 and faster. Or a compromise, 3060 + 4060ti. Or even better just a single used 3090, as it will be still faster than 2x4060ti.

[-]

LanceThunder@reddit (OP)

lol you have to be trolling. i am trying really really hard to take you seriously but i don't understand how you would compare these two cards in a way that you make the 3060 superior. i guess if you are running 7 or 8 3060s that would be better in a lot of ways but it would eat up a lot more power and i only have two PCI slots to work with. continue this conversation with gemini or something. maybe you will believe it more than you believe me.

[-]

AppearanceHeavy6724@reddit

Look you do not understand one thing dude: the only thing that matters when you run LLMs is memory bandwidth , nothing else important. I told you 3060 has higher bandwidth 4060ti, but you seem to understand that. |Yes in games 4060ti is better but not for LLMs.

https://www.reddit.com/r/LocalLLaMA/comments/14gnkfw/think_twice_about_getting_the_rtx_4060_ti/

https://www.reddit.com/r/LocalLLaMA/comments/178gkr0/performance_report_inference_with_two_rtx_4060_ti/

[-]

_hypochonder_@reddit

The bandwidth is not the fastest. (bandwidth 288.0 GB/s)
You want high bandwidth for LLMs for faster interference.
It's mostly scales 1:1.
e. g. RX 7600XT bandwidth with 288.0 GB/s has only 1/3 the interfrence speed as the RX 7900XTX (960.0 GB/s)

[-]

tengo_harambe@reddit

Title says 4060TIs post says 4070?

4060TIs have very low power consumption at factory settings, doubtful you would want to power limit them since they are already slow.

[-]

LanceThunder@reddit (OP)

sorry about that. yeah, i was thinking 4060.

[-]

Put it in your crontab to run 5 minutes after @reboot. Or as a systemd service.

[-]

techmago@reddit

CRONTAB

You could ask for the ai to generate an systemd unit instead a workaround that ugly...

[-]

Stepfunction@reddit

https://forum.level1techs.com/t/how-to-set-nvidia-gpu-power-limit-nvidia-smi/131467