Eco - Friendly Inference
Posted by Qwen30bEnjoyer@reddit | LocalLLaMA | View on Reddit | 18 comments
Hey gents,
Odd question, but from the perspective of a optimist that used to work in oil and gas, it annoys me that this is one of the few industrial processes that can be powered entirely by electricity from solar. Especially since we can buy solar panels for ~$0.20 - $0.40 cents per watt before accounting for other parts of the system, and the cost per kwh amortized across the lifespan of the system, its interesting we don't see more providers advertising using solar.
TripleSecretSquirrel@reddit
Because you need a whole fuckton of solar panels to power an AI datacenter.
I work in commercial and industrial real estate development. I don't build datacenters, but have built industrial facilities that have pretty high energy demands, and even a high-use industrial facility's energy demand absolutely pales in comparison to an AI datacenter.
One current generation GB300 rack from Nvidia consumes 1400W/1.4kW. That's for each server rack in there. Most datacenters getting built now by hyperscalers are looking for 100+mW of power. Depending on where you're at and how much sun you have, you'd need ~200 acres of solar panels to power that. But you'd need way more than that in practice, cause they need to provide 24 hours worth of power with just 8 hours of sunlight. And you need to bump up more to account for cloudy days. Then you need to add space for battery storage so you can access your power 24 hours a day. So in reality, you're looking at like 800-1000 acres of solar panels.
Then you're either buying an extra 1000 acres around your new datacenter on which to plop down your solar panels, which severely constrains where you can build (datacenters want to be very close to backbone dark fiber bundles, which is already a constraint, so finding an extra 1000 acres for cheap is even harder), or you build the solar offsite and have to incur a huge engineering and regulatory costs to then build transmission lines to connect your facility to your solar panels.
Qwen30bEnjoyer@reddit (OP)
Got it, thanks for the perspective! When I think about solar costs its usually in the residential solar context where you don't need to buy new land, but I didn't consider how the land cost would change the economics of it.
StableLlama@reddit
What is your (odd) question?
AI uses huge amounts of electrical energy. Solar is about the cheapest way to produce it, but is only available during day light. Assuming that AI usage is high when people are awake it should be roughly aligned with day light. For the difference wind turbines are handy and also very cheap to run. To balance everything you can use batteries - which are still rather expensive but their price is going down in a huge gradient.
AI revolution is possible without fossils.
Qwen30bEnjoyer@reddit (OP)
I suppose the odd question would be better asked as what is the hang up from a systems engineering perspective. My hunch would be the duck curve created by peak solar production when demand is off-peak neccessitating a level of battery storage a 24/7 datacenter couldn't afford, but then I see Exowatt's claims... maybe the real hangup is operational tempo chilling the conversation around what energy source to use?
Overall I'm just trying to get more perspectives than my own :)
StableLlama@reddit
Putting current shortages aside and looking at the long term, when everything got stable:
You want the AI computation close to the user to minimize latencies. So you need many datacenters all around the world and not a few that are running 24/7 full steam.
User demand follows roughly the day light as people are sleeping at night. So going solar is already following that curve pretty well.
There will be a bit higher demand in the late afternoon and also some for automated systems during the night. That's something where wind turbines are a good match to fill the gap.
Both, solar and wind, can easily produce excess energy at some times, so that must be stored (battery, hydrogen) and fill the gaps when solar and wind isn't enough for the demand.
Renewable energy is actually a much better match for AI power requirements than fossil or nuclear.
And just a word about the "exawatts": these are claims by the big players so that they don't have to disclose what they are actually doing. Out of AI perspective that's a completely wrong way to measure, as scaling power consumption can be easily done by using bad efficiency in hardware and software. But it's the company that can generate the double amount of tokens per watt that is winning and not the other way round.
(ok, to be fair: for a datacenter, in the sense of just the building and it's infrastructure connection, the maximum watt it can supply in energy and cooling is a sensible measurement. But that doesn't relate to the "AI output" you can get out of it)
Qwen30bEnjoyer@reddit (OP)
This is the most interesting comment so far, I didn't consider the idea that token usage patterns actually match the production pattern of solar much closer than the current paradigm of energy use peaking when people get home from work.
Makes me think about the first part of this podcast, when Elon (Don't like the guy, but the episode is interesting for the topics discussed), stated something to the tone that solar was the only scalable energy source for AI.
Just one slight clarification, I did mean exowattthe startup, but that's my bad for not clarifying in the comment.
Forward_Compute001@reddit
Well the ai revolution is not happening if its based on burning fossil fuels. Immagine 10x furrils fules beeing burned.
here solar is transformed into hydrogen not at big scale but we already have a hydrogen powered train, so you can store solar by storing it in matter. And hydro is used when solar is off...green energy is pretty big in europe. Energy autarchy is important and global warming is not just a myth.
Saren-WTAKO@reddit
solar panel to dgx spark and use qwen 122b int4-autoround with mtp=2
cibernox@reddit
I am doing it for my next saas, by running the inference at my own home powered by solar.
I’m doing it to save costs but since my audience is quite environmentally minded, I might advertise it a bit.
Qwen30bEnjoyer@reddit (OP)
If you put up an inference API that's powered by solar, let me know :)
Not enterprise, but I'd be interested.
cibernox@reddit
Lol no. I only have a single consumer GPU.
The just happen to be using very small models where a RAG does the heavy lifting and I can’t get by with my own hardware until a number of users that it would mean I’m profitable and that it would be a good problem to have.
I automatically divert traffic to openrouter if my server is down or at max capacity, but I should be able to handle 90+% of my load for free.
robertpro01@reddit
I'm doing my part, I generate about 25kw per day, that includes 15% extra my total power usage.
ProfessionalSpend589@reddit
Me too! I'm generating around 10Wh in the after hours on sunny days.
qwen_next_gguf_when@reddit
It's electricity and I don't care where it comes from.
Qwen30bEnjoyer@reddit (OP)
Right, which is understandable. I just would expect a bit more overlap between the people that want to own their AI and the people that want to own their electric supply.
Forward_Compute001@reddit
at small scale thats not a problem I guess.
I don't own a fossil fuel distillery if I want to buy bananas that are trasported from far away.
Forward_Compute001@reddit
where I live the majority if the electricity on the grid is green solar hydro or wind.
mr_zerolith@reddit
You gotta buy a giant amount of solar, then giant batteries.. so.. that can really increase the cost of that form of energy.
Also a lot of these providers aren't making a lot of money in the first place. AI is expensive. Solar powered AI is more expensive. Which do you think wins in that kind of market?