24 Gigs on a single slot card... Thank god. Let's get rid of these goofy oversized plastic shrouds so we can get more cards on the mobo.
Multi gpu support for basic inference is all I ask. I have a feeling this upcoming year will be a lot of inference time computer and long generations. Excited for this!
Will continue to grow. Once the “personalAi” craze takes off it will be like the “personal computer” all over again. Wouldn’t be surprised to see Tb vram cards in 4-5 years, def in 10 years. At that point i assume it will be more specialized, possibly not even a video-card anymore. We need the next Bill/Steve/Jeff to put it all together
I would be surprised to see cards reach TB levels. Making very wide (in terms of memory) graphics cards is expensive, and it’s likely going to be cheaper to use a high-speed network to scale. The industry is already working on this.
I often see new promising technologies floating around, memristors, new types of wafers etc., but somehow there hasn't been a mass-production "disruption" still. Hoping for the next year.
This. Imagine if a game has NPC AI you can talk to, and "high settings" for that means more VRAM - games *will* have this, it's a matter of when. Right now, games would have to sacrifice too much in graphics to fit an LLM in a reasonable configuration though.
> it's a matter of when
Having previously worked in ML for the games industry, this is still pretty far off for mainstream games. But I think we'll start to see it slipping into always-online games where they can run the AI workload in the cloud.
I think you could pull off some level of interactions with 1B-3B models. Like a character understanding the basic of what you said, and just choosing between one of several curated courses of actions. LLM doesn't have to be a chatbot *directly*.
Have you tried to this at volume? Comprehension at the 1B-3B is definitely not at this point yet. Beyond conversation (which I think few games are such that users will want to spend more of their time talking than fighting or exploring), is powering AI actions, from enemy planning to NPC everyday routines (to make the world feel more alive).
For this, the smallest borderline acceptable size I've found is 14B unless the game's rules are really simple, with no action requiring reasoning over multiple steps. I'm hoping models released this year in the 3B range get smart enough to power something interesting that a sizeable number of users can run locally.
You definitely don't need 14B. What you need is to rely less on the LLM being a magic prompt-understanding black box and more of a relatively flexible, but focused, decision maker. You can't just show the LLM's output to the user or treat it like lines of dialogue; for that even 14B is far too tiny. But as something like a sentiment classifier, keyword extractor, etc; then small models can do it. Say, a detective game where you have to type what you think happened at the crime scene, but the lines of dialogue are themselves scripted (and thus, much better written than what an AI can make).
For constraining LLM outputs you can use things like [GNBF grammar](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md).
That limits you to a constrained/small class of games where such simple classifiers can be made use of. But I was speaking more generally, such as controlling the AI for a wizard in a complex magic system. Or enemy AI that leverages the environment for strategies against the player. Stuff like that. Conversation is actually one of the less interesting uses for a game designer.
Think of a game like Event\[0\]. That was seen as groundbreaking and impressive at the time. The dialogue was of course scripted, since LLMs weren't even a thing in 2016; but the magic of that game was that you could just talk to the robot with text. All that work they had to put into a custom NLP solution is now trivial to implement with a tiny LLM.
Regarding "AI that leverages the environment for strategies"; honestly even huge LLMs might struggle with this; they have poor spatial reasoning. You're better off using basic algorithms for that (or even a neural network trained for hours on self-play) and just using LLMs for *language*.
Again, that's an example that falls within the same very limited range of games. No spatial reasoning would be needed with the right state/data representation and structures. If the models are not intelligent enough to enhance existing GOAP type AI (it won't be delegating everything to the AI, think of it more like function calling and enhanced graph based reasoning) then they will be useless for most other uses except autocomplete.
It's worth aiming higher; models have been getting better with time. With the latest Qwen2.5-14B the least worst I have tested.
I think if you have such a fixed system (I assume you read in some structured text format with game state like JSON maybe, and ask the model to output JSON), finetuning would be the way to go. You can probably finetune a smaller model entirely locally. You could end up with a finetuned Qwen2.5-3B that outperforms non-finetuned Qwen2.5-14B.
Fine tuning is not helpful. Generally, particularly in a game setting, if we can constrain the problem enough to finetune, there'll be much better reliable approaches available.
I think we'll see smaller indie games experimenting with this in the near future but it's going to be a good while before AAA's are using it. Game dev timelines are really long now and devs will be wary of adding something like this in at the last minute to a game that's releasing soon, especially when the tech is still changing so rapidly. And they won't want to lose out of potential players for a "nice to have" feature if it significantly increases the game's required specs.
Personally, I'd love to see a classic murder-mystery game using LLM powered NPCs. There's a dinner party, someone gets murdered, you have to interview the guests/suspects and explore the house for clues. Each guest has their own backstory, personality, and information about the night in question. The key difference is that you as the player have to come up with the questions base on what you've learned, rather than a case of "I found a knife with the initials S.B, so now when I talk to Steve Banner it gives me a new dialogue option".
What hardware would be needed to do this now? I am not talking about someone making a mass market game but more someone making a simple game with a local LLM.
I think about the lowest configuration anyone capable of generally purchasing videogames has is 6GB VRAM, 16GB RAM (lower than that I imagine they almost only play f2p or pirate). That's obviously too low, but if you can make something that makes the best use of a 3B model or 7B with offloading, you could make it work, and have higher settings modes.
It starts to get good at 16-24GB, where you can run 12B-22B+ models.
Personally, I think a game could make use of chain of thought for characters; make them classify your input, polish the response, double check it, have curated responses, things like that (making small models seem smarter).
Imho it will end up with nvidia and amd just having to release high vram cards to fit their own models in there so videogame devs can just prompt them for their games, instead of bloating each game with their own LLM.
Knowing Nvidia they wouldn't mind if you have to have a subscription to the Omniverse to be able to play your games... $10/month with ads (your Skyrim 25 year edition NPC occasionally recommends a fresh glass of Mountain Dew(tm) and some crispy, crunchy Doritos)
\> wanting to run AI locally
maybe - but most of the software libraries depend on CUDA. Can you even get AI models running without cuda? easily? with good performance?
if I have to do a bunch of extra work and use non-standard tools to get the model running that sucks!
Sure, but if Intel releases a card that's perfect to run AI locally but lacks software tools, then that encourages open source developers to support Intel cards better, which is what Intel wants. They have to offer something that's not just a card that's worse than nVidia's cards in every way.
You can run some pretty powerful models locally with 48GB. I don’t think llama 3.3 70b requires that in some of the smaller quantisation versions.
Would also be cool to run the reasoning models locally for longer, rather than being limited to 50 a week for o1 right now with ChatGPT plus
I was saying B700 will have 24GB and it looks like will happen. Next is a B900 card that i'm hoping 40GB VRAM around 800-900 dollars. If it happens it will brake AI customer market entirely..
I'd pay $1000 for an Intel 48GB card.
Sure it wouldn't have all the Nvidia goodness, but I could get 2 or 3 at a good price for a decent performing large model at home!
Intel is already at a place where their compute is the bottleneck. A 24GB card would be struggling to take advantage of models that require that much VRAM at reasonable speeds. 48GB you're talking about 70b models, and battlemage (combined with poorly-optimized software) isn't up to the task.
You can't, at least not one one, you would been two. With 24GB you can't fit in a 70B model even with IQ2 and 4K context and you don't want that 2.76BPW anyway. Now with two cards the 72B Llama 3.x would run in Q4\_K\_M and 8-16K context, that would be nice and worth the investment into two 24GB Intel cards if they are cheap enough.
Nobody knows because the card doesn't exist. But a theoretical 48GB blackmage card is unlikely to be faster than a pair of old P40s in practice running a 70b model.
a single card without the complexities of cooling and powering that a pair of old P40s have is still pretty damn good if you ask me. after all, you could then go to 96GB with the same complexity as the P40s.
The problem being that you're looking at low-single-digit t/s with P40s, and probably the same on a 48GB intel card. Sure, you *can*. But with such a card surely costing well over a grand for the kind of performance you can get from a pair of P40s costing less than half that, why would you?
given that we don't know what sort of price Intel would go for for the hypothetical 48gb card, but given that we know Intel are looking to gain market share; if they decide to do $USD1000 per card that'd be pricey but not entirely unreasonable as compared to NVIDIA or AMD. And given Intel cards are already very good for media transcoding, they'd have niche outside AI as well, expanding their utility.
Furthermore it'd have warranty and integrated cooling like a brand new A6000 which in Australia goes for 9000+ AUD, so for a patient bugger like me? 48Gb in a single card for a hypothetical price like that would be immensely tempting.
I believe the problem is that the GDDR memory chips don't come in sizes that make it particularly easy to have high density cards without also scaling up all of the other parts of the card.
RTX 5090 is rumored to have 16 x 2 GB GDDR7 modules; I believe Micron and Samsung will make 3GB and 4GB modules, but the JEDEC spec alows for 6GB and 8GB too. Technically, it might be possible some crazy guy makes a frankensteined 64GB 5090 like those Russians and Brazilians that modded previous cards.
I believe the thinking is that we are getting a 512 bit memory bus to make that happen. The 4090 had a 384-bit bus which combined with 2GB chips only allowed for 24GB of vram.
So we are actually seeing them scale up the complexity to make that rumored 32GB possible.
The increased spec for larger Vram chips is what will really pay dividends for home AI long term because it could allow for additional skus that optimize for high capacity with moderate performance without also needing the complexity of a 512 bit memory bus adding to the cost.
You can have a single card with 48GB VRAM using GDDR6 like the Intel card is using, you have that with the pro cards as well. One module is 32bit and the largest size is 2GB. A card with a 384bit bus can have 24GB in a 12 x 2GB at 32bit configuration like the current consumer cards have that we use (3090/Ti, 4090, 7900XTX) or you can have 24 x 2GB at 16bit clamshell mode each (2 RAM chips share the 32bit wide controller in a 2x16bit configuration) like the professional 48GB (or 32GB with 256bit bus) cards do.
I also think it would be great to have a relatively cheap B580 pro 24GB under $500 for local inference.
Asroxk x670 Taichi
Ryzen 7950x
128GB RAM
1500 PSU
Just upgraded all that, actually. Works amazing. Planning to get a second RTX 8000 to get 96GB VRAM. After that I've done all I can to max out my PC on a reasonable budget.
Not bad to get 96G VRAM. Is your RTX 8000 Quadro passively cooled? (Googling it, there appears to be both passive and actively cooled versions.) I have a dual P40 setup in a refurb R730, which was great to get feet wet, but now I'm bit by the LLM bug and want to expand. Also it turns out the R730 is quite loud, and I haven't found a great way to make it quieter (and no easy place to put it out of the way). Very curious the noise level of your setup as well.
I have 4 axial case fans but it also comes with a blower fan. Usually doesn't get past 85C which is within normal operating temps.
However I've had issues previously running models on anything that isn't llama.cpp. I have to be extremely careful to ensure that I don't push the GPU too far because it can overheat extremely fast and the fans would max out, causing the screen to black out.
Strangely enough, the PC still works. I can still hear music and whatnot but get no display. I'm not saying the GPU is fragile or anything but you can accidentally overdo it with some models.
Like, if you're rapidly generating images, trying to clone a voice with an extremely long text input, not giving it time to rest between workload-heavy outputs, all these things can overheat the GPU pretty quickly.
Are the fans controllable? Then maybe just read out all temperatures you can, take the max temperature of all temperatures und use it to controll the fan speed with fancontrol or whatever. That's more or less how I control my fans.
[https://gist.github.com/crashr/bab9d0c6aba238a07bae2b999ee4dad3](https://gist.github.com/crashr/bab9d0c6aba238a07bae2b999ee4dad3)
Yes there is, it is huge. The most popular AI gpu from NVDA the H100 is 80GB. https://www.nvidia.com/en-us/data-center/h100/
DCs have a fixed volume for accelerators, both in power, space, cooling.
Intel has really optimized their software stack and it's already way better than that of AMD. Not completely up to par yet with CUDA, but honestly it's not that far off either. It's basically a non-factor now.
I do training runs as part of my dayjob, that said this comment thread was about inference, hence my comment should only refer to inference performance. Which is terrific on Intel GPUs and works out of the box.
I have an A770+A750 rig. latest llama.cpp compiles/runs fine with dual GPUs. Intel just updated their ollama bult to version 4.6 within the last week too. They have a custom build of ooba. I couldn't get their vllm build running, I think it's unsupported on ubuntu 24.04 (needs 22.04) but I cbf reinstalling as I just use llama.cpp on this rig.
It's ok for a cheap llama.cpp/ollama rig but no tensor parallel, prompt ingestion is slow, can't do xtts2, etc.
Ok well the discussion we were having was some guy telling me it’s “basically already there” but what it sounds like is that it’s all totally experimental even for inference to say nothing of training.
It's solid for llama.cpp inference. llama.cpp seem to be treating syctl as first-class, and a syctl build failure is enough to block an auto-merge.
The other things like ollama are officially supported, but you have to use the specific versoins Intel have built. So rather than experimental, I'd say it's slow to get new versions.
I managed to get vllm working eventually, but it was a lot of fucking around and stuck on an old version.
Training -- I wouldn't get an ARC for it since you're quite limited with what you can use (axolotl build from Intel was old when I looked). Really I'd only recommend it for cheap GPU inference with llama.cpp at this point, and not for dev (you really need cuda if you want to build your own software / hack away at models, etc IMO)
I've been using Intel's custom ollama image using IPEX.
It's been a while since I've run benchmarks as I use it pretty casually, but it's faster than what I need for models in the 30b range with a full 32k context.
And IPEX is using SYCL (https://www.khronos.org/sycl/) with Intel's OneApi, which is also a supported backend in llama.cpp.
See here https://github.com/ggerganov/llama.cpp/blob/master/docs/backend/SYCL.md
I've been able to use the iGPU of a N100 succesfully with it.
I'm not a fan of docker and anything docker+GPU is going to be a PITA.
Probably needs its own post, but I've posted instructions [here](https://www.reddit.com/r/LocalLLaMA/comments/1hfrdos/rumour_24gb_arc_b580/m2e9nd5/) as a comment.
>Goofy oversized plastic shrouds
Right? I'm still on my eVGA 2070 Super. It's dual slot, 2 fans - no real bullshit and its heavily overclocked. +135mhz on the core and memory is at +850mhz. The card doesn't over heat, get loud at full load - it's really stupid how bulky cards have gotten. I'm racist against RGB but each to their own lol
> Multi gpu support for basic inference is all I ask.
That has been around for a good while now. I run 2xA770s with AMD and Nividia GPUs too. I also toss in a Mac for spice. How much more multi do you want?
the plastic shroud isn't the bulk of the size of the card, it's the coolers and radiators. Like, you grab a 3090 or 4090 and it's very heavy from all the copper.
Hope so too, that would seriously shake up the 3090/4090 enthusiast market. If it's $300, then 2x Intel GPUs with 24GB = 48GB VRAM would be less expensive than a used 3090 on ebay.
Dude no. I hate this narrative that gets parroted around. I bought the 3090 at 1600 because it was a good value back then for the performance and vram (needed it for Blender), but the 5090 is already completely out of the running for my next GPU upgrade because of the rumored price. It's ridiculous.
I am waiting for 9070xt benchmarks, but the rumors say it will probably be a side-grade for me if I buy it, so I'll probably just wait another year for my GPU upgrade.
>> Nvidia knows people will pay whatever the msrp is.
> But I won't!!
OK, great, thanks for sharing your opinion. Nvidia's still gonna rake in the cash.
According to what definition of "fairer?" You could just as well argue that scalpers, who can flexibly raise and lower prices, will track the equilibrium price at any given time much more closely than setting an MSRP and only adjusting a few times a year (incl. sales). On the other hand, a price which is many multiples of the manufacturing cost that prices out the best independent researchers in favor of OpenAI/FAANG, regardless of whether the price is set by scalpers or the OEM, is hardly "fair" to those researchers.
Fairer as in I can buy a 5090 from a store if I can afford it, have proper warranty and it's not sold out in perpetuity. That's what happened to the 3090. Had to wait years to buy one (used) for a fair price, because fuck scalpers.
Well it’s still supply and demand. So of course scalpers can do their thing when they a) corner the market by buying all the inventory and b) manufactures not giving a fuck about it because that way they can push the price up themselves. NVIDIA has a huge margin.
Not going to lie, I'd still buy it. There's no other competitors and local AI Inferencing/Gaming are two big passions of mine. Feels bad because this gfx retail price inflation would drive more people into console gaming probably. But at the same time it's the only card that would allow you to run specific models and be top of the line for gaming as well.
At the hardware level it's still two GPUs. At the software level, the simplest way to use them both for inference is to place half the layers of one large model on one card and the other half on the other card. For each token GPU1 computes something, hands the intermediate representation to GPU2, then GPU2 computes the output token. This works well for inference of a single local session because the bottleneck isn't compute, it's memory bandwidth.
For training there are tons of ways to make use of both cards and something like PyTorch has abstractions for the most common approaches: https://pytorch.org/tutorials/beginner/dist_overview.html
https://preview.redd.it/8oueopf6k4ae1.png?width=1322&format=png&auto=webp&s=ee73c1ad03ff6cea478b7a5523d7b93c46f118d4
looks like the bandwidth isn't that good though, it's slower than a Mac Studio with M2 Ultra.
As long as it's under 600$, it would be a good option to consider over a used 3090. And, of course, after they get Pytorch and everything else mainstream running smoothly on all platforms (I'm on Windows, using KoboldCpp for LLMs and ComfyUI for images/video/TTS - it all must work at least as well as with CUDA to make it worth considering an Intel GPU).
Surprised to see this comment so far down.
I'm right there with you - Optimistic, but pragmatic. On the one hand, I'm excited for the prospect of a cheap competitor, but there's a lot of work to be done in SW support before an Intel GPU is ready to battle CUDA.
Nvidia played the long con, cornered us all with that dependency, and I want out.
Having looked at Intels AI/HPC sdks... They are stacked. They have ipex, OpenVino, MKL, MPI and then all the stuff under OneAPI.
Meanwhile AMD can't even get 3 Devs together to implement a stable diffussion on a MiGraphX backend (AMDs equivalent to TensorRT). Had to compile ROCm myself for a decent experience.
I'd just like to say there's a 48GB GDDR6 Vram card out there
[Costs a small fortune though](https://www.newegg.com/p/1FT-000P-004A0?Item=9SIB1PGK8E1530)
Intel stop playing tiddlywinks : give us cards with 32 GB of memory, 48 GB of memory, 64 GB of memory. Speed is much less important than capacity. We don't need pro speed. Consumer speed is fine.
To dear intel. Go big or go home. Make it to be 64gb i wont care about 5090. Even at 4080 speed, it will be faster than mac or sharing layers with cpu.
For their pricing strategy, you're asking the wrong question. "What does Intel think/hope they can get sales numbers up to in xx sector in xx months" is more like it.
Right, but I ask again... who TF is buying an *Arc* GPU for professional use? They would at least get some sales in a consumer price bracket (including those from business), but I can't picture any business putting up with Intel compatibility issues over just buying Nvidia or even AMD.
Maybe there's a niche for video encoding? That seems really small though, and a waste of most of the silicon.
I mean...not every business has META, GOOGL, AAPL, MSFT money.
Some businesses will be like 'ok Joe, here's your budget for the AI model you sold us on in your project plan'. It may not be enough for cards that cost tens of thousands.
7900 XTX is a fair competitor, though has many of the same 'not NVIDIA' issues.
I don't think the average business could get a 4090. Availability is kaput even for someone less 'discerning' about the quality of the seller.
> though has many of the same 'not NVIDIA' issues.
rocm is *way* easier to deal with in 'business' frameworks like vllm, or the venerable llama.cpp if you don't want to fuss with setting up pytorch flash attention.
Exactly. "We could make slightly more money by selling XX of these cards to businesses at $YYYY than we would by selling XXXX cards at $YY to consumers. Therefore, we'll market them to businesses for the higher price and pull down more money for less manufacturing effort".
Not at this tier, right? This is way below a W7800, an L4, or whatever, it's a relatively low tier GPU, where the equivalent Quadros aren't so backordered.
Yeah but OpenCL and Vulkan are also supported.
From that standpoint the intel GPUs have among the most
excellent openness and diversity of compute language / framework support options of any others. First class SYCL/oneAPI. Relatively good OpenCL. And Vulkan too.
You're ahead of me wrt. that. I've been meaning to revisit the a770 performance & dev. soon but am just setting up for that so I'll see.
Do you have any major tips wrt. optimizing use of A770 / vulkan / sycl / whatever these days?
I'm kind of torn with the reported new 24GB ARC in this thread, on the one hand I see from the A770s that they're able to make decent HW for a lower price point than nvidia in midrange consumer GPUs. But the less mature / full support for linux has been frustrating and the slow pace of better SW support in the LLM/AIML/GPGPU ecosystem for better use of any relevant
technologies whether sycl, opencl, vulkan, pytorch limitations / just now getting direct xpu support etc. etc. have been frustrating.
Yeah I think being relatively newer (vs sycl, opencl IIRC) that vulkan is having some growing pains and also being relatively less used for compute vs. graphics.
But given the powerful nature of a free to use open standard that solves the problems of both portable graphics and portable compute I'm sure it'll be getting a lot better on platforms which are open enough & supported enough to be updated / maintained over the next couple years.
FWIW I've seen some of what appear to be (from only a glance at the release announcement summary text) potentially relevantly nice changes in both sycl and vulkan related improvements and fixes lately in llama.cpp's release notes from the past few days / week(s).
https://github.com/ggerganov/llama.cpp/releases
b4382 rpc-server : add support for the SYCL backend (#10934)
b4393 vulkan: multi-row k quants (#10846) (etc.)
b4396 vulkan: Use push constant offset to handle misaligned descriptors
b4397 vulkan: im2col and matmul optimizations...
As far as I know SYCL and Intel's own IPEX code runs best on A770, but I only have the card because I'm developing the llama.cpp Vulkan backend. As far as I know it even beats SYCL in text generation performance in some cases, but prompt processing performance is not good.
I haven't found a good way to optimize for A770, it doesn't behave the same (more predictable) way that Nvidia's and AMD's cards do. As an example: I had a lot of trouble getting the XMX matrix accelerators to work. They just slow the card down on regular mesa versions, only on the latest mesa they kinda start working. But for whatever other reason text generation performance dropped significantly with latest mesa. There's always something.
I just don't have that as much time to divert to Intel as would be needed.
Intel got their gaudi in the mlperf benchmark, AMD didn't even try. Intel is a more serious player than the other company with a DEI CEO. Semi analysis report about amd was brutal. IT body shops in South Asian countries have better engineering leadership than AMD.
https://preview.redd.it/p9kseddams9e1.jpeg?width=1080&format=pjpg&auto=webp&s=9b2357e556708ce9167f27c9f0047793894feafd
Just in case you didn't know, anytime you use the word DEI to degrade a person all we hear is "I am a racist twat" - in case you weren't sure where the down votes were coming from.
I literally put a screenshot of the incompetence and if my labelling DEI triggers someone and in turn label me a racist, there is no difference between them and me. We are same🤷♂️
The racism is in equating DEI with incompetence. This CEO could be 100% incompetent on their own merit. No need to bring background into this. Plenty of other CEOs are really bad at their jobs and still have a job, and wouldn’t be a “diversity hire” which is a crazy take to think that our corporate overlords care a single bit for that. 100% of the times it is who can extract more profits from poor saps like us.
I’d argue that this is a valid take for any one person, not just a CEO. Imagine having a persons entire life and experiences reduced to “DEI hire”. The fact that people fundamentally misunderstand (or willfully misrepresent) the concept of diversity of culture is sort of baffling to me. I am yet to hear one good faith non racist argument against the practice of embracing diversity at the workplace.
Fuck it. I'll throw one at you. In good faith, honest to God. I'm not trying to look down on anyone or be racist. In fact, I'm not even taking a position. I'm going to simply present to you a good faith, non-racist argument.
When hiring, a lot of things need to be factored in to make a great hire. You should really be looking for the quality of the candidate. The best. And if the color of their skin changes that outcome for you, you are the racist. For a fact. If their gender, you're the sexist. I could go on. That's a double-edged sword there. If you choose *not* to move forward with the best candidate based on race, gender, sexuality, etc, that also makes you all those things. It cuts both ways when you judge someone not based on the contents of their character but the color of their skin.
The best thing for any organization is to move forward with whoever is most qualified. Who has the best quality. Obviously. And if you want to argue against that, you're not pounding away in anger at me or my words, but the cold, hard facts. If you don't hire the best, your competition just may, and their job is to put you out of business.
I suspect that dei is really just an overcorrection. An understandable one. For centuries, people were also chosen for their race and gender. They just went the other way with it. Well, now we've all shifted to the complete polar opposite in response. Fine. I get that. It's just not a good long-term solution. It hurts companies and people, too.
You want to be represented? Let your ideas and your contributions do the talking.
All that is exactly what I would say if I was a super duper racist person who had just zero qualms about continuing to step on the minorities that my people have oppressed through the ages.
It takes just a huge amount of willful ignorance of or as is often the case just outright hatred for "others" to not look at the past to see why we are trying to build a different future.
There are countless deeds still on record for houses all around the country that stipulate that those houses are never be sold to anyone but whites. All those "historically black colleges" you've heard about were not thus named because that's where the black people decided to attend school. There are millions of African Americans alive today who would have had to literally sit at the back of the bus or go to separate schools or eat at different restaurants and for whom lynching by local authorities was, and lets be honest, still is, a very real possibility.
DEI isn't about replacing white people with black people, it's an acknowledgement that we have FUCKING SHAT ON AFRICAN AMERICANS and other minorities for centuries through institutionalized and legislated discrimination policies and practices that have left them disadvantaged against their oppressors.
DEI is an opportunity for anyone who is hiring to consider the possibility that the person of color they are interviewing might not meet all the traditional check boxes a hiring manager is looking for but that given the opportunity they have thus far been denied, would flourish in the career they are seeking.
Maybe when a few generations of the oppressed can build up familial wealth, attend the same schools as their white peers, drive down the street and not risk being shot for looking suspicious, cash a check without having the police called... you know... exist on equal terms with white people... maybe then DEI might sound quaint... but more likely, we will have just shifted our oppression to someone else.
So no... it's not fair to just consider the candidate and let your contributions do the talking when your potential contributions were stolen by an entire race of people who mostly are in charge now and looking down on you for how little you have to bring to the table.
IMO of course and from the perspective of a white guy who probably doesn't get it either.
I actually don't disagree with most of that take. Will you allow me to focus on one specific part of it, not as a way to dismiss the rest, but to further expand the thinking behind why people still feel like DEI initiatives are important?
The last part of your take, about wanting to be represented, then one should let one's contributions speak louder? I can for a fact talk about that very personally. I am a an american citizen by birth and by familial right, but grew up in Brazil my entire life. I didn't speak english when I first moved to america, and for the first 2 years of my working career here, I would always be treated differently from my peers due to my heavy accent, in spite of the things that I was saying and contributing to being of similar value as my peers. I'm not speaking hypothetically, I am talking about entire two years of my initial career here having my ideas being passed on because I couldn't represent myself properly. Not for a lack of skills to do so, mind you, but because people dismissed my ideas on the account of my accent. The reason I say it is my accent is that after my accent started reducing, people started listening more.
This is just one example of how institutionalized racism happens without people even noticing. I don't think those people realized they were doing that. I very much would have appreciated having a voice and proper representation. If I broke through that ceiling, it very much wasn't just because of my own merit, but a combination of effort, people taking up for me, and societal norms shifting.
The idea of changing an organization's way of functioning to be more equitable to everyone is something most non-business owners should be rallying behind, not because it makes the organization better for profits, but because it makes it better for people. We shouldn't want what our corporate overlords want for their businesses (they would totally get rid of you or me if they could), and DEI initiatives, are only one small fractional way of making jobs better for the whole of society. It isn't a system of fairness in the meritocratic way, because so far merit has had nothing to do with real effort being put, but rather how much more money have you made for your boss, and that is a pretty poor metric for merit. In a society that has been historically unfair to a large group of people, the only way to make it equitable is to not be fair in the classical way people think of fairness (one for me, one for you), but rather think about a larger picture in which being fair means that things will be uneven for a while until they can even out.
Anyways, that is my take on it. I do appreciate that you took the time to give me an example, and I hope my response doesn't come across as an attack, but rather an expansion of the idea.
But you’re the one who reduced a hyper-qualified MIT PhD computer scientist to “DEI hire.” Seriously, WTF mate.
Then you added a bunch of philosophical posturing for…reasons? The original statement is the problem, and as long as it’s sitting there in the thread no amount of editorializing will make you look any more justified.
I just didn't know if you understood why you were getting the down votes. No one on our side is "triggered", just embarrassed for you is all. Since we all share an interest in AI, I was trying to politely point out why this particular conversation of yours failed here.
"During her tenure as CEO of *AMD*, the market capitalization of *AMD* has grown from roughly $3 billion to more than $200 billion."
\-Quick Wiki search.
Truly incompetent... How dare she?
I have heard that their drivers are better and Intel software support has traditionally been much better than AMD.
When tiny corp tried developing custom AI frameworks to work with Intel and AMD cards, it was comparatively easy on Intel while they had constant crashes on AMD.
This is true only in the GPU market. If you talk about generic accelerators, then the worst offender is Xilinx...
Wait! Xilinx is owned by AMD? Now I understand.
The limits are determined by the size of the memory chips available (currently 2GB for both GDDR6/X and the upcoming GDDR7 while GDDR7 should get 3GB chips later in 2025) and the memory bus width. The memory chips are 32bit and the GPUs have bus width as multiples of 32, so 64, 96, 128, 160, 192, 256, 320, 384 etc. So with 128bit nus you can have 4GB (4x1GB) or 8GB (4x2GB) VRAM etc.
In addition you can run the memory controller and chips in clamshell mode where two chips are connected to the same controller each using 16bits for the total of 32bits and doubling the available capacity. This is how the first batch of 3090Ti was for example because there were no 2GB chips available only 1GB so they had to run 12 on one side and 12 on the other side of the PCB for 24GB total connected to the 384bit bus. The 48GB professional cards from NV or AMD are done the same way, they have the same 384bit bus as the consumer cards and use the same 2GB chips, but they have 2x of them in clamshell mode so you can have az A6000 or A6000 Ada with 48GB VRAM.
This is awesome, but damn, they should pull over the 24GB line for a couple reasons.
1. It would literally differentiate their offering, when searching for a GPU, it would give another memory size that is different from NVDA offerings in the consumer space. It could be 26GB or anything, but it should be *more* than 24.
2. If they wanted to charge premium, they could go with anything larger than 32GB, 36, 40, 48, 52. GPU memory is cheap, GPU memory attached to a GPU is expensive.
By going with 24, it feels like they are going for the "we have 24 at home" for 1/2 the price of NVDA. Like they want people to do a 1=1 price comparison and/or stumble across it when searching for 24GB GPUs. That number is somewhat arbitrary, there is nothing magic in it.
God Intel and AMD are dumb. MBAs have rotten the mind of both organizations.
>That number is somewhat arbitrary, there is nothing magic in it.
I get frustrated and yet also amused by redditors standing on a soapbox without the slightest clue of what they speak.
I just want to point out to you that 26GB isn't really a feasible thing, nor is 30, 31, 33, 38 or whatever other number you come up with. These numbers (capacities) are not arbitrary.
Common Memory Capacities (Aligned with Standards):
4, 6, 8, 12, 16, 20, 24GB etc
These capacities fit standard memory bus widths (e.g., 192-bit, 256-bit, 384-bit).
4GB: Often seen with a 128-bit bus using 4x 1GB chips, while 24GB: Matches a 384-bit bus using 12x 2GB chips.
You can do your own math to see how the in between work out (you won't, but you could).
Capacities like 26GB don’t align well with standard memory buses or chip sizes. They would require uneven or non-standard ***chip configurations***, leading to inefficiencies or higher costs. The proper industry alignment ensures memory is efficiently utilized without leaving gaps or underutilized chips. There are other considerations, but this is already too much effort for someone who will ignore reality.
>God Intel and AMD are dumb.
I know you will not see it, because you still think you are right, even if you read this, but that's really funny. I love it when someone calls something or someone dumb but has no idea what they themselves are talking about. It's delicious irony.
Yes they have to deal with what Samsung and Micron sell. I have read the datasheets that don't require an NDA.
The same admonishment you think you are delivering to me also applies to you.
Instead of arguing about the small stuff, read into the point someone is making and argue against that.
AMD and Intel are fighting a war that is 10 years out of date. AMD with bad software and Intel with a SKU explosion and still thinking it can segment its market and match parity on "features".
Dumb is a volume, not a point on a single dimension. Some of the dumbest people are I know are geniuses. You are probably a genius.
>Instead of arguing about the small stuff, read into the point someone is making and argue against that.
Nice try. Accept the L and move on, you know little of what you speak.
The literal point you made was to stand out and make a 26gb...
You do not get to make a broader point and act righteous ***after*** you have specifically singled something out as a point of contention.
Life pro tip: do not do this in real life, it's leaves an impression and not a good one. Example, I maybe an asshole on reddit but IRL, I do not open my mouth unless I know what I am talking about.
>Some of the dumbest people are I know are geniuses
That I agree with, but it's really universal. Some feel the same about you.
Might be coming in the form of a double slot pro card. The fact this current one exist means we probably won't get any 32gb variant of the B770 which would've been a killer deal for anyone wanting a do everything desktop GPU. If they at least make a 48gb pro card that's a good deal they should be able to snatch a slice from nvidias moat either way.
48gb card. Doesn't need to be pro. It will fuck Nvidia, they will make more profit than they can imagine. Nvidia is too cocky to react to that. Nvidia will stick with their high price until it's too late. They only option they would have is to slash everything by half which they won't do.
Intel is onto something great here. $250 for the b580 which is pretty much 1/2 the performance of the 3090 in ram capacity, gaming FPS, memory bandwidth... if you get 2 for $500 that's a tempting proposition as is to not be using an old card that may have seen mining duty. If we are talking 2x 24gb cards for 3090 bandwidth and compute or 4x cards for 5090 bandwidth and compute but now all of a sudden we have 48 or 96gb of ram for less than the 3090 or 5090 respectively.... I'm sold on as many as I can afford after selling one of my 4090s... do I wish they had a board with 2x the compute for diffusion? Sure. But its worth it for llm batching and larger models securely...
Imagine if they brought back SLI style for gaming with current methodology too...
Suppose it is simply B580 with doubled bandwidth, this will put its VRAM speed at 912GB/s which is slightly slower than 3090. But B580 only has one tenth the TFLOPS of 3090. So I don't have a high hope that it can be replacement of 3090.
Pro cards would not get consumer prices... They are like 10x more expensive... So I doubt we will get it next to nothing like the B580 12GB. But I'm ready to be surprised by Intel. ;)
this is silly, people would buy 2 B580 for $500 if they tried to charge too much for the 24gb model. Also 10x? someone is going to pay in the vicinity of 5k for this card? what shitty businesses are making these decisions?
Better yet, add another $100 or so and just get used 3090. Getting a pair of 3060 12GB is another alternative... I mean, to compete with Nvidia, the competitor need to offer lower prices, given the customers have to deal with worse software support, more bugs and other issues. If Intel offers 24GB card for $500-$600, let alone higher than that, I would never consider buying one. I am not fan of Nvidia at all, and would be happy to support any competitor if they release worthy product at reasonable price, at least 1.5-2 smaller in price than Nvidia products (this can be at the cost of lesser compute and worse software support, but with bigger VRAM).
Please let it be 500\~CAD. I swear to god, I will put up with whatever CUDA bullshit I have to so I won't have to pay over a fucking grand for a 3090, a 2 gen old card.
Another possible "contra" of this is inspired by looking at what people said about the ARC Pro A60 GPU which I guess is the highest one in that product series for ARC A GPU ICs.
People (I just searched) were saying it was vaporware, impossible to find as part of pre-assembled PCs, nearly impossible to find for sale as a DIY upgrade. And people said that kind of stuff for approaching 2 years between late 2022 and early/mid 2024 even though the A7/A5 consumer GPUs were available all that time.
So we'll see what the MSRP is for this but also when and where it is even possible to buy it. I bought A770-16s when they launched and would be looking at this as an upgrade / expansion option but it'd have to be actually easily available retail as an upgrade at the right price point to make sense. Otherwise AMD / NVIDIA 24GB cards exist which are also under consideration, as well as cards with more than 32 GBy integrated (which is what I hoped we could see with a battlemage based card).
Even just an A770 / B580 with 32GB, 48GB of relatively slower VRAM would be very useful. Even more useful would be just fixing the client desktop PC's upper range to offer 400GB/s RAM BW and let the CPU/IGPU/NPU do the work.
If you want a cuda clone you can literally buy AMD right now. What you get with a cuda clone is something that is always several years out of date and performs worse than the original.
218 Comments
SocialDinamo@reddit
mycall@reddit
Paganator@reddit
desmotron@reddit
Mochila-Mochila@reddit
Dead_Internet_Theory@reddit
OkWelcome6293@reddit
martinerous@reddit
OkWelcome6293@reddit
Xanjis@reddit
Dead_Internet_Theory@reddit
WearMoreHats@reddit
Dead_Internet_Theory@reddit
EstarriolOfTheEast@reddit
Dead_Internet_Theory@reddit
EstarriolOfTheEast@reddit
Dead_Internet_Theory@reddit
EstarriolOfTheEast@reddit
Dead_Internet_Theory@reddit
EstarriolOfTheEast@reddit
WearMoreHats@reddit
Dead_Internet_Theory@reddit
MagoViejo@reddit
AntDogFan@reddit
Dead_Internet_Theory@reddit
ReasonablePossum_@reddit
Dead_Internet_Theory@reddit
durable-racoon@reddit
Paganator@reddit
durable-racoon@reddit
Uwwuwuwuwuwuwuwuw@reddit
Adventurous_Train_91@reddit
Unlucky_Ad4879@reddit
Ggoddkkiller@reddit
mycall@reddit
SteveRD1@reddit
SteveRD1@reddit
candre23@reddit
mycall@reddit
tmvr@reddit
candre23@reddit
Thellton@reddit
candre23@reddit
Thellton@reddit
psychicsword@reddit
Dead_Internet_Theory@reddit
psychicsword@reddit
tmvr@reddit
swagonflyyyy@reddit
RobotRobotWhatDoUSee@reddit
swagonflyyyy@reddit
RobotRobotWhatDoUSee@reddit
swagonflyyyy@reddit
RobotRobotWhatDoUSee@reddit
swagonflyyyy@reddit
muxxington@reddit
Twistpunch@reddit
fullouterjoin@reddit
mycall@reddit
TheThoccnessMonster@reddit
Down_The_Rabbithole@reddit
TheThoccnessMonster@reddit
Down_The_Rabbithole@reddit
RebelOnionfn@reddit
Wrong-Historian@reddit
CheatCodesOfLife@reddit
TheThoccnessMonster@reddit
CheatCodesOfLife@reddit
RebelOnionfn@reddit
satireplusplus@reddit
mikaelhg@reddit
satireplusplus@reddit
fallingdowndizzyvr@reddit
ziggo0@reddit
candre23@reddit
camwow13@reddit
esuil@reddit
fallingdowndizzyvr@reddit
Dead_Internet_Theory@reddit
KL_GPU@reddit
satireplusplus@reddit
AiAgentHelpDesk@reddit
Synthetic451@reddit
msaraiva@reddit
epicwisdom@reddit
TheManicProgrammer@reddit
Swashybuckz@reddit
satireplusplus@reddit
epicwisdom@reddit
satireplusplus@reddit
larrytheevilbunnie@reddit
nderstand2grow@reddit
larrytheevilbunnie@reddit
spamzauberer@reddit
Anjz@reddit
Administrative-Air73@reddit
nixed9@reddit
satireplusplus@reddit
nderstand2grow@reddit
satireplusplus@reddit
nderstand2grow@reddit
martinerous@reddit
Buttonskill@reddit
nderstand2grow@reddit
SwanManThe4th@reddit
Secure_Reflection409@reddit
NarrowTea3631@reddit
Unlucky_Ad4879@reddit
Terminator857@reddit
qrios@reddit
Terminator857@reddit
qrios@reddit
Terminator857@reddit
qrios@reddit
Terminator857@reddit
qrios@reddit
sdmat@reddit
Mickey_Arthur@reddit
cloudkiss@reddit
tu9jn@reddit
candre23@reddit
Downtown-Case-1755@reddit
Enturbulated@reddit
Downtown-Case-1755@reddit
SteveRD1@reddit
Downtown-Case-1755@reddit
SteveRD1@reddit
Downtown-Case-1755@reddit
candre23@reddit
OrangeESP32x99@reddit
Downtown-Case-1755@reddit
Cantflyneedhelp@reddit
MoffKalast@reddit
ForsookComparison@reddit
noblex33@reddit
masterlafontaine@reddit
djm07231@reddit
Calcidiol@reddit
Picard12832@reddit
Calcidiol@reddit
Picard12832@reddit
fallingdowndizzyvr@reddit
lordlestar@reddit
SteveRD1@reddit
Puzzleheaded_Swim586@reddit
LostHisDog@reddit
Puzzleheaded_Swim586@reddit
Environmental-Metal9@reddit
Caffdy@reddit
Environmental-Metal9@reddit
Bacon44444@reddit
LostHisDog@reddit
Environmental-Metal9@reddit
staatsclaas@reddit
Environmental-Metal9@reddit
staatsclaas@reddit
Environmental-Metal9@reddit
staatsclaas@reddit
LostHisDog@reddit
stylist-trend@reddit
staatsclaas@reddit
MassiveMissclicks@reddit
burnqubic@reddit
BuildAQuad@reddit
shing3232@reddit
noblex33@reddit
shing3232@reddit
mapppo@reddit
djm07231@reddit
Xanjis@reddit
FullstackSensei@reddit
matteogeniaccio@reddit
newdoria88@reddit
Colecoman1982@reddit
tmvr@reddit
fullouterjoin@reddit
Smile_Clown@reddit
fullouterjoin@reddit
Smile_Clown@reddit
Crafty-Struggle7810@reddit
Elite_Crew@reddit
MoffKalast@reddit
scottix@reddit
big_cock_lach@reddit
segmond@reddit
Cyber-exe@reddit
segmond@reddit
MatlowAI@reddit
Ok_Warning2146@reddit
ttkciar@reddit
Tmmrn@reddit
ttkciar@reddit
FullstackSensei@reddit
Someone13574@reddit
rawednylme@reddit
Successful_Shake8348@reddit
Puzzleheaded_Wall798@reddit
Lissanro@reddit
orrzxz@reddit
gfy_expert@reddit
Cyber-exe@reddit
Healthy-Nebula-3603@reddit
sammcj@reddit
DigThatData@reddit
lurkn2001@reddit
SevenShivas@reddit
Equivalent-Bet-8771@reddit
Calcidiol@reddit
Sylv__@reddit
stddealer@reddit
Dorkits@reddit
PositiveEnergyMatter@reddit
penguished@reddit
Tmmrn@reddit
s101c@reddit
sapperwho@reddit
emprahsFury@reddit
celsowm@reddit