Forgive me in advance if this isn't that clear - I have no experience with computers beyond basic consumer knowledge.
I am inquiring if there is such a product/ business/ service that provides this: I basically want to run an LLM (text based only) locally, maybe run it on my local network so multiple devices can use it. It is a ready/built, plug and play physical server/ piece of hardware that has all the main AI models downloaded on it. The business/ service updates the piece of hardware regularly. So basically a setup to run AI pitched at consumers who just want a ready-to-go local AI. Pitched for individuals/ home use.
I don't have the correct terms to even fully describe what I'm looking for but I would really appreciate if someone could advise on this. Thank you
The Z8 is a very powerful machine! We’ve got a great lineup of AI workstations and software solutions. Feel free to reach out if you have any questions.
Lots of good recommendations here already, here's one more to consider if you aren't up to installing workstations or rackmounts yet. Gigabyte has a new product called AI TOP that's basically a desktop PC for local AI training: www.gigabyte.com/WebPage/1079?lan=en They also sell the mobos/GPUs/SSDs etc. separately but for what it's worth, the pre-built products offer an additional software platform that's supposed to streamline the AI training process even better.
Any PC with recent GPU at 8GB or more can run this. Take a look at Pinokio, install OpenwebUI and ensure in the setting set available on local network.
I think OP is looking for a consumer device he can just plug-in and go, and there's not such a thing (yet). At the minimum, he has to install and configure the software. Not a big deal, I'll grant you, but it can be daunting for people who aren't technically inclined.
Yeah pretty much, I looked into and tried Jan AI but it is more complicated to get it working than I thought, I basically want GPT4o level responses completely locally, for my own private PKM notes etc.
You won't get GPT4o level stuff out of anything consumer level for a little while. The hardware isn't there, but you can get pretty good for under 2k or so of container hardware.
Ok, that's helpful as a benchmark. Do you think eventually there will be consumer level local on-device ai you can buy? Like in a year or two (or more)?
Maybe in 2 years? Random guess, but based on the fact that we're just now reaching the point that open-source 70b models are now as good as what GPT 3.5 was, I'd say in another couple years we might be able to distill that performance and include some novel new architecture to get current 4o performance with a model that can run on phones.
You're better off running ollama on your PC than getting something like this.
If you want something cheap that can run good models, get a graphics card with 12+GB of VRAM. An RTX 3060 12gb used can go for about $250.
Ollama is pretty easy to use, just type ollama run gemma2 or ollama run llama3.2 and it'll (download if needed, and then) run the model. You can interact with it using an API.
I only have a mac laptop, I don't have a PC. And I looked up some tutorials to set up a fully local AI and it seems like you need a lot of storage and proper graphics card etc, and a lot more hidden/ unexpected steps to set it up fully. And I am not sure about API's and how they work, doesn't that mean it connects to the internet?
You can likely find an old gaming rig for sale that you can use. Hook it up to a tv to get Pop!_OS running on it. (I choose Pop due to the native Nvidia support) then install ssh server so you can manage it remotely. After that fine tune the setup to be optimized for using the gpu.
I used to auto start oogabooga and automatic1111 but found the system couldn’t handle both without errors. So I made a script I can run that will start the virtual environment and what ever one I wanted to use at the time. Both were relatively easy to auto start so in your case auto start oogabooga and connect on your network to your own ai. There are modules to add for memory retention too.
You could accomplish this pretty easily with either ChatGPT and or copilot to get the setup help.
And I am not sure about API's and how they work, doesn't that mean it connects to the internet?
TLDR: Nope.
If you want more details...
So it can be a bit confusing and requires context. API stands for Application Programming Interface, and the acronym long predates the proliferation of the internet. Every piece of software that interacts with another piece of software normally does through using an API. This can be a local library that is called through normal dynamic linking; there is not necessarily even a network involved.
Now in the modern era, people are often talking about some form or another of remote API when they say "API". In the context of LLMs, that is typically the case.
Most self-hosting applications provide such an API. They are not connecting to the internet; they are turning your computer into a server that client apps can connect to. You could configure them so you can also connect from other computers on your network, but by default most are not listening for remote connections. It does work essentially the same way a connection to an Internet service would, but you are hosting the service so the traffic never leaves your network.
Regarding what you are looking for, this is a pretty fast moving space. If you buy something 100% preconfigured, you will still need to learn about updating it and how to setup new models. That is, unless you'll be fine with it never improving and therefore never bother. IMO, you should still learn about how to setup and run these things, even if you find some ready to run hardware.
So if I use an API to use chat GPT on Jan AI for example, how does the data I put into the chat and share with the chat interact with Open AI and their servers?
Thanks for the advice at the end there. I think I've learned that there is not quite a ready-to-go solution and that learning more about how these things work would be valuable no matter what the scenario
So if I use an API to use chat GPT on Jan AI for example, how does the data I put into the chat and share with the chat interact with Open AI and their servers?
Presumably you mean if you configure it to use the cloud API?
The minute you involve any cloud service then whatever data you put into the chat is being sent to them. That's really the only way it can work The alternative would essentially be telling the API, "hey give me a text response... I'm not telling you what to respond to, just do it!" As you might imagine, that won't work well.
If you only configure it for use with local models, then your data should be staying on your computer. Of course, that means you'll need hardware powerful enough run a decent model.
There are a few pre-builts that will do that, like TinyBox, but I can tell you that running local AI is actually not nearly as hard as you think. You said you have a Mac, correct? Then you may already be able to run models. In running LLMs, VRAM is king, so the more VRAM you have, the larger the model you can run. (In case you don't know, VRAM is Video RAM, super fast memory that only graphics cards have). Models are measured in billions of parameters (think neurons), so given all other factors are the same, the more parameters a model has, the smarter. However, large models are larger files, and therefore harder to fit into VRAM.
If you have a m series mac. The unified memory can also be treated as VRAM, so if you have like 16 GB of memory, then Apple reserves about 1/3 for the system, you have about 12 GB, which will allow you to run small models like Llama 3.1 8B, or Gemma 2 9B, or Mistral Nemo 12B. At 24GB, you can run medium models like medium models like command R 32B, Gemma 27B, etc. At 48GB you can run Llama 3.1 70B and Qwen 2.5 72B.
You stated in another model, another comment, that you want performance similar to GPT4o. We only have about two models in the open source space that compete with those. One is Llama 3.1 405B, which is impossible to run as a consumer. The other is Mistral Large 123B, which you can run with 64GB VRAM or more.
If you would like to run Mistral Large, the minimum required spec is 2 x RTX 3090, or 48GB VRAM. However, I would recommend 3 X 3090, or 72GB VRAM. So, your options are to either buy a Mac with sufficient vram, though it will be slower than an equivalent PC, or build a PC with 3 X used 3090 at $1800 + other parts so $600-$1000, for a total of $2500-3000. Is it pricey? Of course. But what is the value that you place on your privacy, and the utility you get out of this?
If that seems a bit overwhelming, or daunting at this point in time, then why don't you start with smaller models that you can run right now, and see how you like them? It's possible that you don't need the full intelligence of GPT4o to accomplish the tasks you want. Many people here are satisfied with 70B, some only run 32B, and most of us only have the compute to run 12B or less.
The easiest way to get up and running on an m series Mac is download LM studio, then go to the download model section, and download any model that will fit in your VRAM, and has good performance, like the ones I listed previously. Feel free if you want to ask me any questions!
Thank you, this was helpful and actually makes it seem less daunting. I think you're totally right that i may not need full GPT4o ability and need to test out what works for the core tasks I need right now, and do a bit of playing around.
My current Mac can't run anything rn I downloaded a Llama model from Jan AI and it didn't respond or said I didn't have enough RAM, which is fine, I will definitely keep your advice you mentioned in mind when I get a new laptop or computer.
This looks good, and I think this is pretty much what I am after. What is the model that runs on this and if I am used to using chat GPT4o how would it compare?
I mean no offense to the person selling this, as this is a legitimate product, for a legitimate use case, but the models that this is capable of running, 1B and 3B models are not even remotely close to ChatGPT 4o, to the point they will sound downright brain-dead in comparison. (The b stands for billions of parameters, think neurons, the more they have, the smarter it is, but they're harder to run). The bare minimum that is generally recommended here, and that you can run if you have any normal computer, is 8B models, like Llama 3.1 and Gemma 2 9B. Those are significantly smarter than any tiny model. For reference, the smallest model that is considered in the same class as GPT 4o is Mistral Large 123b, which is over 40x larger than the largest model this hardware can run. I would not recommend buying this hardware for your use case.
Thanks for your input. I just feel overwhelmed with trying to set it up on my computer, but maybe I need to relook at that. I downloaded Jan AI and downloaded Llama but it didn't even respond when I tried to talk to it. I don't think my Mac can run any of these based on the chip I have.. that's why I was looking for something external i can plug in. Sorry if I sound stupid/ like I'm missing something that is actually pretty simple which may be the case. What would you recommend I do? Like someone else said I may not actually need GPT44o capability for the tasks I want to do offline.
You have both llama3.2 the 3B and 1B version.
3B is “smarter” but not as fast as the 1B.
Obviously GPT4o will outperform the smaller models. I can preload all the models, I recommend and make llama3.2 default because it’s the best speed to performance ratio I’ve seen so far.
That being said, once you have it, you can pull all future models as they become available. They’re getting smaller and smarter very fast.
How does it compare if you are used to the large scale consumer chatbot speed?
I basically want to use it to help reorganise and sort notes and writing, just want an idea of the difference in speed.
So do you think that eventually this product would be able to run more capable/ faster models as they become smaller?
Regarding the purchase, after the initial cost what does support/ upgrading/ repairability look like for the customer? Also is there anything you can add to the base model you have to make it faster or more features for a higher cost?
here is a video I posted a few days to show the speed. This is the 3B running, 1B is way faster but accuracy might degrade.
100% yes.
I will continue releasing features, it’s optional to upgrade the software though and I will make the updates available to install manually if users decide to keep the device completely offline. That way you can just upload the update manually using the interface. If not, you will get updates automatically. I will make the code base open source, just need to clean it up and ready for contributions and create docs. I have folks going through it now. The goal is to have an API and have an on device App Store for apps that can use your on-device model(s). A lot of plans like home automation etc…
That sounds really great! I would be way more inclined to buy if I knew that the base thing I am buying isn't just going to be useless or out of date in a year or two when you come out with the next model or something, and that I can continually upgrade.
You’re welcome. A big reason I’m working on this is I want to ship products that work and don’t need to depend on some cloud service for eternity and get bricked eventually.
Yup, you still have all your USB ports. I can also add options for larger storage internally if want to keep it streamlined. Can do up to 4TB. Only difference would be price point
I've asked before, but maybe this time I will get lucky: what are those PCIe risers with flexible cables that you're using? Do they support full PCIe 4.0 speeds? Where can I get them?
I wouldn't call anything with 4x3090 as consumer level. The power requirements of effectively a dedicated circuit is a bit much. That's more hobbyist or small business level.
Think you've still got a bit of an expectation gap there tbh.
Even the solutions like tinybox that are pre-built and specifically intended for this assume a certain level of technical familiarity with the topic, plus some familiarity with linux.
Maybe, I was impressed by https://persys.ai though and going to try it out, willing to accept that it may be slower etc though, but seems pretty user friendly
For now I just need it to do basic commands like sort existing notes and stuff, but eventually when the models get smaller and smarter I'll hopefully be able to keep updating it and get faster and better capabilities. Happy to take the risk and try this out, way less of a risk then getting a gaming PC and going down that rabbit hole with 0 knowledge
Wouldn't this be way over your head when it comes to setting it up? I'm aware you mean it to be plug it in and it should automatically work like some kind of magic.
But you would still need some kind of front end to set everything up, and then prepare your electricity bill.
I don't know of something that you can buy like this, but I am curious about the market for it. If it emitted about 90 dB, required \~400 watts peak, and cost about $2000, would that sell? Because a hobbyist would typically create a build around an NVIDIA 3090 and that's probably about the best one could do. There are certainly ways to cut costs, like having much slower inference, or using less capable models, but what's the specific AI capabilities that are sought (Qwen 2.5 32B level?) and what would people be willing to tolerate? What's the budget?
From my perspective I am just a regular person, little to 0 computer knowledge, am using Chat GPT but want an alternative because I care about privacy, owning my data, supporting other companies with values etc; I am interested in open-source software and AI access and development for all, but also recognise my own limitations and time restrictions and happy to pay a passionate expert's product that does all this for me.
Budget I have no idea. I am paying 20 a month for chat GPT already, I pretty much want that, but offline and local storage etc. Image generation not my priority really, just the text AI. I would pay a decent up front cost if I knew what the setup was i.e how long would it receive updates, support etc... Something like https://persys.ai which was mentioned in another comment, 300-500$ maybe... but open to paying more if it may doubled up as a general home computer/ home server I think... That price could be impossible currently but again I do admit I am new to this so don't even know what a reasonable budget is.
You're asking the right questions. I feel like consumers that value privacy and ownership are willing to cut costs and remove a good few bells and whistles.
Whereabouts are you located? If you're in Europe I can refer you to a system integrator that specializes in plug-n-play AI setups on top of consumer hardware.
They are very affordable, but only do custom orders for now and there is no online store yet.
I think someone can build a system based on Apple M2 Ultra 192GB for that and sell for $7k (orig price $6k), However, since the market is small. I doubt any company will make it in volume.
On the software side, GPT4ALL is about as simple as you can get for running LLMs easily on your computer and is pretty much what you’re looking for.
https://www.nomic.ai/gpt4all
On the hardware side, you could go on Amazon or somewhere and find a prebuilt “gaming PC” that has AT LEAST a “3090” or a “4090” GPU in it. Here’s an example on Amazon.
https://amzn.to/4dJmQov
You can run GPT4ALL and a smaller model LLM on lower spec hardware, but if you want to run good AI models at higher context windows, you’re going to to need the 24GB VRAM that the 4090 GPU provides.
This combo is about as turn-key as you’re going to find right now.
Herr_Drosselmeyer@reddit
What's your budget?
https://www.hp.com/us-en/workstations/z8.html
They'll build you a very potent AI workstation but it'll cost ya.
Even then, you're still expected to install, configure and update yourself. There's no company that'll do that afaik.
Sothan_HP@reddit
The Z8 is a very powerful machine! We’ve got a great lineup of AI workstations and software solutions. Feel free to reach out if you have any questions.
YekytheGreat@reddit
Lots of good recommendations here already, here's one more to consider if you aren't up to installing workstations or rackmounts yet. Gigabyte has a new product called AI TOP that's basically a desktop PC for local AI training: www.gigabyte.com/WebPage/1079?lan=en They also sell the mobos/GPUs/SSDs etc. separately but for what it's worth, the pre-built products offer an additional software platform that's supposed to streamline the AI training process even better.
aguspiza@reddit
tinybox https://tinygrad.org/#tinybox
JFHermes@reddit
I've never actually found a review for one of these machines though. Like, is no one buying them or is it just that those who do don't talk about it?
ozzie123@reddit
Any PC with recent GPU at 8GB or more can run this. Take a look at Pinokio, install OpenwebUI and ensure in the setting set available on local network.
pete_68@reddit
I think OP is looking for a consumer device he can just plug-in and go, and there's not such a thing (yet). At the minimum, he has to install and configure the software. Not a big deal, I'll grant you, but it can be daunting for people who aren't technically inclined.
meeplemop159@reddit (OP)
Yeah pretty much, I looked into and tried Jan AI but it is more complicated to get it working than I thought, I basically want GPT4o level responses completely locally, for my own private PKM notes etc.
1010012@reddit
You won't get GPT4o level stuff out of anything consumer level for a little while. The hardware isn't there, but you can get pretty good for under 2k or so of container hardware.
meeplemop159@reddit (OP)
Ok, that's helpful as a benchmark. Do you think eventually there will be consumer level local on-device ai you can buy? Like in a year or two (or more)?
femio@reddit
As good as 4o?
Maybe in 2 years? Random guess, but based on the fact that we're just now reaching the point that open-source 70b models are now as good as what GPT 3.5 was, I'd say in another couple years we might be able to distill that performance and include some novel new architecture to get current 4o performance with a model that can run on phones.
Journeyj012@reddit
You're better off running ollama on your PC than getting something like this.
If you want something cheap that can run good models, get a graphics card with 12+GB of VRAM. An RTX 3060 12gb used can go for about $250.
Ollama is pretty easy to use, just type ollama run gemma2 or ollama run llama3.2 and it'll (download if needed, and then) run the model. You can interact with it using an API.
meeplemop159@reddit (OP)
I only have a mac laptop, I don't have a PC. And I looked up some tutorials to set up a fully local AI and it seems like you need a lot of storage and proper graphics card etc, and a lot more hidden/ unexpected steps to set it up fully. And I am not sure about API's and how they work, doesn't that mean it connects to the internet?
justme535@reddit
You can likely find an old gaming rig for sale that you can use. Hook it up to a tv to get Pop!_OS running on it. (I choose Pop due to the native Nvidia support) then install ssh server so you can manage it remotely. After that fine tune the setup to be optimized for using the gpu.
I used to auto start oogabooga and automatic1111 but found the system couldn’t handle both without errors. So I made a script I can run that will start the virtual environment and what ever one I wanted to use at the time. Both were relatively easy to auto start so in your case auto start oogabooga and connect on your network to your own ai. There are modules to add for memory retention too.
You could accomplish this pretty easily with either ChatGPT and or copilot to get the setup help.
I
SwordsAndElectrons@reddit
TLDR: Nope.
If you want more details...
So it can be a bit confusing and requires context. API stands for Application Programming Interface, and the acronym long predates the proliferation of the internet. Every piece of software that interacts with another piece of software normally does through using an API. This can be a local library that is called through normal dynamic linking; there is not necessarily even a network involved.
Now in the modern era, people are often talking about some form or another of remote API when they say "API". In the context of LLMs, that is typically the case.
Most self-hosting applications provide such an API. They are not connecting to the internet; they are turning your computer into a server that client apps can connect to. You could configure them so you can also connect from other computers on your network, but by default most are not listening for remote connections. It does work essentially the same way a connection to an Internet service would, but you are hosting the service so the traffic never leaves your network.
Regarding what you are looking for, this is a pretty fast moving space. If you buy something 100% preconfigured, you will still need to learn about updating it and how to setup new models. That is, unless you'll be fine with it never improving and therefore never bother. IMO, you should still learn about how to setup and run these things, even if you find some ready to run hardware.
meeplemop159@reddit (OP)
Right, ok that makes a bit more sense to me
So if I use an API to use chat GPT on Jan AI for example, how does the data I put into the chat and share with the chat interact with Open AI and their servers?
Thanks for the advice at the end there. I think I've learned that there is not quite a ready-to-go solution and that learning more about how these things work would be valuable no matter what the scenario
SwordsAndElectrons@reddit
Presumably you mean if you configure it to use the cloud API?
The minute you involve any cloud service then whatever data you put into the chat is being sent to them. That's really the only way it can work The alternative would essentially be telling the API, "hey give me a text response... I'm not telling you what to respond to, just do it!" As you might imagine, that won't work well.
If you only configure it for use with local models, then your data should be staying on your computer. Of course, that means you'll need hardware powerful enough run a decent model.
ArsNeph@reddit
There are a few pre-builts that will do that, like TinyBox, but I can tell you that running local AI is actually not nearly as hard as you think. You said you have a Mac, correct? Then you may already be able to run models. In running LLMs, VRAM is king, so the more VRAM you have, the larger the model you can run. (In case you don't know, VRAM is Video RAM, super fast memory that only graphics cards have). Models are measured in billions of parameters (think neurons), so given all other factors are the same, the more parameters a model has, the smarter. However, large models are larger files, and therefore harder to fit into VRAM.
If you have a m series mac. The unified memory can also be treated as VRAM, so if you have like 16 GB of memory, then Apple reserves about 1/3 for the system, you have about 12 GB, which will allow you to run small models like Llama 3.1 8B, or Gemma 2 9B, or Mistral Nemo 12B. At 24GB, you can run medium models like medium models like command R 32B, Gemma 27B, etc. At 48GB you can run Llama 3.1 70B and Qwen 2.5 72B.
You stated in another model, another comment, that you want performance similar to GPT4o. We only have about two models in the open source space that compete with those. One is Llama 3.1 405B, which is impossible to run as a consumer. The other is Mistral Large 123B, which you can run with 64GB VRAM or more.
If you would like to run Mistral Large, the minimum required spec is 2 x RTX 3090, or 48GB VRAM. However, I would recommend 3 X 3090, or 72GB VRAM. So, your options are to either buy a Mac with sufficient vram, though it will be slower than an equivalent PC, or build a PC with 3 X used 3090 at $1800 + other parts so $600-$1000, for a total of $2500-3000. Is it pricey? Of course. But what is the value that you place on your privacy, and the utility you get out of this?
If that seems a bit overwhelming, or daunting at this point in time, then why don't you start with smaller models that you can run right now, and see how you like them? It's possible that you don't need the full intelligence of GPT4o to accomplish the tasks you want. Many people here are satisfied with 70B, some only run 32B, and most of us only have the compute to run 12B or less.
The easiest way to get up and running on an m series Mac is download LM studio, then go to the download model section, and download any model that will fit in your VRAM, and has good performance, like the ones I listed previously. Feel free if you want to ask me any questions!
meeplemop159@reddit (OP)
Thank you, this was helpful and actually makes it seem less daunting. I think you're totally right that i may not need full GPT4o ability and need to test out what works for the core tasks I need right now, and do a bit of playing around.
My current Mac can't run anything rn I downloaded a Llama model from Jan AI and it didn't respond or said I didn't have enough RAM, which is fine, I will definitely keep your advice you mentioned in mind when I get a new laptop or computer.
ArsNeph@reddit
May I ask what model of Mac you're running? If you don't mind, I might be able to help you get it up and running
meeplemop159@reddit (OP)
I'll DM you!
ranoutofusernames__@reddit
I’m working on this: https://persys.ai
Based on the Raspberry Pi. You can chat and perform RAG. All local, over WiFi.
meeplemop159@reddit (OP)
This looks good, and I think this is pretty much what I am after. What is the model that runs on this and if I am used to using chat GPT4o how would it compare?
ArsNeph@reddit
I mean no offense to the person selling this, as this is a legitimate product, for a legitimate use case, but the models that this is capable of running, 1B and 3B models are not even remotely close to ChatGPT 4o, to the point they will sound downright brain-dead in comparison. (The b stands for billions of parameters, think neurons, the more they have, the smarter it is, but they're harder to run). The bare minimum that is generally recommended here, and that you can run if you have any normal computer, is 8B models, like Llama 3.1 and Gemma 2 9B. Those are significantly smarter than any tiny model. For reference, the smallest model that is considered in the same class as GPT 4o is Mistral Large 123b, which is over 40x larger than the largest model this hardware can run. I would not recommend buying this hardware for your use case.
meeplemop159@reddit (OP)
Thanks for your input. I just feel overwhelmed with trying to set it up on my computer, but maybe I need to relook at that. I downloaded Jan AI and downloaded Llama but it didn't even respond when I tried to talk to it. I don't think my Mac can run any of these based on the chip I have.. that's why I was looking for something external i can plug in. Sorry if I sound stupid/ like I'm missing something that is actually pretty simple which may be the case. What would you recommend I do? Like someone else said I may not actually need GPT44o capability for the tasks I want to do offline.
ranoutofusernames__@reddit
You have both llama3.2 the 3B and 1B version. 3B is “smarter” but not as fast as the 1B.
Obviously GPT4o will outperform the smaller models. I can preload all the models, I recommend and make llama3.2 default because it’s the best speed to performance ratio I’ve seen so far.
That being said, once you have it, you can pull all future models as they become available. They’re getting smaller and smarter very fast.
meeplemop159@reddit (OP)
How does it compare if you are used to the large scale consumer chatbot speed?
I basically want to use it to help reorganise and sort notes and writing, just want an idea of the difference in speed.
So do you think that eventually this product would be able to run more capable/ faster models as they become smaller?
Regarding the purchase, after the initial cost what does support/ upgrading/ repairability look like for the customer? Also is there anything you can add to the base model you have to make it faster or more features for a higher cost?
ranoutofusernames__@reddit
here is a video I posted a few days to show the speed. This is the 3B running, 1B is way faster but accuracy might degrade.
100% yes.
I will continue releasing features, it’s optional to upgrade the software though and I will make the updates available to install manually if users decide to keep the device completely offline. That way you can just upload the update manually using the interface. If not, you will get updates automatically. I will make the code base open source, just need to clean it up and ready for contributions and create docs. I have folks going through it now. The goal is to have an API and have an on device App Store for apps that can use your on-device model(s). A lot of plans like home automation etc…
meeplemop159@reddit (OP)
That sounds really great! I would be way more inclined to buy if I knew that the base thing I am buying isn't just going to be useless or out of date in a year or two when you come out with the next model or something, and that I can continually upgrade.
Thank you for all the answers
ranoutofusernames__@reddit
You’re welcome. A big reason I’m working on this is I want to ship products that work and don’t need to depend on some cloud service for eternity and get bricked eventually.
SoftItalianDaddy@reddit
This is very interesting but is/will be it possible to have more storage space (eg. connecting an external hard drive)?
ranoutofusernames__@reddit
Yup, you still have all your USB ports. I can also add options for larger storage internally if want to keep it streamlined. Can do up to 4TB. Only difference would be price point
nero10579@reddit
Yes I am actually selling 4x3090 machines like this.
jonahbenton@reddit
I am almost ready to get one. Have a link?
nero10579@reddit
I’m selling this config of machines pre setup with all the software. https://www.reddit.com/r/LocalLLaMA/s/wdUoTJlRHw
MikeRoz@reddit
I've asked before, but maybe this time I will get lucky: what are those PCIe risers with flexible cables that you're using? Do they support full PCIe 4.0 speeds? Where can I get them?
nero10579@reddit
I sold a few in person and through paypal transactions but am going to post them on ebay too. I’ll dm.
1010012@reddit
I wouldn't call anything with 4x3090 as consumer level. The power requirements of effectively a dedicated circuit is a bit much. That's more hobbyist or small business level.
nero10579@reddit
Fair enough lol
AnomalyNexus@reddit
Think you've still got a bit of an expectation gap there tbh.
Even the solutions like tinybox that are pre-built and specifically intended for this assume a certain level of technical familiarity with the topic, plus some familiarity with linux.
meeplemop159@reddit (OP)
Maybe, I was impressed by https://persys.ai though and going to try it out, willing to accept that it may be slower etc though, but seems pretty user friendly
AnomalyNexus@reddit
Certainly seems to have the polish, but that looks like its based on a raspberry pi5?
It's gonna be substantially slower and dumber than chatgpt...if you can live with that go for it.
meeplemop159@reddit (OP)
For now I just need it to do basic commands like sort existing notes and stuff, but eventually when the models get smaller and smarter I'll hopefully be able to keep updating it and get faster and better capabilities. Happy to take the risk and try this out, way less of a risk then getting a gaming PC and going down that rabbit hole with 0 knowledge
civilized-engineer@reddit
Wouldn't this be way over your head when it comes to setting it up? I'm aware you mean it to be plug it in and it should automatically work like some kind of magic.
But you would still need some kind of front end to set everything up, and then prepare your electricity bill.
TheActualStudy@reddit
I don't know of something that you can buy like this, but I am curious about the market for it. If it emitted about 90 dB, required \~400 watts peak, and cost about $2000, would that sell? Because a hobbyist would typically create a build around an NVIDIA 3090 and that's probably about the best one could do. There are certainly ways to cut costs, like having much slower inference, or using less capable models, but what's the specific AI capabilities that are sought (Qwen 2.5 32B level?) and what would people be willing to tolerate? What's the budget?
meeplemop159@reddit (OP)
From my perspective I am just a regular person, little to 0 computer knowledge, am using Chat GPT but want an alternative because I care about privacy, owning my data, supporting other companies with values etc; I am interested in open-source software and AI access and development for all, but also recognise my own limitations and time restrictions and happy to pay a passionate expert's product that does all this for me.
Budget I have no idea. I am paying 20 a month for chat GPT already, I pretty much want that, but offline and local storage etc. Image generation not my priority really, just the text AI. I would pay a decent up front cost if I knew what the setup was i.e how long would it receive updates, support etc... Something like https://persys.ai which was mentioned in another comment, 300-500$ maybe... but open to paying more if it may doubled up as a general home computer/ home server I think... That price could be impossible currently but again I do admit I am new to this so don't even know what a reasonable budget is.
You're asking the right questions. I feel like consumers that value privacy and ownership are willing to cut costs and remove a good few bells and whistles.
Ragecommie@reddit
Whereabouts are you located? If you're in Europe I can refer you to a system integrator that specializes in plug-n-play AI setups on top of consumer hardware.
They are very affordable, but only do custom orders for now and there is no online store yet.
elfish_dude@reddit
Try out Sanctum.ai
Ok_Warning2146@reddit
I think someone can build a system based on Apple M2 Ultra 192GB for that and sell for $7k (orig price $6k), However, since the market is small. I doubt any company will make it in volume.
Porespellar@reddit
On the software side, GPT4ALL is about as simple as you can get for running LLMs easily on your computer and is pretty much what you’re looking for. https://www.nomic.ai/gpt4all
On the hardware side, you could go on Amazon or somewhere and find a prebuilt “gaming PC” that has AT LEAST a “3090” or a “4090” GPU in it. Here’s an example on Amazon. https://amzn.to/4dJmQov
You can run GPT4ALL and a smaller model LLM on lower spec hardware, but if you want to run good AI models at higher context windows, you’re going to to need the 24GB VRAM that the 4090 GPU provides.
This combo is about as turn-key as you’re going to find right now.