Your Experience with Small Language Models

[-]

gionyyy@reddit

We're deploying SLMs in production in manufacturing in offline/ air-gapped environment. They work great and they're fast too. Some SLMs are highly specialized and that's incredibly important to keep in mind. When you do it right it's magical. Indeed, it's more fiddly than sending a one giant prompt to LLMs and expect incredible results. It's only going to get better from now on.

Reply

[-]

GortKlaatu_@reddit

Small models don't know facts, and many times that's generally OK. The only thing I really want them to do and what I want them to do well is to call tools and interpret the results with very little reasoning. I want tiny agents.

Reply

[-]

Echo9Zulu-@reddit

I feel like small models should be good an interpreting data from sensors. That would be a dope tiny agent. Imagine a security agent that sent alerts when a sensor was tripped but also used camera data to describe what was happening

Reply

[-]

_Erilaz@reddit

I'd rather not rely on an LLM to sound any alarms, because it can and eventually will randomly ignore a legit threat, or hallucinate a false one out. Let the sensors themselves and some robust code do their job, an LLM shouldn't deal with anything other than preliminary assessment.

Reply

[-]

Apprehensive_Rub2@reddit

True but also true of people, and i feel like it you could get it to pretty consistently recognise anything suspicious, a few false alarms aren't a big deal.

Reply

[-]

_Erilaz@reddit

It depends on the system we're talking about and the kind of false alarm. A known and well studied failure doesn't really need such an automation at all. And if you're dealing with a complex issue, chances are a small LLM won't do any good, only adding to the confusion. Like, imagine a mishap on a nuclear power plant or some serious malfunction on an airliner. Things rarely go horribly wrong there, but there are millions of things that could go wrong. Say, your left engine's turbine shattered, severing multiple systems in the aircraft and the engine itself is in fire. A mayhem of alarms goes off, and you would think an LLM would help you to prioritize, but here's the catch: the LLM hallucinated the RIGHT engine failure instead because it won a lottery and sampled a 0.00[...]01 probability token. Which wouldn't be that low in a tiny LLM btw, but I digress. If you follow the LLMs advice, you'll turn off the only working engine and still leave the faulty one burning. Statistically speaking, this WILL happen eventually. That's not to say actual pilots didn't commit such mistakes, but this is the reason why people without training aren't supposed to be in the cockpit at all - it is an extremely complex system, which isn't a problem on its own, but if anything goes sideways, you shouldn't have any room for unnecessary ambiguity. And an LLM will be a huge source of it.

Reply

[-]

noprompt@reddit

This

Reply

[-]

250000mph@reddit

Gemma 2 2b is the smallest I'd realistically use. 1.5b's and lower feel a bit too dumb (even Qwen). I use them for casual conversation, like brainstorming and chat. I haven't tried the vision ones yet. How is it?

Reply

[-]

Xer0neXero@reddit

Sorry man

Reply

[-]

Xer0neXero@reddit

How lonely are you ?

Reply

[-]

ronoldwp-5464@reddit

If I were a betting woman, I’m say they’re slightly less lonely than those that revel in the enjoyment of insulting strangers online. This, I hypothesize they’re slightly less lonely than you.

Reply

[-]

gus_the_polar_bear@reddit

For its size, Gemma 2 2b has no business being as good as it is

Reply

[-]

jeremyckahn@reddit

I’m over the moon with how good Llama 3.2 3B is. It runs great without a GPU and has replaced ChatGPT for me.

Reply

[-]

randomqhacker@reddit

I'll second Llama 3.2 3B, but for its prompt adherence which is amazing for a small model. Augment with RAG or multi-shot prompting and it can get the job done.

Reply

[-]

NEEDMOREVRAM@reddit

How would Llama 3.2 3B do if I were to give it an electronic product description from AliExpress and asked it to explain what it does and what I can do with it?

Reply

[-]

jeremyckahn@reddit

Here's a random item I found: https://www.aliexpress.us/item/3256807333679161.html?spm=a2g0o.productlist.main.23.2f1922a2IDAOd9&algo_pvid=4bfc6a02-119b-4c14-9f9d-394ec369798d&aem_p4p_detail=20241115155520255634765343460001104545&algo_exp_id=4bfc6a02-119b-4c14-9f9d-394ec369798d-11&pdp_npi=4%40dis%21USD%215.00%215.00%21%21%215.00%215.00%21%402101c5b117317149199782563e784c%2112000041108970495%21sea%21US%210%21ABX&curPageLogUid=f8MJNsMIIdF6&utparam-url=scene%3Asearch%7Cquery_from%3A&search_p4p_id=20241115155520255634765343460001104545_3 My prompt: > given this product description, tell me what this product is and what I can do with it: > > 20pcs MX25L6433FM2I-08G Brand: MXIC (Wanghong Electronics) Packaging: SOP-8-200mil Brand new imported original authentic product Response: ``` Based on the product description, it appears that this is a small electronic component. The key details are: * "MX25L6433FM2I-08G" is likely the model number of the component. * "MXIC" is the brand name, which suggests that it's a Chinese electronics company. * "SOP-8-200mil" is the package type, which indicates that the component is a small, 8-pin packaged Integrated Circuit (IC). * "20pcs" suggests that the product is a pack of 20 individual components. Given that it's a small IC, it's likely used in electronic devices such as: * Embedded systems * Microcontrollers * Memory chips * Communication devices As for what you can do with it, without more information, it's difficult to say. However, here are a few possibilities: * Use it as a replacement for a similar IC in an existing project or device. * Add it to a DIY project or build a custom electronic device. * Use it to create a custom embedded system or microcontroller project. * Integrate it into a larger circuit or system to enhance its functionality. Keep in mind that working with electronic components can be hazardous, and it's recommended to have some basic knowledge of electronics and safety precautions before handling or soldering the component. ```

Reply

[-]

NEEDMOREVRAM@reddit

Ok, that's doable in my opinion. I'm one of those weirdos who spends 3-4 hours on AliExpress window shopping the latest Chinese tech. Some of it I don't understand so I go window shopping with my LLM. It makes for a good companion to share my weird hobbies. Not going to lie that is kind of weird but whatever. https://www.aliexpress.us/item/3256807563531742.html https://www.aliexpress.us/item/3256807381759331.html That will send you down a rabbit hole tonight if this sort of stuff fascinates you as much as it does me. What model was that again?

Reply

[-]

randomqhacker@reddit

Neat hobby! For your particular use case, though, you may want an LLM with more world knowledge, and the 3B is going to be limited in how much it remembers about specific types of microcontrollers, or subtleties of various technologies. If you want it to "recognize" a certain line of chips, or things about their manufacturer, you would either want to RAG with web search, or use a larger "know-it-all" model that has not been too quantized.

Reply

[-]

jeremyckahn@reddit

Llama 3.2 3B

Reply

[-]

Weary_Stick6473@reddit

It can perform data analysis and eda?

Reply

[-]

randomqhacker@reddit

Doubtful, but I haven't tried. More like extracting relevant information, calling tools, and keeping track of what it's doing enough to maintain a coherent conversation.

Reply

[-]

burtonash@reddit

Please can I ask what your use case is? I've struggled to get any small models to perform tasks well but I'd love to be able to move my use case local.

Reply

[-]

jeremyckahn@reddit

I've been using Llama 3.2 3B a lot for summarization, structuring information, generating plans for work, proofreading, constructing shell commands, and lot of other random stuff. In my experience it's about as good as ChatGPT. For code-related needs I use Qwen 2.5 7/14B.

Reply

[-]

KedMcJenna@reddit

Same - Llama 3.2 3B has been consistent and solid in everything. Just a great al-rounder. And it runs well on old laptops with 8GB RAM. One of the pluses of the currently rumored LLM Plateau (if it comes) is that there’ll be time for the current models to get the attention they deserve. Be nice to wring all the juice we can out of these little marvels before rushing into the next big thing.

Reply

[-]

WaifuEngine@reddit

3B llama is pretty great for it size at q8

Reply

[-]

Sky_Linx@reddit

The only small one I use is StarCoder2 3b just for code auto completions with the Continue extension for VSCode. Since it’s small it’s decently fast for this task although the quality of the completions is not always best. The main models I use are both 32b (regular and coder version of Qwen 2.5).

Reply

[-]

iidealized@reddit

in my experience, they are way over-tuned to benchmarks. Using them for real tasks, they are all much less useful than gpt-4o-mini

Reply

[-]

iamjkdn@reddit

I think more research focus should be there for SLMs since they are more portable and more scalable to deploy

Reply

[-]

noprompt@reddit

With hot swappable LoRAs they’re also more practical.

Reply

[-]

isr_431@reddit

Qwen 2.5 Coder 1.5b/3b are perfect for code completion with Continue.

Reply

[-]

Dark_Fire_12@reddit

Drop phi 3.5, it was useful in a world without qwen2.5 vl or llama 3.2 but we are past that world

Reply

[-]

first2wood@reddit

I have only used them when I played with the GPU Poor LLM Arena. It's terrible in creative writing which feels like reading garbage Japanese MTL because of the mismatched pronouns. But it's okay to just chat with, or simple tasks, like make next week's exercise plan, make a table, mermaid or something like these. But still, from the limited experience, I think it's generally unusable. I would run an API or turn to a computer.

Reply

[-]

Life_Tea_511@reddit

Phi3.5 allucinates a lot. The best was Gemma2 in my experience but still not as good as gpt3.5turbo

Reply

[-]

NowThatHappened@reddit

You have to be much more careful with prompting, and they do go nuts sometimes but not had a problem for specific use cases. I think we still have a tinyllama in production somewhere, doing its thing day after day.

Reply

Your Experience with Small Language Models

Reply to Post

34 Comments

gionyyy@reddit

GortKlaatu_@reddit

Echo9Zulu-@reddit

_Erilaz@reddit

Apprehensive_Rub2@reddit

_Erilaz@reddit

noprompt@reddit

250000mph@reddit

Xer0neXero@reddit

Xer0neXero@reddit

ronoldwp-5464@reddit

gus_the_polar_bear@reddit

jeremyckahn@reddit

randomqhacker@reddit

NEEDMOREVRAM@reddit

jeremyckahn@reddit

NEEDMOREVRAM@reddit

randomqhacker@reddit

jeremyckahn@reddit

Weary_Stick6473@reddit

randomqhacker@reddit

burtonash@reddit

jeremyckahn@reddit

KedMcJenna@reddit

WaifuEngine@reddit

Sky_Linx@reddit

iidealized@reddit

iamjkdn@reddit

noprompt@reddit

isr_431@reddit

Dark_Fire_12@reddit

first2wood@reddit

Life_Tea_511@reddit

NowThatHappened@reddit