There are a number of common formats you'll find open models in, but the most common ones are GGUF and Transformers, if you see .pt or .safetensor files you are usually dealing with Transformers models.

GGUF files can be used in llama.cpp and tools based on it, which include LM Studio, Ollama, and a plethora of other frontends. Transformers models are designed for the Transformers library, and is also supported by a number of other programs, but this tends to be used more for "Professional" purposes, like deploying a model for multi-user use with vLLM or similar.

As a single user formats like GGUF will be the one you should focus on, at least in the beginning. Now your specs is actually very low for LLM use, it's somewhat common for people new to AI to assume that text models are likely easy to run since they just produce text, but in reality LLMs are very demanding, usually more so than Image generation models, at least in terms of size. The first L in LLM literally stands for Large after all.

With just 6GB of VRAM you will be limited to small models (\~8B and lower) and even then you will need to use quantized version. You can think of Quantization as a form of lossy compression, it reduces the size of the model at the cost of some performance. For GGUF the quant level is measured by the Q in the file name. Q8 is a file with little quantization, Q1 is extremely quantized. Generally Q4 is considered the lowest quantization level that still performs somewhat well. Though it does depend a bit on the size of the model in question. Smaller models actually tend to be more sensitive to quantization than bigger ones, so for a 8B model even a Q4 quant will be pretty degraded.

Note that the context (essentially the chat history) also consumes VRAM so with so little VRAM you might be limited in how much text you can actually submit/generate. Especially if you go with a higher quant.

I can't really offer more guided advice without knowing what models you have actually tried and what your goal is. But as far as general advice goes, I'd look at the smaller Gemma and Qwen models. Those tends to be the best amongst the small model class currently.

Also I will mention, just to save you from future downvotes, that the Deepseek model you refer to is a Qwen model distilled from a DeepSeek model, not a proper Deepseek model. This might not mean much to you as a beginner, but trust me when I say that people get annoyed when you claim you are running Deepseek (which is a 671B model) when you are just running one of the small distill models.

Edit: Fixed some typos.

There are a number of common formats you'll find open models in, but the most common ones are GGUF and Transformers, if you see .pt or .safetensor files you are usually dealing with Transformers models.

GGUF files can be used in llama.cpp and tools based on it, which include LM Studio, Ollama, and a plethora of other frontends. Transformers models are designed for the Transformers library, and is also supported by a number of other programs, but this tends to be used more for "Professional" purposes, like deploying a model for multi-user use with vLLM or similar.

As a single user formats like GGUF will be the one you should focus on, at least in the beginning. Now your specs is actually very low for LLM use, it's somewhat common for people new to AI to assume that text models are likely easy to run since they just produce text, but in reality LLMs are very demanding, usually more so than Image generation models, at least in terms of size. The first L in LLM literally stands for Large after all.

With just 6GB of VRAM you will be limited to small models (\~7B and lower) and even then you will need to use quantized version. You can think of Quantization as a form of lossy compression, it reduces the size of the model at the cost of some performance. For GGUF the quant level is measured by the Q in the file name. Q8 is a file with little quantization, Q2 is extremely quantized. Generally Q4 is considered the lowest quantization level that still performs somewhat well. Though it does depend a bit on the size of the model in question. Smaller models actually tend to be more sensitive to quantization than bigger ones, so for a 7B model even a Q4 quant will be pretty degraded.

Note that the context (essentially the chat history) also consumes VRAM so with so little VRAM you might be limited in how much text you can actually submit/generate. Especially if you go with a higher quant.

I can't really offer more guided advice without knowing what models you have actually tried and what your goal is. But as far as general advice goes, I'd look at the smaller Gemini and Qwen models. Those tends to be the best amongst the small model class currently.

Also I will mention, just to save you from future downvotes, that the Deepseek model you refer to is a Qwen model distilled from a DeepSeek model, not a proper Deepseek model. This might not mean much to you as a beginner, but trust me when I say that people get annoyed when you claim you are running Deepseek (which is a 671B model) when you are just running one of the small distill models.