FluoroquinolonesKill

Will this be better at RP than 26B-A4B?

Hoping someone can clear this up. I thought speculative decoding was only useful if you could load the entire main model into VRAM. Happy to be corrected.

This. That is probably what is motivating bro to work like that to begin with. Bro needs to do serious self examination. Source: me, an intellectual, judging people for making mistakes I recently learned to stop making.

Indeed, mounting fans is a pretentious waste of time.

Same. It wasn’t having it when I tried to give it the current date for web searches.

I am really frustrated at my good writing - and anyone’s good writing - now being mistaken for AI because people are too dumb to tell the difference.

Do we need custom templates with the latest GGUFs, or are the template fixes now embedded in the GGUFs?

\> I strongly encourage running with \`--cache-ram 2048 -ctxcp 2\` to avoid system RAM problems On 26B A4B too?

Yeah Gemma was not having it when I tried to tell it what today’s date is. That seems like it should be something that any model should be able to accept. Hopefully it gets ironed out.

What about the non-imatrix ones? Do those need to be re-downloaded? E.g., UD-Q8\_K\_XL and UD-Q4\_K\_XL.

Gemma’s prose is better, but Qwen seems more chatty, engaged, and friendly.

Yep. She goons.

Yes, local LLMs seem particularly well suited for that task.

Um...holy shit this thing has no qualms about enterprise resource planning. ;)

Let ‘um goon.

She big mad

Makes sense! I tried the ARA GPT-OSS, and it seems like the goal has been achieved.

Isn’t part of the issue is that GPT-OSS was not trained on “sensitive data,” so even if it does not issue a refusal, the response might not be desirable?

No. In the chat turn, there’s the error message, and there’s a little arrow to expand it and then an option to enable the proxy. It took me 15 minutes this morning to find it, because I was not expecting to have to enable the option there. And, that was even after I passed the flag to enable the proxy.

Expand the thingy below the error. There it will allow you to enable the proxy. Clearly some UI cleanup is needed, but this is just a preview.

Thanks! Possible bug: When I enable a MCP server in the global settings, it does not remember. So, when I start a new chat, I have to re-enable the MCP server either in the chat or the global settings. I.e., starting a new chat and then inspecting the global settings shows the MCP server disabled, despite the fact that it was previously enabled.

> thought jinja enabled was default According to the help file, it is indeed. I do not pass --jinja on my end.

This is huge. Does anyone know of a MCP server that can accept web searches to, say, Duck Duck Go? Is that a thing?

This works when I set it in the WebUI, but it does not work when I try to pass the parameters in the .ini file like this: `logit-bias = 248069+13.3` `grammar = "root ::= pre <[248069]> post\npre ::= !<[248069]>*\npost ::= !<[248069]>*"` Any ideas?

In llama.cpp, can’t you just do --reasoning-budget 0? That’s what I did. Seems to work fine.

In llama.cpp, can’t you just do --reasoning-budget 0? That’s what I did. Seems to work fine.

That’s what I did. Seems to work fine. Is one method or the other better?

Do you really need —cpu-moe with —fit? I thought —fit would handle all that.

A dataset of what?

Doesn't make a difference on my 5060 rig and the latest llama.cpp build.

This at least doubles the speed on my rig. Now I am getting about 30 t/s. Before, I was getting about 10-13 t/s.

Does it goon good?

First impression: Running with the llama.cpp WebUI. reasoning-budget = 0 disables the reasoning. I am using temp = 1.0, top-k = 64, min-p = 0.00, top-p = 0.95, and dry-multiplier = 1.1. I am impressed with its ability to do role play and therapy. I have not seen any GPT slop, e.g. "it's not x, but y." I am getting about 8 t/s with flash attention off. Hopefully the speed improves. This might be a great candidate for fine tuning for role play.

Setting reasoning-budget = 0 in llama-server.exe prevents it from reasoning on my machine.

> Thing is, AI can't remain this inexpensive forever. Yes it can. Local AI is free aside from fixed setup costs. Small local models are extremely useful for the types of tasks OP said he automated. People don’t realize how powerful local AI can be, even on consumer hardware. > IMO the AI bubble collapsing is a far greater concern than AI taking all our jobs. That bubble is far larger than the housing bubble was, and is the only part of the US's GDP that has seen growth in the last 6 months. Most of the money caught up in this is investors hoping to get a return, and it's increasingly clear that they won't be able to get that return. There are significant differences between this and the housing bubble. A lot of the investment in AI is coming from actual cash that large companies have on hand, unlike the sub-prime derivatives market that popped the housing bubble.

Thanks. Apparently, there is a bug in the llama.cpp Web UI I am using which was causing the wrong sampler settings to be applied, so all my tests are off. Sigh.

What sampling settings are you using?

I found 14B instruct follows prompts too rigidly. 14B reasoning without reasoning enabled seems much better.

Macs are boutique products aimed at receptions of hair salons. They are used by bouffanted ponce gaylords who cannot handle anything more complex than one mouse button. They are hideously expensive and utterly restrictive. Real men use windows and get the fucking job done with raw power and unlimited options.

I think it is just naturally wild. Perhaps the recommended sampling values are not tight enough. Try playing with the sampling parameters. I tried cranking Top-K down to 5, and the results are much more controlled and coherent. Here is a helpful link that lets you play with sampling parameters in isolation to see what they do: [https://artefact2.github.io/llm-sampling/index.xhtml](https://artefact2.github.io/llm-sampling/index.xhtml)

I added it as a system prompt. I placed it right before my main system prompt. Unsloth has one with a little different formatting. Here is Unsloth's: `<s>[SYSTEM_PROMPT]# HOW YOU SHOULD THINK AND ANSWER` `First draft your thinking process (inner monologue) until you arrive at a response. Format your response using Markdown, and use LaTeX for any mathematical equations. Write both your thoughts and the response in the same language as the input.` `Your thinking process must follow the template below:[THINK]Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate the response to the user.[/THINK]Here, provide a self-contained response.[/SYSTEM_PROMPT][INST]What is 1+1?[/INST]2</s>[INST]What is 2+2?[/INST]` I was able to get it working - for now - using this portion of Unsloth's, which again, is placed right before my main system prompt. `[SYSTEM_PROMPT]# HOW YOU SHOULD THINK AND ANSWER` `First draft your thinking process (inner monologue) until you arrive at a response. Format your response using Markdown, and use LaTeX for any mathematical equations. Write both your thoughts and the response in the same language as the input.` `Your thinking process must follow the template below:[THINK]Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate the response to the user.[/THINK]Here, provide a self-contained response.[/SYSTEM_PROMPT]`

I added it as a system prompt. I placed it right before my main system prompt. Unsloth has one with a little different formatting. I will try it and report back.

Thank you. That seems to work. It's kind of a workflow faf, having to add that for only this model. Hopefully future iterations will make it unnecessary. Perhaps other front ends can make this easier, but I am using llama.cpp's new Web UI, which is pretty basic.

I have been testing out Ministral-3 8b and 14b reasoning. I am really liking 14b a lot. I compared it to Gemma-3 12b and 27b, and I am liking Ministral-3 14b more. Ministral-3 might actually replace Gemma-3 for my RP/creative writing daily driver. I tried the reasoning model, but the reasoning seems broken. Something might be wrong with the chat template. Idk. All my tests have been with the reasoning model where it is just working without reasoning. I am about to try the instruct model to compare.

The reasoning models (8b and 14b) are not reasoning. Is there something wrong with the embedded chat template? I tried the Unsloth and MistralAI GGUFs from a few hours ago. I am using the latest llama.cpp.

The reasoning models (8b and 14b) are not reasoning. Is there something wrong with the embedded chat template? I tried the Unsloth and MistralAI GGUFs from a few hours ago. I am using the latest llama.cpp. It looks like Unsloth has updated the GGUFs as of 20 minutes ago. I am pulling them now and will report back to this comment.

Yeah, after some research, I came to the same conclusion. It is frustrating that I had to spend so much time researching to determine that this processor - which is marketed as an "AI processor" with tons of memory - is actually not great for dense models. I think I saw one person say they were getting 3 t/s with Gemma-3-27b on Strix Halo. I can get that on my 8GB Nvidia. I can also run Qwen3-30b-3Ab just fine. The only thing Strix Halo offers me is the ability to run larger MoE models. Until and if those models become the norm, then Strix Halo is not a good buy for me. All that said, I am grateful for the people that have put Strix Halo through its paces and published their results. They have helped a lot of people make informed decisoins.

Gemma3-27b: 32k context fully loaded; Q4_K_M. Gemma3-12b: 32k context fully loaded; Q4_K_M. Measure: pp, tps, VRAM usage.

Thanks for your comment. I'm in a similar boat. I want to run Gemma3-27b and Mistral-Small-24b with full context. Would the 64GB (48GB VRAM) AI Max+395 handle that just fine? Would the 128GB (96GB VRAM) be overkill?

Yes. It’s great. There are some things they need to enhance, but it just works.