KoboldcPP is such a gigantic leap in QoL coming from Oobabooga is just ridiculous.

[-]

jimmyluo@reddit

OGs remember that Oobabooga was here letting us run local text-generation LLMs in their full (quantized) glory with a simple web-UI when most of you whiners were still in diapers.

Reply

[-]

I love it and use it for everything except for coding because I can't figure out how to get syntax highlighting. Normally that's not a problem but since the alternatives I use don't have speculative decoding I've been using kobold because it's so damn fast with that. Open to suggestions.

Reply

[-]

henk717@reddit

One thing you could do is hook Continue up to KoboldCpp (With the OpenAI emulation) so you have it directly in the code editor.

Reply

[-]

LocoLanguageModel@reddit

Thanks Henk, I forgot about the open AI emulation, this is great!

Reply

[-]

Expensive-Paint-9490@reddit

Speculative decoding on kobold? I have to update my version, I think.

Reply

[-]

Organic-Thought8662@reddit

To bypass the check for vocab mismatch, just enable debug mode. Had to do that to get qwen coder instruct 32b working with the 0.5b draft model.

Reply

[-]

silenceimpaired@reddit

Vocab mismatch?

Reply

[-]

Organic-Thought8662@reddit

Have You tried using Sillytavern as a fronted? It does have native support for code blocks and basic syntax highlighting

Reply

[-]

LocoLanguageModel@reddit

Wow that works great thanks so much! A few days ago I briefly googled if silly tavern did that but apparently I gave up too soon.

Reply

[-]

Goldkoron@reddit

Tabby has speculative decoding for exl2 models for a long time

Reply

[-]

LamentableLily@reddit

It's such a godsend that koboldcpp is a single .exe that's easy to run without much thought, but also provides more control via command line if you need it. No docker, no building anything. Just click and go. Chef's kiss.

Reply

[-]

Ulterior-Motive_@reddit

Ironically it did end up becoming the Automatic1111 of LLMs; it hasn't updated in months and has been superseded by other projects.

Reply

[-]

oobabooga4@reddit

[https://github.com/oobabooga/text-generation-webui/commits/dev/](https://github.com/oobabooga/text-generation-webui/commits/dev/)

Reply

[-]

ArsNeph@reddit

Yeah, this is very unfortunate. At least oobabooga gets sampler and tokenizer updates though, Automatic1111 refuses to even support new models. I've been using both since the beginning, and they were excellent, but I always knew they were very temporary. While A1111 has forge carrying the torch, and power users moving on to comfy, I don't really know what to replace ooba with. It's much simpler to load and switch models, including tweaking offload layers than kobold, and it's also the easiest way to access every inference engine. I don't think any other software offers that type of utility.

Reply

[-]

Ulterior-Motive_@reddit

It's sad honestly, both of them were my intros to local AI. I always used ggufs anyway, so switching to a combo of llama.cpp and open-webui wasn't a huge loss, and in many ways a benefit (performance is much better), but it still lacks many of the controls ooba does, the continue function doesn't work as well, and there's no impersonate function (though they recently added autocomplete, which gives mixed results).

Reply

[-]

fish312@reddit

Speaking of A1111, koboldcpp does emulate it, and flux actually works there so it's better than a1111

Reply

[-]

Ill_Yam_9994@reddit

I don't want to diss any open source projects, but I also think KoboldCPP is the best. It has a good balance of freedom/user control and abstraction and the developer is really good about adding support for new models and technologies. Plus it's very lightweight and unbloated.

Reply

[-]

DeweyQ@reddit

I keep hearing such good things about koboldcpp but I can never get it working on my system. Probably due to AVX (have) versus AVX2 (don't have).

Reply

[-]

Psychological-Gold53@reddit

* **Old CPU Compatibility**: If you are having crashes or issues, you can try running in a non-avx2 compatibility mode by adding the `--noavx2` flag. You can also try turning off mmap with `--nommap` or reducing your `--blasbatchssize` (set -1 to avoid batching) have you tried that ?

Reply

[-]

DeweyQ@reddit

Thanks for the suggestion. For whatever reason I couldn't get the command line combinations right. Using kboldcpp-oldcpu.exe and using the UI to mirror the setting I had in ooba... that worked. Not sure it is faster than ooba yet. And for QoL... my struggles mean my ROI is not a wash yet.

Reply

[-]

SomeOddCodeGuy@reddit

KoboldCpp has certainly been my favorite, but I feel like each has some thing that makes them stand out * Koboldcpp is lightweight, easy to use, fast as lightning and generally propogates the Llama.cpp changes the fastest. As a mac user, this is just the perfect API for me * Ollama is extremely new user friendly. If you want to introduce someone to LLMs then 5 steps can have them running Ollama and Open WebUI * Text-Generation-WebUI is all-encompassing in terms of loaders. Its got every kind of loader you could ever need, AND it handles the baked in jinja prompt templates like a champ, so if you want to test out a Chat Completions endpoint? That's the way to go. * LM Studio: While not open source (last I looked), it is free, so tossing it out here too. It has the prettiest UI by far lol.

Reply

[-]

Robot1me@reddit

Oobabooga is only really decent as a backend, change my mind :P

Reply

[-]

Lissanro@reddit

Maybe it was in the past, but at some point it started to lag behind, last time I checked there was no Q6 support for cache, no speculative decoding, no tensor parallelism, some options that are deprecated (like FP8 cache in ExllamaV2, which is worse than Q4, and far worse than Q8 and Q6). It needs a lot of work to get up to date, both as a backend and frontend.

Reply

[-]

10minOfNamingMyAcc@reddit

Ooba worked great for a few months ~ end of 2022 Could only get errors after that. Tried to reinstall a few times but it's just a mess on my windows machine. (Ampere GPU and a new windows install as well.) The only two programs I have always issues with are Oobabooga and kohya_ss

Reply

[-]

mpasila@reddit

Haven't really had issues with it though I've used it by installing it manually on a conda environment. (also you probably mean end of 2023? since the project was created in December 2022)

Reply

[-]

10minOfNamingMyAcc@reddit

That could be very true haha, time flies when you're playing with ai...

Reply

[-]

jacek2023@reddit

I use mostly koboldcpp but I installed text generation web ui because exllama support

Reply

[-]

Daxiongmao87@reddit

Exl2 is so fast. I went from ollama to ooba/text-gen because of it

Reply

[-]

Lissanro@reddit

EXL2 works about half the speed in text-generation-webui (oobabooga), at least this was the case last time I checked due to lack of support of both speculative decoding and tensor parallelism. This is why I ended up migrating to SillyTavern with TabbyAPI (after starting TabbyAPI using `./start.sh --tensor-parallel True` command, I can use [https://github.com/theroyallab/ST-tabbyAPI-loader](https://github.com/theroyallab/ST-tabbyAPI-loader) extension to load models and configure TabbyAPI options).

Reply

[-]

lxe@reddit

Ditto. Been sticking with ooba for exl2. Does the job well.

Reply

[-]

10minOfNamingMyAcc@reddit

tabbyAPI is my preference

Reply

[-]

silenceimpaired@reddit

I have been meaning to try TabbyAPI because KoboldCPP truly seems superior to TextGen by Oobabooga

Reply

[-]

pumukidelfuturo@reddit (OP)

i've read good things about web ui, i should try it.

Reply

[-]

zekses@reddit

I don't see how it's better. I tried using current kobold and in general hated how the settings are spread and the experience as a whole

Reply

[-]

YearZero@reddit

I haven't used ooba for LLM's, but I love the benchmark: [https://oobabooga.github.io/benchmark.html](https://oobabooga.github.io/benchmark.html) It's simple and to the point. Also, it's usually damn quick to test a model when other benches like Livebench or whatever may take days, weeks, or never. It's not the same scale of a bench, but I'm a simple guy, and I like simple things.

Reply

[-]

TurpentineEnjoyer@reddit

If you're getting errors then that's not an ooba problem, it's a you problem. I can install ooba fresh on a Ubuntu 24.04 system in seconds, because my server is set up to be formattable on a whim if I want to start again. Last time I used Kobold about 3 months ago, it was significantly slower - ooba seems to scale the context over time so you get speed with low context which then slows down as context increases. Kobold gets similar performance speed, but it's static to the context size I chose. For that alone I think ooba is better.

Reply

[-]

OutrageousMinimum191@reddit

If you're smart enough to write one command line and save it into bat file, llama.cpp is better than kobold.

Reply

[-]

Ylsid@reddit

Oobabooga got us off the ground and let everyone use the tech easily. Literally named after caveman words because anyone can do it!

Reply

[-]

kexibis@reddit

I use oobabooga web ui through API, settings are also loaded automatically considering the model loaded Just added pc startup loaded and I use models in different uis and copilots remotely. Two , three url-s I connected to domains and I use in copilot or copilot extension in vs code. Also I use them in apps via api... several servers I never added any setting except for context length. So.wt f... are you talking about. Kobold... my a..

Reply

[-]

Qual_@reddit

Laughing in openweb ui

Reply

[-]

pumukidelfuturo@reddit (OP)

i have to try that one, but i'm having errors installing it. So yeah...

Reply

[-]

mrskeptical00@reddit

It’s a pretty simple install if you know your way around Docker. I struggled with it at first and gave up. Went back a few months ago after I played with Docker for something else and it was easy. What OS are you using?

Reply

[-]

RedditDiedLongAgo@reddit

Has anyone in the.thread actually read through koboldcpp.py. Shit's a horrifying mess of globals and sketchy threading hacks. It's amazing that it even works.

Reply

[-]

Pristine_Income9554@reddit

Oh this guy don't saw tabbyAPI with exl2 models

Reply

[-]

silenceimpaired@reddit

TabbyAPI is like comparing apples to oranges with KoboldCpp. They are both fruits (backends) but KoboldCPP loads GGUF files so that means TabbyAPI would do nothing to improve OPs situation unless he redownloads his models … and my guess is it may still not help since OP might be VRAM limited.

Reply

[-]

RedditDiedLongAgo@reddit

Has anyone in their thread actually read through koboldcpp.py? Shit's a horrifying mess of globals and sketchy threading hacks. It's amazing that it even works.

Reply

[-]

wh33t@reddit

KCPP is the goat. it still needs lots of QOL updated but it's by far the most mature tool for text generation imo.

Reply

[-]

rerri@reddit

Picnic.

Reply

[-]

Inevitable-Start-653@reddit

Dude this is a very rude post! From your description of tweaking infinite settings" it sounds like you lack the knowledge or cognition to use the software appropriately. You are shuting on an extremely important open source project, one that I use almost exclusively because of the ability to customize and have access to all the options. The more im writing the more upset im getting, get bent and screw yourself.

Reply

[-]

synth_mania@reddit

Oobabooga is the github username of the developer you dunce. text-generation-webui is the actual name of the project. How rude.

Reply

[-]

doomed151@reddit

What a rude post. oobabooga is the dev's name btw. I think "text-generation-webui" is a perfectly good name for a project.

Reply

[-]

Dundell@reddit

I've used everything I could get my hands on vllm, Aphrodite, tabbyapi, the textgen, llama.cpp, koboldcpp, and simple scripts for direct terminal requests. For my biggest models and deployed chatbots I use TabbyAPI, and a single llama.cpp for the pascal cards. Both now seem good for Q4/Q6 context, speculative decoding, and just simple OAI API Compatible endpoints. But Textgen Webui, I like to use for customization and adding in extensions, editing the Api endpoints for custom versions for voice assistance in interesting way. Seems more like tinkering for projects and loading up some single pascal GPUs for that awesome ad_discord bot project from altoiddealer.

Reply

[-]

mrskeptical00@reddit

I can’t stand either of those, UI on both is not pleasant. Open WebUI is the cleanest and it doesn’t care what backend you use. Msty as a front end for Mac/PC isn’t bad either.

Reply

[-]

TrashPandaSavior@reddit

You can change the UI theme for koboldcpp. The 'corpo' one looks most like chatgpt/open-webui. Even has markdown support.

Reply

[-]

mrskeptical00@reddit

That’s good to know, thanks.

Reply

[-]

silenceimpaired@reddit

Both of the examples he provided could be used as the backend for open webui and his conversation has been about performance, which means his focus is on the backend. So anyone reading the comment above should not run off to use open webui as their new backend because the commentator likes the way it looks.

Reply

[-]

mrskeptical00@reddit

If it’s slow then it shouldn’t be used as a backend. OP seems to be using all-in-ones, he can try different backends llamacpp, Ollama, Koboldcpp - and still have a better UI experience. I’m educating OP as to there being better options out there.

Reply

[-]

Nitricta@reddit

Oobbabooga is the name of the author though? Isn't it just the 'friendly name' that people use for it. Ooba has been pretty good towards me though.

Reply

Reply to Post

58 Comments