Want to set up my own AI thing for RPing (Story Driven)...
Posted by Sufficient-Grape5366@reddit | LocalLLaMA | View on Reddit | 12 comments
However, I know next to nothing technical-wise. What should I start learning? You see, I want to do solo roleplaying and I use to use ChatGBT... However it could not remember details even with giving it the needed data. Not only that, but it seemed to be gimped in many areas (especially censoring things that has no business being censored.) Any help would be appreciated!
theair001@reddit
for the software: quick start: lm studio really getting into it: kobold.cpp, sillytavern, oogabooga, chose your poison, guides are widely available
for the models, start with something small, mistral 7b, then go bigger, mlewd 13b, wizard-vicuna 30b, just to name a few popular ones. make sure to use the q4 for good performance.
if you are able to go bigger, try midnight-miqu 70b, venus 103b, monstral 123b, again just a few popular ones for erp. the bigger work on lower quants too, so dont shy away to try a 120b on q1, maybe you're into it.
don't expect too much. unlike chatgpt, those models come pretty bare and dumb. you'll probably put a lot of time into finding just the right way to prompt the characters you want to the llm to portray. but at least you will control the system prompt, no pesky guiderails except the ones baked into the models.
have fun playing around!
Sufficient-Grape5366@reddit (OP)
Did a little more research on models that are good for 12 gb vram... Decided to try Tiger-Gemma2 9B, will let you know how it goes :3
Sufficient-Grape5366@reddit (OP)
It seems to work to be work so far.
Sufficient-Grape5366@reddit (OP)
Thanks! Attempting Mistral 7b... Any idea how to use it with Kobold?
mustafar0111@reddit
Koboldcpp can do RP stories and gaming with its own front end. Alternative Sillytavern can do it with another app doing inference.
There is sort of a gold rush going on in the gaming space to integrate LLM's into game engines but its still pretty early stages.
Sufficient-Grape5366@reddit (OP)
Trying Koboldcpp now, what model do you suggest?
mustafar0111@reddit
In terms of models Wayfarer 2 is a decent place to start for RP games and dungeon crawlers. I believe its built off of NEMO.
Sufficient-Grape5366@reddit (OP)
Decided to try Gemma-3-27B Abliterated... It generates things quite slowly... I assume that is a ram issue?
mustafar0111@reddit
You want a model that fits completely into GPU VRAM if possible.
Awwtifishal@reddit
Check r/SillyTavernAI for recommendations. You should choose models in a range of sizes that depend on your GPU RAM and your main RAM.
Sufficient-Grape5366@reddit (OP)
Thanks, I'll look into this! I hope it's easy to get into without a lot of coding... I've tried coding classes way back when and it's safe to say that's why I'm balding <<'' Thanks again!
-dysangel-@reddit
story: step sister got stuck in the washing machine again