What do you think makes a good creative writing model?

Posted by Sicarius_The_First@reddit | LocalLLaMA | View on Reddit | 62 comments

Please be specific, stuff like "just write good no slop lol" is not very specific. For example, what abilities, would you like the LLM to have? How does your workflow usually look?

Reply to Post

Reply

62 Comments

[-]

Kep0a@reddit

coherence with visual description. Honestly I think this has messed up RP for me. These models can output pretty sentences but they don't build a visual in my head of a scene, there are tons of errors with every local model I've tried. It fucking sucks, since I think it would just be solved by benchmaxxing for creative writing but none of these companies GAF about it.

Reply

[-]

FunBluebird8@reddit

I agree. Neither model is particularly good at chronological order of events either.

Reply

[-]

Sicarius_The_First@reddit (OP)

that's actually a good point. the thing is benchmaxxing DOES NOT mean the model necessarily smarter. There's SOME correlation, SOMETIMES. a local 14B model can say it outperforms ChatGPT\\Claude all day long, the very idea is simply deceiving (see Phi4 example & benchmarks). Regarding the visual and physical common sense, yes, it's a problem, it's very hard to do for several reasons: data, and model size. We simply do not have good enough data, and there's a limit to what a small 3B-16B model can do. In the range of 20B+ stuff starts to make more sense, but data is crucial.

Reply

[-]

Kep0a@reddit

Is there a data problem? The internet is full of internet novels and roleplay and endless creative literature

Reply

[-]

Sicarius_The_First@reddit (OP)

true. when i say "data problem" i mean both good& well processed data :)

Reply

[-]

Double_Cause4609@reddit

Vocabulary variation: The token, or rather, detokenized word distribution should be zippfian. This is numerically variable. If the same word, bigram, or trigram shows up at an outsized rate, that's an indicator of repetitive vocabulary / phrasing. Unnaturally present trigrams tend to manifest as "slop" to an end user (even if it's a new type of slop that they're not used to, they get used to it \*very\* quickly) Semantic variation: If the embeddings of an LLM across a large number of generations have a high degree of cosine similarity, it's probably a very "by the numbers" model that follows a pretty specific formula. \- Note, the above two have huge implications on the downstream usages, and cover a lot of complaints of "lack of creativity" of models. Plot threads: Given an explicit hook (ie: "I wonder what happens if..." or something in context that could be used to move the story forward...Does it? Foreshadowing/Recontextualization: Is the LLM willing to take themes and reframe them in a creative way? Character Traits: Are they present? (Reasoning models are well represented here) Character Traits: Are they presented well? If they're presented poorly, such as in a way that's too over the top, or unnatural, in some ways that can be as bad as not having them. (Base models and smaller models with continued pre-training usually handle this the best, but special shoutout to Maverick) Intentional deviation: Can the model evolve the character, and deviate from its base definition in a controlled way? Formatting: Does the model pick up on the style? \*Actions\* "Dialogue", or \*Actions\* dialgoue, or \*Actions\* dialogue "inner thoughts" etc. \- Also: most models get it nowadays, but does it handle tense and POV properly? OOC: Related to dialogue, given a prompt, can it exploit OOC commands without them bleeding through in character? OOC is a super powerful tool for directing the story. Consistency: Are there things stated early in the context that have been contracted? \- In particular, things like clothing and equipment are big, but also plot element, world elements, relationships, etc.

Reply

[-]

Double_Cause4609@reddit

Immediate reply due to word limit: Vocabulary variation: The token, or rather, detokenized word distribution should be zippfian. This is numerically variable. If the same word, bigram, or trigram shows up at an outsized rate, that's an indicator of repetitive vocabulary / phrasing. Unnaturally present trigrams tend to manifest as "slop" to an end user (even if it's a new type of slop that they're not used to, they get used to it \*very\* quickly) Semantic variation: If the embeddings of an LLM across a large number of generations have a high degree of cosine similarity, it's probably a very "by the numbers" model that follows a pretty specific formula. \- Note, the above two have huge implications on the downstream usages, and cover a lot of complaints of "lack of creativity" of models. Plot threads: Given an explicit hook (ie: "I wonder what happens if..." or something in context that could be used to move the story forward...Does it? Foreshadowing/Recontextualization: Is the LLM willing to take themes and reframe them in a creative way? Character Traits: Are they present? (Reasoning models are well represented here) Character Traits: Are they presented well? If they're presented poorly, such as in a way that's too over the top, or unnatural, in some ways that can be as bad as not having them. (Base models and smaller models with continued pre-training usually handle this the best, but special shoutout to Maverick) Intentional deviation: Can the model evolve the character, and deviate from its base definition in a controlled way? Formatting: Does the model pick up on the style? \*Actions\* "Dialogue", or \*Actions\* dialgoue, or \*Actions\* dialogue "inner thoughts" etc. \- Also: most models get it nowadays, but does it handle tense and POV properly? OOC: Related to dialogue, given a prompt, can it exploit OOC commands without them bleeding through in character? OOC is a super powerful tool for directing the story.

Reply

[-]

Double_Cause4609@reddit

Consistency: Are there things stated early in the context that have been contracted? \- In particular, things like clothing and equipment are big, but also plot element, world elements, relationships, etc. Impersonation: Does the model impersonate the user on its turn? Pro-activity: Does the model frequently end on its turn with sentences like "waiting to see what {{user}} does" and so on? World embodiment: Can the model confidently introduce or reference details of the world at large in-character, and render the story richer for it? Imagery, simile and metaphor: Is the model capable of dynamically using words not typically associated with the current action / object to create a vivid description of events? Obviously too far goes into purple prose, but a visage crumbling (as though of a mountain), or a "rose blooming in battle" are strong images that I quite like personally, but this is preference. (I'm probably going to get a lot of kickback on this one in particular, but it's worth noting people are probably only kicking back because they've had issues with models with low vocabulary variation using the same imagery in a bunch of different completions, not because it was a bad phrase) Over-embellishment: Related to above, does the model go through excessive effort to avoid using common words like "Such and such articulated" instead of "said", or "orbs" instead of "eyes" etc, to an unnatural degree. Instruction following: Can the model follow system prompt instructions correctly, in-character? Positivity bias: "But everything was alright in the end..." etc, kind of kills the feel of a dark fantasy setting or RP. Personal issues I care about but maybe other people don't: The ability to re-write the assistant's first message, and maybe add an Ali:Chat example to cover the content of the roleplay and summarize it so I can move it to a new context without needing like, 16k, or 32k tokens in context constantly (when a fresh context is totally possible with a summary). \- Also, ability to extract knowledge graphs in a structured way, but again, this is personal preference. Function calling: Personal preference, but The ability of an LLM to call functions is really powerful, and it means you don't need a dedicated function calling LLM loaded too, which matters for the larger models.

Reply

[-]

silenceimpaired@reddit

I wish you could share more details ;) But seriously. Go write a paper and release a sampler. :) What local models come closest to this in your mind?

Reply

[-]

Sicarius_The_First@reddit (OP)

10/10 reply, this is what I was looking for. Very well articulated, good points given and demonstrated. I will take this into account, and talk to some authors too. I think I have a solid idea what and how to make. Thank you again for the excellent suggestions 🙏🏻✍🏻

Reply

[-]

s-i-e-v-e@reddit

The only thing that matters is instruction-following ability. I don't need the model to be creative. My prompt has character sheets, scene outlines, writing samples for author style and a few other things. The model must be able to understand the prompt and follow it exactly as given. It must be able to hit all the points in the scene outline. It must understand the character traits and stay true to the voice of each character.

Reply

[-]

silenceimpaired@reddit

If I may ask, which models do you use that meet that need?

Reply

[-]

Sicarius_The_First@reddit (OP)

what you've describe is only doable by larger models (more layers = better long context and handling complexity). while llama3 70B & mistral 123B and qwen72B CAN do this, they would do it worse than closed source models (much more dry, sloppy, worse instruction following). so i like your perspective very much, as indeed this is something missing in open source, and the community needs.

Reply

[-]

silenceimpaired@reddit

I think you are speaking to s-i-e-v-e and not me. I agree, larger models are more important to this point. In my experience Llama 3.3 70B 8 bit is sufficient to get close enough (for me) to private models of ChatGPT and Claude.

Reply

[-]

Sicarius_The_First@reddit (OP)

Yes, 3.3 was exceptionally good at instruction following (even surpassing all closed models). The problem with it is the dryness. On the other hand more creative model (or any finetune) inevitably hurt the instruction following (some more, some less).

Reply

[-]

Sicarius_The_First@reddit (OP)

interesting. your pov is that of a more professional writer, as opposed to someone else who wanted the model to 1-shot a whole chapter with a single prompt. it's difficult to make a model good at BOTH, but i believe your preference gives more value, i'll consider it.

Reply

[-]

FPham@reddit

Basically, it's the dataset If you feed a model \*too\* much synthetic stuff (which like 90% comes from sharegpt), it starts sounding like ChatGPT 3.5 – seriously, you can feel it. Everytime. You can dress it up however you want, but the illusion always breaks down. I’ve noticed this with pretty much all the Llama 3 finetunes. Eventually, ChatGPT just sneaks in there. It's the way it sings the sentences. It sings like ChatGPT sign. Tadda, dada, da. Dada tada da... I'm really liking Gemma 3 and Mistral Small 22b as the bases – they seem to have avoided a lot of that synthetic sharegpt data, especially gemma. I can actually read its writing without constantly thinking 'AI, AI, AI!' The mistral-small-drummer 22b seems pretty decent as a writing finetune. It's fed Guttengerg dataset. Something like that done to gemma 3 might I think create a TOTL writing that fits to my 3090.

Reply

[-]

AppearanceHeavy6724@reddit

Mistral Small 24b has no synthetics data in training, according to Mistral themselves, yet dry and feel even worse than ChatGPT 3.5. Mistral Small 22b is _massively_ better than 24b, true.

Reply

[-]

Sicarius_The_First@reddit (OP)

they do have a lot of synthetic data in the 24B version (which might explain why the 22b feels "better").

Reply

[-]

AppearanceHeavy6724@reddit

No they do not: > https://mistral.ai/news/mistral-small-3 We’re releasing both a pretrained and instruction-tuned checkpoint under Apache 2.0. The checkpoints can serve as a powerful base for accelerating progress. Note that Mistral Small 3 is neither trained with RL nor synthetic data, so is earlier in the model production pipeline than models like Deepseek R1 (a great and complementary piece of open-source technology!). It can serve as a great base model for building accrued reasoning capacities. We look forward to seeing how the open-source community adopts and customizes it.

Reply

[-]

Sicarius_The_First@reddit (OP)

i'm well aware of their claims, however their claims are incorrect. i'm not trying to argue, this is 100% the case, and i say this since i checked, and was disappointed. for more details see this model card: [https://huggingface.co/SicariusSicariiStuff/Redemption\_Wind\_24B#additional-thoughts-about-this-base](https://huggingface.co/SicariusSicariiStuff/Redemption_Wind_24B#additional-thoughts-about-this-base)

Reply

[-]

AppearanceHeavy6724@reddit

With all due respect, your opinion has little weight. I think it is a misguided belief that lack of or presence of synthetic data in training material has something to do with sloppyness of the model. Mistral Small is simply "overcooked" with stuffing STEM data, same happened to LLama4 - undercooked LLama Maverick (good writer) on LMArena vs LLAma4 Release (awful writer) feels exactly same as Small 22b vs 24b. Meanwhile DS V3 0324 or R1 choke full of synthetic data do not feel particularly sloppy.

Reply

[-]

Sicarius_The_First@reddit (OP)

I can't take your comment seriously, as you clearly didn't read the evidence. If you somehow had a different, logical explanation to the findings, that I could respect, but you didn't even bothered with any of it.

Reply

[-]

AppearanceHeavy6724@reddit

My take is they as well may have lied about not using synthetic data, but I still take their word over yours. Your "evidence" is not convincing at all, besides you completely ignore the fact models that openly use synthetic data are still less sloppy and shitty than 24b.

Reply

[-]

Sicarius_The_First@reddit (OP)

i see. so basically appeal to authority it is then? don't believe your lying eyes.

Reply

[-]

AppearanceHeavy6724@reddit

No I am saying that all your "evidence" is pure speculation. I am almost 100% convinced that slop is artifact of improper training and will appear independently of type of material used to train the model, some convergent function at play. GPT-4, the supposed origin of slop was not trained on its own synthetic output, if you think about. Pixtral Less is less overcooked than Mistral Large and has far more pleasant slop that Mistral Large; both almost certainly used same training data. In this case, yes appeal to authority if the authority is the actual creator of the model; not a some fine-tuner speaking out his gut feelings.

Reply

[-]

Sicarius_The_First@reddit (OP)

Would you mind sharing your suggestion how to train a model properly? I am open to new ideas.

Reply

[-]

AppearanceHeavy6724@reddit

You need to ask Google, they were very successful with their Gemmas, which almost certainly used synthetic data.

Reply

[-]

Kep0a@reddit

I'm honestly thinking of switching back to the 22b version. Have you? Is it just still better?

Reply

[-]

AppearanceHeavy6724@reddit

No I have not, but I'll try tomorrow. My vague recollection is it is a smarter Nemo.

Reply

[-]

Dry-Judgment4242@reddit

To me, it's a model that is resistant to repeatition.

Reply

[-]

Sicarius_The_First@reddit (OP)

that problem was solved a long time ago. my old model would 1 shot 50kb plaintext no problem. is it coherent? hell no. but it does not repeat. can you define more clearly what do you mean?

Reply

[-]

stoppableDissolution@reddit

I think what they mean is not word to word repeptition, but falling into a certain "rhyme". Models tend to form patterns in what they produce - same sentence and paragraph structure, same-ish order of describing things, mentioning the same thing again and again with different wording. It might be less of an issue with longform, but it definitely is the case for turn-based, and I'm not sure its even conceptually solvable.

Reply

[-]

Sicarius_The_First@reddit (OP)

ah, the sloppy writing pattern themselves. got it. yeah it's a hard problem to solve. for now, we only got workarounds (samplers).

Reply

[-]

Expensive-Paint-9490@reddit

How do you get 50k generation with a one-shot prompt?

Reply

[-]

Sicarius_The_First@reddit (OP)

https://preview.redd.it/fbz5vtcpv6xe1.png?width=1443&format=png&auto=webp&s=07e3526b686d41695fa67364b54d8c84587de4b9 by training a model to do so. its not even a big model.

Reply

[-]

Sicarius_The_First@reddit (OP)

again, for a small model coherency will suffer, 100%. also for a big model, even chatgpt, but to a much lesser degree. more layers and more attention heads sure help a lot.

Reply

[-]

AppearanceHeavy6724@reddit

> that problem was solved a long time ago. Mistral Small 2501 and 2503 are extremely repetitive.

Reply

[-]

terminoid_@reddit

number one thing: it should be uncensored and able to authentically reflect the entire human experience, even if it happens to be violent or depressing, etc. strong instruction following so it knows to output 4 sentences, or 1000 words, etc. no markdown or at least configurable. varied sentence structure i think is a prime indicator of quality. some models really like to overdo it with the comma clauses over and over again.

Reply

[-]

AppearanceHeavy6724@reddit

Gemma 3 is censored but way better than all those uncensored finetunes.

Reply

[-]

terminoid_@reddit

i wasn't talking about any model in particular, i was answering the question. Gemma 3 is trivial to uncensor with a prompt, btw. i've got it writing hardcore sex and depraved violence if I want. Gemma 3 still has it's issues though. really tends toward the dramatic, still sloppy, loves commas. luckily the instruction following is so strong that the output can be fixed up quite a bit with specific prompting. i think it could benefit greatly from a focused creative writing tune.

Reply

[-]

Sicarius_The_First@reddit (OP)

99.9% of the times it is indeed the case, ANY RL hurts intelligence, both for uncensoring and for safety. which is why for very hard\\complicated general tasks i always use the base model. However, for a "narrow" domain like writing, i think a finetune can outperform the base model. i say "narrow" because that's a massive, complicated and a hard domain that includes a lot of stuff, but still focused on writing. what would probably happen, if indeed such model is successfully made, it will almost certainty be "dumber" in other areas. that's not a necessarily a bad thing, finetunes are specializations, and the making a generalist model that is EXCEPTIONAL at everything is 2 expensive, so its the domain of massive corpos (google, meta, openai, etc).

Reply

[-]

Majestical-psyche@reddit

Short high quality stories = more diversity and creativity rather than long novels. You can only fit so much in a model. "Lots of diversity" makes a creative writing model Golden. 🪙

Reply

[-]

Sicarius_The_First@reddit (OP)

you're right, its almost always the case, i think that long context is extremely hard for a model to 1-shot. the key is probably using a model to write in phases. "first lets write X paragraphs". "now lets do Y"

Reply

[-]

Majestical-psyche@reddit

It does, look at Gemma 3... It was trained on novels. It has a really good writing style, but it's just not creative. Lots of 'diversity' of good writing is the key.

Reply

[-]

Sicarius_The_First@reddit (OP)

gemma3 got a great potential, it also has that obscure knowledge no other llm has, however it (along with gemini 2.5 pro) got some really annoying biases (writing in italics for emphasize, cannot be prompted out, not even in 2.5 pro). it might be possible to burn it out in the case of gemma3, however gemma3 is still a bitch to tune.

Reply

[-]

TheRealMasonMac@reddit

Depth. A lot of creative model fine-tunes have pretty-looking prose, but it's shallow and becomes boring pretty fast. I guess it works well for the ERP gooners since the scenario is mostly being driven by the user, but not well for genuine fiction writing. RL does significantly improve on this front though at a slight-to-significant cost to instruction following ability. By depth, I mean I could basically give a prompt a few paragraphs long to the model, and it would be able to create the first chapter of Harry Potter (in a hypothetical world where it wasn't previously written).

Reply

[-]

Sicarius_The_First@reddit (OP)

thank you, this is the kind of answer i was looking for. detailed, and well demonstrated. just to clarify, u mean 1 shot in a single prompt a whole chapter?

Reply

[-]

TheRealMasonMac@reddit

Yeah, 1-shot. But if it does a good job at building the chapter of the course of multiple prompts, I'm not gonna complain either -- that's way better than what current models do tbh.

Reply

[-]

Sicarius_The_First@reddit (OP)

1-shotting this will be very hard to achieve, it's possible though. all models, closed models as well, are having huge trouble of doing it. but i do agree that this ability is desirable. such output won't be as coherent, or immediately usable, but it can serve as a base for modification. maybe asking a model to 1-shot 50 paragraphs, and use the output as the general story, then modifying X and Y paragraphs etc...

Reply

[-]

Sicarius_The_First@reddit (OP)

tbh the most powerful, naturally unhinged and creative local model i used is deepseek (without the thinking). only recently closed models caught up with claude in terms of realistic humor understanding.

Reply

[-]

Federal_Order4324@reddit

Have you tried marinaraspaghetti's Gemini preset? It's pretty good and I think it even surpasses gpt 4 in some regards since there seems to be less jailbreaking. I agree that lots of these local models really do lack the actual nuance of characters and people (I remember the lumi and magnum fine-tunes being particularly horny. Have you tried the new qwq (queen reasoning models) fine-tunes like snowdrop? With a detailed reasoning format (breaking down scenario, scene, characters etc.)(I nabbed the reasoning template that marinara put in the Gemini prompt lol) they do seem to get a lot closer to gpt and Gemini overall

Reply

[-]

GregoryfromtheHood@reddit

Something that can follow instructions, hold a narrative, avoid repeating things and also take its time and not rush the story. Something that can come up with its own creative ideas between the plot points. It needs to not get confused at larger context and be able to hold many ideas and world building rules in its working memory and implement them in a smart way into the story. Also something that can actually write well, in decent sized paragraphs and actually fill out chapters. So far the only model that can do this for me is Gemma 3 12b and 27b. They tick all of these boxes. I have tried so many other models, plenty of 70b, 100b+ and none of them can hold a story together like Gemma 3. I'd love suggestions on something bigger and smarter that I could use, but Gemma 3 27b is the absolute best for writing that I've seen, by a lot. I've wanted to find something that could write spicier stories, and so far Fallen Gemma is the best I've found for that, but it has a tendency to make everything dark and grousome and awful rather than fun.

Reply

[-]

secopsml@reddit

For many years my income was directly related to my ability to persuade others to take action. Phishing, head hunting, sales, event organization, (...) I did almost 500 campaigns. Mostly social media automation and email. Record breaking campaigns were touching real problems/needs/desires using relevant, informative, AND *playful* phrases. For every profile/product/audience there is something else that will work the best: - real time marketing / buzzwords, - absurdly misplaced words, - rhymes, - clever phrases, - sometimes smartly noticing details and mixing them with the offer. All that requires general knowledge about humanity. Understanding of life that is less shallow than SEO optimized blog post from 2018. With local models I built simulator of similar conversations using mixtral 8x7b. The solution was to optimize sampling parameters not system prompts. (there are YC backed startups doing the same for sales so you can check) So my advice to you OP is to spend 100h fiddling with one model and the same prompt and test how differently models behave with different sampling parameters. Once you develop basic intuition for sampling params then you'll be able to adjust them properly to the crative scenarios. Top k, min p, max p, repetition and frequency penalties, temperature, (...) Test all. I did my research with oobabooga webui. !RemindMe 2h

Reply

[-]

stoppableDissolution@reddit

Telling one of the most prominent finetune makers on HF to learn basic stuff is... Bold :p

Reply

[-]

Sicarius_The_First@reddit (OP)

tnx for the advice :)

Reply

[-]

RemindMeBot@reddit

I will be messaging you in 2 hours on [**2025-04-26 01:52:54 UTC**](http://www.wolframalpha.com/input/?i=2025-04-26%2001:52:54%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/LocalLLaMA/comments/1k7z4jp/what_do_you_think_makes_a_good_creative_writing/mp27smd/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2FLocalLLaMA%2Fcomments%2F1k7z4jp%2Fwhat_do_you_think_makes_a_good_creative_writing%2Fmp27smd%2F%5D%0A%0ARemindMe%21%202025-04-26%2001%3A52%3A54%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201k7z4jp) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|

Reply

[-]

someguy@reddit

The generated text needs to be creative, entertaining, coherent. So far I've only experienced that with Goliath-120B. I haven't used any, more recent, large models though, and only just ordered a 3090.

Reply

[-]

Sicarius_The_First@reddit (OP)

ah yeah goliath was amazing for the time. this is not the kind of answer i was looking for, but i do cherish the nostalgy :) reminds me the 'ancient days' of might miqu, goliath and the likes. good times. but 2day we're spoiled for choice. time to make a winner.

Reply

[-]

NNN_Throwaway2@reddit

I mean, you want something that's actually...creative. That is, able to come up with things that are unexpected or novel without being unnatural or incoherent. I feel like that's the benefit of using a model for writing, to come up with ideas that you wouldn't have thought of on your own, or to refine them in ways you couldn't have envisioned yourself.

Reply

[-]

Sicarius_The_First@reddit (OP)

can you be more specific? when you feed it a prompt you want to get ideas? paragraphs? a whole chapter?

Reply

[-]

fizzy1242@reddit

i think a model that is proactive and not too shy to do something unless explicitly told to, while still being coherent.

Reply