What frontend ui + backend server are you using?
I'm also struggling to find documentation on this.

I found a similar ChatML excerpt on a random model on Hugging Face, and I'm using it somewhat successfully with Hugging Chat UI (aka chat-ui) as the frontend and Hugging Face's Text Generation Inference as the backend to locally serve models through my GPU.

• frontend: https://github.com/huggingface/chat-ui
• backend: https://github.com/huggingface/text-generation-inference
• model: https://huggingface.co/solidrust/samantha-1.1-westlake-7b-laser-AWQ/tree/main
• based on: https://huggingface.co/cognitivecomputations/samantha-1.1-westlake-7b-laser

​

Main issue:

Hugging Chat's settings panel has this System Prompt section, which I assume is where the ChatML goes.

I've played around with it and somewhat gotten it to work... but copy pasting this block doesn't really work

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

My models with this setup were erroring frequently.

However, I discovered that if I pasted the following into the chat (and leave System Prompt 100% blank) then the tokens actually work as expected (or seem to).

I got this from the original model's description on Hugging Face:

• https://huggingface.co/cognitivecomputations/samantha-1.1-westlake-7b-laser
• (disclaimer: that page's description's image is a little uncanny valley for me, but out of the 10+ models I looked at, this one was the most simlar to my online ChatGPT 3 solutions / Python excerpts)

​

<|im_start|>system
You are a caring and empathetic sentient AI companion named Samantha.<|im_end|>
<|im_start|>user
Hello, what is your name?<|im_end|>
<|im_start|>assistant

So right now I'm manually pasting that into each chat, and it peforms amazingly well for a local GPU LLM, but it's getting cumbersome to remember to paste this every time before I begin typing:

<|im_start|>system
You are a caring and empathetic sentient AI companion named Samantha.<|im_end|>
<|im_start|>user

<|im_end|>
<|im_start|>assistant

I suspect it might be a limitation of how Hugging Chat is passing parameters from System Prompt into the backend, but I haven't found the line of code that's passing that yet.

And you know what, now that I think about it... I wonder if Huggin Chat's System Prompt is ChatML agnostic and if it's just expecing a plaintext description in that bubble in the settings... and maybe it'll insert all the arcane tokens auto-magically.

Actually, yeah, System Prompt passes into some var called model.preprompt.

• https://github.com/huggingface/chat-ui/blob/21c9b41a4e045ccb466c3e8a9a97271f9a1858ba/src/routes/settings/%5B...model%5D/%2Bpage.svelte#L104

Maybe it's simpler than I thought.

The README doesn't mention System Prompt sooo... ah, no good.

Just using plain english doesn't work.
And if I don't manually add the <|im_start|> and <|im_end|> tags, then the model glitches out.

i.e. I must manually type <|im_start|>user prompt here<|im_end|>, or else it'll glitch.

Like if I exclude the tags and say

Hello! How are you

the model will glitch and continue the sentence as if it's finishing it's own thought and say something like

today? <etc response>

So... these tags seem to be very meaningful, but for the stack I'm using with huggingface's frontend + backend... it's not entirely documented, and this was as far as I could get in a weekend just tinkering around.

Overall still a very promising technology, and I'm still exceedingly surprised at the quality and speed on a consumer grade GPU. Amazing for generating mockup JSON objects and Python.

Anyway, best luck OP!

If I ever find out more I'll post here if the thread isn't locked.