I am open sourcing a smart text editor that runs completely in-browser using WebLLM + LLAMA (requires Chrome + WebGPU)

[-]

Equivalent-Bet-8771@reddit

Great project but you lost me at Chrome. I won't put up with Google and their anti-adblocker strategy. You'll pry uBlock Origin from my cold dead hands.

[-]

FriskyFennecFox@reddit

THIS IS GLORIOUS, seems like there's no need for DeepL Write now!

[-]

Duh, here we go with the reason why censorship has absolutely no place in LLMs. Can you please look into using alternative LLMs as an option, u/yyjhao ? The code seems to be using one of the official 3B LLaMAs.

[-]

yyjhao@reddit (OP)

Do you have a small model you like for things like this?

[-]

FriskyFennecFox@reddit

Consider Hermes-3-Llama-3.2-3B, Dolphin3.0-Llama3.2-3B, or Dolphin3.0-Qwen2.5-3b, all should be a suitable drop-in replacement for the official LLaMA 3B and have no corporate alignment applied to them.

I'm not sure how the library you're using would play along though.

[-]

yyjhao@reddit (OP)

I opened a ticket to support Hermes-3-Llama-3.2-3B: https://github.com/mlc-ai/web-llm/issues/650

[-]

Unable-Finish-514@reddit

Wow - I've seen some ridiculous refusals from LLMs, but this one you posted is next level....

[-]

AppearanceHeavy6724@reddit

I'd rather have natively ai-enabled notepad or nano, instead of the trendy, heavy web-based thing.

[-]

thenightsiders@reddit

What kind of features would you want?

I've been considering making something like this, with sync to Git for backup and markdown.

[-]

Inkbot_dev@reddit

For me, the ability to select specific parts of the document and have the model work with just that part, the ability to do a "fill in the middle" type task with some instruction following possible. Getting alternative possibilities for a specific selection's continuation other than the original (at the single token level, as well as multiple tokens out).

All of the LLM interactions should essentially work like merging a diff on a file with conflicts. You should be able to see the original, and proposed solutions side by side, try new proposals, and accept / reject, etc.

I think some default interactions would be good to have shortcuts for (paraphrase this section, write a summary, etc).

[-]

cManks@reddit

Copilot chat let's you use the current selected document as the context to pass to a model (at work we have gp4o, o1, and sonnet available). You can highlight lines in the document and that will restrict the context given to the model to the selected lines.

[-]

rorowhat@reddit

Better error correction, suggestions, options to rewrite the paragraph/sentence etc.

[-]

AppearanceHeavy6724@reddit

hmm, intersting question. Probably autocompletion, first and foremost. Generally something like continue.dev, but with awareness about how llms work internally; essentially hybrid of chat and editor; i sometimes would like to edit the file in chat form, by telling change here and there, and then plant back at earlier point of conversation to reclaim the context, when gets too congested.

[-]

cobbleplox@reddit

The reason for this is not because end-users like web-based things so much. It is just highly practical in this fast-moving space because it will run on any OS/hardware that has a compatible browser.

[-]

AppearanceHeavy6724@reddit

Web-based stuff is never practical beyond simplest uses. It is just newer generation of users grew up with web-based stuff, so they cannot appreciate the use case for local binary native stuff.

[-]

cobbleplox@reddit

I meant it is highly practical for the developer and not driven by what is ideal for the user.

[-]

AppearanceHeavy6724@reddit

Oh absolutely, yes.

[-]

LoaderD@reddit

Neat! Let us know when you open source the program or extension.

[-]

AppearanceHeavy6724@reddit

absolutely.

[-]

Healthy-Nebula-3603@reddit

notepad build in windows has it already ...

[-]

race2tb@reddit

MS office is going to have a run for its money once you can just have an llm in your browser mimic all its functionality.

[-]

countjj@reddit

No Firefox? 😔

[-]

boredcynicism@reddit

It does actually work in Firefox Nightly! I think it's just a matter of forcing on WebGPU support, Settings->Firefox Labs->WebGPU.

[-]

MoffKalast@reddit

Firefox needs to get its shit together. They need to get it all together and put it in a back pack, all their shit, so it's together.

[-]

yyjhao@reddit (OP)

Once Firefox supports WebGPU it will work! I think the work is already in progress and some of the functionalities are enabled in nightly builds

[-]

Pristine-Woodpecker@reddit

Note that for these applications the browser vendors (including Mozilla) are planning to add specific inference engines so you don't have to round trip through WebGPU.

[-]

Inkbot_dev@reddit

Something like this as a standard react component that plugs into a standard OpenAI API would be really cool. Would really allow it to get some traction as a component if it was easily usable in the most popular web development frameworks.

[-]

yyjhao@reddit (OP)

Hmm yeah probably not that much work. Though the loading state/load time can be annoying

[-]

M3GaPrincess@reddit

I'm very impressed. And I'm rarely impressed.

I'd add a button or keyboard shortcut to save the text as a text file. Maybe another to email the text.

I couldn't get it to work, but my "requestAdapter(compat)" fails on the webgpu report, so it's not on your end. I'm on linux, chromium, but did enable both webgpu options in the settings.

Good idea to open-source it. It's a star on your resume. Simple and elegant solution.

[-]

yyjhao@reddit (OP)

Thanks! Will look into the additional functionalities.

[-]

Healthy-Nebula-3603@reddit

even notepad from windows has that feature already ...

[-]

NoCharacter7203@reddit

How's the performance of WebLLM? When I was using another product presumably running WebLLM on browser, it kicked my machine into high gear. Lights were flashing, things were heating up and general experience was a little off

[-]

yyjhao@reddit (OP)

It's using the GPU so kinda depends on the spec of your machine and the size of the model. For this project I am using the 3B model which should be on the smaller side. Bigger models will be more demanding

[-]

DumbRedditorCosplay@reddit

Talking about that. Would LLMs be better at converting OpenOffice documents to docx and vice-versa than OpenOffice itself? Anyone has tried this?

[-]

emteedub@reddit

I am curious on the ease or any notable hangups with implementing WebLLM. Any?

[-]

yyjhao@reddit (OP)

It uses the OpenAI protocol (like most tools nowadays) so wasn't too bad. The issues are more with small models in general. You are also limited to the models that WebLLM provides. You can use your own models but it takes quite some effort to compile it into their format, upload it somewhere (models are big!) and then use them.

[-]

emteedub@reddit

thanks

[-]

Position_Emergency@reddit

This looks cool but I think it would work better as an extension.

Ideally I'd highlight some text I'd written on a web page, right click, "AI Check", and then your editor interface could appear in a small section on the right side of the page with the highlighted text in it.

[-]

yyjhao@reddit (OP)

hey good idea! and it can do probably a lot more, like all the AI stuff like summarizing, q&a etc. (as long as the small models can handle it lol). will give it a try!

[-]

Position_Emergency@reddit

Does WebLLM support loading and switching between different LoRAs?
That's how Apply are getting the most out of small language models.
Lots of LoRAs loaded in memory and switching between them for different tasks.

[-]

yyjhao@reddit (OP)

Not sure, but I also think the Apple Models are way smaller. A >3B model can probably do reasonably well for most simple task without LORA I think.

[-]

ore0s@reddit

Trying to picture the task a workflow of going from the webpage to the small 'injected' editor—would you normally copy & move the content from you've written into the webpage, and then paste into an external notepad app which has a stronger editing experience?

[-]

yyjhao@reddit (OP)

Demo here: https://editor.yyjhao.com

source code: https://github.com/yyjhao/wrait