Looking for software that processes images in realtime (or periodically).

Posted by My_Unbiased_Opinion@reddit | LocalLLaMA | View on Reddit | 5 comments

Are there any projects out there that allow a multimodal llm process a window in realtime? Basically im trying to have the gui look at a window, take a screenshot periodically and send it to ollama and have it processed with a system and spit out an output all hands free. Ive been trying to look at some OSS projects but havent seen anything (or else I am not looking correctly). Thanks for yall help.

Reply to Post

5 Comments

[-]

vasileer@reddit

why should there be a project for such an extreme edge case? it's just cronjob + ffmpeg + ollama, I guess you can get this "project" done from one prompt by any of the frontier models

[-]

My_Unbiased_Opinion@reddit (OP)

this is exactly what I did. Never coded in my life. My mind is blown. Thank you

[-]

throwawayacc201711@reddit

Just build this as a workflow in n8n or any automation pipeline.

[-]

SM8085@reddit

[llm-screenshot.bash](https://github.com/Jay4242/llm-scripts/blob/main/llm-screenshot.bash) is a quick try at it, https://preview.redd.it/wo942c2nez3f1.png?width=1525&format=png&auto=webp&s=1f7f8541bf0736ea4c0c3139d2204074842ceaa8 It took a screenshot of my text editor with itself in it. You can basically feed an ollama vision example like [llm-python-vision-ollama.py](https://github.com/Jay4242/llm-scripts/blob/main/llm-python-vision-ollama.py) and tell the bot to figure it out.

[-]

Calcidiol@reddit

That sounds like a workflow that could / should just be handled by composition. Task scheduler to periodically run the workflow. Step 1 use whatever screenshot utility one wants that can be scripted / configured to take a screenshot. Take the output by file name / pipe / clipboard or whatever and feed it to an agent interface that runs your model with that input plus whatever other prompt / context is needed, then takes the output and does whatever. The individual pieces should already be commonplace as tools used to perform their singular functions. And there are lots of "create your own agent / computer use workflow" UIs / softwares now though even basic scripting should handle a few step process of "always do a,b,c,d" in a given flow no problem.