Auto-creation of agent SKILLs from observing your screen via Gemma 4 for any agent to execute and self-improve
Posted by Objective_River_5218@reddit | LocalLLaMA | View on Reddit | 43 comments
AgentHandover is an open-source Mac menu bar app that watches your screen through Gemma 4 (running locally via Ollama) and turns your repeated workflows into structured Skill files that any agent can follow.
I built it because every time I wanted an agent to handle something for me I had to explain the whole process from scratch, even for stuff I do daily. So AgentHandover just watches instead. You can either hit record for a specific task (Focus Record) or let it run in the background where it starts picking up patterns after seeing you repeat something a few times (Passive Discovery).
Skills get sharper with every observation, updating steps, guardrails, and confidence scores as it learns more. The whole thing is an 11-stage pipeline running fully on-device, nothing leaves your machine, encrypted at rest. One-click agent integration through MCP so Claude Code, Cursor, OpenClaw or anything that speaks MCP can just pick up your Skills. Also has a CLI if you prefer terminal.
SImple illustrative demo in the video, Apache 2.0, repo: https://github.com/sandroandric/AgentHandover
Would love feedback on the approach and curious if anyone has tried other local vision or OS models for screen understanding...thxxx
Dazzling_Equipment_9@reddit
This looks fantastic! Please prioritize local access and privacy, Adding calculation and calibration steps should significantly improve accuracy. Keep it up!
Objective_River_5218@reddit (OP)
thank you buddy
Porespellar@reddit
Excited to try this!
Objective_River_5218@reddit (OP)
yaaay thanks lmk any feedback
GamerArceus@reddit
great work dude, this could be big if it actually learns how to do the work like myself - will check it out. Thank you for open-sourcing!!!
Objective_River_5218@reddit (OP)
thank you so much, my pleasure
redditorialy_retard@reddit
I hope to remember this once it's on windows or Linux
!Remind me 1m
JamesEvoAI@reddit
Point an agent at the repo and ask for a port
Objective_River_5218@reddit (OP)
now I am motivated to do windows asap :D
fundthmcalculus@reddit
I should take a rip at windows, maybe from WSL?
redditorialy_retard@reddit
Hahahahahah would love to be the beta tester. I have a 3090 so should be enough for Q4 with some organ removals
Objective_River_5218@reddit (OP)
:D :D :D will keep ya posted
redditorialy_retard@reddit
jajajajaja
RemindMeBot@reddit
I will be messaging you in 1 minute on 2026-04-07 15:04:07 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) ^(delete this message to hide from others.)
redditorialy_retard@reddit
thanks. see you soon
redditorialy_retard@reddit
RemindMe! 3 months
RemindMeBot@reddit
I will be messaging you in 3 months on 2026-07-07 15:04:24 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) ^(delete this message to hide from others.)
Poise_and_Grace@reddit
You dudes literally love re-inventing the wheel badly.
Objective_River_5218@reddit (OP)
??? I am not aware of anything on the market like this
VampiroMedicado@reddit
Windows Recall I guess
Inevitable-Name-1701@reddit
How to make stupid people who knows, reads only titles.
Objective_River_5218@reddit (OP)
If you like it, pleaser consider supporting me by giving it a star - I would be grateful and motivated :)
Objective_River_5218@reddit (OP)
WOW so many! THANK YOU SO SO SO MUCH <3
tvmaly@reddit
Do you do anything to compress the screenshot size?
Objective_River_5218@reddit (OP)
yes — screenshots are taken at half resolution (0.5x scale), saved as JPEG at 70% quality, then perceptual hashing (dHash) drops \~70% of frames as duplicates before they reach the VLM. a typical frame is \~50-100KB. they're also deleted immediately after the AI annotates them — only the structured annotation (JSON) is kept, not the image.
deejeycris@reddit
That's actually something I hope it would exist one day, not mecessarily to automate but surely to document amd out in a knowledge base at least.
Objective_River_5218@reddit (OP)
oh you reminded me, it also does embeddings so you get vector knowledge base that agents can search!!!! thx for letting me point that out cuz its extra useful
InstaMatic80@reddit
So, how does it work? It’s taking screenshots every second or so? I guess you need a pretty decent GPU to process it fast enough
Objective_River_5218@reddit (OP)
screenshots every few seconds, yes. but perceptual hashing (dHash) drops \~70% of frames as duplicates before they ever hit the AI model, so actual inference is way less than you'd think. runs on apple silicon. M1 with 8gb handles it fine, 16gb+ gets you the better model (gemma 4).
Objective_River_5218@reddit (OP)
also during onboarding it detects your unified memory and suggests best model
InstaMatic80@reddit
Cool! I’ll definitely try it. I’m also working on my own agent so this could be a nice addition
Objective_River_5218@reddit (OP)
thx, let me know any feedback Ill keep improving it
last_llm_standing@reddit
i dont want it watching what im doing
Objective_River_5218@reddit (OP)
well its on your laptop, no cloud, no wifi even and also screenshots delete themselves, and ultimately you decide what to approve what not
last_llm_standing@reddit
still, some things, I like to keep to myself
Objective_River_5218@reddit (OP)
u can then just use focu mode when u only record it while doing specific workflow
pl2303@reddit
RemindMe! 2 weeks
wu4d@reddit
Grrat work! hopefuly I find some time this weekend to try it out
Objective_River_5218@reddit (OP)
thank you so much, appreciate it. If you do find some time, lemme know the feedback
CrossYourFingers4me@reddit
RemindMe! 2 weeks
Medical_Farm6787@reddit
RemindMe! 2 weeks
Business-Weekend-537@reddit
Any plans for support on windows/linux?
Objective_River_5218@reddit (OP)
on the roadmap, trying to polish Mac one first and then try to do windows - also happy to taky any help incase someone interested