I made a tiny world model game that runs locally on iPad
Posted by howthefrondsfold@reddit | LocalLLaMA | View on Reddit | 20 comments
It's a bit gloopy at the moment but have been messing around with training my own local world models that run on iPad. Last weekend I made this driving game that tries to interpret any photo into controllable gameplay. I also added the ability to draw directly into the game and see how the world model interprets it. It's pretty fun for a bit messing around with the goopiness of the world model but am hoping to create a full gameloop with this prototype at some point.
lewd_peaches@reddit
That's awesome! What framework did you use for the iPad deployment? I've been meaning to try something similar on my old Air.
Queasy_Asparagus69@reddit
Cool - no one gives a fuck
z_latent@reddit
That's surprisingly fast, and pretty coherent all things considered. Seems fun to play around with!
Have you seen the models by Ollin Bohan? His models are fast enough to run on the browser, but yours seems more well-structured.
howthefrondsfold@reddit (OP)
Thanks! Yeah, I'm definitely familiar with Ollin Bohan's world models, I think they're some of the funnest to play around with at the moment.
z_latent@reddit
It was pretty magical trying them for the first time. Looking forward to yours next!
howthefrondsfold@reddit (OP)
Yeah I completely agree! If you wanna play it sometime just let me know, I'm looking to get it into peoples hands next week hopefully.
z_latent@reddit
Yeah I'm on your Discord, waiting for the M1 compatible release!
Queasy-Contract9753@reddit
I'm not nearly smart enough to have a meaningful comment. This looks like sorcery to me. You just take a picture and you can play a little game
Just like that.
c64z86@reddit
I always thought world models were huge things that needed a very beefy GPU to even start up with and you've got it all running locally on an iPad! That is pretty freaking fantastic right there!! I can't wait to see where you take this next.
howthefrondsfold@reddit (OP)
I appreciate the kind words! The advancements of world models in the past few years have gotten pretty insane. This was mainly an experiment to see what the lowest end hardware was where I could getting it running and be a playable experience. I didn't honestly expect it to work on iPad either haha!
c64z86@reddit
Yeah it's really crazy! I legit did not expect a full freaking world model to be using less resources than a video/image model to be fully playable on a mobile device, but here we are. Yeah it may be gloopy as it is now but I would play with this all day long. Please keep us updated on where you go with it next! :)
Hefty_Development813@reddit
I wonder what you could do with a real gpu with this same strategy
Waarheid@reddit
This is exquisite and I would happy spend hours playing it in its current gloopy state. Nice work. World models always seem crazy to me.
howthefrondsfold@reddit (OP)
Thanks! The current gloopy state is actually pretty fun hahaha! Probably need to get it into the hands of players though to see if it makes any sense at all!
TyrKiyote@reddit
it just adapts the photo into a prebuilt game engine? I don't understand what the photo is for.
howthefrondsfold@reddit (OP)
yeah kinda, it was more an expermient to see if it could happen. It takes the photo and tries it's best to interpret it based on the game it's trained on. I think a photo of an emtpy ginger shot may not have been the best example hahaha, It's pretty gloopy!
TyrKiyote@reddit
neat thing! a lot of directions this could be built up into. it's very cool, but always want for more documentation to understand it.
howthefrondsfold@reddit (OP)
Thanks! Yeah, I read through lots of popular world model papers to help get here but wanna start sharing my own findings. I was planning to start sharing my process on Discord over time: https://discord.gg/HqU3sz7c
DMmeurHappiestMemory@reddit
That's so dope just out of curiosity because I'm not too familiar with this type of work what type of data are you feeding in to build out the model?
howthefrondsfold@reddit (OP)
Thanks, it's kinda wacky haha! Right now I'm feeding synthetic data from different sources into the model. I'm still experimenting with different techniques but was gonna start sharing my findings more on a Discord that I set up here https://discord.gg/HqU3sz7c