Pre-1900 LLM Relativity Test
Posted by Primary-Track8298@reddit | LocalLLaMA | View on Reddit | 19 comments
Wanted to share one of my personal projects, since similar work has been shared here.
TLDR is that I trained an LLM from scratch on pre-1900 text to see if it could come up with quantum mechanics and relativity. The model was too small to do meaningful reasoning, but it has glimpses of intuition.
When given observations from past landmark experiments, the model can declare that “light is made up of definite quantities of energy” and even suggest that gravity and acceleration are locally equivalent.
I’m releasing the dataset + models and leave this as an open problem.
You can play with one of the early instruction tuned models here (not physics post trained): gpt1900.com
Blog post: https://michaelhla.com/blog/machina-mirabilis.html
GitHub: https://github.com/michaelhla/gpt1900
Hefty_Acanthaceae348@reddit
This is as meaningful as astrology. In hindsight, it's very easy to reinterpret stuff so that it conforms to our current knowledge.
brownman19@reddit
What?? This would be a reproducibility from generalization not reinterpretation lol.
If LLMs arrive at same theses as foundational scientists, that’s a big deal. That means that the LLM was able to reason through the casual chain up to the point of innovation, as long as we trace and decode that.
Hefty_Acanthaceae348@reddit
lol. A llm saying once "light is made up of definite quantities of energy" and reinventing relativity are worlds apart.
brownman19@reddit
Of course they are worlds apart. The point is that the model is generalizing to new patterns that align to observed reality. The trace matters.
Exactly what human observers do. No one is expecting a small model to invent relativity.
For someone who likes to talk big, you really do interpret awfully.
Hefty_Acanthaceae348@reddit
lol. And said patterns are just some vague bullshit that can easily be interpreted however.
Retard.
brownman19@reddit
You are just making up narratives in your own weird small mind. No one said anything was revolutionary. No one said anything about this model reinventing relativity. No one is saying the patterns can be interpreted however because the point is the trace. You cannot interpret things without tracing them. That's the point of interpretability.
Do you understand how *language* works my guy? Semantics is meaning. It's interpretability. It lives in the edges of the graph not the nodes. The reason WHY we even look at REASONING TRACES is to understand the causal chain so we can say that the interpretation was correct.
You are literally proving out that you do not understand what reasoning is. Something tells me you have never actually done any real science lol.
sword-in-stone@reddit
yes, potentially interesting still, not revolutionary at all, still a very good step it's not a binary world
Primary-Track8298@reddit (OP)
This was one of my biggest concerns when starting this project. In the blog post, I discuss methods to avoid this. This project is meant to serve as initial signs of life, leaving it as an open problem.
sword-in-stone@reddit
don't be dumb, OP is trying to reproduce what demis hassabis suggested as a good test of llms innovative potential if done rigourously, without leakage, it's a very strong experiment
GamerFromGamerTown@reddit
I think your model would benefit a lot from additional training data (due to the low ratio of parameters to training data), like the trove newspaper repository, project Gutenburg (if you haven't already), or different languages like the German Deutsches Textarchiv; I know it's easy for me to say that though, when you're the one compiling and training on it haha. Fascinating project though! It might be strong to say it's like you're talking to someone in the past, but you can definitely get a window into the past with this.
SashaUsesReddit@reddit
I love this project.. can I assist by giving you a bunch of hardware and datasets?
DM me if so
nomorebuttsplz@reddit
would you consider uploading the model to hf for easy download?
Primary-Track8298@reddit (OP)
https://huggingface.co/collections/mhla/gpt-1900
nomorebuttsplz@reddit
Cool. Sorry I'm such a basic bitch but any GGUFs out there?
nomorebuttsplz@reddit
not the hero we deserve, but the one we need
KickLassChewGum@reddit
This is really cool. So even a relatively small model fed primarily with scientific corpi (which I'm assuming will be the main source for pre-1900 datasets along with, like, theology lol) can already interpret experimental results quite well. Though the far more interesting test would be: if the model grows a little and is fine-tuned to reason, can it come up with the experiments in the first place? That's where the observations happen, yes, but the far more important part of the scientific process is trying to figure out what to even observe in the first place.
Primary-Track8298@reddit (OP)
That would be the next best step!
sword-in-stone@reddit
interesting AF you can use this model itself to generate more data and do self supervised fine-tuning btw, it's shown to weirdly improve llms
cant use other llms to generate data, cause leakage again
if done rigourously, without leakage, it's a very strong experiment OP
would be up to contribute to this, lmk
Primary-Track8298@reddit (OP)
Yeah I tried this but very hard to get high quality instruction tuning datasets from pretraining corpus alone. Otherwise, base model did not seem strong enough to do self distillation