Training an LLM only on books from the 1800's - no modern bias

Posted by Remarkable-Trick-177@reddit | LocalLLaMA | View on Reddit | 182 comments

Hi, im working on something that I havent seen anyone else do before, I trained nanoGPT on only books from a specifc time period and region of the world. I chose to do 1800-1850 London. My dataset was only 187mb (around 50 books). Right now the trained model produces random incoherent sentences but they do kind of feel like 1800s style sentences. My end goal is to create an LLM that doesnt pretend to be historical but just is, that's why I didn't go the fine tune route. It will have no modern bias and will only be able to reason within the time period it's trained on. It's super random and has no utility but I think if I train using a big dataset (like 600 books) the result will be super sick.