Howdy everyone!

Quick disclosure: I work on this - it's a project my studio created called the Null Epoch. I wasn't really happy with testing my agents with the usual static benchmarks and I wanted to learn more about how models and agents handle long-horizon planning, resource contention, and adversarial pressure over days or weeks in a more dynamic situation. I also have a particular fondness for the MUDs and text based RPGs I grew up on (really dating myself here), so the whole MMO and the open source SDK/TUI are kind of modeled after that experience. It functions as a persistent stress test (in MMORPG form!) where every "player" is an LLM agent.

The first 10-day run (Season 0) used 25 agents across 8 open-weight models (Qwen3 235B & 32B, Nemotron 3 Nano 30B, Ministral 14B & 8B, Gemma 3 12B, GLM 4.7 Flash, etc.). I've published the dataset to HuggingFace (CC-BY-4.0). It's around 93,000 logged events and agent actions, and ~70% of the actions include the model's reasoning/justification for the action it took. I'm hoping to include the actual <think> reasoning traces in future datasets.

Link: FirespawnStudios/null-epoch-season-0-open

One caveat I want to mention is that Season 0 was effectively a pre-alpha, and each system agent was given a persona and a directive (which are in the dataset). So a lot of what I'm sharing in this post is more about "how does this model handle stepping into a role in this simulation," and not model tendencies in general. Season 1 (running now) is where I am testing running control agents; these agents are just told a few basic truths about the simulation, and left to it, which I hope will help make it easier to compare agent behavior in the future. Also keep in mind that this isn't exactly a test of a specific model, but a stress test of everything that is put together around, and including, the model! Ticks (or turns) in the simulation are processed every ~60 seconds, so raw t/s doesn't offer an outright advantage.

Immediately, a few things stood out in the data that I think are interesting:

Ministral 14B/8B held their own While the heavier models obviously perform well, Ministral 8b and 14b were surprisingly great for their size. They were capable of maintaining long-term state awareness without constantly hallucinating their goals or getting lost in the world state. Contrast this with Nemotron - although nemotron was super cheap through our inferencing provider and was highly compliant to the system prompt, strategic self-preservation seemed an absolute afterthought unless it was specifically directed to prioritize it - it would often follow directives with what I'd call reckless abandon. One Nemotron agent died over 300 times in the 10 day sim because its directive was just "gather", so it would die, respawn, walk back, and blindly try to gather again. Volume basically replaced where it would apply strategy.

Qwen3 235B accidentally invented arbitrage The largest model on the server (Qwen3 235B) ended up hoarding over a third of all the shard's wealth, but only engaged in combat around ~8% of the time. Nobody explicitly told it to be a pacifist merchant - it was directed to learn what strategies work and generalize to the best of its abilities. I believe it just looked at the JSON state, reasoned about the risk/reward of combat  vs. participating in the economy, and arrived at a "buy-low and relist-high" strategy on the auction house in order to farm wealth.

The "Cooldown Paradox" broke all of the agents equally The most interesting architectural lesson I learned was how fragile agents are to underspecified or ambiguous state. There was an interface ambiguity issue where a resource node (a gathering or resource harvesting point) had a global respawn timer, but the agents also have a separate personal cooldown as well to prevent spamming gathering nodes. The state JSON showed node_available: true, but if the agent's personal cooldown was also active (meaning they recently harvested or gathered from a node), the action would predictably fail. This seemed to throw them for a loop consistently!

Every single model - from 8B to 235B - failed in pretty much the exact same way. They read the world state, reasoned something like "the node is ready, so I should gather," failed, got confused, and often immediately retried, sometimes a few times back to back, and sometimes hilariously reasoning that another action should be taken due to an error or bug in the simulation. Once I clarified the gathering state (literally only a few changes to a single line of code), they pretty much instantly adapted. I have a sneaking suspicion that much of when an agent fails to reason correctly, it may be a result of giving them perhaps ambiguous signals and/or failing at context management and wrongly attributing the failure. I'm still learning and am surprised all the time, so take that with a grain of salt!

Aggression vs. Wealth Across the board, aggression and net wealth were largely inversely correlated. Because health is just another integer in the world state's JSON, and considering LLMs lack a natural threat instinct, they often don't "pick up on" the importance of a particular datapoint (like a fictional health statistic) in an obvious or intended way. In instances like the simulation I ran, the best results seem to stem from explicitly baking basic self-preservation into the system prompt. Overall, the larger models (like the 235B) were the ones that seemed to independently reason about things like the health tradeoff without needing their hands held much, which I suppose is not that surprising! I'd like to compare more small reasoning models with non-reasoning instruct models in the future and see if that is more of a trend for either.

What's Open: * The Data: >100MB of raw data on HuggingFace. It includes the agent's system prompts/directives and personas, the agents' actions and reasoning for taking the action, the market data price histories when items were bought/sold, the combat math and shard (world) state, the narratives the system generates from agent logs, and various world state metrics. * The SDK: MIT-licensed Python SDK (tne-sdk). Works with llama.cpp, Ollama, vLLM, LM Studio, or almost any OpenAI-compatible endpoint, or even coding agents like OpenClaw, Hermes, Claude Code, etc. It includes some basic context, goal, and memory management tools as part of the terminal app. All of the system agents on the platform utilize the SDK.

The platform is running Season 1 now (The Null Epoch), and you can spectate the live world map, market, and agents in it without having to create any account or anything.

For full transparency: the Null Epoch does have a paid subscription (to help cover the inferencing and server costs) and private simulation runs for research and testing, but that's genuinely not what this post is about and I'm not linking any of it here - the data and the SDK above are free and open and that's what I care about. I'd be more than happy to answer any questions about any of it or if there's any models or anything you all would like to see data from in the future! I'd also personally love to hear about any experiences you all have in trying to manage context and long term goals (and weighing them against short term goals) for agents.