The Singularity Gate - new benchmark for AI predicting paradigm-breaking post-cutoff scientific discoveries
Posted by king_malebolgia_@reddit | LocalLLaMA | View on Reddit | 3 comments
I just released a new benchmark called The Singularity Gate. Tests whether frontier AI can predict paradigm-breaking scientific discoveries published after their training cutoff.
Top score: 17.75% (partial credit, Opus 4.7).
Fully-correct outcome rate: 0% across all respondents.
Our claim here is simple. Passing the Singularity Gate is necessary, though not sufficient, for autonomous AI-driven discovery. A model that can predict paradigm-breaking discoveries isn't necessarily Einstein-level. But a model that can't is definitely not.
These are partial-credit scores. No model fully predicts a discovery yet. I'm happy to discuss methodology, related work, or the framing in the comments.
Paper: https://doi.org/10.5281/zenodo.20358378
Site: https://singularitygate.org
ttkciar@reddit
This is off-topic for LocalLLaMA.
ortegaalfredo@reddit
We need a meta-benchmark that measures the ability of a benchmark to get VC funding for a model.
New_Comfortable7240@reddit
This graph is sooo misleading, my statistic teacher would rip this angrily back in uni