Standard reinforcement learning is kinda dumb.
We humans reason about the mistakes we have done, and update in a more directed rational fashion.
Basically using understanding of the state space not just to solve problems but to also increase the speed of our learning as well.
Resolves YES iff before 2027 it will >50% likely that there exists such a model and has state of the art performance (in case of internal models). And for it to qualify it must explicitly have extra reasoning space to figure out how to update it's weights. Just like humans reason about what mistakes they made.
Karpathy mentioning something in this direction very recently:
Update 2025-05-11 (PST) (AI summary of creator comment): The creator has further clarified the requirements for 'extra reasoning space' and 'human-like reasoning about mistakes' as outlined in the market description:
The 'rational reinforcement learning' must involve a process distinct from, and going beyond, learning achieved through mere token prediction.
The qualifying model must engage in a process of reasoning about why an event or error occurred. This reasoning is used to update its internal models or understanding of the relevant context (analogous to a human analyzing another individual's actions to update their mental model of that person).
Standard learning mechanisms such as backpropagation or simple reinforcement learning updates, by themselves, are not considered sufficient to constitute the required 'extra reasoning space'. The model must possess a more deliberative process for understanding and integrating lessons from its experiences or mistakes, rather than solely undergoing a standard algorithmic update.
Update 2025-05-11 (PST) (AI summary of creator comment): The creator has further specified the process for determining if it is '>50% likely that such a model exists' by 2027, a key condition for market resolution:
This evaluation will be made by consulting AI models.
The AI models will be provided with the context of this market for their assessment.
The creator intends to use all state-of-the-art (SOTA) AI models available at the time of resolution (specific models to be determined in the future).
The determination will be based on an average across at least 10 answers obtained from these SOTA models.
Update 2025-05-15 (PST) (AI summary of creator comment): Regarding the AI models that will be consulted for determining the market's resolution:
They will be given the ability to search the internet.
This search capability will include all sources mentioned in this market.
1,000
3.001. So if I estimate there is >50% probability that it exists I resolve YES.
2. Well if you want to learn programming language, you look at compile errors for instance and understand the causality behind the issue and then you're able to avoid this in different context simply because you read the error message. AIs don't read the error message and even if they did their learning wouldn't be a function of it. (as in the wouldn't put attention towards the error message and then look internally and try to update their internal models of how the language works)
@patrik i like the idea of giving the resolution to an LLM, I suggest you choose a specific LLM, tell us the specific prompt you'll choose, and maybe prompt the LLM 5 times for the resolution and resolve against the average probability mentioned (ensure you use an anonymous conversation that doesn't make use of your previous conversations)
@patrik (specifically regarding your answer to num 2, I see no reason why our learning should be different than LLMs: in their training material they probably encountered error messages that updated their weights, and when they see error messages in context they can also learn similarly to how we humans do)
@SimoneRomeo Yes don't worry, I'll prompt it in the most unbiased way possible.
Well but that learning is just token prediction, which is maybe what happens when you just read the documentation or something as human. But this goes deeper, suppose that bob makes a bad action towards alice, then alice will spend time thinking why bob made that action. Once she figures it out she updates her mental model of bob. LLMs don't have this process of reasoning in order to update their mental models to reflect reality. Right now it's just backpropagation after a given token prediction (or reinfoecemenent but that again is just a simple update, no space to reason why it happened).
@patrik could you write the prompt and your chosen LLM in the description? I don't doubt that you'll be unbiased but I'm assuming that the answer will depend a lot on how the actual question is phrased and it's an important matter to bet in this market
@SimoneRomeo I'll give the AI context of this market and by 2027 I think AIs will be much better at evaluating such things that different prompts shouldn't matter a lot.
Also I'll ask all the state of the art models and average it out across at least 10 answers. Can't tell you future LLM names now tho.
@ProjectVictory you're free to not bet. Just reminding you tho that this question resolves at the end of 2026
@patrik I know, and that's fairly soon. I personally wouldn't bet on LLMs becoming perfectly reliable in under two years.
@ProjectVictory Yep and no one is saying that. But they are sufficiently reliable that when averaged and given ability to search internet including all sources mentioned in this market they are capable of deciding whether this market should resolve YES or NO to a sufficient degree that it makes sense for people to just bet on what they believe will happen and ignore the resolution criteria. Which is the entire point of this. Since we need unbiased probability estimate.