What is the strongest mechanistic argument that next token prediction plus reinforcement learning can produce open-ended general intelligence? [D]

Certainly not the best argument, but an intuitive one that I find at least somewhat compelling: When doing next-token prediction for prompts that ask reasoning questions, during training the model can minimize its training loss by 1) overfitting and memorizing the answer to every data point in the training set, 2) learn a set of heuristics that functionally looks like reasoning on the training set but is in reality a bunch of shortcuts that won't generalize OOD, or 3) become a model that is capable of reasoning so that it can fill in the likely next token without having to memorize the entire training set. Regularization / not being overparameterized w.r.t. the training data can make 1) less likely. Frontier models train on far more tokens than they have parameters, mostly seeing each example once, so wholesale memorization isn’t really an option. Grokking kind of shows something like this; e.g. models memorize modular arithmetic first, then shift to an interpretable generalizing algorithm under weight decay.

On the other hand, learning some heuristics that allow the model to do well on the training data without actually being "reasoning", where these heuristics might not generalize to OOD data, is also possible. It comes down to how "easy" learning reasoning is vs how easy learning a set of satisfactory heuristics is. I don't really know the answer to that.

But if the transformer architecture is capable of modeling reasoning, fitting the model with next token prediction wouldn't necessarily be a terrible approach at least. 

This is definitely my favorite one I've seen so far! I think this is a great argument for being able to capture statistical structure that is actually present and available to the machine via training data or RL.

It still doesn't *quite* explain how that process produces open-ended general intelligence. I think this is a great mechanistic explanation for a powerful tool, though.

You need to clarify what you mean by “open-ended general intelligence”. Not nitpicking, there is no nontrivial account of this thing - in my view no robust, self contained account whatsoever - and that’s pretty much the whole space where substantive discussion would take place, so you can’t really take for granted here whatever loose notion you might have in your head. Especially since you’re apparently demanding highly specific mechanistic accounts of others. Without question begging, or relying on an underspecified reference system, what exactly are the properties that systems you count as “open ended general intelligences” satisfy that other (putative) intelligences do not satisfy?

Regularization doesn't necessarily separate your second and third cases, since a shortcut can be cheaper to represent than the underlying rule. If both predict the training examples equally well, the loss provides no reason to prefer the one that survives a distribution shift. The missing piece is what makes the training data expose those shortcuts often enough to rule them out.

I can try & explain what I think the answer is by starting with a question.

What is the strongest mechanistic argument that next token prediction plus reinforcement learning can produce an AI system that is capable of adding 12 digit numbers together, carrying on a conversation, writing arbitrary computer code that works very well, and explaining why a joke is funny?

These are all things that current AI systems can do very well, but fundamentally we don't have a mechanistic explanation for how they do so well. We can rule out some approaches (we know they can't just be regurgitating inputs because the model physically can't have seen a substantial fraction of all 12 digit additions, we know they can't just be crystallizing some simple program like a calculator because that can't write arbitrary computer code or explain why a joke is funny).

There is a hypothesis borne out by some data, which is that (as long as you avoid overparameterisation) to predict the next token successfully the model needs to learn to simulate the environment that produced the tokens in the first place, which in the case of human-authored tokens means simulating the human mind, which we know by definition is a general intelligence. We can rule out that it's simulating any given real human mind perfectly (let alone all of them) because the models aren't large enough to contain a physical simulation of the human brain, but we have no idea how much of that physical simulation is necessary for the intelligence "algorithms" we actually use to be simulated, and thus we don't know how much of our intelligence "algorithm" is successfully simulated by the model in its attempt to predict the next token. Under this hypothesis, you would think that generally the model can't get any more intelligent than the sum of the human beings who must be simulated to produce the tokens in the training data, and it can be very hard to elicit behaviour that's beyond what any individual token-producing mind could do (other than switching simulated minds at will, as with the models incredible proficiency with languages). To make the model usable as a product, RLHF is done to make the AI system preferentially model one particular simulated mind, an "AI assistant" with stipulated qualities that vary between the labs but which generally include the ability to access all knowledge held by the model as a whole (just because they try to achieve this doesn't mean it works perfectly, elicitation can still be difficult).

The most effective way we've found so far to go beyond the limits of any individual human mind simulated through predicting the training data is RLVR. You take the weird egregore of simulated human & machine intelligence you got from pretraining on a bunch of data, and you make it the agent in a standard RL setup, rather than the agent being initialised with random weights or some other method. You have to be careful & judicious with how you do this, because it's possible to just lose a lot of the gains you got from pretraining if you RL too hard on a single environment (catastrophic forgetting). As an example of how to avoid this, you can have a lot (thousands or tens of thousands) of different task environments + your original pretraining goal of predicting the next token, weight success on each, and treat the whole suite as one big target for hill-climbing, so the model/agent is dissuaded from forgetting things it's already learned or making narrow improvements. The specifics of how to do this are secret sauce that are mostly only known by lab employees (although interlab movement is so ubiquitous that I doubt they have many secrets from each other).

Does that method work to go beyond human performance? Pretty unambiguously the answer is yes, the math proofs released the other day show that, you can also try various things like chess with the newest models, or if you have access to one of the cyberdefender programs finding vulnerabilities. Will it work to go beyond human performance in any domain, without a requirement to make specific RL environments for that domain? It's an active debate that's hard to settle as to whether cross domain generalisation occurs in this way, but that may not end up mattering. The reason it may not end up mattering is that creating RL environments is itself a verifiable task you can hillclimb on, as is the general process of training AI models. If we can make the models superhuman at training better & more general models like we made them superhuman at math proofs, the theory is that we can just use those better models to keep creating more higher quality RL environments, make better architectural & engineering decisions in training, with no obvious blocker that prevents any specific task from being hill-climbed by smarter and smarter model training AIs. An assumption of this approach leading to fully general intelligence is that tasks which are truly unverifiable (rather than just difficult to construct a good RL environment for) are not required for achieving any specific real-world goal. As an example of what I mean by unverifiable, it is probably the case that it's impossible to construct an RL environment that could train a model to feel the colour red being a particular sensation identical to the sensation a given human feels when perceiving the colour red, but as long as the model can accurately perceive the colour red & write sentences as if it experiences the sensation, the actual qualia may be unnecessary to completing tasks in the real world, more a matter of philosophy than anything that could affect outcomes.

So the closest we have to a mechanistic explanation is something like "Pretraining an LLM makes a coarse simulation of many human intelligences available as an AI system, and by taking that system as a base you can use RL to improve its performance beyond human level at any task you can make an environment for. Because you can make an environment for training AI models and making RL environments, you can make an AI system that is superhuman at training AI models and making RL environments, leading to recursive self-improvement at the task of training AI models & increased generality through an expanding number of higher quality RL environments". You may notice it's not really mechanistic, it's just a description of what has empirically worked so far and what the labs think will work next. That is because we don't really have a mechanistic understanding of why these practices lead to intelligence. The closest you'll find in the realm of theory is the quite removed older stuff about prediction being compression and compression being intelligence (like Solomonoff or Hutter), but I get the sense you are looking for a more direct mechanical understanding that we just don't have because we don't understand these systems that well.

Even though you couldn’t quite finish it off, this was still an excellent explanation. Thank you very much!

Would this argument hold if suppose we did pretraining with a training cutoff of say 5000 BCE(or at the very beginning of language) assuming we had enough training data hypothetically and used RL to get good at say whatever the frontier was in math of that time. Where would this hypothetical model construct future goals and rewards from? The model may have solved a lot of arithmetic and some geometry but how do we move further , it seems like there is no straightforward way for the model to arbitrarily keep creating better goals and rewards without an experience of the physical world .

I think you would run into a couple of issues. One is that there is nothing in this hypothesis which says the program can start arbitrarily small, like the old concept of a "seed AI"; a requirement for the initial LLM is that it is able to simulate a human mind/minds to some unknown degree of fidelity. So you need a certain size of model to be able to run a simulation of a mind, and then you need so much more data than the size of that model that overparameterisation isn't an issue. If you chose a point in time where language had just begun, I suspect there literally would not be enough language in existence to train a model large enough. I don't even know if an LLM the size of Talkie (trained on all the pre-1930 text the authors could find) would be big enough in principle, but I certainly think running experiments with RL posttraining on Talkie to see if we can push Talkie's performance beyond human level is a fantastic idea to explore the post-training realm.

Another issue is that one of the tasks you need to RL the model to do is research taste, which in the real world models are only starting to get good at now. In 5000 BCE we were thousands of years away from the enlightenment and the scientific method, there's a very reasonable argument we weren't actually making any progress at that time in terms of science improving, just expanding like smarter animals. It's not clear to me that there was any activity humans were doing in 5000 BCE that could teach the model how to choose promising avenues of computer science. Humans eventually got there through a historically complicated path that involved many wars, resource management etc and it's not clear that you get that from just simulating 5000 BCE humans and doing RL on berry math.

As for your point about real world data, I absolutely think real world data will be a necessary input. Obviously the pretraining stage has to include a ton of real world data, but designing a good RL environment generally looks like "Observe the kind of intelligence you are lacking at work in the real world & identify the smallest possible tasks that something skilled at that task successfully solves but which current models can't, then construct a simulated environment to test for those tasks with intermediate steps for grading. Once you've tried training with that RL environment, go back and check if the model can do the task now, and if it can't check for sim2real issues or add other aspects of the task as RL goals, then try again." To my mind, this process of creating more and more RL environments of higher and higher quality necessarily involves interacting with the real world because that's how you find out if your RL was successful.

I had no idea about talkie. https://talkie-lm.com/chat

Super interesting. Could talkie viably come up with 20th century inventions if it gets to work on research long enough? Surely this is what the original authors were trying to do right?

It's the best working recipe we have at the moment.