What is the strongest mechanistic argument that next token prediction plus reinforcement learning can produce open-ended general intelligence? [D]

Certainly not the best argument, but an intuitive one that I find at least somewhat compelling: When doing next-token prediction for prompts that ask reasoning questions, during training the model can minimize its training loss by 1) overfitting and memorizing the answer to every data point in the training set, 2) learn a set of heuristics that functionally looks like reasoning on the training set but is in reality a bunch of shortcuts that won't generalize OOD, or 3) become a model that is capable of reasoning so that it can fill in the likely next token without having to memorize the entire training set. Regularization / not being overparameterized w.r.t. the training data can make 1) less likely. Frontier models train on far more tokens than they have parameters, mostly seeing each example once, so wholesale memorization isn’t really an option. Grokking kind of shows something like this; e.g. models memorize modular arithmetic first, then shift to an interpretable generalizing algorithm under weight decay.

On the other hand, learning some heuristics that allow the model to do well on the training data without actually being "reasoning", where these heuristics might not generalize to OOD data, is also possible. It comes down to how "easy" learning reasoning is vs how easy learning a set of satisfactory heuristics is. I don't really know the answer to that.

But if the transformer architecture is capable of modeling reasoning, fitting the model with next token prediction wouldn't necessarily be a terrible approach at least. 

This is definitely my favorite one I've seen so far! I think this is a great argument for being able to capture statistical structure that is actually present and available to the machine via training data or RL.

It still doesn't *quite* explain how that process produces open-ended general intelligence. I think this is a great mechanistic explanation for a powerful tool, though.

I can try & explain what I think the answer is by starting with a question.

What is the strongest mechanistic argument that next token prediction plus reinforcement learning can produce an AI system that is capable of adding 12 digit numbers together, carrying on a conversation, writing arbitrary computer code that works very well, and explaining why a joke is funny?

These are all things that current AI systems can do very well, but fundamentally we don't have a mechanistic explanation for how they do so well. We can rule out some approaches (we know they can't just be regurgitating inputs because the model physically can't have seen a substantial fraction of all 12 digit additions, we know they can't just be crystallizing some simple program like a calculator because that can't write arbitrary computer code or explain why a joke is funny).

There is a hypothesis borne out by some data, which is that (as long as you avoid overparameterisation) to predict the next token successfully the model needs to learn to simulate the environment that produced the tokens in the first place, which in the case of human-authored tokens means simulating the human mind, which we know by definition is a general intelligence. We can rule out that it's simulating any given real human mind perfectly (let alone all of them) because the models aren't large enough to contain a physical simulation of the human brain, but we have no idea how much of that physical simulation is necessary for the intelligence "algorithms" we actually use to be simulated, and thus we don't know how much of our intelligence "algorithm" is successfully simulated by the model in its attempt to predict the next token. Under this hypothesis, you would think that generally the model can't get any more intelligent than the sum of the human beings who must be simulated to produce the tokens in the training data, and it can be very hard to elicit behaviour that's beyond what any individual token-producing mind could do (other than switching simulated minds at will, as with the models incredible proficiency with languages). To make the model usable as a product, RLHF is done to make the AI system preferentially model one particular simulated mind, an "AI assistant" with stipulated qualities that vary between the labs but which generally include the ability to access all knowledge held by the model as a whole (just because they try to achieve this doesn't mean it works perfectly, elicitation can still be difficult).

The most effective way we've found so far to go beyond the limits of any individual human mind simulated through predicting the training data is RLVR. You take the weird egregore of simulated human & machine intelligence you got from pretraining on a bunch of data, and you make it the agent in a standard RL setup, rather than the agent being initialised with random weights or some other method. You have to be careful & judicious with how you do this, because it's possible to just lose a lot of the gains you got from pretraining if you RL too hard on a single environment (catastrophic forgetting). As an example of how to avoid this, you can have a lot (thousands or tens of thousands) of different task environments + your original pretraining goal of predicting the next token, weight success on each, and treat the whole suite as one big target for hill-climbing, so the model/agent is dissuaded from forgetting things it's already learned or making narrow improvements. The specifics of how to do this are secret sauce that are mostly only known by lab employees (although interlab movement is so ubiquitous that I doubt they have many secrets from each other).

Does that method work to go beyond human performance? Pretty unambiguously the answer is yes, the math proofs released the other day show that, you can also try various things like chess with the newest models, or if you have access to one of the cyberdefender programs finding vulnerabilities. Will it work to go beyond human performance in any domain, without a requirement to make specific RL environments for that domain? It's an active debate that's hard to settle as to whether cross domain generalisation occurs in this way, but that may not end up mattering. The reason it may not end up mattering is that creating RL environments is itself a verifiable task you can hillclimb on, as is the general process of training AI models. If we can make the models superhuman at training better & more general models like we made them superhuman at math proofs, the theory is that we can just use those better models to keep creating more higher quality RL environments, make better architectural & engineering decisions in training, with no obvious blocker that prevents any specific task from being hill-climbed by smarter and smarter model training AIs. An assumption of this approach leading to fully general intelligence is that tasks which are truly unverifiable (rather than just difficult to construct a good RL environment for) are not required for achieving any specific real-world goal. As an example of what I mean by unverifiable, it is probably the case that it's impossible to construct an RL environment that could train a model to feel the colour red being a particular sensation identical to the sensation a given human feels when perceiving the colour red, but as long as the model can accurately perceive the colour red & write sentences as if it experiences the sensation, the actual qualia may be unnecessary to completing tasks in the real world, more a matter of philosophy than anything that could affect outcomes.

So the closest we have to a mechanistic explanation is something like "Pretraining an LLM makes a coarse simulation of many human intelligences available as an AI system, and by taking that system as a base you can use RL to improve its performance beyond human level at any task you can make an environment for. Because you can make an environment for training AI models and making RL environments, you can make an AI system that is superhuman at training AI models and making RL environments, leading to recursive self-improvement at the task of training AI models & increased generality through an expanding number of higher quality RL environments". You may notice it's not really mechanistic, it's just a description of what has empirically worked so far and what the labs think will work next. That is because we don't really have a mechanistic understanding of why these practices lead to intelligence. The closest you'll find in the realm of theory is the quite removed older stuff about prediction being compression and compression being intelligence (like Solomonoff or Hutter), but I get the sense you are looking for a more direct mechanical understanding that we just don't have because we don't understand these systems that well.

Even though you couldn’t quite finish it off, this was still an excellent explanation. Thank you very much!

It's the best working recipe we have at the moment.

That seems to be the most popular answer I hear. I'm really looking for something rooted in causal mechanisms and first principles, though.

I'm really looking for something rooted in causal mechanisms and first principles, though.

The real answer is that we do not have a strong enough understanding of the causal mechanisms or first principles to make such statements. You can make plausible sounding speculation about what might be the case, but I'd argue those examples are not useful because they're not backed up with any kind of rigour.

The only area where we actually have the data and have enough information on how models behave is in their macro level behaviour and scaling based off of that. We think that AGI is possible because we've made machines that have been able to exhibit significant intelligence and have shown significant improvements in that domain with further refinement. That is the concrete factual basis for which we can rely on and base further analysis.

I think the question you pose is an example of a fairly common cognitive bias in ML, which is a tendency to assume that any argument grounded in principles or sounds vaguely mathsy is somehow more valid that practical ones. But the reality is that a lot of that stuff is pure puffery, we don't have a good theoretical or first principle understanding of even basic deep neural networks, let alone anything made in the last two decades. ML is an applied science first a foremost, and good arguments should be rooted in demonstrable applied results and not speculative maths terms.

There is none because "general intelligence" is ill-defined.