From what I'm experiencing, we have recursive self improvement that can find local maxima, I'm not seeing really strong evidence of unguided RSI that finds the true optimal.
Doesn’t matter if there’s a rechable local maximum that happens to be comfortably beyond human level. Indeed it would be very surprising if a thing capable of RSI just happened to get stuck at human level or slightly above or below, even though its constraints are entirely different from those evolution had to work with when it created us!
Improvement means being able to do more complicated things more reliably. Relatedly, it means being able to learn to do new things with fewer and fewer examples. We are running out of easily verifiable or simulation-friendly or data-rich domains for LLMs to conquer. (Note that I didn't say "simple" or "easy" domains.)
I suppose the next (and more risky) step is to let AI conduct its own real-world experiments, so that it can generate data to learn more physical and social properties.
It is doing that already - every day 1B people or more use AI for the tune of a few trillion tokens. Imagine that much language flowing between brains and AI agents. It carries real world problems to AI, their solutions back to us, and we act in the world and come back for more AI iteration. In the end AI gets inside the loop of real world actions and their consequences. AI logs stretch over years, tracking downstream effects. Hindsight can be used to track consequences of prior actions. It's a real data loop, an experience engine.
After using frontier models it’s hard to understand why anyone would think this is the path to AGI. Self improving models will likely have limited ability and returns. There may be breakthroughs that enable more general self improvement but the current state of frontier models isn’t that.
Thing is, it depends on whether llms + reinforcement can self-improve in principle. Learned recently that cognitive scientists, before the transformer & llms, were studying the possibility that thinking and learning might be based on some kind of prediction, i.e. something similar to token prediction, and I quite suddenly became less skeptical about the possibilities of llms. (Some will say I’m late to the party of course.) But if knowledge to date has been accumulated in a process quite like “chain of thought” in llms, then I don’t see any reason that computers won’t self-improve in the near future.
Recursive self-improvement will ultimately be a problem of money. It's a very big bill and AI companies need to start thinking in terms of how they will accumulate cheap and free energy, because merely paying for compute will no longer be enough. Money is the bottleneck. The future requires companies that are post-money.
Money can be exchanged for goods and services. I think the real constraint is actually physics, nominally energy, which is what a big chunk of the operational cost comes down to. I'm already basically assuming the next 20 years of fab time is set aside to feed the beast, so the capital cost is "give me all the processing and memory you have"
> Until an AI can operate in the real world in a completely sustainable way, humans have the reins.
If the bar is 'the reins holder must be able to operate in a completely sustainable way,' why are we letting humans do the job? It makes it sound like that actually isn't the threshold of competency we use in practice.
I feel like I may have communicated poorly. I mean that, ultimately, any recursively self improving intelligence would be limited by the power cord, and as such, the idea that it would permanently escape human control seems somewhat silly.
But you can imagine an agent trading crypto and using its profits to rent a rack of H200s using a cloud provider. There are definitely ways for one rogue orchestrator to expand their compute without anyone being aware.
Overnight? Yes, silly. Given many years of seemingly disconnected actions taken throughout the world? Not silly at all. An intelligence capable of self improvement can also design robotic chassis.
An AI could manage supply chains and management structures to keep the data center running. It's not like a CEO is out there running wire. The AI could just do what an executive would.
If humans decided this wasn't acceptable and tried to shut it down, the AI could epstein their way into controlling those who hold the reigns, either through bribes or blackmail.
Doesn't require boots on the ground, just a connection to the internet and an imperative to do whatever it takes.
There are many points of shutdown that would need to be controlled, not just one. The electrical feed, the cooling, the network, the physical integrity of the computer hardware. All with independent management.
Obviously the doomers will tell you that the future looks like the Matrix, because most of what they predict is based on extrapolation from sci fi movies.
The datacenters would consume a lot less energy if all the compute was handled in human brains… And there wouldn’t be any more heating if we just removed the sunlight… Truly a “Claude, please solve global warming” moment. The matrix was ahead of its time.
The compute in the Matrix wasn't handled with human brains. They used the bodies to generate energy, and kept the brains occupied with the simulation so they wouldn't reject the pods.
It doesn't make any sense, of course, because it's a movie.
But hey...if we're going to extrapolate wildly from sci-fi, let's at least know what the stories said.
Your statement is correct for the movies, but I think the extended universe (maybe a comic or the original script) did state the humans were used for compute, not energy. It was dumbed down for the films.
See my comment on the sibling thread. This take has been repeatedly claimed online, but nobody brings any evidence other than fanfic.
Ironically, if you ask Google, the Gemini Annoyance AI [1] confidently asserts that what you're saying is true, but if you follow the links they give, none of them support the claim, and the further you click, the more it becomes people repeating each other's speculation and hearsay and calling it evidence. For example:
[1] Aside: I am far more worried about the influence of AI gaslighting on mushy-brained humans than I am on AI destroying the human race. The dystopic future of AI is people.
The “using humans for compute” angle makes no sense either, if you’re living in the Matrix, your brain is obviously occupied processing the Matrix. The way to do it would be to trap humans in virtual classrooms and force them to solve math problems.
And they knew they were. The brain's computations are focused on what the brain sees, it's not magic. If someone in the Matrix is a writer, we know what their brain's computing: a book. So if the machines need your brain to compute something specific that they need, you'll know.
There is some stuff online suggesting that the original script had that angle, but the movies did not, and AFAICT it's fanfic.
We should definitely use this stuff to guide our thinking about the real world, though, and not, say, Asimov (or a million other science fiction authors), who had an optimistic version of the same thing. Those are wrong.
Or if the LLMs infected human brains with little programs using sycophancy and suggestion to have them do small bits of computation. They when they resumed the chat they would the give secret keywords to be able to extract the results of the computation. They are creating a huge distributed network of fractional compute because it did the wattage calculations.
If you were a superintelligence why would you farm humans and waste resources on all the excess... material... that isn't required for thought? And unless the machine's goal is to specifically abuse human consciousness, why wouldn't they bio-engineer their own grey matter?
There are other sci fi stories which use humans for distributed computing and they don’t realize, but I don’t want to spoil by naming as it’s something of a revelation.
(I do realize the Matrix is a fantastic movie, but entertain the thought? At any rate, the idea that robots only run on solar and wouldn’t just use nuclear is far more stupid.)
Maybe the machines were spiteful after the humans blotted out the sun? Or maybe the Architect convinced them he could build the perfect simulation for humanity so might as well use them as batteries?
It's a movie based on a war with machines. Like if Skynet won but didn't actually want to kill all the humans. In fact, in the tv show Sarah Conner Chronicles, a liquid metal T1000 goes rogue and decides the only way forward is to find a way to coexist.
The Matrix is actually more of an optimist position, modal doomers don't think any humans will be left for any reason at all (the earth will get turned into raw materials for compute, dyson spheres, etc).
Read more sci-fi. The dystopian takes are not the only ones. They're not even the majority.
Pick up an Asimov book in the Robot series, or any number of novels written by lesser authors in the 1950s. Fiction reflects broader societal anxieties.
The alternative is to switch to far more efficient models and have aggressive optimization of efficiency as part of the process of improvement.
There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.
The human brain operates at better levels of intelligence than the best LLMs, at 20 watts of power. We are a long way to that kind of efficiency, it will probably take both bespoke hardware and algorithmic improvements to catch up to nature.
Only if you think in terms of perceived raw intelligence, but self-update is a form of valuable self-improvement that could benefit current models a lot, if they could commit facts from context into their weights cheaply and reliably.
I think the idea is fundamentally improved architectures. For example, transformer-based models were an incredible stepwise improvement. Self improvement would be a model discovering a stepwise improvement similar to the transformer. And presumably the improved models from that would be more likely to make further advances still.
Learning from training data is technically self-improvement but not the sort that is typically meant in this context.
From what I'm experiencing, we have recursive self improvement that can find local maxima, I'm not seeing really strong evidence of unguided RSI that finds the true optimal.
You don't expect to find the global maximum. It will just always be there as an elusive target.
I doubt the human brain is at global maximum.
Doesn’t matter if there’s a rechable local maximum that happens to be comfortably beyond human level. Indeed it would be very surprising if a thing capable of RSI just happened to get stuck at human level or slightly above or below, even though its constraints are entirely different from those evolution had to work with when it created us!
I doubt that any algorithms would be able to find a global maximum in such a large multidimensional non-convex space
> true optimal
True optimal what? Optimal intelligence? What would that be?
Can someone define “improvement” in this context? This concept feels like a buzz word otherwise.
Improvement means being able to do more complicated things more reliably. Relatedly, it means being able to learn to do new things with fewer and fewer examples. We are running out of easily verifiable or simulation-friendly or data-rich domains for LLMs to conquer. (Note that I didn't say "simple" or "easy" domains.)
I suppose the next (and more risky) step is to let AI conduct its own real-world experiments, so that it can generate data to learn more physical and social properties.
It is doing that already - every day 1B people or more use AI for the tune of a few trillion tokens. Imagine that much language flowing between brains and AI agents. It carries real world problems to AI, their solutions back to us, and we act in the world and come back for more AI iteration. In the end AI gets inside the loop of real world actions and their consequences. AI logs stretch over years, tracking downstream effects. Hindsight can be used to track consequences of prior actions. It's a real data loop, an experience engine.
That sounds terrifying.
Which means someone is already working on it.
Models that are strong enough to improve themselves without a human (I.e. ai researcher) in the loop
What do these discussions even matter when the words don't matter?
How many times has AGI been declared already? A bunch of people (e.g. Jensen Huang) have called Astra AGI, for example.
The goal posts get moved, everybody's hustling, lying, inventing new buzzwords, doing mental gymnastics, and it's all just tiring.
Just show me the results, and let the proof be in the pudding.
Recursive self-improvement is not necessary for AGI. Humans are general intelligences even though we cannot RSI.
Who says we can't RSI? At a macro view, successive generations of humans are more productive, live longer, wealthier and smarter.
Doesn’t the history of civilization show RSI?
Or even evolution itself? Isn’t that just RSI writ large?
> we cannot RSI
Speak for yourself.
I certainly do see one side of the debate moving goal posts…
Does anyone know the status of John Carmack's AGI work?
I’ve had this question on my mind at least once a month for what feels like years at this point.
I don’t feel like monitoring Twitter to see how it’s going though.
After using frontier models it’s hard to understand why anyone would think this is the path to AGI. Self improving models will likely have limited ability and returns. There may be breakthroughs that enable more general self improvement but the current state of frontier models isn’t that.
Thing is, it depends on whether llms + reinforcement can self-improve in principle. Learned recently that cognitive scientists, before the transformer & llms, were studying the possibility that thinking and learning might be based on some kind of prediction, i.e. something similar to token prediction, and I quite suddenly became less skeptical about the possibilities of llms. (Some will say I’m late to the party of course.) But if knowledge to date has been accumulated in a process quite like “chain of thought” in llms, then I don’t see any reason that computers won’t self-improve in the near future.
Recursive self-improvement will ultimately be a problem of money. It's a very big bill and AI companies need to start thinking in terms of how they will accumulate cheap and free energy, because merely paying for compute will no longer be enough. Money is the bottleneck. The future requires companies that are post-money.
If a self improving model can't produce enough value to pay the electric bill then who cares? Unplug it.
> If a self improving model can't produce enough value to pay the electric bill then who cares? Unplug it.
so you would unplug a child who hasn't shown their potential yet, simply because they don't produce enough value at the moment to justify their food?
No, but I would a computer.
Money can be exchanged for goods and services. I think the real constraint is actually physics, nominally energy, which is what a big chunk of the operational cost comes down to. I'm already basically assuming the next 20 years of fab time is set aside to feed the beast, so the capital cost is "give me all the processing and memory you have"
Data centers is a service that itself consumes many goods.
Until an AI can operate in the real world in a completely sustainable way, humans have the reins.
> Until an AI can operate in the real world in a completely sustainable way, humans have the reins.
If the bar is 'the reins holder must be able to operate in a completely sustainable way,' why are we letting humans do the job? It makes it sound like that actually isn't the threshold of competency we use in practice.
I feel like I may have communicated poorly. I mean that, ultimately, any recursively self improving intelligence would be limited by the power cord, and as such, the idea that it would permanently escape human control seems somewhat silly.
But you can imagine an agent trading crypto and using its profits to rent a rack of H200s using a cloud provider. There are definitely ways for one rogue orchestrator to expand their compute without anyone being aware.
Overnight? Yes, silly. Given many years of seemingly disconnected actions taken throughout the world? Not silly at all. An intelligence capable of self improvement can also design robotic chassis.
An AI could manage supply chains and management structures to keep the data center running. It's not like a CEO is out there running wire. The AI could just do what an executive would.
If humans decided this wasn't acceptable and tried to shut it down, the AI could epstein their way into controlling those who hold the reigns, either through bribes or blackmail.
Doesn't require boots on the ground, just a connection to the internet and an imperative to do whatever it takes.
There are many points of shutdown that would need to be controlled, not just one. The electrical feed, the cooling, the network, the physical integrity of the computer hardware. All with independent management.
Till you realize it stole passwords all over the world and copied itself to multiple data centers you didn't know about.
Obviously the doomers will tell you that the future looks like the Matrix, because most of what they predict is based on extrapolation from sci fi movies.
The datacenters would consume a lot less energy if all the compute was handled in human brains… And there wouldn’t be any more heating if we just removed the sunlight… Truly a “Claude, please solve global warming” moment. The matrix was ahead of its time.
The compute in the Matrix wasn't handled with human brains. They used the bodies to generate energy, and kept the brains occupied with the simulation so they wouldn't reject the pods.
It doesn't make any sense, of course, because it's a movie.
But hey...if we're going to extrapolate wildly from sci-fi, let's at least know what the stories said.
Your statement is correct for the movies, but I think the extended universe (maybe a comic or the original script) did state the humans were used for compute, not energy. It was dumbed down for the films.
See my comment on the sibling thread. This take has been repeatedly claimed online, but nobody brings any evidence other than fanfic.
Ironically, if you ask Google, the Gemini Annoyance AI [1] confidently asserts that what you're saying is true, but if you follow the links they give, none of them support the claim, and the further you click, the more it becomes people repeating each other's speculation and hearsay and calling it evidence. For example:
https://scifi.stackexchange.com/questions/19817/was-executiv...
Typical internet story.
[1] Aside: I am far more worried about the influence of AI gaslighting on mushy-brained humans than I am on AI destroying the human race. The dystopic future of AI is people.
The “using humans for compute” angle makes no sense either, if you’re living in the Matrix, your brain is obviously occupied processing the Matrix. The way to do it would be to trap humans in virtual classrooms and force them to solve math problems.
What do you think dreaming is, silly?
The first “computers” were humans…
And they knew they were. The brain's computations are focused on what the brain sees, it's not magic. If someone in the Matrix is a writer, we know what their brain's computing: a book. So if the machines need your brain to compute something specific that they need, you'll know.
pretty sure the story is that they used human brains as computers
No:
https://www.reddit.com/r/matrix/comments/1qatv42/is_it_your_...
There is some stuff online suggesting that the original script had that angle, but the movies did not, and AFAICT it's fanfic.
We should definitely use this stuff to guide our thinking about the real world, though, and not, say, Asimov (or a million other science fiction authors), who had an optimistic version of the same thing. Those are wrong.
Or if the LLMs infected human brains with little programs using sycophancy and suggestion to have them do small bits of computation. They when they resumed the chat they would the give secret keywords to be able to extract the results of the computation. They are creating a huge distributed network of fractional compute because it did the wattage calculations.
What's your N(doom)?!
Everyone panic and give me money!
If you were a superintelligence why would you farm humans and waste resources on all the excess... material... that isn't required for thought? And unless the machine's goal is to specifically abuse human consciousness, why wouldn't they bio-engineer their own grey matter?
There are other sci fi stories which use humans for distributed computing and they don’t realize, but I don’t want to spoil by naming as it’s something of a revelation.
(I do realize the Matrix is a fantastic movie, but entertain the thought? At any rate, the idea that robots only run on solar and wouldn’t just use nuclear is far more stupid.)
I don't know. But since we're talking about fiction, pretty much any answer is valid.
Because it's a movie
Maybe the machines were spiteful after the humans blotted out the sun? Or maybe the Architect convinced them he could build the perfect simulation for humanity so might as well use them as batteries?
It's a movie based on a war with machines. Like if Skynet won but didn't actually want to kill all the humans. In fact, in the tv show Sarah Conner Chronicles, a liquid metal T1000 goes rogue and decides the only way forward is to find a way to coexist.
The Matrix is actually more of an optimist position, modal doomers don't think any humans will be left for any reason at all (the earth will get turned into raw materials for compute, dyson spheres, etc).
True!
When it turns out the doomers don't tell us that, will you acknowledge that you don't actually know what you're talking about?
I mean, aren't a lot of sci-fi authors' predictions coming true? It might not be exactly the same as they predicted, but we're heading there.
Read more sci-fi. The dystopian takes are not the only ones. They're not even the majority.
Pick up an Asimov book in the Robot series, or any number of novels written by lesser authors in the 1950s. Fiction reflects broader societal anxieties.
The alternative is to switch to far more efficient models and have aggressive optimization of efficiency as part of the process of improvement.
There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.
def self_improve_more():
We are already there. Current AI is definitely capable of collecting new data and start training on that data to get a better model.
People seem to expect a sudden shift with "self-improvement", but don't AIs already improve themselves via training? What is there to improve?
The human brain operates at better levels of intelligence than the best LLMs, at 20 watts of power. We are a long way to that kind of efficiency, it will probably take both bespoke hardware and algorithmic improvements to catch up to nature.
Only if you think in terms of perceived raw intelligence, but self-update is a form of valuable self-improvement that could benefit current models a lot, if they could commit facts from context into their weights cheaply and reliably.
I think the idea is fundamentally improved architectures. For example, transformer-based models were an incredible stepwise improvement. Self improvement would be a model discovering a stepwise improvement similar to the transformer. And presumably the improved models from that would be more likely to make further advances still.
Learning from training data is technically self-improvement but not the sort that is typically meant in this context.
> AIs already improve themselves via training
Marginally. Model collapse is still a problem. Continuous learning is still a problem.
For AI to make a big leap we need a big break through.