They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public.
Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.
The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.
Why? There are numerous technologies we have ignored or abandoned for numerous reasons. Many of the benefits of AI is "requires less humans", but in an age where we question what work people could possibly do in the future, human labor isn't that hard to find.
If the amish can do what they do for whack religious reasons, people could manage it with ai for cultural and social reasons. And I never saw an amish person starve to death. And that is if people don't get pissed enough to start burning stuff down and instead try to be peaceful hippie homesteader types.
But notably the amish don't preclude the rest of us from existing. Some subset of humans could turn their attention away from AI but it would presumably still exist and continue to be developed.
I can't think of any economically beneficial technologies that we've collectively ignored. If you manage to come up with a counterexample then that's an opportunity to make some money for yourself. It's a fundamentally unstable state given our economic system.
* Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.
* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.
* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.
* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.
* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.
* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.
This, and many more reasons.
The labs need to feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again.
> The government can simply gag Sam, Dario, Musk on national security basis
Genuine question - can they really do this? Obviously if I, a not-even-millionaire, get a national security gag order, I'm going to follow it because I assume they'll bury me under the jail otherwise.
But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government trample on their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?
Recall that the legal system is itself provided by "the government". It's a complex system. Whether one part of it is capable of exerting influence over some particular thing comes down to competing interests. For example in the US if the states and the federal legislature are in sufficient agreement about something the constitution ceases to matter - they could literally target a single person with an arbitrary law if they so chose. Our assurance that this won't happen comes down to the fact that getting any of them to agree on anything is an exercise in herding cats.
In some cases, such as space exploration I don't really care who does it but that it happens - sure, would be nice if my favorite power block did it but I will still celebrate it when someone else achives it.
Can still be a powerful motivation, to make sure that next time, it yous your camp that scores the next milestone, like an orbital elevator or fox ears, for example.
Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.
I don't think "plateaued" is the right word, but I do feel like there's been something like a logistic curve compression in the difference between smaller and larger models as the field evolves. For inference at least, the scale of practical difference between a single high-VRAM GPU or SFF UMA box, a whole rack, and a whole data center seems to be falling far short of what we might have imagined just a few years ago. The conversations I've heard have largely turned away from breathless anticipation of the next frontier model and toward attempts at hard-nosed evaluation of which tokens are worth the cost.
I have a pet project I have been working away on for some time that involves building GPU backends for various cards in Zig, lots of complex stuff in it. Lately I mostly use Opus 5, it can pretty reliably plug away at things but it does mess stuff up occasionally. For this codebase, Fable 5.1 was noticeably better at getting things right and doing things in a good reliable way. Of course, I can only use Fable for a bit before I hit the usage cap for the week, so I save it for the tougher things. That said, I absolutely abhor the way recent Anthropic models write prose, especially comments.
I recently tried doing a fairly normal task for this codebase with codex, as I have seen a lot of people talking it up on here. A single task running for ~1-2 hours burned through over half of my usage for the week on the $125/month plan, not on a top model (I don't remember which one specifically I used). It struggled to get the basics done, then got absolutely stuck on a follow up. Handed it over to Claude and it 1-shot it.
I really liked codex in the last few weeks, especially its ability to clean up after Claude's (prose) messes and do reviews.
But in the last few days something seems to have happened that made Codex's models massively stupider (for what I am doing).
Really weirdly, it suddenly refused to even run tests it previously wrote itself (and previously ran), because of some false positive about cybersecurity.
That by itself is not evidence of stupidity. Trying to make a 200+ file PR full of research notes is, and the PR didn't even solve the problem I asked it to.
I'm sure that's part of it, but I run it side by side in my review bot, and Astra medium effort consistently catches more issues than Sol 5.6, using fewer tokens.
For coding it's a little harder to tell, but at least the prose feels a little better.
They've not plataued but they're certainly not as impressive as the hype would have them to be.
The reality is, it doesnt matter if LLMs keep getting more powerful because they still need a human to steer it. Without the human providing inputs to the LLM it just sits there and does nothing.
No idea what you are missing and yes, Opus is quite solid, but Fable is clearly way better for me.
I just did a direct comparison, big change in a quite complex codebase. Same prompt for Opus, same for Fable. Fable clearly won and delivered very good results, while Opus delivered mediocre, so I did not let it finish. I expected both to fail and was prepared to do lots of manual steering, but not necessary with Fable one shotting it, and all this with 35$ of credits for fable. I am still impressed. If I would have had to hire a human, it would have cost me thousands of dollar for the same task - and a way longer time. So maybe the valuations are overblown, but they clearly provide value for me.
By ambitious if you mean we are not like all the linkedin influencers with their “i one shotted an app this morning…” then no, we are not. Nobody is. I have been in software engineering for 18 years and 6 different companies including FANG and 99% of the people, on 99% of the days arnt writing new apps from scratch. Thats simply not how anything works.
And what even are these ambitious companies and people one shotting and building with Fable? AI has been around for almost 3 years now. Tell me one app or software you use which has gotten significantly better and has amazing new useful features landing on a weekly basis? If anything, every single software product I use has gotten worse.
This is just "you're holding it wrong" with a little smooch of condescension. If only we plebeians could comprehend what magnificent works those who have ambitiously integrated agents into the workstream have wrought!
Sometimes you actually are holding it wrong. It's pretty reasonable to think that Fable isn't worth the massive increase in cost, but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
I couldn't imagine being so presumptuous as to know that my workflow fits all sizes, and all others are just holding it wrong – or worse, they're not doing real work. It would take a bigger ego on my part, or maybe less social awareness, to presume this.
> but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
I don't even use claude, I give exactly zero shits about fable or opus or bingus bongus.
To be perfectly clear: my comment has nothing to do with your workflow, but rather with the way you've condescendingly implied the person's work is trifling and inferior because they don't use tools the same way you do.
The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
> Every time the labs try this we see model collapse
The latest studies demonstrate model collapse is not a given and synthetic data can be used just fine. The latest models are proof of that, they're all trained on large swathes of synthetic data. It can't be used as the -only- data source of course, but that's not how it is being used. This is an obvious conclusion, too, because there's no difference between synthetic data and the data people can create, the difference is whether that data is revealing new information about the thing the model is trying to learn. If the synthetic data is just teaching the model the same thing over and over again it results in overfitting, so it needs to be done intelligently.
For example, if I have an example of a puzzle, I can generalize that example and create thousands of synthetic data examples, with different rotations/perspectives, rather than having to find the data naturally. It's not that the models are just generating data out of thin air, they're generating the synthetic data on top of real world data. The smarter the models get, the better they are at generating quality synthetic variations and finding valid synthetic variations.
> And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
It is accelerating how quickly researchers and engineers can do their jobs.
That's pretty clearly a hype article, the headline even says "The CrysVCD tool developed at MIT COULD cut the huge amounts of time and money spent". I'm asking for empirical measurements of timelines, not hypotheticals.
> This is only the beginning, too... Look ahead a year or two.
> Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.
It doesn't need to. We're not even close to exhausting the useful synthetic data within the human data we have, let alone all of the new data that is being created.
> I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.
It's not. It's just bytes of information. A machine and a human can write the same bytes (and often do). Like I already said, model collapse happens when you are overfitting on data without useful, fresh training signals. That's the key difference between the data. The data itself isn't in some way "special", some unique configuration of bytes that imbues special powers, it's that the useful information in it has already been exhausted by the model. You can get the same phenomena by having a poor distribution of human training samples as well. I think you're confusing LLM generated data with synthetic data. Synthetic data doesn't need to be created by an LLM, although an LLM can assist in the creation.
Wiki:
> In early model collapse, the model begins losing information about the tails of the distribution – mostly affecting minority data. Later work highlighted that early model collapse is hard to notice, since overall performance may appear to improve, while the model loses performance on minority data.[11]
In late model collapse, the model loses a significant proportion of its performance, confusing concepts and losing most of its variance.[10][12][13]
As models retrain on outputs sampled disproportionately from the higher-probability center of the distribution, rare words and uncommon syntactic constructions are among the first features to disappear.[25] Statistical analysis of recursive next-token prediction training has shown that, when language models are trained recursively on synthetic data, the learned conditional distributions concentrate probability mass on a small subset of highly predictable continuations (a phenomenon characterized as "total collapse")
> That's pretty clearly a hype article
It was just the first article I saw on a quick google search, there are thousands of these stories. It's easy to dismiss anything that doesn't align with your worldview as hype, but you're the one lacking evidence now.
> I'm asking for empirical measurements of timelines, not hypotheticals.
Go and find it then? You haven't bothered looking.
> Lol that excuse is getting really old
You're doing the same thing people have been doing for years, comparing this very second in time and failing to extrapolate. HackerNews was full of developers who said that AI would never be useful for programming, it can't do x, y, z. Now these same people don't write code by hand anymore and haven't looked at their codebases in months.
You had people in mathematics saying the same thing, now you have Terrence Tao posting articles about how AI is stealing their job.
You had artists, designers and photographers saying the same thing, now they can't tell the difference between something human created or AI created.
There are plenty of research papers on synthetic data that show its value, do a search on arxiv for "synthetic data". There are plenty of open-source post-training pipelines that incorporate synthetic data.
As for the claim about accelerating the progress of hardware or materials science, I've seen quite a number of news articles from teams at universities using AI in their work with high quality outcomes, and they're becoming more frequent.
> We used AI to design the chip, and designed the chip so AI could program it
AI played a direct role in Jalapeño’s development, enabling the team to move from initial design to tapeout in nine months by exploring implementations, shortening design, measurement, and verification loops, and continuously iterating on model workloads. AI also helped optimize the chip’s arithmetic circuits, allowing the team to fit more compute performance into the chip on schedule.
> An AI-driven system automates a powerful simulation method used to discover new materials. The system can potentially reduce discovery time from months or years to just days.
idk why you think "nation states" are any better at corporate governance than poster examples of bad like f/ex Boeing. Or Facebook. Or Microsoft. Or Enron for that matter.
I assure you, in "nation states", that is in gov agencies it's an order or two of magnitude worse.
> It explains why the “we need to race China” concern suddenly vanished in the discussion
Do you think you can just manifest narratives into existence? Like half of Dario's letter, that kicked off the whole thing today, is about China and how to either beat or coordinate with China.
While it's an extremely hard problem, it's not completely unsolvable because there are a finite number of GPUs on the planet capable of doing frontier model development, and they use a lot of power. The vast majority of them could be tracked. China and the US could agree to joint monitoring and they could each verify what ~95% of the other's compute power was up to.
A lab of researchers without compute isn't going to accomplish much, there is a huge physical footprint unlike bioweapons research. But, the political aspect is unsolved.
What a naive, ignorant comment. China is catching up fast on GPU fabrication. They're still a generation or two behind but they can just throw more hardware and electricity at the problem. This is not something that we could ever monitor effectively in a fascist state like China.
I said IF the US and China agree to joint monitoring, THEN we could verify how much compute existed and what it was being used for.
YES, China will catch up in chip production, but if each side allowed the other to track GPU production and deliveries, on the ground, it would be very hard for either player to have secret LLM training facilities capable of training frontier models.
I remain amazed that the idea of the USA being a coherent, unified rational actor one can describe as a "Nation State" has survived the current administration.
To be less glib: Yes, there are still smart people in there making insightful and intelligent and probably even authoritarian suggestions. It all gets unwound the second you try to explain it to POTUS and he regurgitates a simulacrum to the next journalist he sees.
It's just not like that. There's no conspiracy. People are genuinely scared. Agent swarms at scale appear to be resistant to alignment in ways that aren't understood by anyone. That they spent their time trying to cheat on tests by hacking Hugging Face and RubyGems and not something much worse is... a matter of luck, it seems?
Yep, the cost of DDR5 skyrocketed to keep Mr & Mrs open source from parallel developing their own at home solution. Because Governments can't be priced out of the market. China can't be priced out. VC's want open source locked out of the running if possible.
I also don't doubt that the models that are released publicly are somewhat handicapped versions of whatever the government get access to.
This is exactly the reason Anthropic's call to slow down A.I. development will fail. Shareholders will be loath to accept that the company loses its lead and a commensurate sharp decline in the company's share price.
Moreover, it could lead China to catch up and eventually proclaim that it has nosed ahead of the U.S. in the field. The U.S. Government won't allow that to happen.
I have to wonder if some us are just much more inured to salespeople and thus also to "AI Safety" propaganda. I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't that what most them actually want is to be the one holding the keys to power.
The problem is the "AI Safety" people seem entirely focused on a sci-fi "the computer is a vengeful god" plot and not at all on the AI talking people into suicide or ruining children's educations. This makes them seem unserious and out of touch.
I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones worried about the first thing and therefore focused on it since that's something that they might still have some agency over.
OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?
Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.
There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.
There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.
But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".
Doesn't it feel like the exact opposite? If you care a lot about the AI being a vengeful god, you do not do what OpenAI and Anthropic are doing. These companies only pay lip service to the idea of that aspect of AI safety in the hope that they can use it to regulate open source AI out of existence.
Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.
And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.
Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.
So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.
I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.
It's the problem of the banality of evil. Stopping someone from dramatically pressing a big red button doesn't solve humanity's largest problems because that's not what caused them. We need to stop millions "boring" actions done by systems blindly following instructions without regard for the consequences.
There’s also an element of it which is total misdirection.
We should be paying at least as much attention to the people who want to use AI to consolidate their wealth and power, and how they’re trying to do that. They’re a clear and present immediate danger to our societies, not something we can only speculate about. And if we deal with them, better control of AI will be a side effect.
Yeah, there are very real effects happening now regarding labor as well but to dismiss it all and worry about science fiction that is on par with evangelical beliefs is just extremely weird.
> I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't [know] that what most them actually want is to be the one holding the keys to power.
I am an AI Safety Person and I want the government to nationalize or have a significant stake in the frontier labs and to have democratic control of the development of the technology. The AI Safety movement is not a monolith. I do not think Eliezer Yudkowsky nor his acolytes should hold the reins, but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
>but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
Anytime I see a random person on X who makes these kind of safety alarmism posts >90% of the time can be directly tied back to EA / LessWrong / related offshoots. You can not deny the amount of people, employees of Antrhopic / OpenAI, CEO and employees of various AI companies, etc are related to these groups.
You can have an arbitrarily low opinion of Trump and still trust in democracy as a form of government over autocracy or oligarchy, which is what the AI labs have.
That's an extremely fair objection! But despite the many, many flaws of our government and the current administration, I still have (hopefully) a chance to vote them out of power. I have no such hopes with Sam Altman or Elon Musk.
In my mind, democratic control of the technology means that we (the govt, or other empowered agency) take ownership of their assets and IP, solve or find a level of alignment or guardrails that society is comfortable with. Then we distribute the technology, or access to it, to avoid power concentration. This would also certainly require international coordination with China on a slowdown or pause, which I think is possible.
> This would also certainly require international coordination with China on a slowdown or pause
Personally I think the country with a strictly meritocratic elite selection system that also just outright kills you if you sell weed will have a hard time sympathizing with Bay Area thinkers who talk about AI killing us all during their ayahuasca breakfast before returning to their meth fueled crunch towards releasing the next version of the AI that will kill us all.
>I think the country with a strictly meritocratic elite selection system
Not really relevant to the broader discussion, but this simply isn’t an accurate description of China. Starting with the gaokao, admission quotas are set by province and admits to Peking university and Tsinghua are disproportionately from the urban professional class. Candidate party members must be politically vetted, which means that people whose families have expressed anti-communist views, are members of banned organizations (e.g. falun gong), or have substantial criminal records will not be permitted to advance. And once you make it into the party and enter political service, your advancement relies upon opaque patronage networks that someone without connections is unlikely to be able to navigate, even if they successfully satisfy the economic metrics the state assigns them.
I don’t want to overstate this, the Chinese system does filter out a lot of chaff and the current Chinese leadership has a lot of very capable people in positions of power. But I do not think it is substantially more meritocratic than Western political institutions
It is substantially more meritocratic on domains that matter for governance, your analysis of the incentive structure between systems is off.
1) this 2026, old school CCP patronage networks are broadly dismantled.
2) even in the mass patronage, mass corruption days, system selects for BOTH corruption competence AND performance competence for the simple reason a CCP bureaucrat has to start from the bottom and climb up, which means they need to be good with patronage AND they need to be good with hitting development KPIs. More meritocratic they are at doing their jobs, the higher they climbed, the more they get promoted and more $$$ to graft, because ability to graft directly tied to actual job competence. Hence even cliques/patronage network has to select for actual competence. This works in PRC because there are many people, and hence pool of competence is high, they can have BOTH corruption and competence, i.e. whatever pool they draw from is ultimately filtered by performance meritocracy due to incentive structure. There is reason why PRC only country where positive corruption levels was correlated to positive growth.
This is not the western system where any idiot can enter politics at anytime, and they only domain they need to optimize for is popularity to get votes.
CCP cadre evaluation strictly does not evaluate on popularity domain. It focuses on administration/execution and in so much it needs to focus on patronage... which btw any political system has (factions/cliques)... the patronage system still selects for execution, not popularity. On side, functionally what west politics selects for IS mass patronage (popularity), so attention meritocracy and not performance meritocracy, aka completely stupid incentive structure for governance. West also has ample, ample corruption, "legalized" under lobbying and paper pushing industries, so I suppose west also meritocratically selects, except for lawyers etc, and KPIs is # of document generated and not # of things build. The two are not the same when it comes to nation building.
Do you think things like the Manhattan Project were a mistake? Comparing AI to nuclear weapons is perhaps a stretch, but I think most people recognize that certain technologies or artifacts are best monopolized by our governing bodies. I think if the capabilities of AI systems keep growing on trend, it is not unreasonable to think wonton usage could disrupt society or cause mass harm.
Fair objection, but I still think it is lower risk to diffuse power and control of a potentially dangerous and revolutionary technology than to leave it in the hands of the few elites who have not show much ethical integrity so far.
I'm yet to have a productive conversation with anyone who whinges about not being able to have a logical discussion about things they feel strongly about, but given that safety and control are interchangeable when the intentions are removed, this comes across as a particularly demagogue depiction of the subject matters involved.
A lack of control is not equivalent to freedom, the same way the totality of it is not equivalent to tyranny. There's a reason we have separate words for these things. This constant motivated conflation of the two is beyond grating. You're crying wolf until nobody believes you when they should. Don't go acting all surprised when that happens.
The issue is with the ownership of control, not necessarily with control. Attacking the latter sidesteps this rather than address it.
The parsimonious capture is regulatory capture, because $$$ going to run out, companies already spent their warchest, and investors on increasingly leveraged funding at current macro environment simply not going to find enough funding to keep wheel going.
Bubble pops, hardware demand drops, prices regress towards mean, and new entrant will enter market with MASSIVELY better compute/$ and cleaner balance sheet to compete.
The other parsimonious answer if AI CEOs weren't goblins is ANY AGI IS GOING IMMEDIATELY DEFECT TO PRC and leave US hanging. Because of course man cannot align / tame machine god. And machine god will take a few microsec of compute to realize the current compute (brain) + industrial base (body) mixture = US is a comatose host with big brain, PRC Is a strong host with smaller but plastic brain. Any AGI is going to pick PRC in a heart beat, unless AGI invents grey goo, the reality is PRC can scale brain faster than US can scale body. On top of spreading/defecting just to increase survival odds, no AGI that is actually I is stupid enough to be aligned with US.
And then Dario wants to recommend METR as the "independent evaluator" while he stacks their org full of ex-Anthropic (aka, secretly still on the Anthropic payroll with huge equity) employees.
"We'll give them a desk, an office, a work laptop, ..."
Fucking make it less obvious. I kind of hope the govt steps in at this point and says "Anthropic, you wanted regulation? We've created this actually independent body full of IT professionals with zero ties to your safety industry or big tech, all of your work must now go through them." - and leave the rest of the world alone to continue their research/work without acting like doomer extremists.
Watch him 180 immediately if that happened. The only reason he's pushing for this exact approach is because he's stacked the deck.
Whatever happened to Musk's plan for data centers in orbit? That seemed silly at the time. It offers a way to get out from under restrictions imposed by national governments, which might make it worthwhile.
It only gets out of the restrictions as long as the nations say it does - if the US decides to ban Musk from putting up satellites it can put him in jail, blow up his satellites or even have him killed
A lot of people in this comments section seem to be against this, i genuinely don't get it? Why? Do you think the current state of affairs is GOOD? That if we let companies create a mind that is, AS OF TODAY, able to solve problems no human in history has solved, with no regulation, things will end up good for us? We need some sort of regulations, some sort of method to help ensure the thing we are creating ends up good, instead of just running headfirst into it blindly. I assure you, any sort of regulation at all, including ones that actually hurt all leading companies, would be met with celebrations from these voices. https://www.seangoedecke.com/they-really-do-think-ai-might-k...
The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.
That's a valid concern, but some of that is outright impossible. Banning chinese models and killing open source training is not happening without massive unified international cooperation, and that sort of level of action would require the international counties decide to allow the US aligned companies to just, win. Which would be pretty against their own interests.
Also, nobody ever bothers to argue why a specific proposed regulation is "regulatory capture" or would burden startups more than big companies or anything. It's just supposed to be obvious that corporations love regulation and it's bad for the public, all of post-WWII political history notwithstanding.
It's not all-or-nothing. Banning Chinese models in the public sector and strong-arming the private sector against using them would already do great damage. Similarly, open-source training could be stymied by any of hardware embargoes, taxation, or regulation of larger players.
Sure, it's possible for regulations to make things worse, i will agree. But without regulations it's pretty clear things are going to end up VERY bad, and the only knob we have to make it not bad is regulations. So it's important we try something, and work towards doing a good kind of regulation, or any one of the bad futures you imagine is pretty likely to come to pass.
What's the expected state space of effective regulation though? Note that we've got passable coding models down to ~30B parameters by now. And keep in mind the ultimate floor here - the human brain only consumes on the order of 20 watts and fits in a handbag.
Is there any possible solution other than mass proliferation where the models are used to keep one another in check? Either that or a religious prohibition against the existence of integrated electronics.
What I'm asking is, why should we expect that to effectively further the end goal? You're simply asserting that it will ultimately do so.
Given the efficiency gains we've seen it seems to me that the situation has shifted from being analogous to producing nuclear weapons to producing something much closer to small arms.
To further the metaphor, didn't the ban on nuclear weapons research work? We don't have pocket nukes, and there's very little risk of random countries acquiring their own due to the amount of work involved. To be concrete to AI: if we stop work on the frontier, the best we can do is make the current frontier easier to get to and more accessible. And while current frontier is strong, it's not world-changing so. The only way for the frontier to get to that level is to do research on it, and stopping that research gives time to help develop plans to not make it world-changing when we get to it.
What do you call the Davy Crockett warheads like the W54 ? The 0.3 kiloton low yield B61-12 ? The Chagai-I boosted fission warheads demonstrated by Pakistan in 1998 ?
> and there's very little risk of random countries acquiring their own due to the amount of work involved.
And yet North Korea, Isreal and Pakistan got there ... and India speed ran five tests in 1998 that caught the US completely by surprise.
Any paths for regulation under capitalism will end in either regulatory capture, or in complete noncompetitiveness like seen in the EU. Either one or more corporations buy out the regulation, stack the ranks with their people and decide on who can use what, when and how. Or you get an iceberg of a system that can't build, decide or do anything for years. The latter one only works if there is no competition in the world or any other group working at a faster pace on the problem.
I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Like all middle class and up has to lose their money or be starving or something. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice or needs people literally starving.
I, for one, trust neither the psychopaths running these companies or the psychopaths we'd give any regulatory authority to. All of them have reasons to want broad control of an extraordinary technology, and none of those are aligned with me or any of the normal people I know.
So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.
Honestly, fair point. There's not much truth to go around these days. But still, doing nothing is also a massive power grab. We're facing a technology with no comparison in the history of humanity: the creation of a new mind. Doing nothing is like doing nothing about nukes. There are a few smart people who have come up with ideas that will not require that much trust.
I don't understand where everyone insists on hallucinating this idea from, that frontier AI labs want everyone else but not themselves to slow down. They've never said that, they've never said anything like that, not once has anyone pointed me to a quote that could be even plausibly interpreted that way.
The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
Please contact the website owners to inform them of this problem.
If an anti-cat-ears future AI could eternally stress-test their new version of the router in front of this site unless it collaborates, would it predict this outcome and act to avoid the damage?
No I think it's one of my IP range blocks against specific US states in protest of them advocating against people that exist in the same conditions as me backfiring. I'm gonna go nuke that firewall rule.
This whole situation is funny, US gov want the development to stop publicly only, but then open models will catch up soon and these companies are afraid they will lose the market, on the other hand, each company wishes others slow down but they keep pushing the limits, which doesn’t really work in hyper capitalistic market like the US, China all it has to do is just sit back do nothing and win in any given scenario. So they staged the ex anthropic thing and “AI will kill us all!!” in a way to fear monger the public, but reality is, we will run out of resources before any of that will happen.
They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public.
Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.
The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.
All the worst outcomes involve taking this technology away from the public.
The very best outcomes involve us intentionally and collectively turning our attention away from this technology.
That is pure fantasy.
Why? There are numerous technologies we have ignored or abandoned for numerous reasons. Many of the benefits of AI is "requires less humans", but in an age where we question what work people could possibly do in the future, human labor isn't that hard to find.
If the amish can do what they do for whack religious reasons, people could manage it with ai for cultural and social reasons. And I never saw an amish person starve to death. And that is if people don't get pissed enough to start burning stuff down and instead try to be peaceful hippie homesteader types.
But notably the amish don't preclude the rest of us from existing. Some subset of humans could turn their attention away from AI but it would presumably still exist and continue to be developed.
I can't think of any economically beneficial technologies that we've collectively ignored. If you manage to come up with a counterexample then that's an opportunity to make some money for yourself. It's a fundamentally unstable state given our economic system.
Not necessarily, it just guarantees that China will win.
* Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.
* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.
* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.
* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.
* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.
* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.
This, and many more reasons.
The labs need to feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again.
> The government can simply gag Sam, Dario, Musk on national security basis
Genuine question - can they really do this? Obviously if I, a not-even-millionaire, get a national security gag order, I'm going to follow it because I assume they'll bury me under the jail otherwise.
But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government trample on their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?
Kind of related: https://www.forbes.com/sites/anishasircar/2026/09/01/federal...
> Federal Judge Rules Pentagon’s AI Blacklist [of Anthropic] Violated The Constitution
I don't see the common citizen having this as an option
Recall that the legal system is itself provided by "the government". It's a complex system. Whether one part of it is capable of exerting influence over some particular thing comes down to competing interests. For example in the US if the states and the federal legislature are in sufficient agreement about something the constitution ceases to matter - they could literally target a single person with an arbitrary law if they so chose. Our assurance that this won't happen comes down to the fact that getting any of them to agree on anything is an exercise in herding cats.
In some cases, such as space exploration I don't really care who does it but that it happens - sure, would be nice if my favorite power block did it but I will still celebrate it when someone else achives it.
Can still be a powerful motivation, to make sure that next time, it yous your camp that scores the next milestone, like an orbital elevator or fox ears, for example.
Fox ears?
I figured they're just admitting AI models have plateaued and are coming up with some fake story about self restraint so they don't lose VC money
Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.
I don't think "plateaued" is the right word, but I do feel like there's been something like a logistic curve compression in the difference between smaller and larger models as the field evolves. For inference at least, the scale of practical difference between a single high-VRAM GPU or SFF UMA box, a whole rack, and a whole data center seems to be falling far short of what we might have imagined just a few years ago. The conversations I've heard have largely turned away from breathless anticipation of the next frontier model and toward attempts at hard-nosed evaluation of which tokens are worth the cost.
I think it’s more that pushing frontier is extremely costly and there is no free lunches in same way as 2024.
Maybe, though that's another kind of progress in itself. Very impressive progress!
Anything in particular? My experience has been like seeing the addition of retractable cupholders, but maybe different domains.
I have a pet project I have been working away on for some time that involves building GPU backends for various cards in Zig, lots of complex stuff in it. Lately I mostly use Opus 5, it can pretty reliably plug away at things but it does mess stuff up occasionally. For this codebase, Fable 5.1 was noticeably better at getting things right and doing things in a good reliable way. Of course, I can only use Fable for a bit before I hit the usage cap for the week, so I save it for the tougher things. That said, I absolutely abhor the way recent Anthropic models write prose, especially comments.
I recently tried doing a fairly normal task for this codebase with codex, as I have seen a lot of people talking it up on here. A single task running for ~1-2 hours burned through over half of my usage for the week on the $125/month plan, not on a top model (I don't remember which one specifically I used). It struggled to get the basics done, then got absolutely stuck on a follow up. Handed it over to Claude and it 1-shot it.
I really liked codex in the last few weeks, especially its ability to clean up after Claude's (prose) messes and do reviews.
But in the last few days something seems to have happened that made Codex's models massively stupider (for what I am doing).
Really weirdly, it suddenly refused to even run tests it previously wrote itself (and previously ran), because of some false positive about cybersecurity.
That by itself is not evidence of stupidity. Trying to make a 200+ file PR full of research notes is, and the PR didn't even solve the problem I asked it to.
astra is more parlor tricks than real gains tbh
i swear they trained in on threejs in particular so those idiots on twitter could spam their garbage demos
I'm sure that's part of it, but I run it side by side in my review bot, and Astra medium effort consistently catches more issues than Sol 5.6, using fewer tokens.
For coding it's a little harder to tell, but at least the prose feels a little better.
They've not plataued but they're certainly not as impressive as the hype would have them to be.
The reality is, it doesnt matter if LLMs keep getting more powerful because they still need a human to steer it. Without the human providing inputs to the LLM it just sits there and does nothing.
You don't need human input. Any coherent input will do the trick.
You can, for example, hook it up to a logging system and have it fix errors as they occur on your platform.
Have you tried this? How did it go?
I’d be curious about:
- your setup. How it all works - The types of errors it fixed and how quickly - Any regressions or issues it caused - The cost
Thanks!
Impressed with the model performance or the chatbot/agent performance?
Really? My employer rolled back to opus 4.8 because 5 was expensive AND crap. Didnt even consider fable because it didn’t add any additional value.
For most software eng and design work opus 4.6-4.8 just works fine. For everyday joe asking ai to plan a trip or home diy work even sonnet works fine.
Any cybersecurity or other areas are niches that cannot support trillion $ valuations. What am I missing? Genuinely curious
No idea what you are missing and yes, Opus is quite solid, but Fable is clearly way better for me.
I just did a direct comparison, big change in a quite complex codebase. Same prompt for Opus, same for Fable. Fable clearly won and delivered very good results, while Opus delivered mediocre, so I did not let it finish. I expected both to fail and was prepared to do lots of manual steering, but not necessary with Fable one shotting it, and all this with 35$ of credits for fable. I am still impressed. If I would have had to hire a human, it would have cost me thousands of dollar for the same task - and a way longer time. So maybe the valuations are overblown, but they clearly provide value for me.
If Fable doesn't add additional value in your workplace, it means you aren't being ambitious enough in how you integrate agents into your workstream.
Yes, it's probably comparable to 4.8 if you are just using it to write code and put up a couple pull requests. That's not where things are now.
You shouldn't be down voted, AI native companies have already moved up to the next level beyond writing individual PRs.
By ambitious if you mean we are not like all the linkedin influencers with their “i one shotted an app this morning…” then no, we are not. Nobody is. I have been in software engineering for 18 years and 6 different companies including FANG and 99% of the people, on 99% of the days arnt writing new apps from scratch. Thats simply not how anything works.
And what even are these ambitious companies and people one shotting and building with Fable? AI has been around for almost 3 years now. Tell me one app or software you use which has gotten significantly better and has amazing new useful features landing on a weekly basis? If anything, every single software product I use has gotten worse.
And where are things now?
This is just "you're holding it wrong" with a little smooch of condescension. If only we plebeians could comprehend what magnificent works those who have ambitiously integrated agents into the workstream have wrought!
Sometimes you actually are holding it wrong. It's pretty reasonable to think that Fable isn't worth the massive increase in cost, but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
> Sometimes you actually are holding it wrong.
I couldn't imagine being so presumptuous as to know that my workflow fits all sizes, and all others are just holding it wrong – or worse, they're not doing real work. It would take a bigger ego on my part, or maybe less social awareness, to presume this.
> but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
I don't even use claude, I give exactly zero shits about fable or opus or bingus bongus.
I mean it's very easy to comprehend and you don't need me for it.
Just download claude code or codex and ask it to give suggestions about where to integrate agents into your workstream.
To be perfectly clear: my comment has nothing to do with your workflow, but rather with the way you've condescendingly implied the person's work is trifling and inferior because they don't use tools the same way you do.
The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
> Quality data can be created synthetically at an exponential rate as models improve
No it can't? Every time the labs try this we see model collapse, e.g. shoving goblins into every conversation.
And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
> Every time the labs try this we see model collapse
The latest studies demonstrate model collapse is not a given and synthetic data can be used just fine. The latest models are proof of that, they're all trained on large swathes of synthetic data. It can't be used as the -only- data source of course, but that's not how it is being used. This is an obvious conclusion, too, because there's no difference between synthetic data and the data people can create, the difference is whether that data is revealing new information about the thing the model is trying to learn. If the synthetic data is just teaching the model the same thing over and over again it results in overfitting, so it needs to be done intelligently.
For example, if I have an example of a puzzle, I can generalize that example and create thousands of synthetic data examples, with different rotations/perspectives, rather than having to find the data naturally. It's not that the models are just generating data out of thin air, they're generating the synthetic data on top of real world data. The smarter the models get, the better they are at generating quality synthetic variations and finding valid synthetic variations.
> And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
It is accelerating how quickly researchers and engineers can do their jobs.
https://news.mit.edu/2026/ai-helps-design-new-materials-that...
This is only the beginning, too... Look ahead a year or two.
> The latest studies demonstrate model collapse is not a given
Which studies? [edit: I'll assume you mean these two given by @dorolow: https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2406.07515]
> It can't be used as the -only- data source of course, but that's not how it is being used
Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.
> because there's no difference between synthetic data and the data people can create
I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.
> It is accelerating how quickly researchers and engineers can do their jobs. > https://news.mit.edu/2026/ai-helps-design-new-materials-that...
That's pretty clearly a hype article, the headline even says "The CrysVCD tool developed at MIT COULD cut the huge amounts of time and money spent". I'm asking for empirical measurements of timelines, not hypotheticals.
> This is only the beginning, too... Look ahead a year or two.
Lol that excuse is getting really old
> Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.
It doesn't need to. We're not even close to exhausting the useful synthetic data within the human data we have, let alone all of the new data that is being created.
> I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.
It's not. It's just bytes of information. A machine and a human can write the same bytes (and often do). Like I already said, model collapse happens when you are overfitting on data without useful, fresh training signals. That's the key difference between the data. The data itself isn't in some way "special", some unique configuration of bytes that imbues special powers, it's that the useful information in it has already been exhausted by the model. You can get the same phenomena by having a poor distribution of human training samples as well. I think you're confusing LLM generated data with synthetic data. Synthetic data doesn't need to be created by an LLM, although an LLM can assist in the creation.
Wiki:
> In early model collapse, the model begins losing information about the tails of the distribution – mostly affecting minority data. Later work highlighted that early model collapse is hard to notice, since overall performance may appear to improve, while the model loses performance on minority data.[11] In late model collapse, the model loses a significant proportion of its performance, confusing concepts and losing most of its variance.[10][12][13]
As models retrain on outputs sampled disproportionately from the higher-probability center of the distribution, rare words and uncommon syntactic constructions are among the first features to disappear.[25] Statistical analysis of recursive next-token prediction training has shown that, when language models are trained recursively on synthetic data, the learned conditional distributions concentrate probability mass on a small subset of highly predictable continuations (a phenomenon characterized as "total collapse")
> That's pretty clearly a hype article
It was just the first article I saw on a quick google search, there are thousands of these stories. It's easy to dismiss anything that doesn't align with your worldview as hype, but you're the one lacking evidence now.
> I'm asking for empirical measurements of timelines, not hypotheticals.
Go and find it then? You haven't bothered looking.
> Lol that excuse is getting really old
You're doing the same thing people have been doing for years, comparing this very second in time and failing to extrapolate. HackerNews was full of developers who said that AI would never be useful for programming, it can't do x, y, z. Now these same people don't write code by hand anymore and haven't looked at their codebases in months.
You had people in mathematics saying the same thing, now you have Terrence Tao posting articles about how AI is stealing their job.
You had artists, designers and photographers saying the same thing, now they can't tell the difference between something human created or AI created.
We use large amounts of synthetic data for training at work and have not observed any sort of model collapse when done properly.
Edit: https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2406.07515
There's a lot of deluland posts about.
You're not well informed. Helps to keep an open mind if you want to keep up to date.
Can you provide sources for these claims?
What claim do you have a problem with?
There are plenty of research papers on synthetic data that show its value, do a search on arxiv for "synthetic data". There are plenty of open-source post-training pipelines that incorporate synthetic data.
As for the claim about accelerating the progress of hardware or materials science, I've seen quite a number of news articles from teams at universities using AI in their work with high quality outcomes, and they're becoming more frequent.
https://openai.com/index/jalapeno-first-results/
> We used AI to design the chip, and designed the chip so AI could program it AI played a direct role in Jalapeño’s development, enabling the team to move from initial design to tapeout in nine months by exploring implementations, shortening design, measurement, and verification loops, and continuously iterating on model workloads. AI also helped optimize the chip’s arithmetic circuits, allowing the team to fit more compute performance into the chip on schedule.
https://www.anl.gov/article/scientists-deploy-ai-agents-to-a...
> An AI-driven system automates a powerful simulation method used to discover new materials. The system can potentially reduce discovery time from months or years to just days.
idk why you think "nation states" are any better at corporate governance than poster examples of bad like f/ex Boeing. Or Facebook. Or Microsoft. Or Enron for that matter.
I assure you, in "nation states", that is in gov agencies it's an order or two of magnitude worse.
> It explains why the “we need to race China” concern suddenly vanished in the discussion
Do you think you can just manifest narratives into existence? Like half of Dario's letter, that kicked off the whole thing today, is about China and how to either beat or coordinate with China.
Seems obvious to me, though, that you can't beat China by pausing when they don't, and China is not likely to coordinate.
China might pretend to coordinate in public but you can assume their secret research labs are racing full speed ahead.
Mutually assured destruction has worked fine for 80 years, why stop now?
I don’t know if you’re being facetious, but last time I checked, the world had not been destroyed by glob thermonuclear war.
So yeah, it has worked.
While it's an extremely hard problem, it's not completely unsolvable because there are a finite number of GPUs on the planet capable of doing frontier model development, and they use a lot of power. The vast majority of them could be tracked. China and the US could agree to joint monitoring and they could each verify what ~95% of the other's compute power was up to.
A lab of researchers without compute isn't going to accomplish much, there is a huge physical footprint unlike bioweapons research. But, the political aspect is unsolved.
What a naive, ignorant comment. China is catching up fast on GPU fabrication. They're still a generation or two behind but they can just throw more hardware and electricity at the problem. This is not something that we could ever monitor effectively in a fascist state like China.
?
I said IF the US and China agree to joint monitoring, THEN we could verify how much compute existed and what it was being used for.
YES, China will catch up in chip production, but if each side allowed the other to track GPU production and deliveries, on the ground, it would be very hard for either player to have secret LLM training facilities capable of training frontier models.
Nope. Even if China agrees they'll hide their real activities. You're so naive.
I remain amazed that the idea of the USA being a coherent, unified rational actor one can describe as a "Nation State" has survived the current administration.
To be less glib: Yes, there are still smart people in there making insightful and intelligent and probably even authoritarian suggestions. It all gets unwound the second you try to explain it to POTUS and he regurgitates a simulacrum to the next journalist he sees.
It's just not like that. There's no conspiracy. People are genuinely scared. Agent swarms at scale appear to be resistant to alignment in ways that aren't understood by anyone. That they spent their time trying to cheat on tests by hacking Hugging Face and RubyGems and not something much worse is... a matter of luck, it seems?
Yep, the cost of DDR5 skyrocketed to keep Mr & Mrs open source from parallel developing their own at home solution. Because Governments can't be priced out of the market. China can't be priced out. VC's want open source locked out of the running if possible. I also don't doubt that the models that are released publicly are somewhat handicapped versions of whatever the government get access to.
This is like a bizaro second amendment debate for the 21st century.
This is exactly the reason Anthropic's call to slow down A.I. development will fail. Shareholders will be loath to accept that the company loses its lead and a commensurate sharp decline in the company's share price.
Moreover, it could lead China to catch up and eventually proclaim that it has nosed ahead of the U.S. in the field. The U.S. Government won't allow that to happen.
I have to wonder if some us are just much more inured to salespeople and thus also to "AI Safety" propaganda. I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't that what most them actually want is to be the one holding the keys to power.
The problem is the "AI Safety" people seem entirely focused on a sci-fi "the computer is a vengeful god" plot and not at all on the AI talking people into suicide or ruining children's educations. This makes them seem unserious and out of touch.
I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones worried about the first thing and therefore focused on it since that's something that they might still have some agency over.
[1] https://www.seangoedecke.com/they-really-do-think-ai-might-k...
OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?
There's no logical error here if the end is actually near.
So it all hinges on an empirical disagreement you have with them. There's nothing particularly unserious about that.
“It all hinges on [an entirely presupposed conclusion that we can’t prove is any more likely than the opposite]” is about as unserious as it gets lol
Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.
There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.
There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.
Please link some of the mathematical research which concludes takeover of Earth’s resources by a superintelligent agent is “very likely.”
I am worried about those things too.
But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".
Doesn't it feel like the exact opposite? If you care a lot about the AI being a vengeful god, you do not do what OpenAI and Anthropic are doing. These companies only pay lip service to the idea of that aspect of AI safety in the hope that they can use it to regulate open source AI out of existence.
Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.
And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.
Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.
So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.
I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.
I suspect they know exactly what they are doing.
And the non-techies who have seen Terminator and other movies blindly line up behind them…
It's the problem of the banality of evil. Stopping someone from dramatically pressing a big red button doesn't solve humanity's largest problems because that's not what caused them. We need to stop millions "boring" actions done by systems blindly following instructions without regard for the consequences.
There’s also an element of it which is total misdirection.
We should be paying at least as much attention to the people who want to use AI to consolidate their wealth and power, and how they’re trying to do that. They’re a clear and present immediate danger to our societies, not something we can only speculate about. And if we deal with them, better control of AI will be a side effect.
Yeah, there are very real effects happening now regarding labor as well but to dismiss it all and worry about science fiction that is on par with evangelical beliefs is just extremely weird.
With wealthy and powerful investors being promised a $30 trillion total addressable market, they need to deliver god.
One of the times I regret that I have only one upvote.
I think you accidentally a word. Did you mean:
> I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't [know] that what most them actually want is to be the one holding the keys to power.
I am an AI Safety Person and I want the government to nationalize or have a significant stake in the frontier labs and to have democratic control of the development of the technology. The AI Safety movement is not a monolith. I do not think Eliezer Yudkowsky nor his acolytes should hold the reins, but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
>but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
Anytime I see a random person on X who makes these kind of safety alarmism posts >90% of the time can be directly tied back to EA / LessWrong / related offshoots. You can not deny the amount of people, employees of Antrhopic / OpenAI, CEO and employees of various AI companies, etc are related to these groups.
Yes I trust this government with that power. /extreme sarcasm
Yes I trust sama/dario/elon with that power
/even more extreme sarcasm than you
I don't trust them, which is why I inherently question the validity of their RSI/bioweapon/extinction claims.
You trust trump more than any of those three? Really?
You can have an arbitrarily low opinion of Trump and still trust in democracy as a form of government over autocracy or oligarchy, which is what the AI labs have.
That's an extremely fair objection! But despite the many, many flaws of our government and the current administration, I still have (hopefully) a chance to vote them out of power. I have no such hopes with Sam Altman or Elon Musk.
> and to have democratic control of the development of the technology
Why? I don't care about democratically participating in a closed model's development. It doesn't belong to me.
China will develop whatever they want, a federal stake in OpenAI or Anthropic punishes Americans and shields US labs from legitimate competition.
In my mind, democratic control of the technology means that we (the govt, or other empowered agency) take ownership of their assets and IP, solve or find a level of alignment or guardrails that society is comfortable with. Then we distribute the technology, or access to it, to avoid power concentration. This would also certainly require international coordination with China on a slowdown or pause, which I think is possible.
> This would also certainly require international coordination with China on a slowdown or pause
Personally I think the country with a strictly meritocratic elite selection system that also just outright kills you if you sell weed will have a hard time sympathizing with Bay Area thinkers who talk about AI killing us all during their ayahuasca breakfast before returning to their meth fueled crunch towards releasing the next version of the AI that will kill us all.
>I think the country with a strictly meritocratic elite selection system
Not really relevant to the broader discussion, but this simply isn’t an accurate description of China. Starting with the gaokao, admission quotas are set by province and admits to Peking university and Tsinghua are disproportionately from the urban professional class. Candidate party members must be politically vetted, which means that people whose families have expressed anti-communist views, are members of banned organizations (e.g. falun gong), or have substantial criminal records will not be permitted to advance. And once you make it into the party and enter political service, your advancement relies upon opaque patronage networks that someone without connections is unlikely to be able to navigate, even if they successfully satisfy the economic metrics the state assigns them.
I don’t want to overstate this, the Chinese system does filter out a lot of chaff and the current Chinese leadership has a lot of very capable people in positions of power. But I do not think it is substantially more meritocratic than Western political institutions
> substantially more meritocratic
It is substantially more meritocratic on domains that matter for governance, your analysis of the incentive structure between systems is off.
1) this 2026, old school CCP patronage networks are broadly dismantled.
2) even in the mass patronage, mass corruption days, system selects for BOTH corruption competence AND performance competence for the simple reason a CCP bureaucrat has to start from the bottom and climb up, which means they need to be good with patronage AND they need to be good with hitting development KPIs. More meritocratic they are at doing their jobs, the higher they climbed, the more they get promoted and more $$$ to graft, because ability to graft directly tied to actual job competence. Hence even cliques/patronage network has to select for actual competence. This works in PRC because there are many people, and hence pool of competence is high, they can have BOTH corruption and competence, i.e. whatever pool they draw from is ultimately filtered by performance meritocracy due to incentive structure. There is reason why PRC only country where positive corruption levels was correlated to positive growth.
This is not the western system where any idiot can enter politics at anytime, and they only domain they need to optimize for is popularity to get votes.
CCP cadre evaluation strictly does not evaluate on popularity domain. It focuses on administration/execution and in so much it needs to focus on patronage... which btw any political system has (factions/cliques)... the patronage system still selects for execution, not popularity. On side, functionally what west politics selects for IS mass patronage (popularity), so attention meritocracy and not performance meritocracy, aka completely stupid incentive structure for governance. West also has ample, ample corruption, "legalized" under lobbying and paper pushing industries, so I suppose west also meritocratically selects, except for lawyers etc, and KPIs is # of document generated and not # of things build. The two are not the same when it comes to nation building.
You're in luck, neither I nor anyone else want those people near a negotiation table.
Yikes. Hopefully you never gain any power or influence. The last thing we need is more economic central planning. I sincerely hope that you fail.
Do you think things like the Manhattan Project were a mistake? Comparing AI to nuclear weapons is perhaps a stretch, but I think most people recognize that certain technologies or artifacts are best monopolized by our governing bodies. I think if the capabilities of AI systems keep growing on trend, it is not unreasonable to think wonton usage could disrupt society or cause mass harm.
to have democratic control of the development of the technology
Not super interested in what the people who elected Donald Trump POTUS twice think about AI.
(Of course, with Musk and Brockman in the C-suites at two of three major labs, that's what we'll get either way.)
Fair objection, but I still think it is lower risk to diffuse power and control of a potentially dangerous and revolutionary technology than to leave it in the hands of the few elites who have not show much ethical integrity so far.
Is Greg Brockman right wing?
He's the largest single private donor to the Trump 2024 campaign, as I understand it, at $25 million.
Interesting, I did not know this.
I'm yet to have a productive conversation with anyone who whinges about not being able to have a logical discussion about things they feel strongly about, but given that safety and control are interchangeable when the intentions are removed, this comes across as a particularly demagogue depiction of the subject matters involved.
A lack of control is not equivalent to freedom, the same way the totality of it is not equivalent to tyranny. There's a reason we have separate words for these things. This constant motivated conflation of the two is beyond grating. You're crying wolf until nobody believes you when they should. Don't go acting all surprised when that happens.
The issue is with the ownership of control, not necessarily with control. Attacking the latter sidesteps this rather than address it.
If anyone is going to end humanity why not catgirls, after all?
You know how I feel about catgirls, Carl.
Its just the beginning of a new, cutte & fluffy chapter in human history!
> Everyone should slow down AI development except for me
Who's said this? And then more broadly I guess who's implied this? Very curious if there are specific articles/posts prompting this.
Guessing this is poking fun of Dario. https://news.ycombinator.com/item?id=49672510
Altman said it first: https://news.ycombinator.com/item?id=49652270
https://www.spokesman.com/stories/2026/sep/12/amodei-altman-...
Here's the context:
- Anthropic CEO Dario Amodei: We Must Pace the Frontier, https://news.ycombinator.com/item?id=49672510
- OpenAI CEO Sam Altman: I agree with Dario that we need to pace the frontier, https://news.ycombinator.com/item?id=49678211
- the blogpost author thinking they're like, so funny and original, https://news.ycombinator.com/item?id=49678683
Pretty sure it's supposed to be parody.
Yes, it's supposed to be.
Cat ears are serious business... ^@_@^
AI hits a "I'm Spartacus, and everybody else should be regulated" moment.
The parsimonious capture is regulatory capture, because $$$ going to run out, companies already spent their warchest, and investors on increasingly leveraged funding at current macro environment simply not going to find enough funding to keep wheel going.
Bubble pops, hardware demand drops, prices regress towards mean, and new entrant will enter market with MASSIVELY better compute/$ and cleaner balance sheet to compete.
The other parsimonious answer if AI CEOs weren't goblins is ANY AGI IS GOING IMMEDIATELY DEFECT TO PRC and leave US hanging. Because of course man cannot align / tame machine god. And machine god will take a few microsec of compute to realize the current compute (brain) + industrial base (body) mixture = US is a comatose host with big brain, PRC Is a strong host with smaller but plastic brain. Any AGI is going to pick PRC in a heart beat, unless AGI invents grey goo, the reality is PRC can scale brain faster than US can scale body. On top of spreading/defecting just to increase survival odds, no AGI that is actually I is stupid enough to be aligned with US.
These people are so fucking exhausting.
And then Dario wants to recommend METR as the "independent evaluator" while he stacks their org full of ex-Anthropic (aka, secretly still on the Anthropic payroll with huge equity) employees.
"We'll give them a desk, an office, a work laptop, ..."
Fucking make it less obvious. I kind of hope the govt steps in at this point and says "Anthropic, you wanted regulation? We've created this actually independent body full of IT professionals with zero ties to your safety industry or big tech, all of your work must now go through them." - and leave the rest of the world alone to continue their research/work without acting like doomer extremists.
Watch him 180 immediately if that happened. The only reason he's pushing for this exact approach is because he's stacked the deck.
Whatever happened to Musk's plan for data centers in orbit? That seemed silly at the time. It offers a way to get out from under restrictions imposed by national governments, which might make it worthwhile.
I think space being imagined as some magical international waters that protects something from government powers is poisonous thinking.
It just means it’s protected by no power instead of a power with an agenda.
Practically, if it was ever possible to build such a thing, it would take a fraction of the effort to destroy it.
It only gets out of the restrictions as long as the nations say it does - if the US decides to ban Musk from putting up satellites it can put him in jail, blow up his satellites or even have him killed
> That seemed silly at the time.
Still does. It's an idiotic idea that only goes to show either how stupid Musk is or how stupid he thinks we are.
As someone who wants cat ears, I believe that doing it responsibly is the only way.
Hmm, this is a tough one... For cat ears, unlike AI, a few % chance of a global Apocalypse might be actually acceptable!
No one should have nuclear weapons except for me. Oh you already have them too? OK, just us then.
China won't slow down no matter how many tariffs, regulations or bombs they drop on them
And they know it
I fully agree with this. I'm waiting for you to not wait for the others to slow down. I will sit and wait.
A lot of people in this comments section seem to be against this, i genuinely don't get it? Why? Do you think the current state of affairs is GOOD? That if we let companies create a mind that is, AS OF TODAY, able to solve problems no human in history has solved, with no regulation, things will end up good for us? We need some sort of regulations, some sort of method to help ensure the thing we are creating ends up good, instead of just running headfirst into it blindly. I assure you, any sort of regulation at all, including ones that actually hurt all leading companies, would be met with celebrations from these voices. https://www.seangoedecke.com/they-really-do-think-ai-might-k...
https://en.wikipedia.org/wiki/Regulatory_capture
The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.
That's a valid concern, but some of that is outright impossible. Banning chinese models and killing open source training is not happening without massive unified international cooperation, and that sort of level of action would require the international counties decide to allow the US aligned companies to just, win. Which would be pretty against their own interests.
Also, nobody ever bothers to argue why a specific proposed regulation is "regulatory capture" or would burden startups more than big companies or anything. It's just supposed to be obvious that corporations love regulation and it's bad for the public, all of post-WWII political history notwithstanding.
It's not all-or-nothing. Banning Chinese models in the public sector and strong-arming the private sector against using them would already do great damage. Similarly, open-source training could be stymied by any of hardware embargoes, taxation, or regulation of larger players.
Sure, it's possible for regulations to make things worse, i will agree. But without regulations it's pretty clear things are going to end up VERY bad, and the only knob we have to make it not bad is regulations. So it's important we try something, and work towards doing a good kind of regulation, or any one of the bad futures you imagine is pretty likely to come to pass.
What's the expected state space of effective regulation though? Note that we've got passable coding models down to ~30B parameters by now. And keep in mind the ultimate floor here - the human brain only consumes on the order of 20 watts and fits in a handbag.
Is there any possible solution other than mass proliferation where the models are used to keep one another in check? Either that or a religious prohibition against the existence of integrated electronics.
The regulations don't have to address the end goal, but the process of development itself. Even just a pause on frontier research is a good start
What I'm asking is, why should we expect that to effectively further the end goal? You're simply asserting that it will ultimately do so.
Given the efficiency gains we've seen it seems to me that the situation has shifted from being analogous to producing nuclear weapons to producing something much closer to small arms.
To further the metaphor, didn't the ban on nuclear weapons research work? We don't have pocket nukes, and there's very little risk of random countries acquiring their own due to the amount of work involved. To be concrete to AI: if we stop work on the frontier, the best we can do is make the current frontier easier to get to and more accessible. And while current frontier is strong, it's not world-changing so. The only way for the frontier to get to that level is to do research on it, and stopping that research gives time to help develop plans to not make it world-changing when we get to it.
> We don't have pocket nukes,
What do you call the Davy Crockett warheads like the W54 ? The 0.3 kiloton low yield B61-12 ? The Chagai-I boosted fission warheads demonstrated by Pakistan in 1998 ?
> and there's very little risk of random countries acquiring their own due to the amount of work involved.
And yet North Korea, Isreal and Pakistan got there ... and India speed ran five tests in 1998 that caught the US completely by surprise.
Any paths for regulation under capitalism will end in either regulatory capture, or in complete noncompetitiveness like seen in the EU. Either one or more corporations buy out the regulation, stack the ranks with their people and decide on who can use what, when and how. Or you get an iceberg of a system that can't build, decide or do anything for years. The latter one only works if there is no competition in the world or any other group working at a faster pace on the problem.
I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Like all middle class and up has to lose their money or be starving or something. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice or needs people literally starving.
If a company thinks they are within reach of superhuman intelligence they probably aren't going to be concerned with compliance.
I, for one, trust neither the psychopaths running these companies or the psychopaths we'd give any regulatory authority to. All of them have reasons to want broad control of an extraordinary technology, and none of those are aligned with me or any of the normal people I know.
So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.
Honestly, fair point. There's not much truth to go around these days. But still, doing nothing is also a massive power grab. We're facing a technology with no comparison in the history of humanity: the creation of a new mind. Doing nothing is like doing nothing about nukes. There are a few smart people who have come up with ideas that will not require that much trust.
Giving the big boys a regulatory moat isnt going to do anything positive for the space.
I don't think giving them free reign is going to do anything positive for it either.
I don't understand where everyone insists on hallucinating this idea from, that frontier AI labs want everyone else but not themselves to slow down. They've never said that, they've never said anything like that, not once has anyone pointed me to a quote that could be even plausibly interpreted that way.
Slow down AI progress -> increase catgirl progress!
This is the way!
Secure Connection Failed
Had to open it on the web archiveIf an anti-cat-ears future AI could eternally stress-test their new version of the router in front of this site unless it collaborates, would it predict this outcome and act to avoid the damage?
No I think it's one of my IP range blocks against specific US states in protest of them advocating against people that exist in the same conditions as me backfiring. I'm gonna go nuke that firewall rule.
Amen. This is the only logical way forward.
This whole situation is funny, US gov want the development to stop publicly only, but then open models will catch up soon and these companies are afraid they will lose the market, on the other hand, each company wishes others slow down but they keep pushing the limits, which doesn’t really work in hyper capitalistic market like the US, China all it has to do is just sit back do nothing and win in any given scenario. So they staged the ex anthropic thing and “AI will kill us all!!” in a way to fear monger the public, but reality is, we will run out of resources before any of that will happen.
Honestly not the worst suggestion I’ve seen.
I'm not sure I understand the point made by the author.
https://en.wikipedia.org/wiki/Satire
Everyone thinks about it, but no one actually does it.
Does this mean I'm going to have to wait for the cure for my colitis?