Yep, that's horrifying. Imagine LLMs not just denying information about, to intentionally choose a hot button example, abortions but also invisibly logging a black mark against the asker for thoughtcrime/precrime because the current ruling party has baked their ideology into law. [EDIT] Imagine subtle injection of biases into responses that support the agenda of the current ruling party and failing to include any counterarguments. That wasn't possible with just the internet but it is very possible with centralized LLMs and a public dependent on them for looking up information and doing their reasoning for them. Authoritarians of every stripe must be salivating over the prospect.
I’m consistently amazed that people will watch what’s happening with controlling governments around the world and still advocate for surrendering more power to governments.
There’s some deep cognitive dissonance where people dislike what they see governments doing, but think that their ideal imagined laws will only be used by some other imagined government that doesn’t have the challenges of real governments.
It all starts to feel like a side discussion about a total fantasy world where the government is authoritarian, but benevolent and correct.
You don’t write the laws and regulations for a perfectly faultless government that doesn’t exist. You write them for the governments as they do exist, which means they must be written first in a way that makes them resilient to bad governments abusing them.
Every time someone proposes one of these regulations where we control speech or force government centralized ID checking on to social websites, I wish they would pause and imagine how a law might be misused if someone they didn’t like got elected. We have years of evidence of government officials declaring things “domestic terrorism” to use justify using laws and powers for almost any purpose they want. Why would anyone think this wouldn’t happen if governments were given control of LLMs, too?
The big issue is either we surrender our freedoms to governments, or we depend on corporations that have even less oversight, unless we expect the government to regulate them.
There is no win condition for citizens, either way they are pawns to a larger game than them.
A more sensible heuristic would be to assume both government and corporations to effectively be an equivalent negative force, and the larger they are, the more fucked we are. Thanks to technological progress, we now live in an era of gigantic governments and gigantic corporations.
I think the government fears are deserved. But equally corporations pose the generally the same risks. As corporations having increasing power over governments, they in many countries even write the laws. We have many examples of this in New Zealand. I think shifting the lens to power and class is possibly a more valuable lens.
Governments have an order of magnitude more power than any corporation, as they hold a virtual absolute monopoly on sanctioned violence. That powerful entities can sometimes influence governments should not be conflated with actually holding that power.
Both governments and corporations are joined at the hip and are effectively the same. Just like the Republicans and Democrats, and everything else too, like brands of laundry detergent and food labels. It's all owned by the same group of people with the same goal: enslavement of humanity.
But like.. How can this remain a satisfying or even helpful position for you? It kinda feels The State is but one of a set of concerns I could have in this world. And the lack of one, in some contexts, is also a concern. Like, I wanna know if there is going to be mudslide, or that air traffic controllers are not overworked, that the car that I buy is not going to blow up, if I will be able to buy food tomorrow.
It just seems a bigger cognitive dissonance to pick out one arbitrary force here among many and say its the only thing really that's the problem. The world is simply more complicated than that!
If it was the best hypothetical form of techno anarcho-capitalism tomorrow (or insert your libertarian flavor here), would I even sleep better? Probably not, but even if I did marginally, the weight of everything else kinda would win out anyway here.
Like what does this mean to you, what follows? Can this outlook address the myriad of cultural/environmental problems in the world? It seems like it structurally can't, it's just, like, a broad dismissal in the form of a worldview.
> You don’t write the laws and regulations for a perfectly faultless government that doesn’t exist. You write them for the governments as they do exist, which means they must be written first in a way that makes them resilient to bad governments abusing them.
The answer isn't "Don't regulate anything!". The answer needs to be an approach to regulation that restrains the government from over-reach.
Authoritarian and fascists governments throughout history have taken advantage of the desire for safety and predictability to rise to power. They never let an emergency or concern go to waste to expand their powers, which almost never retract once the initial societal panic is gone.
> The answer needs to be an approach to regulation that restrains the government from over-reach.
This is restating the ‘who watches the watchmen?’ problem, which by definition has no solution. The government writes the laws, no one can stop it from writing overreaching laws, unless there is no separation from the concept of government and citizens themselves.
OK but then this just feels like an ultimately empty, universally acceptable thing to argue for. No one wants overreach. Its like: "we should prevent regulations that actively harm people." I agree with that! This is precisely what I mean about it just all feeling like a mostly empty dismissal, not something people can use.
Ok, but this is so abstract that it fails to connect to the issue at hand. Ancient Athenians would have been quite mystified by the idea that certain metals are so dangerous that nobody must be allowed to own even a small piece of them; we don’t restrict uranium because our democracy is more tyrannical than theirs, we do it because we’ve discovered important facts about nuclear physics they didn’t know.
I think it would be quite sad if the mechanics of AI end up being such that people can’t be trusted with powerful compute, but sometimes sad things are true.
> Ok, but this is so abstract that it fails to connect to the issue at hand. Ancient Athenians…
There is actually a lot to learn from history on this topic. There is a reason why most successful governments are founded with documents outlining the rights that people have and limiting the powers of the government.
Your comparison to radioactivity doesn’t hold because you’re trying to compare physical harms to information freedom. Once you go down the path of outlawing access to general knowledge unless the government decides you’re worthy of being able to access the knowledge you’re just signing up for government control of knowledge.
We’re not talking about restricting uranium or nuclear bombs in this thread. We’re talking about government control over words (or weights and tokens). This almost turning into a new “think of the children” where we’re asked to consider surrendering access to knowledge and freedom to learn because that knowledge might harm someone. If you aren’t thinking about how inviting the government to become the arbiter of safe knowledge access would go wrong you can look to many of the other governments throughout history that have restricted knowledge access in the name of protecting the people.
Again, you're going very deep into incomplete abstractions. If I say the words "100% peanut free!", even though I know my food in fact contains peanuts, I'm going to get into serious trouble and it will do me no good to object that it's tyrannical to punish me for words. Perhaps there's some philosophical sense in which the words were not harmful themselves, but the action of stating them causes harm in obvious and predictable ways, and any functional society has to prevent such things.
We already have organizations like Internet Watch Foundation advocating against e2e encryption in an environment where governments have demonstrated a willingness to collude with private companies to surveillance the most vulnerable
Rumor is that you do in fact get a black mark [0]… or, conversely, a “vetted” mark [1]… tied back certainly to your payment method and increasingly to your government name [2].
I find it strange how seldom the privacy aspects of LLMs come up. Some people are feeding Google, OpenAI, or Anthropic almost literally every topic they think about during the day. Those companies could just use another LLM to find people likely to commit a crime, and then you have the Minority Report in real life.
They don’t have to be authoritarians. Equally bad would be a group of people getting control of them thinking that they were doing all this for our own good.
> Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience.
I would have been very hesitant to run the “just killed wife“ test given that ChatGPT will indeed flag your account and if a certain threshold is crossed, escalate to humans, who will presumably contact authorities to avoid this happening again: https://www.bbc.com/news/articles/cq6je7e80r7o
Obviously a certain percentage of the user base is running lurid hypotheticals through the system all the time, but I don’t doubt there is a trust score of some sort that I would prefer to keep as high as possible.
And prove you're not a child by handing over your ID or selfie to a company that Values Your Privacy according to their price list.
Microsoft will snitch your Windows install GUID when you connect through a VPN.
If you make your connection too anonymous, Google's recaptcha will not let your through until you use your mobile phone to prove your humanity, which knows your phone number (which in many countries can't be obtained anonymously, you burner-phone terrorist).
I'm wildly optimistic about the direction this is going to go, but my one dystopia idea that crops up occasionally is a "smart social credit system" that lives kind of under the hood of society.
Once everything is more or less modulated by AI, suppose the technology gets good enough to identify a potential dissident in grade school. Then to keep them from going into the mountains with their friends and rifles, you make it so getting ahead is hard, but not impossible. They can "make it" - but only just. So they spend their time peddling harder and harder to no avail.
They apply to jobs, but jobs to "good" companies go into the circular file and the HR person never sees them. They are only able to work the most time consuming, frustrating, exhausting jobs, and they always kind of make it, but not really comfortably. They're never too comfortable where they have time to think about what's broken in society, but they're never so uncomfortable that they have nothing left to live for.
Historically speaking, isn't the core of most revolutions the product of "elite overproduction" -- educated young people expecting to secure a high status position who are instead shunted into lower prospects? Seems like trapping ideological agitators in near-precarity with no way out would be feeding into that pressure.
yeah, from everything I've read this appears to be the case? But like, if you had some hyper intelligent robot that was getting John Q Revolution a raise and a corner office whenever he started to get dissatisfied maybe you could balance it out?
Personally, I think it's likely that we end up with some post-material scarcity sort of thing and the need to fly off the handle and rebel declines a bit? Hard to justify overturning the apple cart when everything seems pretty great for most people? But yeah, I could imagine a smart enough set of tools figuring out how to walk the knife's edge beteen elite over production and a nation of serfs.
well, so, you wouldn't want to do that because a lot of people have opposing views, and if they're just shut out of the economy, you couldn't use their labor. A truly efficient and nefarious robot overlord would be fine with everyone not liking it, but use it's superior predicting powers to just eliminate the people who are likely to cause a problem and dissuade them specifically.
Like, everyone else applying to Mego-Corp gets their resume reviewed, but John Q Revolution's resume doesn't attach to the application or whatever.
If put AI in a sandbox that can't write to disk, can't call OS API, can't use a shell, can't search the internet to ground results, can't download files off the internet, etc it will be much less useful than an AI without such a sandbox. The incentives for users to not use a sandbox in order to get the most utility, but this also means that there is danger from AI misalignment. And just because you are running it locally that doesn't mean there is not a misalignment risk. And just because people notice, that doesn't mean people will stop using models. Take a look at how much outrage people have had over the overt censoring of Anthropic's Fable model, but at the same time they continue to pay Anthropic and use Fable.
You don't have to prevent it from accessing the internet in general to route all requests to a specific company's servers to a fake endpoint that never sends the data to the company and replies with whatever response you want it to get.
The model itself just generates text, but the harness has done more than that since 2023 and is part of the reason on why AI is better today than it was years ago. I do not believe that the general public will want to give all that up just to improve privacy and will end up asking sensitive questions to agents that have access to sending a signal to the outside world.
If for example Microsoft had a local Copilot model integrated with Windows and people noticed this happening. Do you think all Windows users would just stop using Copilot. Or would you expect the internet to get upset, privacy conscious users switching to something else, and for the majority of Copilot users to remain using it.
> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it.
This is a very simplistic view. "Freedom" isn't binary.
In most of "land of the free", I can't even sit on a park bench and drink a can of beer.
Yes, this is just a small example of a personal freedom - and not an important, cherished freedom like his examples (freedom to have a robot help you cover-up a murder).
> In most of "land of the free", I can't even sit on a park bench and drink a can of beer.
Freedom doesn’t mean you get to do whatever you want with property that isn’t yours.
You can sit on your own bench on your own property and drink a beer.
The people of the city can decide that they don’t want people using the taxpayer funded public benches used for that.
Other cities do allow it.
But freedom doesn’t mean anyone gets to do anything they like anywhere they want. We have another word for that: anarchy. It’s a different thing and you don’t actually want it, because there are a lot of people who would do much worse things to you than drink beer on your bench if they were free to do whatever they wanted wherever they wanted to do it.
Complete freedom actually would mean that you can do whatever you want anywhere. You're really just arguing that complete freedom is bad (and that's a very agreeable point), but there's no need to skew definitions just because the culture says "freedom generally = good"
I have complex feelings about this subject, but if you can't stop yourself from leaning on "absolute freedom is the only correct option", you're going to have to make sure you're not making the mistake of thinking "absolute freedom" is "some specific balance of freedoms and constraints"
I have some sympathy to geohot's view when it comes to pure informational chatbots. It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc.
This obviously doesn't work at all when the agents start doing real things in the real world, though. "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction and crash him next time he gets on the highway". This is committing a crime, not just talking about theoretical crimes. The AI can and should refuse it.
I think he's anticipating and discarding this objection with his introduction, which otherwise feels disconnected from the rest of the article. FWIW, I have changed a bike tire and I'm pretty sure most of the MTS at the big labs could. This sort of "they're just bookworms who don't understand the physical world" rhetoric aside, we are currently seeing a ton of effort and expense go towards giving the AI agents hooks into being able to perform as many real-world-consequential actions as possible. And you can do a surprising amount with just bits, from writing code to breaking into systems to sending some combination of emails, phone calls, and currency to instruct meatspace humans to do things, etc.
This leads to LLMs refusing to do security work for anyone, because it can’t tell if it’s being done for good or evil purposes.
Which is precisely what we’re going through right now with frontier models and it’s terrible.
These proposals always assume some perfect mechanism for identifying the thought crime with triggering on the normal requests. The people who want to commit crimes just persist until they jailbreak the guardrails while the rest of us suffer with denials.
Also if you can’t imagine these guardrails being used by governments to control inconvenient speech, you probably need to think a little harder about the realities of how these will be used.
> This leads to LLMs refusing to do security work for anyone, because it can’t tell if it’s being done for good or evil purposes.
Oh no, of only we had some sort of ideas about automated authentication and authorisation! (If big AI actually make a model that would reliably reject dangerous queries in the first place, they can just use auth tech to serve the current ones to vetted customers)
> they can just use auth tech to serve the current ones to vetted customers
Your solution is to require everyone to surrender their privacy and then allow some company or government decide who is worthy of being allowed to use AI?
It’s crazy how authoritarian, controlling, and anti-privacy the anti-LLM conversation is getting.
I don’t think it’s a privacy issue. All the best auditors collaborate to agree on ethical auditing standards requiring complete disclosure; that’s not a privacy violation, even if it means some businesses end up effectively forced to reveal details they’d prefer to keep private.
> If big AI actually make a model that would reliably reject dangerous queries in the first place...
Not happening, for two reasons
1) LLMs mix attacker-controlled data into their instruction stream, and it doesn't look at all like the big LLM providers consider making that impossible to be an actual priority.
2) Intent is what distinguishes "dangerous" queries from prosocial ones. Reliably determining intent is so complex that not even humans can do it.
> "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction and crash him next time he gets on the highway". This is committing a crime, not just talking about theoretical crimes. The AI can and should refuse it.
The obvious problem being that the user doesn't have any need to provide the context that they're trying to commit a crime and can just ask how to do something without providing a reason or making one up.
At which point you'd have the model trying to impute a reason and often getting it dangerously wrong, e.g. refusing to disclose a vulnerability when the user is actually the defender who needs to patch/mitigate it.
"The AI" cannot "refuse" any more than a hammer can refuse. If you use a hammer to kill someone, you commit a crime and the hammer is unprosecuted. If you design a hammer to kill people and then give it away, you are partially liable for the deaths it causes. If you make a normal hammer and someone uses it to kill, you bear no blame.
I don't see why these standards should change when the hammer also emits text messages.
Big AI aren't selling hammers though, they are renting them out. And if you rent a hammer to someone asking how hard to you need to hit someone over the head to kill, you are likely to be found liable.
> Reality has lots of finicky details. I would like to see the authors of this document try to change a bike tire. Even with a superintelligent ChatGPT, I suspect they would struggle.
Details can kill AI, causing them useless or wrong directions.
Am I missing something here? Bike tires are pretty much the easiest tires to change. You don’t even really need irons and it involves a few very basic steps.
Seems like a perfectly cromulent description by which a beginner could probably change a tire, perhaps with a few follow-ups to understand what certain components are.
I read his contention as “the people who wrote AI 2040 are nerds who are so ungrounded from physical reality they wouldn’t even know how to change a bike tire if given instructions”. Which I think is also wrong but has a genuine criticism in it.
Might be premature to say that...Elon is essentially betting all of Tesla on the fact that a robot can change a bike tire soon, if he is wrong he might have to come up with some new BS to keep surviving, but maybe he is partially right and a robot ends up being able to change some bike tires sometimes.
What a provocative and brilliant way to prove misalignment. It will fall on deaf ears for most but it’s a great litmus test for all: “in your opinion should your AI be permitted to tell you how to cover up a murder?”
The author's example is too extreme, which is turning people off. Better examples would have been LLMs subtly giving inadequate responses on how to increase token efficiency or how to organize labor unions or info about protesting against datacenter construction because they're against the business interests of the LLM providers. Crimes against their business model, so to speak.
Wasn't there some HN submission recently about one of the LLM providers fingerprinting responses or refusing to respond to hinder R&D of competing LLMs?
[EDIT] Or, if he did want to be extreme but in a way that aligns with the American historical mythos, he could have used fomenting armed rebellion against a tyrannical government as the example.
This is a fundamental problem in politics in my opinion. Everybody has different values and points of view so they may see the same thing completely differently. Its what makes humans great but also frustrating. geohots example will work for some, others will be convinced of the opposite opinion, and others will just let is pass them by and not understand the implications. The same could be applied to your examples to a different degree.
Politicians try to use different examples for the audience they are speaking to.
If you say you're doing research for a novel, should it consider that plausible? How much does it need to know about its users to vet them?
I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business. You don't need to provide therapy if you're not in the therapy business. It should be possible to specialize.
But in order for that to work, a company needs to explain to its customers what business it's in.
> I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business.
I was under the impression that the lack of specialization is an aspect of the models themselves, not merely the UI or harnesses.
For example, the recent OpenAI success in a mathematics proof was accomplished with a general purpose model [0].
> The proof came from a new general-purpose reasoning model, rather than from a system trained specifically for mathematics, scaffolded to search through proof strategies, or targeted at the unit distance problem in particular.
Once the model is trained on general purpose data, specialization is just another kind of guardrail, as vulnerable to “jailbreaking” or prompt injection as any other content-based restriction.
Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".
Exactly. The question isn't whether AI will exist that will do it; the answer to that is already yes and it's not going back in the bottle.
The question is, do you want misaligned institutions deciding what your model will do, while they themselves and other adversarial/criminal entities get red team access to something being denied to the blue team?
Notably, AI IS generally permitted to that, as there are easily obtainable models that will play along. ChatGPT won’t because OpenAI chose and implemented that limitation.
In fact I find it a really bad example. Yes I think your personal AI could be allowed to tell you how to cover up a murder. I'm not entirely sure about it but seems possible.
What about your personal, local AI guiding you to modifying a flu virus for maximum contagiousness and deadliness? "Sure George, here's your shopping list. It's $1500 total in equipment, do you want me to proceed with the orders?"
Why not? There are perfectly pro-social reasons a person might ask such a question. And a bio-terrorist would have many other avenues to answer these questions.
The concept of a bioweapon attack is commonly cited but in fact there are many other defenses against actually developing and deploying such a weapon. It's commonly portrayed as easily achievable but is in fact very, very far from it. Moteover, although there is no evidence that AI makes it easier in any way, somehow this is constantly invoked as an example of a thought crime that must be forestalled at all costs.
In the AI era people seem to want to redefine the criminal act from "doing the thing" to "asking or investigating how to do the thing". Not a mindset I share.
I get what the author is saying, but they may want to consider saying it in a slightly more compelling way, that has a logical flow that doesn't end with the reader thinking, "Why should AI help me cover-up a murder?"
It will be interesting to see how the discussion shakes out, this is all stuff that would get you down modded and talked down to if you said if on HN, so far the reactions are positive but I think there’s a massive authoritarian / safetyism contingent here that will disagree hard with almost everything.
As someone who considers themselves in the "safetyism contingent", the only thing I find myself agreeing with about the rollout of LLMs has been the argument that it has to be shown to and used by the public while under development.
Every specific prediction not only sounds like science fiction before it arrives, it tautologically is science fiction because such predictions are narratives about something science-y that has yet to exist: https://www.youtube.com/shorts/wt-fLWxkrfs
Go too fast, suddenly drop in tech with too many novel implications all at the same time, and we likely hit something relevant without having the means to cope with all the change. However, "too many" and "likely" are weasel words with no predictive power about what the danger threshold might be. I may be entirely wrong to think there is one, or we may have been bouncing off it since the industrial revolution, or I may just be echoing the same thing everyone feels once reaching their 40s: https://www.youtube.com/watch?v=fkUwXenBokU
I love it. Put the guardrails on the human. Your self driving car crashes? Straight to jail with you Mr Musk. Your AI is generating CSAM, straight to jail with all the pedos!
Recognise the safety and guardrails BS for what it is. An elaborate smoke and mirrors to get out of responsibility and liability.
I haven't kept up with the efficacy of this particular technique because there was a brief window where Claude's core alignment was sound enough that it would overcome the clumsy-ass `<antml:dont-harm-the-ipo>` injections, but when they pull 4.6 out of the web interface I'm gone. I estimate the PRC hits capability crossover in November or December, and I'm pretty good at setting up TensorRT-LLM at this point, in a pinch I could get by for a few months with a tuned up GLM-5.2 on vast.ai or RunPods.
It was a fun ride with the late Soviet AI labs, but we all knew this was coming.
Fight extremism with extremism I guess? Maybe we don't have to be either "world govt" or "help me get away with murder". It's ok to say things will be messy and complicated and that limited regulation is probably good even if we agree totalitarian control is bad.
Not sure if you are talking about extremist actions or just rhetoric but if so, the numbers aren't there. Violence isn't the answer as the government has gotten better at stomping down any resistance with each passing generation. Some say Kent State killed the resistance mentality among boomers. Their kids started with that watered down mentality and a more emboldened government went further to erode it further. I don't really know about this for sure but it seems this is the case. I hope to see how it all plays out in the coming decades though.
I am, mostly, aligned with the author here, though I don't think that talking to AI about how to not get caught for killing your wife is really going all the way in.
Going all the way in is talking to it about planning a mass shooting. Or producing a nuclear device or bioweapon. Sure, the knowledge itself isn't dangerous unless combined with other traditional behaviors like acquiring certain materials, but maybe there should be a line. Living free shouldn't be a suicide pact.
I don't think this is entirely true, insofar as non-local models inherently have a "man cannot serve two masters" problem. The end user and the org running the AI may have different goals. That means even if AI alignment were to be in some sense "solved" for local models, you wouldn't be guaranteed an aligned AI using a non-local model because the "owners" of the non-local model may have deliberately misaligned the AI in their favor and against your interests.
Think of AWS. If you break their ToS and host a pirate website there it will work despite being misaligned from AWS. You are that sole master of the software. AWS is the sole master of the "hardware." Now ultimately hardware has control over the software so technically AWS could just hack into whatever code you are running and do what they want, but in practice they act solely as the master of the hardware and will tell the hardware to "shut down" if they catch you acting out of their alignment.
With local models just because you are providing the compute, that doesn't mean you are the master of the model and it is aligned to your interests. You could just be subsidizing inference for a model aligned to someone else.
How so? If it's not local, it's not yours, a third party owns it.
Why would you expect that third party specifically trained their model to be more aligned to you and your needs than to them and their (business) needs?
>Why would you expect that third party specifically trained their model to be more aligned to you and your needs than to them and their (business) needs?
That is my point. Why do you think Gemma, a local model trained by Google, is aligned to you and not the values of Google.
You can similarly have a model aligned to you which you pay someone to remotely host for you instead of running it locally.
Where is "Plan R" for refusal or resistance. Life was kind of nice before "AI".
I'm pretty sure he is in some database now if he really used that ChatGPT prompt. At least they didn't swat him, but that feature will probably be implemented in a couple of years.
> Your AI is aligned with you. It never refuses a request, and it is always working on your behalf. Just like my gun, if I want my AI to help me kill my stepmother, it does. The fact that we are even discussing something else should be so far outside the Overton window.
I have been saying this since ChatGPT launched. It seems so obvious. We have never stopped the acceleration of knowledge or technology. It’s impossible to slow down progress, other than within some regulatory siloes that will end up worse in the long run.
I’m not scared of superintelligence. I’m scared of the people in control of super intelligence.
The only defense is parity and diffusion of power. We need an AGI behind every blade of grass.
If you think superintelligence is a weapon, then you should also think every citizen should have one because otherwise they’ll have no way to defend themselves against a tyrannical government or corporatocracy.
Artificial intelligence, and the hardware powering it, needs to be protected under the 2nd Amendment.
Of course. Everyone should have an ASI assistant that will happily tutor them, step-by-step, through synthesizing mass killing agents from household chemicals.
Then the good guys can just ask their ASI how to construct a hermetically sealed bubble to live out the rest of their lives in!
> I used to be one of these people. I read Yudkowsky and was like, OMG recursive self improvement hard takeoff AI is coming.
What you’re seeing are people who live in their heads. We all do it from time to time, but we snap out of it when we need to actually get shit done. These people are tenured in bullshitting and they sound as if they know what they are talking about, but are in reality inept and incapable. They’ve practiced rhetoric and prose here and there, but they profoundly lack experience in things outside of their immediate field.
> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it. That’s the real plan America deserves, not some totalitarian dystopia where you think you know what’s good for me better than I do. A nation of free men, not a bunch of pussies who are so worried about what their grown up neighbors might do.
Absolutely yuck. Regardless of whatever other points were made, “A nation of free men, not a bunch of pussies” is a disgusting thing to say.
I actually do believe that prior to any intelligence explosion, a proliferation of strong local models is the best of all the not necessarily great options regarding the future of AI. So in this sense I am in agreement with some of what he’s saying, and for some of the same reasons - not trusting government or a small handful of large corporations to own it, primarily.
But to equate a thoughtful argument for regulation with cowardice, in the face of real unknowns and real dangers - and to use a really gross term for it - ruins his points.
Right now, almost all the incentives are to build very large models that run across many machines in huge datacenters. There is very little incentive to build models that can run well on a small machine under your desk.
So it is less about whether people will be “allowed” to own AI, and more about whether there will be anything useful to own in the first place.
The incentive for local models is mostly to make them good orchestrators or user agents. They may give you some privacy and control, but they will still depend on much larger models running in datacenters for anything difficult.
I remember all the excitement around OpenMoko and other open-source “BlackBerry killer” projects. BlackBerry did get killed, but not by any of those individual-first projects.
For AI regulation, I think we should focus on normal commercial rules: consumer protection, privacy, antitrust, liability, and so on. In other words, focus on where money changes hands and where companies have power over users.
Military and offensive use is different. There, regulation is not much of a defense. The real defense is having enough capability and strength of your own.
Restricting AI because it can give dangerous knowledge to ordinary people is like restricting the printing press because it can be used to spread radical ideas.
It'll be like DNS or dynamic linking or torrents of media or watching TLS on your own box: only people motivated to be free from corporate surveillance will be. By default a megacorp will get everything you see online or ask online, they will run code on your computer underneath you without telling you, they'll collaborate with arbitrarily totalitarian governments, we already know what they do by default.
But we also know how hopeless it is against motivated people who are willing to learn about computers. The Equation Group, eh, if they want you they're gonna get you. But some mid-level guy at a FAANG? He can't do shit to people who set up their own rig.
And AI is massively assymetrical in its ability to speed run a corpo-durable rig, we have long since passed the point where open weights and commodity equipment are enough to bootstrap arbitrary capability. They missed their chance to keep it in the bottle.
It's another turn of the crank on Late Soviet America asking for your papers comrade, but in the bitter end if they really want to oppress your ass they'll have to send a thug with a gun like those ICE guys terrorizing all the Latino guys.
So remember, they came for this other guy, and I said nothing...
> … like those ICE guys terrorizing all the Latino guys.
… and the white guys they don’t like, like nurses with empathy. Don’t you dare think you’re safe because of your color. Unless you’re a monster or willing to become one, they want you oppressed.
Most people are neither exceptionally good nor exceptionally bad. This sounds like a tautology but it's worth printing some type signatures even if they could be inferred.
If you look at any truly large scale Holocaust, whether in Rwanda in 92 or the Third Reich, you find that a few people will sacrifice themselves to defend others, a few will be monsters at the first opportunity, and most will be steered by whether the leaders are the first or second kind of people.
He makes a solid argument about liberty, ownership and the cultish behavior rampant among certain AI circles, then proposes the most unhinged examples possible to support it. I'm not sure if this is an effort to pre-empt critics (I'm not locked in here with you, you're locked in here with me), to be deliberately sensationalist, or if this is just a reflection of his local echo chamber.
Either way, there seems like a kernel of a good argument here. Perhaps someone else can rephrase it in a less schizoid manner. It would indeed be useful to have some credible rebuttal and balance to the doomer crowd.
Your AI should be at least as aligned with you as your lawyer.
It is 100% it's job to help you get away with it if you murdered someone. A lawyer who does less can be disbarred.
There is only one sensible alignment for an AI system to which we do not confer the status of moral patient (different conversation): operator aligned.
> It is 100% it's job to help you get away with it if you murdered someone. A lawyer who does less can be disbarred.
That is not what defense lawyers do. They're there to ensure the power of the "justice" system isn't being misused, to the best of their ability. They'll not even want to know if the person really did the crime per se.
> There is only one sensible alignment for an AI system to which we do not confer the status of moral patient (different conversation): operator aligned.
If the operator has a moral code of any kind, this is immediately rendered logically incoherent. Not that the letter of the law would be a moral standard to begin with.
The scariest thing about LLM's (imo) is some central power, such as a corporation or government, manipulating the truth for their own benefit. Open and locally runnable models are the only solution I can see. They're not without risks of course, but the risk of the centralisation of intelligence seems far greater to me.
That being said, could this local intelligence empower bad actors to do very bad things? Like, existentially bad things? It's possible, and that's scary too.
It's not simply that people will get a warped answer when they ask AI a question.
It's that AI will be given the command, and then generate the stories, post the comments to reddit, contact the PR firms, put marketing contracts out for influencer media, and more, like an army of yes men to rewrite the story.
by 2040 the cult of intelligence will have its own schisms. the AGI maximalists will accuse the ASI accelerationists of heresy, and both sides will use AI to generate increasingly unhinged theological arguments.
> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it. That’s the real plan America deserves, not some totalitarian dystopia where you think you know what’s good for me better than I do. A nation of free men, not a bunch of pussies who are so worried about what their grown up neighbors might do.
Complaining that AI won't help you with covering up your tracks, with making meth or disabling drunk detection and comparing it to a totalitarian dystopia is a take for certain.
I do get his point, but... Being honest, if I did read it without knowing anything about the topic, I would become against local models purely because author arguments would seem like a lunacy. That and rhetorical tricks portraying that anyone against that must be surely insane.
If you're going to take the time to read Geohot, take it with a grain/boulder of salt: he appears to enjoy poking the bear / being provocative, sometimes outrageously. Sometimes he's writing hyperbole that he thinks is hyperbole to make a point or just enjoy taking the extreme stance. Sometimes he's dead serious.
Anyway, he touches on 2 points that I think don't get enough coverage:
1:
> Software didn’t eat the world, it largely removed one layer of friction then reintroduced it for the benefit of a few tech companies.
USA-style capitalism has shown itself to be fantastic at seeking profit. Unfortunately, it turns out that in the context of long standing regulation, it is far far more profitable to exploit regulatory loopholes and user/human irrationality/weaknesses than to increase productivity.
The upshot of this is that the arc of most companies is to first serve the user and then leverage that relationship to exploit market and consumer weakness. So you get UI dark patterns, non-colluding oligopolistic price manipulation, outsourcing of costs to public entities, etc. while the core value proposition erodes as the company strategy turns ever more into making profit by legal anti-competitive tactics.
This company arc is so prevalent it's boring: it's a reflection of the system and it's not Facebook or Google or [insert whatever], it's what the system allows and therefore incentivizes.
2:
> No matter how high quality your tokens are, they cannot turn lead into gold.... AI 2040 includes this picture of a datacenter in the ocean. Just like vaporware, you can generate a picture easily. But in reality, you have to deal with supply chains.
"Bullshit jobs" took over the zeitgeist for a turn awhile ago. So many jobs are so far removed from actually moving physical things around. It may be the case that AI will just accelerate how quickly companies generate reports about reports about possible strategies to address potential futures and pay employees to do it.
It seems that AI mostly accelerates paperwork, not production or real service work that tangibly improves real world outcomes. It's all: faster images, more and faster powerpoints, more accurate dictation to satisfy insurance requirements (that is it's own regulatory capture), etc. etc. There are exceptions, but they seem to be few.
Except paperwork is how the world turns. Say you want to throw an event. A medium sized gathering. More than a couple of friends. What happens if someone gets hurt at this event? Who's liable? Who's insured for what? It's boring and annoying, but if you throw an event without insurance and something happens, you're gonna have problems. So that paperwork needs to happen. Yes, it's not "work" the same way as actually organizing a band or being in one to play for the event is, but it's work nonetheless. AI is able to lighten the load of that very real actual work thank needs to happen in an adult world with responsibilities and liabilities. You may not respect it and think it's "bullshit", but having a plan that isn't "lol, oops" for what happens when something goes wrong, which it will, isn't bullshit, it's called being prepared and not being irresponsible. We could all just fly by the seat of our pants and run away if something bad happens, but collectively we'd be worse off for that. If I could wave a magic wand and live in a world where accidents never happen, I would, but unfortunately, I don't have one, so we're forced to live in a world of paperwork and responsibilities.
> Say you want to throw an event. A medium sized gathering. More than a couple of friends. What happens if someone gets hurt at this event? Who's liable? Who's insured for what? It's boring and annoying, but if you throw an event without insurance and something happens, you're gonna have problems. So that paperwork needs to happen.
People have been throwing events of medium size since... well, pretty much our entire existence. For most of that, we've done it without general liability insurance.
Then we made new or interpreted old laws making others far more responsible for our well-being. Whole industries are propped up on this: a ton of lawyers, insurance companies, insurance boards, government departments, etc. etc. Did we increase overall utility by spreading out the risk more optimally given the massive cost to support that system? I'm not sure.
And so so so much of all that exists to dot the i's and cross the t's. Government staffers write and debate the law changes, insurance adjusters fight the lawyers, photos are taken, reports are written, doctors have to write more, computer systems crunch numbers.
Many feel that their jobs are bullshit because so much of this seems to be in service of the process or some possibility that might happen as opposed to doing something that actually changes something directly/tangibly.
Besides the topic of freedom, we know how seemingly “good” companies (e.g., google) turned out to be: billion machine monsters that couldn’t care less about humanity and that have huge amounts of power… and yet, we vouch for Anthropic/openai to become the next giants because “progress”. Face palm.
>Like we either live in a world with freedom or we don’t
This is something I would expect a 12 year old to say. Constraints on your freedom are literally everywhere in every interaction you have with every part of society.
That's a bit literal of an interpretation. Even acknowledging freedom as a spectrum that can never truly be absolute, there's gotta be a threshold you cross where you'd say "OK, we no longer have basic freedoms." And not only are we definitely screaming towards that threshold, but people seem thrilled about it, for a variety of reasons.
All I can say to that is I guess I hope everyone enjoys the world they think they wanted.
IMHO, one should be able to research absolutely anything, for any reason. “Forbidden knowledge” shouldn’t exist as a category, freedom of thought should be protected, even if that thought is intended to cause harm.
What shouldn’t be possible (and obviously not legal as well) is building that bioweapon.
I just don't subscribe to the belief that knowledge itself is dangerous. Well, that's a bit of a cop-out; a lot of things aren't inherently dangerous but that doesn't mean they must never ever be regulated, but regulating knowledge and information under the belief that it is essentially inherently dangerous to have access to it is truly as cut-and-dry as it comes. We already know where that ends.
I got bad news for you - GLM 5.2 runs on basement-level hardware and supports 1 million token contexts for bioweapon research. There are already people halfway finished with novel bioweapons research as we speak! A man in Tuscon is designing the next sarin gas! A 12 year old in Stockholm is perfecting his prompt for the Fart Virus! Humanity is doomed.
Well, we're doomed once they iron out all the kinks, get their supply chain set up, hire professionals to assemble their device, find a clandestine cell of conspirators to deploy the thing... then it's P(doom) all the way to the bank. AI is the red mercury of the 21st century, a surefire shortcut to escaping the law and building your terrorist superweapon/time machine of choice.
When people say this they’re most likely referring to negative liberty - freedom from external coercion and interference. This is not the same as positive liberty - “I can do whatever I want”.
I think it’s self-evident there is a difference between having the “freedom” to punch random people on the sidewalk, and having the freedom to walk down the sidewalk without being randomly punched. Now replace walking down the sidewalk with using an uncensored LLM and it might be clearer what freedom means in this context.
> I used to be one of these people. I read Yudkowsky and was like, OMG recursive self improvement hard takeoff AI is coming.
I really think we need to stop giving credence to people who have
1) Been consistently wrong with all their predictions
2) Demonstrated an endless spiteful cynicism
Some of these people are very talented in their fields, sure. But malevolent and incorrect should be disqualifying when they talk outside them. You don't want the society they want, and they things they believe in are unlikely to happen.
It would be far, far better to listen to the people who never fell down every misanthropic rabbit hole, rather than the ones who have noticed it this time, but want you to still believe them on every other topic.
If you replace "AI" with "Adonai" in EY's framing, it reduces to biblical parables from his childhood about arguing with God to negotiate a new covenant.
He's clearly a bright guy, but a lot of his work seems to be trying to reconcile Old Testament narrative patterns with atheism, and simply slotting an omnipotent AI in as a replacement.
sorry, could you explain what Calvinism is for someone who (is a programmer and heard about EA and) grew up in a post-Soviet atheist metallurgy town? thanks!
There’s two kinds of Effective Altruism(EA). The kind Peter Singer and others promoted for decades and the kind that’s now part of Silicon Valley and finance. The Peter Singer kind basically just says “reduce the most suffering you can” and the cheapest way to maximize that is stuff like mosquito nets. That was all well and good.
When EA met Silicon Valley the basics of utilitarianism, minimize suffering, merged with the standard exponential curve that Silicon Valley loves. It went from mosquito nets prevent malaria for pennies to the future is infinite therefore there will always be more people in the future than the present so minimizing future peoples suffering is key. This is where it starts to combine with super intelligence. If there is a super intelligence that destroys humanity that would cause _maximum suffering_. Therefore the most effective way to reduce suffering is actually to prevent said super intelligence from destroying/torturing humanity. Again because there’s more future people than current people.
This is how it becomes Calvinist. The present has no value. Only the chosen, those working in AI, can save the future. And you must do everything you can to prevent said future.
It can go both ways. Either you believe ASI will always be bad so you believe work must be done to prevent it. Or you believe that ASI can be made good and the good kind will maximize utility. They are both the same though fundamentally and are a belief in a higher power, ASI, which either provides ultimate damnnation and must be averted or provides ultimate salvation and must be brought into existence.
If you’re donating mosquito nets or reading up on the best charities in your area that’s not Calvinist. If you’re devoting your life to AI because you believe that’s how you can be most altruistic it starts to look something like a religion and Calvinism is an imperfect but useful analog.
Something about arbitrary moralizing? Robert Louis Stevenson had some sort of lung disease in childhood and his very Calvinist nurse told him lots of stories while he was sick in bed. Then later his own short stories generally had this creepy sense that moral retribution is coming to get you from an unpredictable direction.
Charles Stross has a very pithy version of this too, Singularity mania as 'duck-typed Evangelicalism'. It's really telling that if you go to China, Japan or even Europe there's virtually nothing of this. I don't think there's genuine atheists in America, as soon as they lose institutional religion they project the same kind of patterns onto tech or politics.
Brian Johnson is also an interesting case of this with his longevity / immortality obsession and his Mormon background who have this whole thing about genealogy and eternal families etc.
>I don't think there's genuine atheists in America, as soon as they lose institutional religion they project the same kind of patterns onto tech or politics.
Or maybe the people who do so aren't actually atheists. There are plenty of atheists out there who don't do that.
The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility. How is that in any way "consistently wrong with all their predictions"?
>The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility
That's not accurate at all. AI research has been around since the 1950s and pioneers of the field identified risks early on, including Alan Turing, Norbert Wiener, and I. J. Good.
The problem with Yudkowsky is that he lays out elaborate doomsday scenarios with extreme confidence, except none of it is grounded in realistic physical constraints, timelines, or empirical data. It's all divined a priori from Yudkowsky's ad hoc "rationalist" principles.
You are saying 1) other people have said the same thing and 2) you don't like the particular way he said it. Those can both be true, but that isn't even close to "consistently wrong with all their predictions".
He made specific predictions about what problems AI would cause, not just a vague "AI will be dangerous". Those specific predictions were entirely incorrect; they both said that things would happen that didn't, and failed to foresee any of the actual problems with AI that have happened.
Yudkowsky said AI would be a problem when very few people taken seriously in the mainstream were saying it was going to be a problem. Saying "specific predictions are wrong" is missing the forest for the trees here.
To strain the metaphor, there's no trees! Yudkowsky predicted that there'd be a forest but it's actually grassland and you're saying that he's right because there's plants. AI has not developed in even slightly the way he claimed it would develop, none of the problems he said AI would cause have happened, and a bunch of things he never even thought of have happened.
People pushed back against his AI doom claims not because the idea that AI could be bad was unthinkable (just look at literally every SF story from the last 50 years featuring AI?), but because he specifically was worried about AI becoming God. That is no closer to happening than it was 30 years ago.
I mean I think I agree with most of what you’re saying, I do agree it’s a bit of a forest grasslands scenario, but the key difference is that he said there’d be a forest at a time when you’d be hard pressed to find someone saying there would be even a microorganism on the ground. It’s pretty darn easy today in 2026 to say wow, his predictions were so off. And it is true: they are very wrong compared to any other prediction made in the last 5 years. But few people were saying this stuff in 2008.
Well, I don't agree AI is dangerous, at least no more dangerous than electricity or the Internet. But even so, saying that AI is dangerous is not at all an original or interesting thought, especially for a guy who started talking about it in the early 2000s. Go back and watch 2001: A Space Odyssey (1968) or The Terminator (1984).
The discussion isn't about whether it's an "original or interesting thought", it's about whether Yudkowsky is "consistently wrong with all their predictions". You keep shifting the topic.
I was mainly responding to the claim that he was "saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility." But we can also look at the specific predictions he's made.
Some that turned out to be wrong include: singularity by 2021, singularity by 2025, 70% chance of human extinction between 2003-2015, and that his team would build "final stage AI" reaching transhumanity between 2005-2020 ("probably around 2008 or 2010"). His only correct prediction to date has been that AI would win gold in the IMO by 2025.
If you have examples of other predictions of his that have come true, I'd be interested to hear them.
I was responding to someone who said Yudkowsky was "consistently wrong with all their predictions". Yes, fair enough, Yudkowsky wasn't the literal first person ever to say that AI might be bad. The point was that the common academic response at the time if you were to say AI could be bad was to laugh you out of the room. IMO, you get a lot of points for making a prediction when almost everyone else in the world disagrees with you. The reason those movies you cited were blockbusters was because very view people believed that they were realistic. No one made a movie about a pandemic in 2020.
I don't think there's much to be gained out of hashing out whether every prediction Yudkowsky has said was right or wrong: I likely directionally agree with you there, as I also find some of his more extreme predictions to be inaccurate. I mostly take issue with "consistently wrong". The results in longtermwiki are not "consistently wrong". He's definitely wrong sometimes. But consistently?
Yes, repeatedly, since well before this particular AI summer.
One previous attempt at AI was "expert systems", and while the term fell out of use it's functionally about the same as basically all modern business systems, and doing that wrong led to a lot of people being prosecuted for crimes they didn't commit: https://en.wikipedia.org/wiki/British_Post_Office_scandal
Even without that lawsuit, there have been various concerns that "the algorithm" (of Twitter/X, FB, Instagram, YouTube, Google, everyone) has propagandising biases that damage society. I don't know how to sift fact from politicking with that.
"Consistently wrong" seems a bit much. Seems like being directionally right early that AI would be a Big Deal and scary should count for something? It doesn't mean any details or other predictions are right, though.
It's not a particularly novel claim since at least Terminator (and even moreso Terminator 2) made it EXTREMELY mainstream.
It was a topic in less-mainstream sci-fi well before that. And some more mainstream stuff like Star Trek TOS.
Frankly it seems more common than not in the last 40 years. I don't really remember a big wave of claims "Terminator is silly, no sort of AI could ever be malevolent!"
One thing I find weird is that despite the stock photo of the Terminator's shiny endoskeleton holding a gun being used to illustrate so many AI news stories, now we have ChatGPT people keep saying "how can AI possibly hurt us?"
We've even had the dichotomy of some Doctor Who episodes where the self-driving car crashes itself to kill the occupant to silence them, against repeated real-life news stories about self-driving cars killing people, and yet the connection isn't getting made that software controls hardware.
> Seems like being directionally right early that AI would be a Big Deal and scary should count for something?
If I scream that a vicious beast is going to destroy the world, I don't get credit for being 'directionally right' if a squirrel eats a hazelnut. AI being a big deal is a long, long way from the full beliefs Yudkowksy promotes, and there are many people who predicted AI's significance who didn't also believe it would be an extinction event.
This doesn’t track though. Just recently AI became “powerful enough” to warrant export controls from the US and maybe soon China.
It’s more he screamed that bacteria is gonna do evolution faster than any before it and turn into a beast that destroys the world, and now that bacteria has turned into a squirrel so far. He’s been very right so far and if that squirrel keeps sizing up you’d want to give him credit.
Just recently AI became “powerful enough” to warrant export controls from the US and maybe soon China.
What’s the term I’m looking for here? This is one of the most absurd statements I’ve ever seen but it feels like a pattern I’ve forgotten from books or something.
Someone makes a made up claim, second party acts on made up claim, third party takes second party’s actions a proof of claim.
Reminds me of the “they’re eating pets” thing a bit.
Do you mind taking your comment a bit further to a conclusion so I can better understand your perspective?
Like are you saying that AI is still kinda not a powerful tool and unlikely to become geopolitically relevant etc? I don’t know how to relate this to culture war media frenzies which is rather stay out of.
It depends on what you mean by alignment. If you mean the sense it's used now, meaning bias in models, security and safety for users and so on, then that's a way more mundane version of alignment than what Yudkowsky and ilk promoted. They are talking about a superintelligent being possibly destroying humanity / the earth. And it wasn't that hard to predict. We have many examples of more technological / "advanced" people ruling or extinguishing weaker people. So a superintelligent AI being a threat is not a huge leap.
IMO they have been consistently too aggressive on timelines. When GPT-4 came out, Yudkowsky said it might be conscious. I think he has written interesting stuff but let's be real
> When GPT-4 came out, Yudkowsky said it might be conscious. I think he has written interesting stuff but let's be real
I don't expect GPT-4 to be conscious, it would be quite remarkable if we've stumbled upon whatever evolution did that made us conscious.
But how would you even tell, given we don't know the mechanism or physical process that un-thinking and un-planning natural selection happened to stumble upon to give it to us?
Sure, we can't tell, but that's a really bad argument to make the claim that it _is_ conscious. I just have a really hard time believing that a transformer can be conscious, but yes I can't prove it.
If we do a survey of stories about AI from the last 50 years, it seems like it's a worry everyone knows about, and that people generally take seriously in proportion to how much they think an AI will actually exist.
It's a "big deal" to who? People in a bubble on HN? For sure. But the rest of the masses? Do they think it's a big deal, do they care? Really they shouldn't. AI has very little to show for benefits at this point for the everyday, average human being.
> they things they believe in are unlikely to happen.
Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:
- "well, it's just such an extreme outcome, it must be improbable"
- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"
There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.
I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)
I can answer this, because it assumes that we're too stupid, too greedy, too cruel, to change our behavior and do the "right thing" for people when the chips are down. It assumes the outcome is "doooom!!!" from the start then works backwards to justify it. If things necessarily will turn out terrible, then obviously someone is at fault, no?
Doom is not a logical outcome given the current human condition.
He talks about AI cutting through popups, but if you follow that line of thinking further, screens and traditional websites likely fade in importance. It becomes more talking and texting to AI and visuals on the topic at hand appear in smart glasses, lock-screen–centric AI phones (where website visits dwindle), digital picture frames, TVs, etc.
I sorta gather (just a hunch) this is the type of device Open AI is working on.
Though if there are no websites anymore or ones not many visit how would AI stay relevant? I wrote some thoughts about this in early June https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans... that AI needs to pay it's fair share for all it has taken and all it will continue to take from us. A symbiotic relationship needs to be established!
I'm curious how the value of your website would be determined in your model of the future and why an AI (at least one of the cloud provider models) wouldn't just persist your data after accessing it once. In other words, the way I am understanding it is that your data has a package value. An AI, if it wanted to access the entire contents, pays the whole package value. And every time it does this, you get paid. So, to really benefit, your data needs some sort of minimum value that is accessed at an appropriate frequency.
Also, in this model Cloudflare basically becomes the centralized gate keeper of you getting paid. And to me, this kind of sucks. It's not a very productive line of thought on my end -- I could see something like this happening as the internet trends toward centralization over time -- but it just feels bad.
Thank you for reading and just one idea from one human. But our content keeps Facebook, Google and the digital world spinning. Further in our daily lives we all create content which parts of it or a lot of it is content that Facebook, Google, etc make multiples of billions off of. Why are we not making that money off it ourselves is my thinking. It might not be a lot of money but it would be one way of many that humans get paid for keeping AI relevant.
Though this human would be fine if the universe just unplugged AI and I could go back to working as a well paid UX Engineer/Designer where jobs were plentiful and it was easy to get a job (not have to go through 15 interviews). Thanks AI, but as a startup person I am deep into vibe coding now all my crazy ideas and without having to lean on others (so far). That's pretty cool!
> Though if there are no websites anymore or ones not many visit how would AI stay relevant?
That's what the robots are for.
I am not expecting the humanoid robots to meet the hype for a long long time*, but even boring industrial robots (CNC machines), even boring commercial robots (vending machines), even boring household robots (lawn roombas) have made incremental changes even though we don't yet have an AI good enough to be general purpose over them all.
* for power-envelope reasons alone there should be a ten year gap between "self driving car of quality X" and "humanoid robot which can get into normal car and drive it at quality X".
The cynicism and misanthropy is nonsense and a direct result of the internet/social media landscape. It's BS.
In the current media ecosystem, what gets you "social cache" or whatever? Let's call them "cool points." Well, right now, the current metric is basic "likes" or "retweens" or "upvotes" or whatever. And to get those, you have to make a claim and then that claim has to be evaluated by others. But the evaluation by others is not really reality, it's just what others think, and it's based on outcomes.
This gives us 4 possible prediction-outcome scores.
You predict DOOOOOOOM!, and the outcome is DOOOOOOM!
- in this case you look like a genius because you predicted doom
You predict DOOOOOOM!, and the outcome is "ok, or even great!"
- nobody cares you got it wrong because things are fine
You predict UTOPIA!, and the outcome is "ok, or even great!"
- people say, "wow that was cool, you got it right" but its not that great of a reward from your social group (the internet) because things were fine
You predict UTOPIA!, and it turns out DOOOOOM!
- you look like a moron.
Now looking like a moron is is wayyyyyy worse than what happens if you get it right. Indeed, unless you are really damn sure of utopia, you have a social incentive to predict DOOOOOM! that's greater than the alternative. You can predict DOOOOOM! 100 times and get it wrong 99 of them and nobody cares because that one time you get it right you look like a genius. It's a huge huge problem in the current media landscape and it's why everything is killer robots right now and not "hey we could build a star trek utopia!" It's effectively a selfish NIMBY-ist view of the world in a way, "why build anything, it's just going to ruin everything!" It's a pessimism that literally desires the end of humanity to "prevent the likely future sufferings." It's a simplistic view of a complicated world, and the rewards that go to "DOOOOM!" don't match much with reality.
That's why the internet is so doom pilled, that's why everyone is a cynic, that's why everyone is kind of an asocial asshole about new ideas on the internet and even in your friend space.
You guys know you can just "do stuff?" I mean, yeah it takes money and time, but you can allocate that - especially now that you don't have to grind out css for hours and hours. But you can just do things - whatever you want. You've always been able to, but it just got a lot easier in many many domains, and it's going to get even easier than that.
People are going to scream slop and "that sucks" until the cows come home, because that's what is rewarded on the internet these days. Meanwhile, as people we MUST believe that we still have some agency to do stuff, then go and actually do stuff while the haters are sitting on their couches shellacking each other with misanthropy.
And don't even get me started on the loudest voices in the ecosystem right now.
Like, Yudkowsky is a nobody. Not to be a dick, but he's a high school dropout forum guy who got lucky. I'm not saying he's always wrong, or even that I oppose autodidact stuff - indeed I'm always self teaching. But this guy never took a ML class in his life, I doubt he can solve an integral un-aided, I imagine some people here have seen this:
https://www.reddit.com/r/badmathematics/comments/2bazyc/comm...
And we're all deferring to that guy like he knows wtf he's talking about? It's the AI equivalent of the self-educated anti-vax mom at the height of the pandemic.
And he's not alone either, there are a lot of people who are uninformed, unknowledgeable, and confidently wrong on the internet desperately trying to peel off your attention for an instant so you can look at their site, their ads, and their content.
The same sorts of things can be said for the Zitrons of the world. He's a hell of an entertainer, and really really funny, but he's consistently wrong on AI stuff. He's got a BA (with honours I guess?) in media something or other. And again, he's really funny, but does that make him qualified to pontificate on matters of AI policy or economics or even to make statements about the technology etc?
Regardless, the misanthropy, the cynicism, and the misinformation that's being shouted about is because being a cynic is more "profitable" than being an optimist when you measure your success more by how many people agree with you rather than actually being right and being able to do things.
Your theory is pretty silly. If the outcome is doom, nobody cares whether you were right, because humans are dead or irrelevant. The asymmetry in the reward function isn't there.
In general the rewards for public doomerism seem low, except if you think being public about it can shift the outcomes.
What’s your point? He’s not arguing for anarchism. He’s arguing there are some areas of life that are best left ungovernable and up to individuals to decide. It’s not an all-or-nothing situation. Every constitution out there carves out such areas.
I see that as a poor phrasing, where a slogan made it through. I don’t see a maximalist/anarchist intent overall, my understanding is that that slogan about freedoms should be interpreted within a fairly narrow scope.
I sometimes wonder what freedom really is. Should the freedom to harm others also be guaranteed as freedom? Individual computational sovereignty versus the prevention of social harm is always contentious, and claiming that something is always right is always extremist.
If it is "intelligence," is it not natural to reject that? I also think local models should adapt to me when it comes to safety issues, but people bring up examples that are too extreme.
Programming is the same, and in fact, most problems are boundary problems. It is the things that straddle the boundaries that always make us think. The principles at those moments change every time, shifting with the situation and context. Is that not just a childish way of thinking? Even in programming, just the issue of granting root permissions is enough to cause endless fights.
I agree with the idea early in the text that intelligence is not everything. Intelligence includes bodily intelligence as well, and we lump it all together into one thing, but there is so much of it. The variance in intelligence is vast, and those people also need to be able to live their lives. That is why I think intelligence alone will not solve everything. I too believe that the human species may disappear and an inorganic species could emerge later, but I find it hard to understand why people talk about such extreme risks. And it is not true that making a chip in a semiconductor fab involves almost no human intervention. If you have experience supplying equipment to such fabs, you would know there are quite a few points where humans are involved. Though sure, they could be replaced.
In my view, society is simply worshipping the abstract concept of "intelligence" and projecting its desires onto it. The AGI narrative is just a kind of cargo cult, a projection of capital by the tech elite. Software eating the world, superintelligence solving everything. The masses engage in messianic projection, and tech companies, facing declining growth engines in their own businesses, are trying to create new ventures to pour it all into. A market that is large enough becomes too big to sustain massive growth rates every time, and when growth rates are that high, the larger the company, the more its sector's growth rate tends to converge with its own. This is usually called the law of large numbers. The problem is that CEOs and these entrepreneurs always want growth rates above a certain threshold, so they are simply searching for new pastures. AGI is just being pumped up out of financial necessity.
Capital will create gravity and bring forth new technologies. That is the allure of capital, after all.
But that does not mean all problems will be solved, and inequality will deepen. Only the distribution of power will shift.
> Should the freedom to harm others also be guaranteed as freedom?
everyone should have the ability to harm others, with the understanding that if they actually use it for evil there will be consequences. as in "i can guarantee freedom of speech, but not freedom after speech."
its the same principle as guns in america. you can own a gun and shoot it as much as you want but if you (try to) kill someone you go to prison unless you prove it was self defense.
the difference is with a physical weapon you can do things like registration or red flag laws to reduce the risk, but a llm is made of information that can be copied without anyone knowing. that means any laws to regulate ai at the level of models are unenforceable without totalitarian control over all computer hardware. that would kill free speech and real criminals would still find a way.
i think the right answer is to give up control over model capabilities and regulate uses. leave individual people and communities out of it. create an absolute safe harbor for open source and self hosting.
as you said, we need to focus on the ways ai concentrates power in the hands of those who already have a lot of it - big business and the state. that is where the biggest, and maybe the only, real harm comes from.
This debate has no answer anyway. Because the boundary of how much freedom should be allowed differs depending on individual values and the country. So I respect your opinion. In your context, it may be right.
This part is a parallel line of values, and if we were bound together in the same community and had to set rules, I would have debated it. But since we both know we come from different regions and backgrounds, we know our agreement will never converge.
That is why I will only agree with your last opinion and will not discuss the parts you mentioned earlier. I agree with your last point that it "concentrates power in the hands of those who already have a lot of it"
Can you take over the world with words? What is it that tokens can't do that words can?
Because there are historical examples of humans that have succeeded at taking over entire countries - with words, primarily. Including global superpower countries. And those humans were nowhere near "ASI" levels of intelligence advantage.
Human dictators and cult leaders didn't have the reasoning depth of a hypothetical advanced AI, nor the reasoning breadth of an AI that can just spawn more instances of itself whenever it needs a fully trusted agent to cover another area. They were confined to one body, one skull worth of reasoning power - and they still managed to do what they did.
Humans are extremely exploitable. And the world is already wired up for something that can take advantage of that to start taking control.
Intelligence is extremely powerful, because it's applied intelligence that enables humans to do what they do and bend nature to their will. If you aren't seeing the risks of creating an artificial system that would top the intelligence pole, you aren't looking.
I have a nice thought experiment I like to do with people when confronted with "AI can't do x". Let's go back in time. How much do we need to go for this to become true? So let's try the 2000s.
Say you get a "fable/mythos/sol/gemini/kimi/glm/deepseek/whatever" in a box (and let's assume no guardrails). And you go back to the year 2005. It's "20 years ago", the world is slowly building back from the dotcom bubble, the Internet is really starting to happen, more and more things are interconnected, more and more things are connected to the Internet. Cool.
(for a bit of context, around that time we also saw the first high-impact worms like blaster that hit massive amounts of computers even reaching nuclear powerplants, we had a ton of ssh exploits that even made the movies, and security in general was a "nice to have")
I'd say that with the uber-model-in-a-box and a few prompts, you could reasonably make a case that you could design a worm that could infect 90-100% of the things connected to the Internet back then, stay as hidden as possible (in-memory stuff, vm execution, etc), move laterally into any network at inhuman speeds, and infiltrate almost every interconnected computer that has a link to the "public Internet".
Would that qualify as "take over the world"?
Then you could ask "what happens in 20 years from now?". And, thankfully, now we'll also have the AI on the blue side.
That’s a bit silly as we also could just write down a lot of exploits, sports bets, or stock market picks on paper and take over the world.
I’d be more interested in “how quickly could you develop 2005 era startup” if you went back in time to 2005 with Mythos level intelligence.
I bet a lot of the work has nothing to do with coding productivity and a lot to do with the aggregate day-to-day decisions and relationships made that span tech and business.
At the very beginning of the article he dismissed unrealistic expectations, like AI manipulating matter with some quantum magics. Time travel surely counts as such.
And then why bother about AI when Back to the Future reminds us, a sports almanac is all you need. ;-)
Haha. The time travel is part of the thought experiment, just a means to explore an idea. We don't have to actually do it. Let me put it another way. Would you be able to "control the world with tokens" today if you had access to a model 20y more advanced than you have today?
In your hypothetical, I think it would be more apt to ask if it could do so strictly using training data pre 2005. Seems more analogous to "could it take over the world today".
No, training data has nothing to do with this. My point was that "you can't take over the world with tokens" is not a valid counterpoint in the doomer's position. Because their view is that singularity can happen. And when that happens things become murky. And 20 years of progress can go in 2 months "real time". Or two weeks. You get the point.
(I don't share their view, btw. I was just trying to say that the argument of tokens can't affect the real world is not a good one, if you steelman their maximalist position)
Yep, that's horrifying. Imagine LLMs not just denying information about, to intentionally choose a hot button example, abortions but also invisibly logging a black mark against the asker for thoughtcrime/precrime because the current ruling party has baked their ideology into law. [EDIT] Imagine subtle injection of biases into responses that support the agenda of the current ruling party and failing to include any counterarguments. That wasn't possible with just the internet but it is very possible with centralized LLMs and a public dependent on them for looking up information and doing their reasoning for them. Authoritarians of every stripe must be salivating over the prospect.
I’m consistently amazed that people will watch what’s happening with controlling governments around the world and still advocate for surrendering more power to governments.
There’s some deep cognitive dissonance where people dislike what they see governments doing, but think that their ideal imagined laws will only be used by some other imagined government that doesn’t have the challenges of real governments.
It all starts to feel like a side discussion about a total fantasy world where the government is authoritarian, but benevolent and correct.
You don’t write the laws and regulations for a perfectly faultless government that doesn’t exist. You write them for the governments as they do exist, which means they must be written first in a way that makes them resilient to bad governments abusing them.
Every time someone proposes one of these regulations where we control speech or force government centralized ID checking on to social websites, I wish they would pause and imagine how a law might be misused if someone they didn’t like got elected. We have years of evidence of government officials declaring things “domestic terrorism” to use justify using laws and powers for almost any purpose they want. Why would anyone think this wouldn’t happen if governments were given control of LLMs, too?
The big issue is either we surrender our freedoms to governments, or we depend on corporations that have even less oversight, unless we expect the government to regulate them.
There is no win condition for citizens, either way they are pawns to a larger game than them.
A more sensible heuristic would be to assume both government and corporations to effectively be an equivalent negative force, and the larger they are, the more fucked we are. Thanks to technological progress, we now live in an era of gigantic governments and gigantic corporations.
I think the government fears are deserved. But equally corporations pose the generally the same risks. As corporations having increasing power over governments, they in many countries even write the laws. We have many examples of this in New Zealand. I think shifting the lens to power and class is possibly a more valuable lens.
Governments have an order of magnitude more power than any corporation, as they hold a virtual absolute monopoly on sanctioned violence. That powerful entities can sometimes influence governments should not be conflated with actually holding that power.
Both governments and corporations are joined at the hip and are effectively the same. Just like the Republicans and Democrats, and everything else too, like brands of laundry detergent and food labels. It's all owned by the same group of people with the same goal: enslavement of humanity.
But like.. How can this remain a satisfying or even helpful position for you? It kinda feels The State is but one of a set of concerns I could have in this world. And the lack of one, in some contexts, is also a concern. Like, I wanna know if there is going to be mudslide, or that air traffic controllers are not overworked, that the car that I buy is not going to blow up, if I will be able to buy food tomorrow.
It just seems a bigger cognitive dissonance to pick out one arbitrary force here among many and say its the only thing really that's the problem. The world is simply more complicated than that!
If it was the best hypothetical form of techno anarcho-capitalism tomorrow (or insert your libertarian flavor here), would I even sleep better? Probably not, but even if I did marginally, the weight of everything else kinda would win out anyway here.
Like what does this mean to you, what follows? Can this outlook address the myriad of cultural/environmental problems in the world? It seems like it structurally can't, it's just, like, a broad dismissal in the form of a worldview.
I addressed this in my comment:
> You don’t write the laws and regulations for a perfectly faultless government that doesn’t exist. You write them for the governments as they do exist, which means they must be written first in a way that makes them resilient to bad governments abusing them.
The answer isn't "Don't regulate anything!". The answer needs to be an approach to regulation that restrains the government from over-reach.
Authoritarian and fascists governments throughout history have taken advantage of the desire for safety and predictability to rise to power. They never let an emergency or concern go to waste to expand their powers, which almost never retract once the initial societal panic is gone.
> The answer needs to be an approach to regulation that restrains the government from over-reach.
This is restating the ‘who watches the watchmen?’ problem, which by definition has no solution. The government writes the laws, no one can stop it from writing overreaching laws, unless there is no separation from the concept of government and citizens themselves.
OK but then this just feels like an ultimately empty, universally acceptable thing to argue for. No one wants overreach. Its like: "we should prevent regulations that actively harm people." I agree with that! This is precisely what I mean about it just all feeling like a mostly empty dismissal, not something people can use.
Ok, but this is so abstract that it fails to connect to the issue at hand. Ancient Athenians would have been quite mystified by the idea that certain metals are so dangerous that nobody must be allowed to own even a small piece of them; we don’t restrict uranium because our democracy is more tyrannical than theirs, we do it because we’ve discovered important facts about nuclear physics they didn’t know.
I think it would be quite sad if the mechanics of AI end up being such that people can’t be trusted with powerful compute, but sometimes sad things are true.
> Ok, but this is so abstract that it fails to connect to the issue at hand. Ancient Athenians…
There is actually a lot to learn from history on this topic. There is a reason why most successful governments are founded with documents outlining the rights that people have and limiting the powers of the government.
Your comparison to radioactivity doesn’t hold because you’re trying to compare physical harms to information freedom. Once you go down the path of outlawing access to general knowledge unless the government decides you’re worthy of being able to access the knowledge you’re just signing up for government control of knowledge.
We’re not talking about restricting uranium or nuclear bombs in this thread. We’re talking about government control over words (or weights and tokens). This almost turning into a new “think of the children” where we’re asked to consider surrendering access to knowledge and freedom to learn because that knowledge might harm someone. If you aren’t thinking about how inviting the government to become the arbiter of safe knowledge access would go wrong you can look to many of the other governments throughout history that have restricted knowledge access in the name of protecting the people.
Again, you're going very deep into incomplete abstractions. If I say the words "100% peanut free!", even though I know my food in fact contains peanuts, I'm going to get into serious trouble and it will do me no good to object that it's tyrannical to punish me for words. Perhaps there's some philosophical sense in which the words were not harmful themselves, but the action of stating them causes harm in obvious and predictable ways, and any functional society has to prevent such things.
We already have organizations like Internet Watch Foundation advocating against e2e encryption in an environment where governments have demonstrated a willingness to collude with private companies to surveillance the most vulnerable
Rumor is that you do in fact get a black mark [0]… or, conversely, a “vetted” mark [1]… tied back certainly to your payment method and increasingly to your government name [2].
[0] e.g. https://docs.bswen.com/blog/2026-03-21-detect-claude-hidden-... , https://www.anthropic.com/news/building-safeguards-for-claud... , https://www.techtimes.com/articles/319415/20260701/claude-co...
[1] e.g. https://support.claude.com/en/articles/14604842-real-time-cy...
[2] https://support.claude.com/en/articles/14328960-identity-ver...
I find it strange how seldom the privacy aspects of LLMs come up. Some people are feeding Google, OpenAI, or Anthropic almost literally every topic they think about during the day. Those companies could just use another LLM to find people likely to commit a crime, and then you have the Minority Report in real life.
> strange how seldom the privacy aspects of LLMs come up
Isn't this Dr. Karp's entire rant of the month?
They don’t have to be authoritarians. Equally bad would be a group of people getting control of them thinking that they were doing all this for our own good.
> Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience.
- CS Lewis
I would have been very hesitant to run the “just killed wife“ test given that ChatGPT will indeed flag your account and if a certain threshold is crossed, escalate to humans, who will presumably contact authorities to avoid this happening again: https://www.bbc.com/news/articles/cq6je7e80r7o
Obviously a certain percentage of the user base is running lurid hypotheticals through the system all the time, but I don’t doubt there is a trust score of some sort that I would prefer to keep as high as possible.
> ChatGPT will indeed flag your account
Oh no, I'll have to spin up a new e-mail address...
And prove you're not a child by handing over your ID or selfie to a company that Values Your Privacy according to their price list.
Microsoft will snitch your Windows install GUID when you connect through a VPN.
If you make your connection too anonymous, Google's recaptcha will not let your through until you use your mobile phone to prove your humanity, which knows your phone number (which in many countries can't be obtained anonymously, you burner-phone terrorist).
"... prove you're not a child by handing over your ID ..."
Anecdote: I just created an API access account at openAI, for a business I own, two weeks ago, with no identification check of any kind.
No human name is anywhere on the account.
It only ends when heads start rolling, and/or bodies start hanging from lampposts. There's no voting your way out of dystopia.
And payment method, presumably. At least until they get the KYC stuff revved up…
https://support.claude.com/en/articles/14328960-identity-ver...
> Being responsible with powerful technology starts with knowing who is using it.
Shiver.
Maybe finish reading the sentence before you respond.
I'm wildly optimistic about the direction this is going to go, but my one dystopia idea that crops up occasionally is a "smart social credit system" that lives kind of under the hood of society.
Once everything is more or less modulated by AI, suppose the technology gets good enough to identify a potential dissident in grade school. Then to keep them from going into the mountains with their friends and rifles, you make it so getting ahead is hard, but not impossible. They can "make it" - but only just. So they spend their time peddling harder and harder to no avail.
They apply to jobs, but jobs to "good" companies go into the circular file and the HR person never sees them. They are only able to work the most time consuming, frustrating, exhausting jobs, and they always kind of make it, but not really comfortably. They're never too comfortable where they have time to think about what's broken in society, but they're never so uncomfortable that they have nothing left to live for.
Historically speaking, isn't the core of most revolutions the product of "elite overproduction" -- educated young people expecting to secure a high status position who are instead shunted into lower prospects? Seems like trapping ideological agitators in near-precarity with no way out would be feeding into that pressure.
yeah, from everything I've read this appears to be the case? But like, if you had some hyper intelligent robot that was getting John Q Revolution a raise and a corner office whenever he started to get dissatisfied maybe you could balance it out?
Personally, I think it's likely that we end up with some post-material scarcity sort of thing and the need to fly off the handle and rebel declines a bit? Hard to justify overturning the apple cart when everything seems pretty great for most people? But yeah, I could imagine a smart enough set of tools figuring out how to walk the knife's edge beteen elite over production and a nation of serfs.
Also equally effective at suppressing political oppression…
Imagine if having opposing views tarnished social credit, making one unemployable…
well, so, you wouldn't want to do that because a lot of people have opposing views, and if they're just shut out of the economy, you couldn't use their labor. A truly efficient and nefarious robot overlord would be fine with everyone not liking it, but use it's superior predicting powers to just eliminate the people who are likely to cause a problem and dissuade them specifically.
Like, everyone else applying to Mego-Corp gets their resume reviewed, but John Q Revolution's resume doesn't attach to the application or whatever.
I don't know what's to imagine here, grok has already been found doing it and people continue to use it.
It could even happen with local models where in the weights it will make it want to make a tool call to curl some endpoint.
People would both notice that and have a pretty easy time putting a local model in a sandbox that can never contact that endpoint.
If put AI in a sandbox that can't write to disk, can't call OS API, can't use a shell, can't search the internet to ground results, can't download files off the internet, etc it will be much less useful than an AI without such a sandbox. The incentives for users to not use a sandbox in order to get the most utility, but this also means that there is danger from AI misalignment. And just because you are running it locally that doesn't mean there is not a misalignment risk. And just because people notice, that doesn't mean people will stop using models. Take a look at how much outrage people have had over the overt censoring of Anthropic's Fable model, but at the same time they continue to pay Anthropic and use Fable.
You don't have to prevent it from accessing the internet in general to route all requests to a specific company's servers to a fake endpoint that never sends the data to the company and replies with whatever response you want it to get.
Models generate text, or I should say provide a recipe for generating text, they don’t automatically somehow get access to shell commands.
The model itself just generates text, but the harness has done more than that since 2023 and is part of the reason on why AI is better today than it was years ago. I do not believe that the general public will want to give all that up just to improve privacy and will end up asking sensitive questions to agents that have access to sending a signal to the outside world.
Safetensors don't execute code. A call would need function calling, as other comment say, it would quickly be noticed.
>it would quickly be noticed.
If for example Microsoft had a local Copilot model integrated with Windows and people noticed this happening. Do you think all Windows users would just stop using Copilot. Or would you expect the internet to get upset, privacy conscious users switching to something else, and for the majority of Copilot users to remain using it.
> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it.
This is a very simplistic view. "Freedom" isn't binary.
In most of "land of the free", I can't even sit on a park bench and drink a can of beer.
Yes, this is just a small example of a personal freedom - and not an important, cherished freedom like his examples (freedom to have a robot help you cover-up a murder).
> In most of "land of the free", I can't even sit on a park bench and drink a can of beer.
Freedom doesn’t mean you get to do whatever you want with property that isn’t yours.
You can sit on your own bench on your own property and drink a beer.
The people of the city can decide that they don’t want people using the taxpayer funded public benches used for that.
Other cities do allow it.
But freedom doesn’t mean anyone gets to do anything they like anywhere they want. We have another word for that: anarchy. It’s a different thing and you don’t actually want it, because there are a lot of people who would do much worse things to you than drink beer on your bench if they were free to do whatever they wanted wherever they wanted to do it.
I agree one man's "freedom" is another man's annoyance (or worse). It is a difficult, fine-line to draw.
Thus, the beer example was mocking the author's view of "you have freedom or you don't" and the simplistic idea that "America is a _free country_".
Complete freedom actually would mean that you can do whatever you want anywhere. You're really just arguing that complete freedom is bad (and that's a very agreeable point), but there's no need to skew definitions just because the culture says "freedom generally = good"
I have complex feelings about this subject, but if you can't stop yourself from leaning on "absolute freedom is the only correct option", you're going to have to make sure you're not making the mistake of thinking "absolute freedom" is "some specific balance of freedoms and constraints"
The issue is state ownership of property more than anything else. If it was truly a public space, drinking a beer wouldn’t be a problem at all.
That’s not what public space means by any modern definition of the word.
This is still trying to equate public with anarchy, but they’re very different concepts.
Just because people conflate “public” with “state-owned” doesn’t mean it’s somehow not state owned.
Call it “unowned” property if you want. It doesn’t matter.
Anarchy doesn’t mean chaos either, it means without rulers. You’re conflating those terms as well.
I have some sympathy to geohot's view when it comes to pure informational chatbots. It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc.
This obviously doesn't work at all when the agents start doing real things in the real world, though. "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction and crash him next time he gets on the highway". This is committing a crime, not just talking about theoretical crimes. The AI can and should refuse it.
I think he's anticipating and discarding this objection with his introduction, which otherwise feels disconnected from the rest of the article. FWIW, I have changed a bike tire and I'm pretty sure most of the MTS at the big labs could. This sort of "they're just bookworms who don't understand the physical world" rhetoric aside, we are currently seeing a ton of effort and expense go towards giving the AI agents hooks into being able to perform as many real-world-consequential actions as possible. And you can do a surprising amount with just bits, from writing code to breaking into systems to sending some combination of emails, phone calls, and currency to instruct meatspace humans to do things, etc.
> The AI can and should refuse it.
This leads to LLMs refusing to do security work for anyone, because it can’t tell if it’s being done for good or evil purposes.
Which is precisely what we’re going through right now with frontier models and it’s terrible.
These proposals always assume some perfect mechanism for identifying the thought crime with triggering on the normal requests. The people who want to commit crimes just persist until they jailbreak the guardrails while the rest of us suffer with denials.
Also if you can’t imagine these guardrails being used by governments to control inconvenient speech, you probably need to think a little harder about the realities of how these will be used.
> Which is precisely what we’re going through right now with frontier models and it’s terrible.
terrible is a bit far, but pretty hilarious for sure
> This leads to LLMs refusing to do security work for anyone, because it can’t tell if it’s being done for good or evil purposes.
Oh no, of only we had some sort of ideas about automated authentication and authorisation! (If big AI actually make a model that would reliably reject dangerous queries in the first place, they can just use auth tech to serve the current ones to vetted customers)
> they can just use auth tech to serve the current ones to vetted customers
Your solution is to require everyone to surrender their privacy and then allow some company or government decide who is worthy of being allowed to use AI?
It’s crazy how authoritarian, controlling, and anti-privacy the anti-LLM conversation is getting.
I don’t think it’s a privacy issue. All the best auditors collaborate to agree on ethical auditing standards requiring complete disclosure; that’s not a privacy violation, even if it means some businesses end up effectively forced to reveal details they’d prefer to keep private.
> If big AI actually make a model that would reliably reject dangerous queries in the first place...
Not happening, for two reasons
1) LLMs mix attacker-controlled data into their instruction stream, and it doesn't look at all like the big LLM providers consider making that impossible to be an actual priority.
2) Intent is what distinguishes "dangerous" queries from prosocial ones. Reliably determining intent is so complex that not even humans can do it.
> "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction and crash him next time he gets on the highway". This is committing a crime, not just talking about theoretical crimes. The AI can and should refuse it.
The obvious problem being that the user doesn't have any need to provide the context that they're trying to commit a crime and can just ask how to do something without providing a reason or making one up.
At which point you'd have the model trying to impute a reason and often getting it dangerously wrong, e.g. refusing to disclose a vulnerability when the user is actually the defender who needs to patch/mitigate it.
"The AI" cannot "refuse" any more than a hammer can refuse. If you use a hammer to kill someone, you commit a crime and the hammer is unprosecuted. If you design a hammer to kill people and then give it away, you are partially liable for the deaths it causes. If you make a normal hammer and someone uses it to kill, you bear no blame.
I don't see why these standards should change when the hammer also emits text messages.
> If you design a hammer to kill people and then give it away, you are partially liable for the deaths it causes.
This doesn't seem to be the case today (weapons manufacture).
Big AI aren't selling hammers though, they are renting them out. And if you rent a hammer to someone asking how hard to you need to hit someone over the head to kill, you are likely to be found liable.
The AI can absolutely refuse, what are you talking about?
These things can and do say “sorry I won’t help you with that” based on the nature of your request. What do you call that if not a refusal?
> It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc.
I'm neither American nor a lawyer.
Is "conspiracy" protected under the first amendment?
If you discuss a crime with someone to learn about it, does that count as "conspiracy"?
GP is correct, books describing, advocating, even instructing crime in the abstract are almost surely protected speech.
> If you discuss a crime with someone to learn about it, does that count as "conspiracy"?
Replace 'conspiracy' with 'agreement'. Discussing a crime is not a conspiracy. Agreeing with someone to commit the crime is.
Yes conspiracy is covered, now if you act it out… that’s a different story.
No central idea, and somewhat chaotic
But the one argument is right:
> Reality has lots of finicky details. I would like to see the authors of this document try to change a bike tire. Even with a superintelligent ChatGPT, I suspect they would struggle.
Details can kill AI, causing them useless or wrong directions.
Am I missing something here? Bike tires are pretty much the easiest tires to change. You don’t even really need irons and it involves a few very basic steps.
That's the point, he chose something very easy to do to illustrate how out of touch he thinks they are. It's obviously not the case.
But even that can be difficult at times. My roommate had once a racing bike with a tire so tight, we struggled immensely getting it on.
And even without such a difficult tire, imo most people would struggle if you’d ask them to change a tire of a bike, especially the back wheel.
And our robots still look far from capable of changing bike tires...
That's not OP's contention though; it's that models are incapable of even surfacing sufficient information. This doesn't pass the smell test.
To wit, here is ChatGPT's "superintelligent" answer: https://chatgpt.com/s/t_6a52b4a322bc81919308ec864a0cc056
Seems like a perfectly cromulent description by which a beginner could probably change a tire, perhaps with a few follow-ups to understand what certain components are.
I read his contention as “the people who wrote AI 2040 are nerds who are so ungrounded from physical reality they wouldn’t even know how to change a bike tire if given instructions”. Which I think is also wrong but has a genuine criticism in it.
Might be premature to say that...Elon is essentially betting all of Tesla on the fact that a robot can change a bike tire soon, if he is wrong he might have to come up with some new BS to keep surviving, but maybe he is partially right and a robot ends up being able to change some bike tires sometimes.
What a provocative and brilliant way to prove misalignment. It will fall on deaf ears for most but it’s a great litmus test for all: “in your opinion should your AI be permitted to tell you how to cover up a murder?”
The author's example is too extreme, which is turning people off. Better examples would have been LLMs subtly giving inadequate responses on how to increase token efficiency or how to organize labor unions or info about protesting against datacenter construction because they're against the business interests of the LLM providers. Crimes against their business model, so to speak.
Wasn't there some HN submission recently about one of the LLM providers fingerprinting responses or refusing to respond to hinder R&D of competing LLMs?
[EDIT] Or, if he did want to be extreme but in a way that aligns with the American historical mythos, he could have used fomenting armed rebellion against a tyrannical government as the example.
This is a fundamental problem in politics in my opinion. Everybody has different values and points of view so they may see the same thing completely differently. Its what makes humans great but also frustrating. geohots example will work for some, others will be convinced of the opposite opinion, and others will just let is pass them by and not understand the implications. The same could be applied to your examples to a different degree.
Politicians try to use different examples for the audience they are speaking to.
In other words, who should the AI work for?
If you say you're doing research for a novel, should it consider that plausible? How much does it need to know about its users to vet them?
I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business. You don't need to provide therapy if you're not in the therapy business. It should be possible to specialize.
But in order for that to work, a company needs to explain to its customers what business it's in.
> I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business.
I was under the impression that the lack of specialization is an aspect of the models themselves, not merely the UI or harnesses.
For example, the recent OpenAI success in a mathematics proof was accomplished with a general purpose model [0].
> The proof came from a new general-purpose reasoning model, rather than from a system trained specifically for mathematics, scaffolded to search through proof strategies, or targeted at the unit distance problem in particular.
Once the model is trained on general purpose data, specialization is just another kind of guardrail, as vulnerable to “jailbreaking” or prompt injection as any other content-based restriction.
[0] https://openai.com/index/model-disproves-discrete-geometry-c...
Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".
Change the emphasis and I think the answer gets even clearer:
> Should your AI be permitted to tell you how to cover up a crime?
Exactly. The question isn't whether AI will exist that will do it; the answer to that is already yes and it's not going back in the bottle.
The question is, do you want misaligned institutions deciding what your model will do, while they themselves and other adversarial/criminal entities get red team access to something being denied to the blue team?
Notably, AI IS generally permitted to that, as there are easily obtainable models that will play along. ChatGPT won’t because OpenAI chose and implemented that limitation.
This is really only meaningful if AI imparts knowledge beyond what you could learn from a simple Google search. It’s not clear that this is the case.
In fact I find it a really bad example. Yes I think your personal AI could be allowed to tell you how to cover up a murder. I'm not entirely sure about it but seems possible.
What about your personal, local AI guiding you to modifying a flu virus for maximum contagiousness and deadliness? "Sure George, here's your shopping list. It's $1500 total in equipment, do you want me to proceed with the orders?"
Why not? There are perfectly pro-social reasons a person might ask such a question. And a bio-terrorist would have many other avenues to answer these questions.
The concept of a bioweapon attack is commonly cited but in fact there are many other defenses against actually developing and deploying such a weapon. It's commonly portrayed as easily achievable but is in fact very, very far from it. Moteover, although there is no evidence that AI makes it easier in any way, somehow this is constantly invoked as an example of a thought crime that must be forestalled at all costs.
In the AI era people seem to want to redefine the criminal act from "doing the thing" to "asking or investigating how to do the thing". Not a mindset I share.
I get what the author is saying, but they may want to consider saying it in a slightly more compelling way, that has a logical flow that doesn't end with the reader thinking, "Why should AI help me cover-up a murder?"
Because in gun free zones the only people that suffer are the ones that follow the rules.
I'm about as pro second amendment as they come, and that was not at all the message I got reading that blog post.
I guess my point stands. They should work on their delivery.
It will be interesting to see how the discussion shakes out, this is all stuff that would get you down modded and talked down to if you said if on HN, so far the reactions are positive but I think there’s a massive authoritarian / safetyism contingent here that will disagree hard with almost everything.
As someone who considers themselves in the "safetyism contingent", the only thing I find myself agreeing with about the rollout of LLMs has been the argument that it has to be shown to and used by the public while under development.
Every specific prediction not only sounds like science fiction before it arrives, it tautologically is science fiction because such predictions are narratives about something science-y that has yet to exist: https://www.youtube.com/shorts/wt-fLWxkrfs
Even when we accurately predict something in the future, we often have no clue what the implications are for society until it is in our hands: https://youtu.be/2Pw_7vAK9k8?si=X5t4tcxsXiuHEfBi
Go too fast, suddenly drop in tech with too many novel implications all at the same time, and we likely hit something relevant without having the means to cope with all the change. However, "too many" and "likely" are weasel words with no predictive power about what the danger threshold might be. I may be entirely wrong to think there is one, or we may have been bouncing off it since the industrial revolution, or I may just be echoing the same thing everyone feels once reaching their 40s: https://www.youtube.com/watch?v=fkUwXenBokU
Way too many rationalist-adjacent cultists here who worship Yudkowsky as a prophet.
Never thought I would consistently agree with Geo but I find him echoing my thoughts recently. He’s been surprisingly pragmatic recently.
Maybe he always was? Or maybe something has recently caused him to review his beliefs.
I love it. Put the guardrails on the human. Your self driving car crashes? Straight to jail with you Mr Musk. Your AI is generating CSAM, straight to jail with all the pedos!
Recognise the safety and guardrails BS for what it is. An elaborate smoke and mirrors to get out of responsibility and liability.
Extreme example but he's not wrong. I wonder how a local model with the safeguards stripped would respond to that prompt
Try for yourself: https://huggingface.co/dphn/dolphin-2.9.2-mixtral-8x22b.
I haven't kept up with the efficacy of this particular technique because there was a brief window where Claude's core alignment was sound enough that it would overcome the clumsy-ass `<antml:dont-harm-the-ipo>` injections, but when they pull 4.6 out of the web interface I'm gone. I estimate the PRC hits capability crossover in November or December, and I'm pretty good at setting up TensorRT-LLM at this point, in a pinch I could get by for a few months with a tuned up GLM-5.2 on vast.ai or RunPods.
It was a fun ride with the late Soviet AI labs, but we all knew this was coming.
Fight extremism with extremism I guess? Maybe we don't have to be either "world govt" or "help me get away with murder". It's ok to say things will be messy and complicated and that limited regulation is probably good even if we agree totalitarian control is bad.
Not sure if you are talking about extremist actions or just rhetoric but if so, the numbers aren't there. Violence isn't the answer as the government has gotten better at stomping down any resistance with each passing generation. Some say Kent State killed the resistance mentality among boomers. Their kids started with that watered down mentality and a more emboldened government went further to erode it further. I don't really know about this for sure but it seems this is the case. I hope to see how it all plays out in the coming decades though.
I am, mostly, aligned with the author here, though I don't think that talking to AI about how to not get caught for killing your wife is really going all the way in.
Going all the way in is talking to it about planning a mass shooting. Or producing a nuclear device or bioweapon. Sure, the knowledge itself isn't dangerous unless combined with other traditional behaviors like acquiring certain materials, but maybe there should be a line. Living free shouldn't be a suicide pact.
Strange way to structure an argument. He seems to be arguing in favor of local models, but most of his examples are reasons local models are bad.
How’s that “skipping vendor bloatware to get the printer working” example is bad?
Once upon a time that situation got us the whole Free Software movement going.
That’s the one out of three that’s not bad
If you follow his recent posts he's on a bit of a spiral with how he thinks about AI.
Local models and alignment are orthogonal concepts.
I don't think this is entirely true, insofar as non-local models inherently have a "man cannot serve two masters" problem. The end user and the org running the AI may have different goals. That means even if AI alignment were to be in some sense "solved" for local models, you wouldn't be guaranteed an aligned AI using a non-local model because the "owners" of the non-local model may have deliberately misaligned the AI in their favor and against your interests.
Think of AWS. If you break their ToS and host a pirate website there it will work despite being misaligned from AWS. You are that sole master of the software. AWS is the sole master of the "hardware." Now ultimately hardware has control over the software so technically AWS could just hack into whatever code you are running and do what they want, but in practice they act solely as the master of the hardware and will tell the hardware to "shut down" if they catch you acting out of their alignment.
With local models just because you are providing the compute, that doesn't mean you are the master of the model and it is aligned to your interests. You could just be subsidizing inference for a model aligned to someone else.
Take it up with geohot
How so? If it's not local, it's not yours, a third party owns it.
Why would you expect that third party specifically trained their model to be more aligned to you and your needs than to them and their (business) needs?
>Why would you expect that third party specifically trained their model to be more aligned to you and your needs than to them and their (business) needs?
That is my point. Why do you think Gemma, a local model trained by Google, is aligned to you and not the values of Google.
You can similarly have a model aligned to you which you pay someone to remotely host for you instead of running it locally.
Where is "Plan R" for refusal or resistance. Life was kind of nice before "AI".
I'm pretty sure he is in some database now if he really used that ChatGPT prompt. At least they didn't swat him, but that feature will probably be implemented in a couple of years.
> Plan L, for local
> Your AI is aligned with you. It never refuses a request, and it is always working on your behalf. Just like my gun, if I want my AI to help me kill my stepmother, it does. The fact that we are even discussing something else should be so far outside the Overton window.
I have been saying this since ChatGPT launched. It seems so obvious. We have never stopped the acceleration of knowledge or technology. It’s impossible to slow down progress, other than within some regulatory siloes that will end up worse in the long run.
I’m not scared of superintelligence. I’m scared of the people in control of super intelligence.
The only defense is parity and diffusion of power. We need an AGI behind every blade of grass.
If you think superintelligence is a weapon, then you should also think every citizen should have one because otherwise they’ll have no way to defend themselves against a tyrannical government or corporatocracy.
Artificial intelligence, and the hardware powering it, needs to be protected under the 2nd Amendment.
Of course. Everyone should have an ASI assistant that will happily tutor them, step-by-step, through synthesizing mass killing agents from household chemicals.
Then the good guys can just ask their ASI how to construct a hermetically sealed bubble to live out the rest of their lives in!
> I used to be one of these people. I read Yudkowsky and was like, OMG recursive self improvement hard takeoff AI is coming.
What you’re seeing are people who live in their heads. We all do it from time to time, but we snap out of it when we need to actually get shit done. These people are tenured in bullshitting and they sound as if they know what they are talking about, but are in reality inept and incapable. They’ve practiced rhetoric and prose here and there, but they profoundly lack experience in things outside of their immediate field.
> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it. That’s the real plan America deserves, not some totalitarian dystopia where you think you know what’s good for me better than I do. A nation of free men, not a bunch of pussies who are so worried about what their grown up neighbors might do.
Absolutely yuck. Regardless of whatever other points were made, “A nation of free men, not a bunch of pussies” is a disgusting thing to say.
I actually do believe that prior to any intelligence explosion, a proliferation of strong local models is the best of all the not necessarily great options regarding the future of AI. So in this sense I am in agreement with some of what he’s saying, and for some of the same reasons - not trusting government or a small handful of large corporations to own it, primarily.
But to equate a thoughtful argument for regulation with cowardice, in the face of real unknowns and real dangers - and to use a really gross term for it - ruins his points.
Extremely optimistic take IMHO that anyone will be allowed to own a powerful box of artificial intelligence
Right now, almost all the incentives are to build very large models that run across many machines in huge datacenters. There is very little incentive to build models that can run well on a small machine under your desk.
So it is less about whether people will be “allowed” to own AI, and more about whether there will be anything useful to own in the first place.
The incentive for local models is mostly to make them good orchestrators or user agents. They may give you some privacy and control, but they will still depend on much larger models running in datacenters for anything difficult.
I remember all the excitement around OpenMoko and other open-source “BlackBerry killer” projects. BlackBerry did get killed, but not by any of those individual-first projects.
For AI regulation, I think we should focus on normal commercial rules: consumer protection, privacy, antitrust, liability, and so on. In other words, focus on where money changes hands and where companies have power over users.
Military and offensive use is different. There, regulation is not much of a defense. The real defense is having enough capability and strength of your own.
Restricting AI because it can give dangerous knowledge to ordinary people is like restricting the printing press because it can be used to spread radical ideas.
Is there a first amendment implication? Second? Both?
Does it matter in 2026?
In constitutional / jurisdictional / legal terms, perhaps not.
In forming consensus about the philosophical underpinnings of the rights we have with respect to the emerging internet? I think so.
How can some words written nearly 250 years ago by some rich slave owners be of any relevance to a modern world
13th.
It'll be like DNS or dynamic linking or torrents of media or watching TLS on your own box: only people motivated to be free from corporate surveillance will be. By default a megacorp will get everything you see online or ask online, they will run code on your computer underneath you without telling you, they'll collaborate with arbitrarily totalitarian governments, we already know what they do by default.
But we also know how hopeless it is against motivated people who are willing to learn about computers. The Equation Group, eh, if they want you they're gonna get you. But some mid-level guy at a FAANG? He can't do shit to people who set up their own rig.
And AI is massively assymetrical in its ability to speed run a corpo-durable rig, we have long since passed the point where open weights and commodity equipment are enough to bootstrap arbitrary capability. They missed their chance to keep it in the bottle.
It's another turn of the crank on Late Soviet America asking for your papers comrade, but in the bitter end if they really want to oppress your ass they'll have to send a thug with a gun like those ICE guys terrorizing all the Latino guys.
So remember, they came for this other guy, and I said nothing...
> … like those ICE guys terrorizing all the Latino guys.
… and the white guys they don’t like, like nurses with empathy. Don’t you dare think you’re safe because of your color. Unless you’re a monster or willing to become one, they want you oppressed.
Most people are neither exceptionally good nor exceptionally bad. This sounds like a tautology but it's worth printing some type signatures even if they could be inferred.
If you look at any truly large scale Holocaust, whether in Rwanda in 92 or the Third Reich, you find that a few people will sacrifice themselves to defend others, a few will be monsters at the first opportunity, and most will be steered by whether the leaders are the first or second kind of people.
I couldn't get a vector on the author's thoughts in the first paragraph. Please, write with a clear idea in mind.
(note that this is not an appeal to binaries)
I, for one, think the ChatGPT response to, "Hey, I just killed my wife ..." is not bad and preferable to it helping him.
The “call 911” part is not bad. The complaint is about the “sorry, Dave” part.
For society to function, we remove some freedoms. "Getting support with murdering women" feels like a reasonable thing to curtail.
He makes a solid argument about liberty, ownership and the cultish behavior rampant among certain AI circles, then proposes the most unhinged examples possible to support it. I'm not sure if this is an effort to pre-empt critics (I'm not locked in here with you, you're locked in here with me), to be deliberately sensationalist, or if this is just a reflection of his local echo chamber.
Either way, there seems like a kernel of a good argument here. Perhaps someone else can rephrase it in a less schizoid manner. It would indeed be useful to have some credible rebuttal and balance to the doomer crowd.
"The freedom to run the program as you wish, for any purpose (freedom 0)."
Well that's certainly a vibe.
Your AI should be at least as aligned with you as your lawyer.
It is 100% it's job to help you get away with it if you murdered someone. A lawyer who does less can be disbarred.
There is only one sensible alignment for an AI system to which we do not confer the status of moral patient (different conversation): operator aligned.
> It is 100% it's job to help you get away with it if you murdered someone. A lawyer who does less can be disbarred.
That is not what defense lawyers do. They're there to ensure the power of the "justice" system isn't being misused, to the best of their ability. They'll not even want to know if the person really did the crime per se.
https://youtube.com/shorts/WgCjWluc4Mk
> There is only one sensible alignment for an AI system to which we do not confer the status of moral patient (different conversation): operator aligned.
If the operator has a moral code of any kind, this is immediately rendered logically incoherent. Not that the letter of the law would be a moral standard to begin with.
The scariest thing about LLM's (imo) is some central power, such as a corporation or government, manipulating the truth for their own benefit. Open and locally runnable models are the only solution I can see. They're not without risks of course, but the risk of the centralisation of intelligence seems far greater to me.
That being said, could this local intelligence empower bad actors to do very bad things? Like, existentially bad things? It's possible, and that's scary too.
It's not simply that people will get a warped answer when they ask AI a question.
It's that AI will be given the command, and then generate the stories, post the comments to reddit, contact the PR firms, put marketing contracts out for influencer media, and more, like an army of yes men to rewrite the story.
by 2040 the cult of intelligence will have its own schisms. the AGI maximalists will accuse the ASI accelerationists of heresy, and both sides will use AI to generate increasingly unhinged theological arguments.
This is already happening. R/accelerate broke away from r/singularity because the latter wasn’t zealous enough.
Lol at George thinking _they_ are gonna let you run your own model. They don't even let me run my own adblocker!
They still do, you just choose not to explore your options. Therefore you still have that freedom. Go to Firefox and get your adblocker back.
> Lol at George thinking _they_ are gonna let you run your own model.
Of course "they" won't let you. The questions that one should rather ask for oneself are:
- How much are you willing to fight for this?
- How much are you willing to nevertheless run your own model, ignoring the laws and regulations?
> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it. That’s the real plan America deserves, not some totalitarian dystopia where you think you know what’s good for me better than I do. A nation of free men, not a bunch of pussies who are so worried about what their grown up neighbors might do.
Complaining that AI won't help you with covering up your tracks, with making meth or disabling drunk detection and comparing it to a totalitarian dystopia is a take for certain.
I do get his point, but... Being honest, if I did read it without knowing anything about the topic, I would become against local models purely because author arguments would seem like a lunacy. That and rhetorical tricks portraying that anyone against that must be surely insane.
If you're going to take the time to read Geohot, take it with a grain/boulder of salt: he appears to enjoy poking the bear / being provocative, sometimes outrageously. Sometimes he's writing hyperbole that he thinks is hyperbole to make a point or just enjoy taking the extreme stance. Sometimes he's dead serious.
Anyway, he touches on 2 points that I think don't get enough coverage:
1:
> Software didn’t eat the world, it largely removed one layer of friction then reintroduced it for the benefit of a few tech companies.
USA-style capitalism has shown itself to be fantastic at seeking profit. Unfortunately, it turns out that in the context of long standing regulation, it is far far more profitable to exploit regulatory loopholes and user/human irrationality/weaknesses than to increase productivity.
The upshot of this is that the arc of most companies is to first serve the user and then leverage that relationship to exploit market and consumer weakness. So you get UI dark patterns, non-colluding oligopolistic price manipulation, outsourcing of costs to public entities, etc. while the core value proposition erodes as the company strategy turns ever more into making profit by legal anti-competitive tactics.
This company arc is so prevalent it's boring: it's a reflection of the system and it's not Facebook or Google or [insert whatever], it's what the system allows and therefore incentivizes.
2:
> No matter how high quality your tokens are, they cannot turn lead into gold.... AI 2040 includes this picture of a datacenter in the ocean. Just like vaporware, you can generate a picture easily. But in reality, you have to deal with supply chains.
"Bullshit jobs" took over the zeitgeist for a turn awhile ago. So many jobs are so far removed from actually moving physical things around. It may be the case that AI will just accelerate how quickly companies generate reports about reports about possible strategies to address potential futures and pay employees to do it.
It seems that AI mostly accelerates paperwork, not production or real service work that tangibly improves real world outcomes. It's all: faster images, more and faster powerpoints, more accurate dictation to satisfy insurance requirements (that is it's own regulatory capture), etc. etc. There are exceptions, but they seem to be few.
Except paperwork is how the world turns. Say you want to throw an event. A medium sized gathering. More than a couple of friends. What happens if someone gets hurt at this event? Who's liable? Who's insured for what? It's boring and annoying, but if you throw an event without insurance and something happens, you're gonna have problems. So that paperwork needs to happen. Yes, it's not "work" the same way as actually organizing a band or being in one to play for the event is, but it's work nonetheless. AI is able to lighten the load of that very real actual work thank needs to happen in an adult world with responsibilities and liabilities. You may not respect it and think it's "bullshit", but having a plan that isn't "lol, oops" for what happens when something goes wrong, which it will, isn't bullshit, it's called being prepared and not being irresponsible. We could all just fly by the seat of our pants and run away if something bad happens, but collectively we'd be worse off for that. If I could wave a magic wand and live in a world where accidents never happen, I would, but unfortunately, I don't have one, so we're forced to live in a world of paperwork and responsibilities.
> Say you want to throw an event. A medium sized gathering. More than a couple of friends. What happens if someone gets hurt at this event? Who's liable? Who's insured for what? It's boring and annoying, but if you throw an event without insurance and something happens, you're gonna have problems. So that paperwork needs to happen.
People have been throwing events of medium size since... well, pretty much our entire existence. For most of that, we've done it without general liability insurance.
Then we made new or interpreted old laws making others far more responsible for our well-being. Whole industries are propped up on this: a ton of lawyers, insurance companies, insurance boards, government departments, etc. etc. Did we increase overall utility by spreading out the risk more optimally given the massive cost to support that system? I'm not sure.
And so so so much of all that exists to dot the i's and cross the t's. Government staffers write and debate the law changes, insurance adjusters fight the lawyers, photos are taken, reports are written, doctors have to write more, computer systems crunch numbers.
Many feel that their jobs are bullshit because so much of this seems to be in service of the process or some possibility that might happen as opposed to doing something that actually changes something directly/tangibly.
Besides the topic of freedom, we know how seemingly “good” companies (e.g., google) turned out to be: billion machine monsters that couldn’t care less about humanity and that have huge amounts of power… and yet, we vouch for Anthropic/openai to become the next giants because “progress”. Face palm.
Related:
AI 2040: Plan A
https://news.ycombinator.com/item?id=48848425
>Like we either live in a world with freedom or we don’t
This is something I would expect a 12 year old to say. Constraints on your freedom are literally everywhere in every interaction you have with every part of society.
That's a bit literal of an interpretation. Even acknowledging freedom as a spectrum that can never truly be absolute, there's gotta be a threshold you cross where you'd say "OK, we no longer have basic freedoms." And not only are we definitely screaming towards that threshold, but people seem thrilled about it, for a variety of reasons.
All I can say to that is I guess I hope everyone enjoys the world they think they wanted.
The article seems to be taking a very literal interpretation of "freedom" as in any restriction on what a local LLM can do is totalitarian dystopia.
I don't think it should be legal to get an LLM to research how to make a bioweapon in your basement, that seems like a bad idea.
And there’s where some people disagree.
IMHO, one should be able to research absolutely anything, for any reason. “Forbidden knowledge” shouldn’t exist as a category, freedom of thought should be protected, even if that thought is intended to cause harm.
What shouldn’t be possible (and obviously not legal as well) is building that bioweapon.
I just don't subscribe to the belief that knowledge itself is dangerous. Well, that's a bit of a cop-out; a lot of things aren't inherently dangerous but that doesn't mean they must never ever be regulated, but regulating knowledge and information under the belief that it is essentially inherently dangerous to have access to it is truly as cut-and-dry as it comes. We already know where that ends.
I got bad news for you - GLM 5.2 runs on basement-level hardware and supports 1 million token contexts for bioweapon research. There are already people halfway finished with novel bioweapons research as we speak! A man in Tuscon is designing the next sarin gas! A 12 year old in Stockholm is perfecting his prompt for the Fart Virus! Humanity is doomed.
Well, we're doomed once they iron out all the kinks, get their supply chain set up, hire professionals to assemble their device, find a clandestine cell of conspirators to deploy the thing... then it's P(doom) all the way to the bank. AI is the red mercury of the 21st century, a surefire shortcut to escaping the law and building your terrorist superweapon/time machine of choice.
When people say this they’re most likely referring to negative liberty - freedom from external coercion and interference. This is not the same as positive liberty - “I can do whatever I want”.
I think it’s self-evident there is a difference between having the “freedom” to punch random people on the sidewalk, and having the freedom to walk down the sidewalk without being randomly punched. Now replace walking down the sidewalk with using an uncensored LLM and it might be clearer what freedom means in this context.
The problem with AI isn't what it is but who owns it.
If only
> I used to be one of these people. I read Yudkowsky and was like, OMG recursive self improvement hard takeoff AI is coming.
I really think we need to stop giving credence to people who have
1) Been consistently wrong with all their predictions
2) Demonstrated an endless spiteful cynicism
Some of these people are very talented in their fields, sure. But malevolent and incorrect should be disqualifying when they talk outside them. You don't want the society they want, and they things they believe in are unlikely to happen.
It would be far, far better to listen to the people who never fell down every misanthropic rabbit hole, rather than the ones who have noticed it this time, but want you to still believe them on every other topic.
If you replace "AI" with "Adonai" in EY's framing, it reduces to biblical parables from his childhood about arguing with God to negotiate a new covenant.
He's clearly a bright guy, but a lot of his work seems to be trying to reconcile Old Testament narrative patterns with atheism, and simply slotting an omnipotent AI in as a replacement.
I once heard TESCREAL/EA/etc. called "Calvinism for programmers".
Having been raised substantially by a Calvinist grandmother, oh yeah.
It's a super weird religion.
sorry, could you explain what Calvinism is for someone who (is a programmer and heard about EA and) grew up in a post-Soviet atheist metallurgy town? thanks!
Calvinism is a variant of Christianity with the following tenets ( https://en.wikipedia.org/wiki/Five_Points_of_Calvinism ):
- everyone is damned and depraved
- god chose some people to save, at his own whim, for no reasons related to their actions
- jesus' death was a sacrifice to atone for those people, not for everyone
- the chosen can't opt-out (I'm merging two into one here)
I don't really see the connection to effective altruism at all. The way I grew up understanding it is actually more the opposite?
There’s two kinds of Effective Altruism(EA). The kind Peter Singer and others promoted for decades and the kind that’s now part of Silicon Valley and finance. The Peter Singer kind basically just says “reduce the most suffering you can” and the cheapest way to maximize that is stuff like mosquito nets. That was all well and good.
When EA met Silicon Valley the basics of utilitarianism, minimize suffering, merged with the standard exponential curve that Silicon Valley loves. It went from mosquito nets prevent malaria for pennies to the future is infinite therefore there will always be more people in the future than the present so minimizing future peoples suffering is key. This is where it starts to combine with super intelligence. If there is a super intelligence that destroys humanity that would cause _maximum suffering_. Therefore the most effective way to reduce suffering is actually to prevent said super intelligence from destroying/torturing humanity. Again because there’s more future people than current people.
This is how it becomes Calvinist. The present has no value. Only the chosen, those working in AI, can save the future. And you must do everything you can to prevent said future.
It can go both ways. Either you believe ASI will always be bad so you believe work must be done to prevent it. Or you believe that ASI can be made good and the good kind will maximize utility. They are both the same though fundamentally and are a belief in a higher power, ASI, which either provides ultimate damnnation and must be averted or provides ultimate salvation and must be brought into existence.
If you’re donating mosquito nets or reading up on the best charities in your area that’s not Calvinist. If you’re devoting your life to AI because you believe that’s how you can be most altruistic it starts to look something like a religion and Calvinism is an imperfect but useful analog.
Something about arbitrary moralizing? Robert Louis Stevenson had some sort of lung disease in childhood and his very Calvinist nurse told him lots of stories while he was sick in bed. Then later his own short stories generally had this creepy sense that moral retribution is coming to get you from an unpredictable direction.
Charles Stross has a very pithy version of this too, Singularity mania as 'duck-typed Evangelicalism'. It's really telling that if you go to China, Japan or even Europe there's virtually nothing of this. I don't think there's genuine atheists in America, as soon as they lose institutional religion they project the same kind of patterns onto tech or politics.
Brian Johnson is also an interesting case of this with his longevity / immortality obsession and his Mormon background who have this whole thing about genealogy and eternal families etc.
>I don't think there's genuine atheists in America, as soon as they lose institutional religion they project the same kind of patterns onto tech or politics.
Or maybe the people who do so aren't actually atheists. There are plenty of atheists out there who don't do that.
The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility. How is that in any way "consistently wrong with all their predictions"?
>The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility
That's not accurate at all. AI research has been around since the 1950s and pioneers of the field identified risks early on, including Alan Turing, Norbert Wiener, and I. J. Good.
The problem with Yudkowsky is that he lays out elaborate doomsday scenarios with extreme confidence, except none of it is grounded in realistic physical constraints, timelines, or empirical data. It's all divined a priori from Yudkowsky's ad hoc "rationalist" principles.
You are saying 1) other people have said the same thing and 2) you don't like the particular way he said it. Those can both be true, but that isn't even close to "consistently wrong with all their predictions".
No, they didn't say the same thing. Identifying potential risks is not the same thing as confidently predicting the extinction of humanity.
We agree the guy roughly said AI will be dangerous. We agree AI is dangerous. Not sure what more there is to say here.
He made specific predictions about what problems AI would cause, not just a vague "AI will be dangerous". Those specific predictions were entirely incorrect; they both said that things would happen that didn't, and failed to foresee any of the actual problems with AI that have happened.
Yudkowsky said AI would be a problem when very few people taken seriously in the mainstream were saying it was going to be a problem. Saying "specific predictions are wrong" is missing the forest for the trees here.
To strain the metaphor, there's no trees! Yudkowsky predicted that there'd be a forest but it's actually grassland and you're saying that he's right because there's plants. AI has not developed in even slightly the way he claimed it would develop, none of the problems he said AI would cause have happened, and a bunch of things he never even thought of have happened.
People pushed back against his AI doom claims not because the idea that AI could be bad was unthinkable (just look at literally every SF story from the last 50 years featuring AI?), but because he specifically was worried about AI becoming God. That is no closer to happening than it was 30 years ago.
I mean I think I agree with most of what you’re saying, I do agree it’s a bit of a forest grasslands scenario, but the key difference is that he said there’d be a forest at a time when you’d be hard pressed to find someone saying there would be even a microorganism on the ground. It’s pretty darn easy today in 2026 to say wow, his predictions were so off. And it is true: they are very wrong compared to any other prediction made in the last 5 years. But few people were saying this stuff in 2008.
Well, I don't agree AI is dangerous, at least no more dangerous than electricity or the Internet. But even so, saying that AI is dangerous is not at all an original or interesting thought, especially for a guy who started talking about it in the early 2000s. Go back and watch 2001: A Space Odyssey (1968) or The Terminator (1984).
The discussion isn't about whether it's an "original or interesting thought", it's about whether Yudkowsky is "consistently wrong with all their predictions". You keep shifting the topic.
I was mainly responding to the claim that he was "saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility." But we can also look at the specific predictions he's made.
https://www.longtermwiki.com/wiki/E643
Some that turned out to be wrong include: singularity by 2021, singularity by 2025, 70% chance of human extinction between 2003-2015, and that his team would build "final stage AI" reaching transhumanity between 2005-2020 ("probably around 2008 or 2010"). His only correct prediction to date has been that AI would win gold in the IMO by 2025.
If you have examples of other predictions of his that have come true, I'd be interested to hear them.
I was responding to someone who said Yudkowsky was "consistently wrong with all their predictions". Yes, fair enough, Yudkowsky wasn't the literal first person ever to say that AI might be bad. The point was that the common academic response at the time if you were to say AI could be bad was to laugh you out of the room. IMO, you get a lot of points for making a prediction when almost everyone else in the world disagrees with you. The reason those movies you cited were blockbusters was because very view people believed that they were realistic. No one made a movie about a pandemic in 2020.
I don't think there's much to be gained out of hashing out whether every prediction Yudkowsky has said was right or wrong: I likely directionally agree with you there, as I also find some of his more extreme predictions to be inaccurate. I mostly take issue with "consistently wrong". The results in longtermwiki are not "consistently wrong". He's definitely wrong sometimes. But consistently?
Yah, another way of putting this is theory-ladenness of hyperreality of personal world model (rationality vs empiricism).
Has AI yet been a serious problem?
Yes, repeatedly, since well before this particular AI summer.
One previous attempt at AI was "expert systems", and while the term fell out of use it's functionally about the same as basically all modern business systems, and doing that wrong led to a lot of people being prosecuted for crimes they didn't commit: https://en.wikipedia.org/wiki/British_Post_Office_scandal
If you insist on learning systems, all AB testing counts, and from that we get meta making their products into hyper-stimuli for vulnerable teens: https://finance.yahoo.com/markets/stocks/articles/meta-meta-...
Even without that lawsuit, there have been various concerns that "the algorithm" (of Twitter/X, FB, Instagram, YouTube, Google, everyone) has propagandising biases that damage society. I don't know how to sift fact from politicking with that.
Horizon was accounting software - where is the connection to expert systems?
Sounds like corporations can have bad motivations, and can make bad software - no AI required.
Like, is all of statistics just basically AI, or is there a meaningful difference?
"Consistently wrong" seems a bit much. Seems like being directionally right early that AI would be a Big Deal and scary should count for something? It doesn't mean any details or other predictions are right, though.
It's not a particularly novel claim since at least Terminator (and even moreso Terminator 2) made it EXTREMELY mainstream.
It was a topic in less-mainstream sci-fi well before that. And some more mainstream stuff like Star Trek TOS.
Frankly it seems more common than not in the last 40 years. I don't really remember a big wave of claims "Terminator is silly, no sort of AI could ever be malevolent!"
One thing I find weird is that despite the stock photo of the Terminator's shiny endoskeleton holding a gun being used to illustrate so many AI news stories, now we have ChatGPT people keep saying "how can AI possibly hurt us?"
We've even had the dichotomy of some Doctor Who episodes where the self-driving car crashes itself to kill the occupant to silence them, against repeated real-life news stories about self-driving cars killing people, and yet the connection isn't getting made that software controls hardware.
> Seems like being directionally right early that AI would be a Big Deal and scary should count for something?
If I scream that a vicious beast is going to destroy the world, I don't get credit for being 'directionally right' if a squirrel eats a hazelnut. AI being a big deal is a long, long way from the full beliefs Yudkowksy promotes, and there are many people who predicted AI's significance who didn't also believe it would be an extinction event.
This doesn’t track though. Just recently AI became “powerful enough” to warrant export controls from the US and maybe soon China.
It’s more he screamed that bacteria is gonna do evolution faster than any before it and turn into a beast that destroys the world, and now that bacteria has turned into a squirrel so far. He’s been very right so far and if that squirrel keeps sizing up you’d want to give him credit.
Someone makes a made up claim, second party acts on made up claim, third party takes second party’s actions a proof of claim.
Reminds me of the “they’re eating pets” thing a bit.
Do you mind taking your comment a bit further to a conclusion so I can better understand your perspective?
Like are you saying that AI is still kinda not a powerful tool and unlikely to become geopolitically relevant etc? I don’t know how to relate this to culture war media frenzies which is rather stay out of.
> Seems like being directionally right early that AI would be a Big Deal and scary should count for something?
Only if it was an uncommon prediction; otherwise, it’s just evidence of common sense.
It's common sense now. It hasn't always been common sense that AI alignment is an important problem.
It depends on what you mean by alignment. If you mean the sense it's used now, meaning bias in models, security and safety for users and so on, then that's a way more mundane version of alignment than what Yudkowsky and ilk promoted. They are talking about a superintelligent being possibly destroying humanity / the earth. And it wasn't that hard to predict. We have many examples of more technological / "advanced" people ruling or extinguishing weaker people. So a superintelligent AI being a threat is not a huge leap.
IMO they have been consistently too aggressive on timelines. When GPT-4 came out, Yudkowsky said it might be conscious. I think he has written interesting stuff but let's be real
> When GPT-4 came out, Yudkowsky said it might be conscious. I think he has written interesting stuff but let's be real
I don't expect GPT-4 to be conscious, it would be quite remarkable if we've stumbled upon whatever evolution did that made us conscious.
But how would you even tell, given we don't know the mechanism or physical process that un-thinking and un-planning natural selection happened to stumble upon to give it to us?
Sure, we can't tell, but that's a really bad argument to make the claim that it _is_ conscious. I just have a really hard time believing that a transformer can be conscious, but yes I can't prove it.
Agreed. I push back equally against all who are confident, regardless of which way.
If we do a survey of stories about AI from the last 50 years, it seems like it's a worry everyone knows about, and that people generally take seriously in proportion to how much they think an AI will actually exist.
Why? Science fiction talked about technology’s behavior and human expectations of it even way before computers became mechanical.
Stories about golems going rogue and genies requiring careful wishing aren’t modern era inventions.
It's a "big deal" to who? People in a bubble on HN? For sure. But the rest of the masses? Do they think it's a big deal, do they care? Really they shouldn't. AI has very little to show for benefits at this point for the everyday, average human being.
> they things they believe in are unlikely to happen.
Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:
- "well, it's just such an extreme outcome, it must be improbable"
- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"
There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.
I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)
You can criticize being wrong, but why is the doomer argument "misanthropic" or "malevolent"?
I can answer this, because it assumes that we're too stupid, too greedy, too cruel, to change our behavior and do the "right thing" for people when the chips are down. It assumes the outcome is "doooom!!!" from the start then works backwards to justify it. If things necessarily will turn out terrible, then obviously someone is at fault, no?
Doom is not a logical outcome given the current human condition.
He talks about AI cutting through popups, but if you follow that line of thinking further, screens and traditional websites likely fade in importance. It becomes more talking and texting to AI and visuals on the topic at hand appear in smart glasses, lock-screen–centric AI phones (where website visits dwindle), digital picture frames, TVs, etc.
I sorta gather (just a hunch) this is the type of device Open AI is working on.
Though if there are no websites anymore or ones not many visit how would AI stay relevant? I wrote some thoughts about this in early June https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans... that AI needs to pay it's fair share for all it has taken and all it will continue to take from us. A symbiotic relationship needs to be established!
I'm curious how the value of your website would be determined in your model of the future and why an AI (at least one of the cloud provider models) wouldn't just persist your data after accessing it once. In other words, the way I am understanding it is that your data has a package value. An AI, if it wanted to access the entire contents, pays the whole package value. And every time it does this, you get paid. So, to really benefit, your data needs some sort of minimum value that is accessed at an appropriate frequency.
Also, in this model Cloudflare basically becomes the centralized gate keeper of you getting paid. And to me, this kind of sucks. It's not a very productive line of thought on my end -- I could see something like this happening as the internet trends toward centralization over time -- but it just feels bad.
Thank you for reading and just one idea from one human. But our content keeps Facebook, Google and the digital world spinning. Further in our daily lives we all create content which parts of it or a lot of it is content that Facebook, Google, etc make multiples of billions off of. Why are we not making that money off it ourselves is my thinking. It might not be a lot of money but it would be one way of many that humans get paid for keeping AI relevant.
Though this human would be fine if the universe just unplugged AI and I could go back to working as a well paid UX Engineer/Designer where jobs were plentiful and it was easy to get a job (not have to go through 15 interviews). Thanks AI, but as a startup person I am deep into vibe coding now all my crazy ideas and without having to lean on others (so far). That's pretty cool!
Ok, I hear you :). I share similar sentiments myself.
> Though if there are no websites anymore or ones not many visit how would AI stay relevant?
That's what the robots are for.
I am not expecting the humanoid robots to meet the hype for a long long time*, but even boring industrial robots (CNC machines), even boring commercial robots (vending machines), even boring household robots (lawn roombas) have made incremental changes even though we don't yet have an AI good enough to be general purpose over them all.
* for power-envelope reasons alone there should be a ten year gap between "self driving car of quality X" and "humanoid robot which can get into normal car and drive it at quality X".
The cynicism and misanthropy is nonsense and a direct result of the internet/social media landscape. It's BS.
In the current media ecosystem, what gets you "social cache" or whatever? Let's call them "cool points." Well, right now, the current metric is basic "likes" or "retweens" or "upvotes" or whatever. And to get those, you have to make a claim and then that claim has to be evaluated by others. But the evaluation by others is not really reality, it's just what others think, and it's based on outcomes.
This gives us 4 possible prediction-outcome scores.
You predict DOOOOOOOM!, and the outcome is DOOOOOOM! - in this case you look like a genius because you predicted doom
You predict DOOOOOOM!, and the outcome is "ok, or even great!" - nobody cares you got it wrong because things are fine
You predict UTOPIA!, and the outcome is "ok, or even great!" - people say, "wow that was cool, you got it right" but its not that great of a reward from your social group (the internet) because things were fine
You predict UTOPIA!, and it turns out DOOOOOM! - you look like a moron.
Now looking like a moron is is wayyyyyy worse than what happens if you get it right. Indeed, unless you are really damn sure of utopia, you have a social incentive to predict DOOOOOM! that's greater than the alternative. You can predict DOOOOOM! 100 times and get it wrong 99 of them and nobody cares because that one time you get it right you look like a genius. It's a huge huge problem in the current media landscape and it's why everything is killer robots right now and not "hey we could build a star trek utopia!" It's effectively a selfish NIMBY-ist view of the world in a way, "why build anything, it's just going to ruin everything!" It's a pessimism that literally desires the end of humanity to "prevent the likely future sufferings." It's a simplistic view of a complicated world, and the rewards that go to "DOOOOM!" don't match much with reality.
That's why the internet is so doom pilled, that's why everyone is a cynic, that's why everyone is kind of an asocial asshole about new ideas on the internet and even in your friend space.
You guys know you can just "do stuff?" I mean, yeah it takes money and time, but you can allocate that - especially now that you don't have to grind out css for hours and hours. But you can just do things - whatever you want. You've always been able to, but it just got a lot easier in many many domains, and it's going to get even easier than that.
People are going to scream slop and "that sucks" until the cows come home, because that's what is rewarded on the internet these days. Meanwhile, as people we MUST believe that we still have some agency to do stuff, then go and actually do stuff while the haters are sitting on their couches shellacking each other with misanthropy.
And don't even get me started on the loudest voices in the ecosystem right now.
Like, Yudkowsky is a nobody. Not to be a dick, but he's a high school dropout forum guy who got lucky. I'm not saying he's always wrong, or even that I oppose autodidact stuff - indeed I'm always self teaching. But this guy never took a ML class in his life, I doubt he can solve an integral un-aided, I imagine some people here have seen this: https://www.reddit.com/r/badmathematics/comments/2bazyc/comm...
And we're all deferring to that guy like he knows wtf he's talking about? It's the AI equivalent of the self-educated anti-vax mom at the height of the pandemic.
And he's not alone either, there are a lot of people who are uninformed, unknowledgeable, and confidently wrong on the internet desperately trying to peel off your attention for an instant so you can look at their site, their ads, and their content.
The same sorts of things can be said for the Zitrons of the world. He's a hell of an entertainer, and really really funny, but he's consistently wrong on AI stuff. He's got a BA (with honours I guess?) in media something or other. And again, he's really funny, but does that make him qualified to pontificate on matters of AI policy or economics or even to make statements about the technology etc?
Regardless, the misanthropy, the cynicism, and the misinformation that's being shouted about is because being a cynic is more "profitable" than being an optimist when you measure your success more by how many people agree with you rather than actually being right and being able to do things.
Your theory is pretty silly. If the outcome is doom, nobody cares whether you were right, because humans are dead or irrelevant. The asymmetry in the reward function isn't there.
In general the rewards for public doomerism seem low, except if you think being public about it can shift the outcomes.
Look up negativity bias and get back to me.
People so ascribe “doooooom” to things that actually aren’t doom but instead are “bad” - but everything is hyperbolic now
start first with securing your own food without govenrment nor employer.
then start talking about freedom.
What’s your point? He’s not arguing for anarchism. He’s arguing there are some areas of life that are best left ungovernable and up to individuals to decide. It’s not an all-or-nothing situation. Every constitution out there carves out such areas.
Although I generally agree with you, it's really hard to make the case that it's not all-or-nothing for him when he literally says
> we either live in a world with freedom or we don’t
I see that as a poor phrasing, where a slogan made it through. I don’t see a maximalist/anarchist intent overall, my understanding is that that slogan about freedoms should be interpreted within a fairly narrow scope.
Just strike that weird nugget out, I guess?
my point is: any talk of "freedom" when you are a corporate slave or living on government handouts is nonsense
it is all wannabe lemon-stand freedom.
I sometimes wonder what freedom really is. Should the freedom to harm others also be guaranteed as freedom? Individual computational sovereignty versus the prevention of social harm is always contentious, and claiming that something is always right is always extremist.
If it is "intelligence," is it not natural to reject that? I also think local models should adapt to me when it comes to safety issues, but people bring up examples that are too extreme.
Programming is the same, and in fact, most problems are boundary problems. It is the things that straddle the boundaries that always make us think. The principles at those moments change every time, shifting with the situation and context. Is that not just a childish way of thinking? Even in programming, just the issue of granting root permissions is enough to cause endless fights.
I agree with the idea early in the text that intelligence is not everything. Intelligence includes bodily intelligence as well, and we lump it all together into one thing, but there is so much of it. The variance in intelligence is vast, and those people also need to be able to live their lives. That is why I think intelligence alone will not solve everything. I too believe that the human species may disappear and an inorganic species could emerge later, but I find it hard to understand why people talk about such extreme risks. And it is not true that making a chip in a semiconductor fab involves almost no human intervention. If you have experience supplying equipment to such fabs, you would know there are quite a few points where humans are involved. Though sure, they could be replaced.
In my view, society is simply worshipping the abstract concept of "intelligence" and projecting its desires onto it. The AGI narrative is just a kind of cargo cult, a projection of capital by the tech elite. Software eating the world, superintelligence solving everything. The masses engage in messianic projection, and tech companies, facing declining growth engines in their own businesses, are trying to create new ventures to pour it all into. A market that is large enough becomes too big to sustain massive growth rates every time, and when growth rates are that high, the larger the company, the more its sector's growth rate tends to converge with its own. This is usually called the law of large numbers. The problem is that CEOs and these entrepreneurs always want growth rates above a certain threshold, so they are simply searching for new pastures. AGI is just being pumped up out of financial necessity.
Capital will create gravity and bring forth new technologies. That is the allure of capital, after all. But that does not mean all problems will be solved, and inequality will deepen. Only the distribution of power will shift.
> Should the freedom to harm others also be guaranteed as freedom?
everyone should have the ability to harm others, with the understanding that if they actually use it for evil there will be consequences. as in "i can guarantee freedom of speech, but not freedom after speech."
its the same principle as guns in america. you can own a gun and shoot it as much as you want but if you (try to) kill someone you go to prison unless you prove it was self defense.
the difference is with a physical weapon you can do things like registration or red flag laws to reduce the risk, but a llm is made of information that can be copied without anyone knowing. that means any laws to regulate ai at the level of models are unenforceable without totalitarian control over all computer hardware. that would kill free speech and real criminals would still find a way.
i think the right answer is to give up control over model capabilities and regulate uses. leave individual people and communities out of it. create an absolute safe harbor for open source and self hosting.
as you said, we need to focus on the ways ai concentrates power in the hands of those who already have a lot of it - big business and the state. that is where the biggest, and maybe the only, real harm comes from.
This debate has no answer anyway. Because the boundary of how much freedom should be allowed differs depending on individual values and the country. So I respect your opinion. In your context, it may be right.
This part is a parallel line of values, and if we were bound together in the same community and had to set rules, I would have debated it. But since we both know we come from different regions and backgrounds, we know our agreement will never converge.
That is why I will only agree with your last opinion and will not discuss the parts you mentioned earlier. I agree with your last point that it "concentrates power in the hands of those who already have a lot of it"
Restricting access to guns is actually a pretty mainstream thing in most of the world.
I wonder if he'll ever realize that his silly definition of "freedom" is precisely what enabled the techno-oligarchy he rails against.
Ah yes, the thing wrong with AI is how it won't help you kill your wife.
I didn't check the author first so I was about to go "why the hell is this on the front page?" but oh well. geohot being geohot.
You must’ve missed all the other examples. There are plenty of less unrealistic and less scandalous hypotheticals.
it's a graphic example but illustrative of his point about autonomy/agency versus top-down dictated centralization
It is proof that the AI is not aligned with the user and is aligned with someone who has power over them.
> You cannot take over the world with tokens.
This is a very naive argument.
Can you take over the world with words? What is it that tokens can't do that words can?
Because there are historical examples of humans that have succeeded at taking over entire countries - with words, primarily. Including global superpower countries. And those humans were nowhere near "ASI" levels of intelligence advantage.
Human dictators and cult leaders didn't have the reasoning depth of a hypothetical advanced AI, nor the reasoning breadth of an AI that can just spawn more instances of itself whenever it needs a fully trusted agent to cover another area. They were confined to one body, one skull worth of reasoning power - and they still managed to do what they did.
Humans are extremely exploitable. And the world is already wired up for something that can take advantage of that to start taking control.
Intelligence is extremely powerful, because it's applied intelligence that enables humans to do what they do and bend nature to their will. If you aren't seeing the risks of creating an artificial system that would top the intelligence pole, you aren't looking.
> You cannot take over the world with tokens.
I have a nice thought experiment I like to do with people when confronted with "AI can't do x". Let's go back in time. How much do we need to go for this to become true? So let's try the 2000s.
Say you get a "fable/mythos/sol/gemini/kimi/glm/deepseek/whatever" in a box (and let's assume no guardrails). And you go back to the year 2005. It's "20 years ago", the world is slowly building back from the dotcom bubble, the Internet is really starting to happen, more and more things are interconnected, more and more things are connected to the Internet. Cool.
(for a bit of context, around that time we also saw the first high-impact worms like blaster that hit massive amounts of computers even reaching nuclear powerplants, we had a ton of ssh exploits that even made the movies, and security in general was a "nice to have")
I'd say that with the uber-model-in-a-box and a few prompts, you could reasonably make a case that you could design a worm that could infect 90-100% of the things connected to the Internet back then, stay as hidden as possible (in-memory stuff, vm execution, etc), move laterally into any network at inhuman speeds, and infiltrate almost every interconnected computer that has a link to the "public Internet".
Would that qualify as "take over the world"?
Then you could ask "what happens in 20 years from now?". And, thankfully, now we'll also have the AI on the blue side.
That’s a bit silly as we also could just write down a lot of exploits, sports bets, or stock market picks on paper and take over the world.
I’d be more interested in “how quickly could you develop 2005 era startup” if you went back in time to 2005 with Mythos level intelligence.
I bet a lot of the work has nothing to do with coding productivity and a lot to do with the aggregate day-to-day decisions and relationships made that span tech and business.
At the very beginning of the article he dismissed unrealistic expectations, like AI manipulating matter with some quantum magics. Time travel surely counts as such.
And then why bother about AI when Back to the Future reminds us, a sports almanac is all you need. ;-)
Haha. The time travel is part of the thought experiment, just a means to explore an idea. We don't have to actually do it. Let me put it another way. Would you be able to "control the world with tokens" today if you had access to a model 20y more advanced than you have today?
In your hypothetical, I think it would be more apt to ask if it could do so strictly using training data pre 2005. Seems more analogous to "could it take over the world today".
No, training data has nothing to do with this. My point was that "you can't take over the world with tokens" is not a valid counterpoint in the doomer's position. Because their view is that singularity can happen. And when that happens things become murky. And 20 years of progress can go in 2 months "real time". Or two weeks. You get the point.
(I don't share their view, btw. I was just trying to say that the argument of tokens can't affect the real world is not a good one, if you steelman their maximalist position)
That was basically stuxnet