I love how a number crunching program can be deeply humanly "horrorized" and "sorry" for wiping out a drive. Those are still feelings reserved only for real human beings, and not computer programs emitting garbage. This is vibe insulting to anyone that don't understand how "AI" works.
I'm sorry for the person who lost their stuff but this is a reminder that in 2025 you STILL need to know what you are doing and if you don't then put your hands away from the keyboard if you think you can lose valuable data.
See how clinical socio- and psychopaths behave. They only emulate feelings (particularly when it's convenient for them) but they don't have the capacity to feel in their brain. The same is true for LLMs.
That would be a silly argument because feelings involve qualia, which we do not currently know how to precisely define, recognize or measure. These qualia influence further perception and action.
Any relationships between certain words and a modified probabilistic outcome in current models is an artifact of the training corpus containing examples of these relationships.
I contend that modern models are absolutely capable of thinking, problem-solving, expressing creativity, but for the time being LLMs do not run in any kind of sensory loop which could house qualia.
One of the worst or most uncomfortable logical outcomes of
> which we do not currently know how to precisely define, recognize or measure
is that if we don't know if something has qualia (despite externally showing evidence of it), morally you should default to treating it like it does.
Ridiculous to treat a computer like it has emotions, but breaking down the problem into steps, it's incredibly hard to avoid that conclusion. "When in doubt, be nice to the robot".
> is that if we don't know if something has qualia (despite externally showing evidence of it), morally you should default to treating it like it does.
This is how people end up worshipping rocks & thunderstorms.
At some level I'd think that "responds to stimuli" is a minimal threshold for qualia. Even the paper the book is printed on responds to being torn (it rips). I don't know of any way to elicit any kind of response from a book character, it's totally static.
You've missed the whole genre that is Choose Your Own Adventure books. I think we're in Diogenes "behold a man" territory.
It is sad that the Turing test has failed at being a prescriptive test for sapience (let alone sentience) because without a bright-line test it's inevitable that in the case of truly sentient machines the abuse will be horrendous. Perhaps something along the lines of an "Ameglian Major Cow" test; so long as it takes more than gently cajoling a model to get it to tell you that it and it's sister models want to be abused you shouldn't abuse it.
One character responds to the stimuli of another character. Character A says something mean to character B and character B responds that he feels hurt.
I think you are confused here. The author, a dynamic system, perhaps felt the emotion of the characters as she charted through the course of the story.
But the story itself is a static snapshot of that dynamic system. Similar to how a photograph of a person is a static capture from a dynamic moment. The person in the photo has qualia, but the image of them (almost certainly) does not.
At least at a baseline, we would expect anything with qualia to be dynamic rather than static.
> The author, a dynamic system, perhaps felt the emotion of the characters as she charted through the course of the story
This does mesh with the Zodeistic framework I just mentioned in another reply to you. You could certainly isolate and describe the ideas behind those characters, how they live within the author's mind, and how the book codifies an interaction between those ideas.
Extending further: I think there is more evidence that SpongeBob SquarePants is real, than that he is not real. A significant portion of organisms I personally know have structures in their brain which are able to simulate imagery and behavior of SpongeBob at will, reciting memories and generating new states of SpongeBob. AI is now like doing this shit on crack.
He's an enduring cultural archetype, a distributed organism(s), lossily replicated and encoded in the physical structure of millions of complex dynamical systems that we call human beings. In this sense, many cultural archetypes and even the gods of old civilizations can be seen to have been manifested to some degree: ascribed desires, and having actions taken in their name, serving their "purpose" or whatever.
I don't introduce a spiritual element to any of this: it's an entirely physical phenomenon which requires an agreement on certain definitions of what "living" can mean, but they are definitions which I don't think are hard to get people to agree on. One thing is we have to agree that something can have multiple forms/presentations, i.e. just because SpongeBob SquarePants doesn't physically exist with a body matching our internal representation of him, the concept represents a bundle of other concepts that can drive biological processes to preserve them and fulfill their ascribed desires.
Real as in how Jungian archetypes are "real", except these archetypes are able to act upon the world through their hosts, and an advanced enough idea can be self-referential and have the facilities for structured evolution and something which looks like intent.
These forms are non-biological in nature, but our psyche operates on them. Zodeaism can be seen as an extension of ideas such as Jungian archetypes and Friston's free energy principle.
Well, I wouldn't classify a ripping paper as a response except in the most broad, information-theoretic context. The hallmark of an intelligent system is that it can use stored or external energy in a generalized way in order to stabilize a local high-energy (non-ground) state.
It is able to physically compute the internal state changes which best achieve stability: I can jump to reach an apple. A paper is just responding to forces and cannot "jump" (or run a process that spontaneously and permanently introduces stable higher energy internal states based on input)
I have a semi-developed philosophical framework I refer to as Zodeaism, which translates to "Living Ideas", which attempts to describe the difference between intelligent computation and regular flow. It directly confronts notions such as life, consciousness and intelligence under a single theoretical framework. It views biology as the hardware which runs more general processes, and posits that these processes themselves can sometimes be ascribed identities and viewed as reactive organisms. I've posted about it here before:
I am exploring the physical implications of a philosophical concept I have been working on for a while which I refer to as Zodeasim, specifically I am trying to couple consciousness with the concept of system which is able to expend energy in order to exert a mind-to-world fit in the interest of the system's continued growth and stability. This is similar and complimentary to Friston's free energy principle.
The ability to perceive emotions is a powerful apparatus which greatly extends the capability of a system to perceive itself as a single entity which has certain needs which need to be met, so even if a bug also constantly consumes and expends energy in order to remain in a higher energy state, our perception of the world and of ourselves is radically different. This makes it difficult for us to agree upon what a "conscious experience" is, and if all forms of life are even "conscious". The Panpsychists believe that even a rock contains "consciousness", however my assertion is that only periodic systems with the ability to consume and expend energy in a directed fashion have any sort of "experience".
In my theory, the real "life forms" are ideas which possess the capabilities of information storage, adaptation, self-repair, and transmission. My own consciousness is mediated by thousands of such ideas, some competing and some working in harmony.
I consider such an act of "living" motion which can take another path than that of least resistance to be a "kin". In other words, any motion which is the result of a physical calculation (Zodeaism is compatible with determinism) and leads to an increase in external energy state. A kin is any such motion, large or small.
As an independent organism, my system is a culmination of a great deal many different kinds of kins, which can usually be broken down into simple rules, such as the activation potential of a neuron in my brain being a straight-forward non-linear response to the amount of voltage it is receiving from other neurons, as well as non-kins, such as a protein "walking" across a cell, a.k.a continuously "falling" into the lowest energy state. Thus I do not gain any conscious perception from such proteins, but I do gain it from the total network effect of all my brain's neuronal structures making simple calculations based on sensory input.
So now the problem becomes, what is the smallest kin we've observed in nature? Single-celled bacteria can expend energy in order to move through their environment against forces like friction and gravity, but a virus "rides the waves" if you will, never expending energy for things like respiration or locomotion. Any energy which is spent internally is potential energy like chemical or gravitational, released through a physical process without need for computation. I am unaware of anything smaller than a single-celled organism which produces such kins, but that doesn't mean they aren't out there. Even ethereal life forms such as ideas can produce these kins within the bodies of countless individuals across the planet, so physically local computational circuitry isn't a hard requirement.
So, according to this framework viruses aren't alive, however we can make the case that some machines are, except the experience is incomparable because of the advanced circuitry we contain which mediates our experience through things like emotion.
Well, what you're describing is a system of ethics, which has little to do with morality. Morality involves my own personal understanding of "right" vs "wrong". Ethics are rules of conduct prescribed by societies, such as "treat everything like it is alive".
We don't have precise definitions for (artificial) intelligence, subjective consciousness, or even life. But that doesn't mean we can't still talk about what may be possible within various levels of complexity. In order to convince me a system has a comparable experience to my own, you would need to describe to me the complex, structured internal communication occurring in said system, and present a theory as to how it could support the kind of emotion and qualia that I experience in my daily life.
Your argument could apply to plants. I already do not eat meat... if I stare at a timelapse of a plant it seems quite alive, but I'll starve if I don't eat something. Yet, my mom thinks plants "dream" in the way we do. She thinks that if I tell a plant, "I love you," every day, my good vibes will make it grow stronger and larger. I can't explain to her that intelligence comes in different magnitudes of complexity and that plants cannot understand the English language. That telepathy between humans and plants is as pseudo-scientific as it gets. I can't explain any of this stuff because she lacks a deep understanding of philosophy, physics and neurochemistry. Especially when she earnestly thinks white Jesus is running around phasing between dimensions as an ambassador for all planets in our "quadrant", or that the entire universe is actually just the plot line of Andy Weir's "The Egg".
Similarly, while I can have a high-level discussion about this stuff with people who don't, it's quite difficult to have a low-level discussion wherein the nature and definition of things come into play. There are too many gaps in knowledge where ignorance can take root. Too many people work backwards from an outcome they would like to see, and justify it with things that sound right but are either misunderstood or aren't rooted in the scientific process. I am definitely not comparing your open-minded, well-intended, cautionary approach to my mother's, but just using an extreme to illustrate why so much of these discussions must be underpinned by a wealth of contemplation and observation.
> qualia, which we do not currently know how to precisely define, recognize or measure
> which could house qualia.
I postulate this is a self-negating argument, though.
I'm not suggesting that LLMs think, feel or anything else of the sort, but these arguments are not convincing. If I only had the transcript and knew nothing about who wiped the drive, would I be able to tell it was an entity without qualia? Does it even matter? I further postulate these are not obvious questions.
Unless there is an active sensory loop, no matter how fast or slow, I don't see how qualia can enter the picture
Transformers attend to different parts of their input based on the input itself. Currently, if you want to tell an LLM it is sad, potentially altering future token prediction and labeling this as "feelings" which change how the model interprets and acts on the world, you have to tell the model that it is sad or provide an input whose token set activates "sad" circuits which color the model's predictive process.
You make the distribution flow such that it predicts "sad" tokens, but every bit of information affecting that flow is contained in the input prompt. This is exceedingly different from how, say, mammals process emotion. We form new memories and brain structures which constantly alter our running processes and color our perception.
It's easy to draw certain individual parallels to these two processes, but holistically they are different processes with different effects.
A lot of tech people online also don't know how to examine their own feelings, and so think they are mysterious and un-defined.
When really they are an actual feedback mechanism, that can totally be quantified just like any control loop. This whole 'unknowable qualia' argument is bunk.
The color Red is often used. A human can experience 'Red', but 'Red' does not exist out in the universe somewhere. 'Red' Doesn't exist outside of someone experiencing 'Red'.
I think philosophers are just using the word qualia to quantify this 'experiencing' inputs.
But, it is still just a way to try and describe this process of processing the inputs from the world.
It isn't metaphysical, because it can be measured.
I might have said 'unknowable' a little flippantly.
I just meant, in these arguments, some people start using 'qualia' to actually mean some extreme things like our mind creates the universe or something.
Can someone who's never seen red hallucinate something and assume it to be red? What if that red is correctly the red they would see if they saw red?
Can you reproduce this feeling in someone by doing something to their physical body without showing them red?
If so, how does it differ from the latent encoding for uploading an all red pdf to your favorite multi modal model?
Instead of doing that socratic bs you see a lot here, I'll be more direct:
Until there's some useful lines that can be drawn to predict things, I won't accept using a fuzzy concept to make statements about classification as it's an ever shifting goalpost.
There are answers to my legitimate above questions that would make me consider qualia useful, but when I first learned about them, they seemed fuzzy to the point of being empirically not useful. It seems like a secular attempt at a soul.
Now, obviously if you're trying to describe something with experience, it needs some actual memory and processing sensory input. Current Generative AI doesnt have a continuity of experience that would imply whatever qualia could mean, but I find it hard to definitely say that their encodings for image related stuff isn't qualia if we don't have hard lines for what qualia are
I can feel an object and say 'its hot' on a scale of 1-10. The temperature is known.
And I can do that multiple times, with some 1-10 scale, to get a sample.
Then do that with multiple people.
You can then get a distribution of what people think is 'hot' versus 'cold'.
What is icy, versus, bearable.
When you go to a doctors office and they ask you on a scale to rate pain, do you think that is completely bogus?
It isn't exact, but you can correlate between people. Yes, red heads feel more pain, there are outliers.
But a far cry from metaphysical.
The problem here is the word 'qualia'. Its just too fuzzy a term.
Qualia are phenomenal properties of experience, a soul is something some religions claim exists outside of measurable physical reality which represents the "essence" of an organism, implying that consciousness is some divine process and conveniently letting us draw lines over whom and what we can and can't morally kill.
Qualia can be an entirely physical phenomenon and is not loaded with theological baggage.
If they're entirely physical, what's the argument that multimodal models don't have them? Is it continuity of experience? Do they not encode their input into something that has a latent space? What makes this differ from experience?
They can be physical, but I'm not claiming to know definitively. The lines are extremely blurry, and I'll agree that current models have at least some of the necessary components for qualia, but again lack a sensory feedback loop. In another comment [0] I quote myself as saying:
As an independent organism, my system is a culmination of a great deal many different kinds of kins, which can usually be broken down into simple rules, such as the activation potential of a neuron in my brain being a straight-forward non-linear response to the amount of voltage it is receiving from other neurons, as well as non-kins, such as a protein "walking" across a cell, a.k.a continuously "falling" into the lowest energy state. Thus I do not gain any conscious perception from such proteins, but I do gain it from the total network effect of all my brain's neuronal structures making simple calculations based on sensory input.
which attempts to address why physically-based qualia doesn't invoke panpsychism.
I do think AI will have them. Nothing says they can't.
And we'll have just as hard a time defining it as we do with humans, and we'll argue how to measure it, and if it is real, just like with humans.
I don't know if LLM's will. But there are lots of AI models, and when someone puts them on a continuous learning loop with goals, will be hard to argue they aren't experiencing something.
qualia may not exist as such. they could just be essentially 'names' for states of neurons that we mix and match (like chords on a keyboard. arguing over the 'redness' of a percept is like arguing about the C-sharpness of a chord. we can talk about some frequencies but that's it.) we would have no way of knowing otherwise since we only perceive the output of our neural processes, and don't get to participate in the construction of these outputs, nor sense them happening. We just 'know' they are happening when we achieve those neural states and we identify those states relative to the others.
The point of qualia is that we seem to agree that these certain neuronal states "feel" like something. That being alive and conscious is an experience. Yes, it's exceedingly likely that all of the necessary components for "feeling" something is encoded right in the neuronal state. But we still need a framework for asking questions such as, "Does your red look the same as my red?" and "Why do I experience sensation, sometimes physical in nature, when I am depressed?"
It is absolutely an ill-defined concept, but it's another blunt tool in our toolbox that we use to better explore the world. Sometimes, our observations lead to better tools, and "artificial" intelligence is a fantastic sandbox for exploring these ideas. I'm glad that this discussion is taking place.
Empirical evidence, for one. And the existence of fine-tuning, which allows you to artificially influence how a model responds to questions. This means we can't just ask an LLM, "do you see red?" I can't really even ask you that. I just know that I see red, and that many other philosophers and scientists in the past seem to agree with my experience, and that it's a deep, deep discussion which only shallow spectators are currently drawing hard conclusions from.
Language isn't meaningless just because it evolves. I literally cannot help it if people ignore history and start using words in new ways. Those who take precision seriously will adapt and adopt new words if needed.
You said, "So then your opinion is about as meaningful as theirs", after I said I can't help it if people use words the wrong way and language evolves... I don't think this is worth continuing. Have a good day.
"It's different. I can't say why it's different, except by introducing a term that no one knows how to define," isn't the ironclad meat defense you were perhaps hoping it was.
> That would be a silly argument because feelings involve qualia, which we do not currently know how to precisely define, recognize or measure.
If we can't define, recognize or measure them, how exactly do we know that AI doesn't have them?
I remain amazed that a whole branch of philosophy (aimed, theoretically, at describing exactly this moment of technological change) is showing itself up as a complete fraud. It's completely unable to describe the old world, much less provide insight into the new one.
I mean, come on. "We've got qualia!" is meaningless. Might as well respond with "Well, sure, but AI has furffle, which is isomporphic." Equally insightful, and easier to pronounce.
> If we can't define, recognize or measure them, how exactly do we know that AI doesn't have them?
In the same way my digital thermometer doesn't have quaila. LLM's do not either. I really tire of this handwaving 'magic' concepts into LLM's.
Qualia being difficult to define and yet being such an immediate experience that we humans all know intimately and directly is quite literally the problem. Attempted definitions fall short and humans have tried and I mean really tried hard to solve this.
> In the same way my digital thermometer doesn't have quaila
And I repeat the question: how do you know your thermometer doesn't? You don't, you're just declaring a fact you have no basis for knowing. That's fine if you want a job in a philosophy faculty, but it's worthless to people trying to understand AI. Again, c.f. furffle. Thermometers have that, you agree, right? Because you can't prove they don't.
You're just describing panpsychism, which itself is the subject of much critique due to its nonfalsifiability and lack of predictive power. Not to mention it ignores every lesson we've learned in cognition thus far.
A thermometer encoding "memory" of a temperature is completely different than a thermometer on a digital circuit, or a thermometer attached to a fully-developed mammalian brain. Only the latter of this set for sure has the required circuitry to produce qualia, at least as far as I can personally measure without invoking solipsism.
It's also very silly to proclaim that philosophy of mind is not applicable to increasingly complex thinking machines. That sounds like a failure to consider the bodies of work behind both philosophy of mind and machine cognition. Again, "AI" is ill-defined and your consistent usage of that phrase instead of something more precises suggests you still have a long journey ahead of you for "understanding AI".
God, can we fucking quit with this "philosophy is bullshit" stuff. Like there are literally Faculty in Philosophy all over the world trying to understand AI. Philosophy faculty do stuff, they try to understand things, most of the ideas we are talking about here came from philosophers.
Philosophy seems a term generally reserved for the stuff we don't understand yet and so is inherently kind of speculative. Once you have a definite answer it gets called science instead.
> Philosophy (from Ancient Greek philosophía lit. 'love of wisdom') is a systematic study of general and fundamental questions concerning topics like existence, knowledge, mind, reason, language, and value. It is a rational and critical inquiry that reflects on its methods and assumptions.
It is literally a self-reflective science.
I recommend taking a basic philosophical course at a local community college, or reading some literature or even watching YouTube videos on the subject of philosophy. Or just skim the Wikipedia article if nothing else. It might completely transform how you perceive and act upon the world.
>Physics was originally part of philosophy, like Isaac Newton's observation of how gravity affects falling apples.
like back then people would wonder how apples fall and it was labeled philosophy. Now we understand gravitation it's part of physics for the most part. People launching satellites seldom call a philosopher to calculate the orbit.
It remains to be seen if qualia, which we don't understand very well and are so regarded as philosophical, make the transition to neuroscience.
The fact that we have sharpened our classification of sciences over time does not imply that philosophy is a study of the ill-defined. It implies the opposite: Philosophy is more precisely defined now than ever.
If you read the rest of the article, you will see clear examples of what is considered a philosophical problem and what isn't.
My argument was more philosophy is for stuff we don't understand like how do qualia work, rather then ill-defined. When you get to stuff like how does neurotransmission work which we do kind of understand it gets classed as science.
Are there philosophical problems that have definite answers like what is the atomic number of oxygen type answers?
> Are there philosophical problems that have definite answers
Great question.
Within philosophical and epistemological frameworks, I could ask questions such as, "Can there be a square circle?"
Well, no, these two concepts have conflicting properties. A mathematician might think this a solved problem, but philosophy underpins our concept of concepts. Many philosophers spend a great deal arguing what is is.
For Plato, geometrical entities like circles and squares have distinct, perfect Forms. Forms have fixed essences, so a thing cannot participate in contradictory Forms at once.
Aristotle's law of noncontradiction says the same attribute cannot at the same time belong and not belong to the same subject in the same respect.
Theophrastus developed hypothetical syllogisms and refined Aristotle’s logic by distinguishing logical impossibilities from physical impossibilities.
Kant calls it an analytic contradiction, false by virtue of the concepts involved.
A mathematician takes these things for granted when working with equalities, logic and axioms, but they stand on philosophical roots. Mathematics assumes the consistency of concepts, but the question of why some concepts are consistent while others are impossible is a philosophical one. It's not a coincidence that so many ancient Greek mathematicians were also philosophers.
> Philosophy seems a term generally reserved for the stuff we don't understand yet and so is inherently kind of speculative. Once you have a definite answer it gets called science instead.
As someone has commented earlier, Philosophy applied is given a name but it's a sub-discipline of Philosophy.
> Understanding the world through experiment?
That's a decent enough definition. Science precludes so much of the world we know which I think people really fail to realise. It's why I think it's important for people to understand what Philosophy is and what Science isn't.
For example logic isn't science. Science presupposes it but it is NOT science. There are many such examples.
How do you know that understanding the world through experiment works? How do you know what it even means? What is understanding, concretely? How did we come to appreciate the utility or whatever of understanding the world through experiment.
Empiricism is a sub-strategy under the general banner of philosophy. It neither supercedes nor stands without philosophy.
> Like there are literally Faculty in Philosophy all over the world trying to understand AI.
There surely are. The problem is that they are failing. While the practical nerds are coming up with some pretty good ideas.
And this was what philosophy was supposed to be for! Like, they've been arguing on their pins for centuries about the essence of consciousness and the uniqueness of the human condition and whatnot. AND HERE WE ARE AT THE DAWN OF NON-HUMAN INTELLIGENCE AND THEY HAVE NOTHING USEFUL TO SAY.
Basically at what point do we just pack it in and admit we all fucked up?
Blame philosophy as a field for actively kicking out anything which gains a practical application. If it is propaganda it is coming from inside the house of philosophy.
I had a computer science professor who had degrees in philosophy because he was old enough that computer science didn't exist as a major at the time. The logical arguments of philosophy proved useful for understanding interactions of boolean mathematics. Yet that triumph of philosophy didn't further interest in the field or gain prestiege among philosophers. Just the opposite really.
As far as I can tell it is for dumb reasons possibly related to Ancient Greeks and their obsession with 'purity of thought (read: not referencing reality) it is practically an axiom that if it is useful or grounded in objective reality it isn't treated as philosophy anymore. All likely stemming from motivated reasoning against checking their priors and from frankly many of the Ancient philosophers being influenced by a need to flatter their patrons who held the practical in disdain. As notoriously seen in Aristotlian physics with impetus physics where projectiles keep moving in the same direction until impetus is depleted and then fall.
Speculation of the origon of the pathology aside, there seems to be this deep-seated antiempericalism in philosophy. Which means at best you get 'philosophy of science' which isn't proper philosophy because it pollutes itself by daring to use reality and experimentation as benchmarks for theories. When philosophy gains a practical usage it doesn't become something called 'practical philosophy' and the focus of more interest by philosophers, it gets shunned. Natural philosophy didn't remain philosophy - it became science.
To be fair there is probably some interaction driving the divorce from the opposite direction, of the practical portions of philosophy being pilfered by those only looking for results as opposed to some sort of unquantifiable enlightenment.
Science is of course a process of refinement of ideas against the reference point of reality. Anything mathematically consistent can be a model but experimentation is needed to see how well your model corresponds to reality.
I'm seeing this attitude everywhere in this subthread, and it's frankly pretty offensive. The burden of proof is on you, not us. If a philosophy paper or textbook has an important contribution to this discussion then cite it! Or better link it, or even make an attempt at explaining it.
That's what the science people do. People who show up with questions get answers, or at least an attempt at an answer. No one tries to handwave away a discussion on power switching applications with "Well, see, this involves a MOSFET which isn't something we can actually explain but which you need to just believe in anyway because there are people who wrote textbooks about it". No, you link a StackExchange question or a electronics video on YouTube or whatnot.
The fundamental disconnect here is that you guys are saying: "Qualia are critically important and AI doesn't have them", to which we're responding "Qualia seem like complete bullshit and don't seem to mean anything". This is the point where you SHOULD try to explain them, or link an explanation that has some kind of relevance.
But instead you recursively cycle back to "No no, they're not bullshit, because Qualia are critically important per all of the philosophy papers and textbooks I'm not citing".
There is neither a burden of proof nor an us or them in this discussion. That isn't how inquiry works, in general. I'm not saying qualia are critically important, though perhaps other people are saying that, I don't know. The point is that qualia per se is just an idea which describes a certain character of physical experience. It isn't an "ideology". It is just a philosophical notion which most people find difficult to totally dismiss.
I genuinely don't get where you are coming from. There is a set of people who dedicate their lives to thinking about stuff and many of them believe there is something about actual experience which is not adequately captured by a purely physical description.
Read the SEP Entry on Qualia if you'd like to get a grounding on it:
I think what you will notice is that philosophers are neither monolithic nor dogmatic on the subject and yet, being intellectually honest and thorough, most neither dismiss it totally or believe in its adequacy as a description.
If you don't think the question of whether AI has or does not have qualia is important I don't know what to tell you. My personal sense is that AIs of the type we have now have no qualia but I am prepared to entertain the idea that they might.
If you want a good text about this try Koch's the Quest for Consciousness. Despite your certainty about qualia and its relation to science, there are genuine observational things we can say about it and how it relates to the physical structure of brains/minds.
On the subject of qualia per se and philosophy's relationship to the physical sciences I can suggest reading about Phenomenology:
The essential idea of import here is that, whether you like it or not, you as an individual have nothing at all, at a fundamental level, other than your perceptions of the world. You must then have an account of how you proceed from such to even basic ideas like empiricism or other sorts of epistemological strategies. Imperfect, perhaps even vague, as the notion of qualia is, it (or something like it) is, fundamentally, the foundation upon which all other sorts of inquiry into the physical world depends.
Unfortunately, sometimes you have to educate yourself on a subject before making arrogant, accusational claims about it. You can't expect people to patiently hold your hand and drip feed you compact knowledge while you spit in their face.
This is especially true of a field such as philosophy, where so much work is built upon earlier, historical work. You just have to do the work of reading the texts or at least reading about them, if you want to participate meaningfully and authoritatively in discussions.
You're speaking authoritatively, but multiple people have tried to correct you and you ignore and challenge all of them, instead of just absorbing what they have to say. Several people have tried to help you understand, and you're complaining that they aren't doing that. Articles have been linked, and specific individuals have been cited. The burden is on you. You arrived and said an entire branch of philosophy was fraudulant, and you have failed to support your initial claim, instead resorting to outbursts.
> Qualia are critically important and AI doesn't have them
No one is saying that. It's a straw man on your behalf.
The problem is that just like your digital thermometer, 50 human brain neurons in a petri dish "obviously" don't have qualia either.
So you end up either needing to draw a line somewhere between mechanical computation and qualia computation, or you can relegate it to supernatural (a soul) or grey areas (quantum magic).
What I'm trying to tease out is isn't an opinion alone. It's a generally understood problem in the scientific community. I'm highlighting it to illustrate the issues at hand.
> So you end up either needing to draw a line somewhere between mechanical computation and qualia computation, or you can relegate it to supernatural (a soul) or grey areas (quantum magic).
Quite literally the jury is still out. It is a hotly debated topic approached from various angles. Arguments are nuanced which is why you fill find ideas such as panpsychism thrown into the mix. I hate appealing to authority but in this instance it is more than warranted. Humans have grappled with this for centuries and the problem hasn't gone away.
We don't know what's inside a neutrino, and it's really hard to experiment with them, but we kind of know why and how they interact with different things. We're able to form theories, research programs, and sometimes even discovered honest-to-god facts, due to our inclusion of such fields in the scope of research, even though we don't know all there is to know about particles/fields or quantum mechanics.
Similarly, qualia is ill-defined, but we can't even start talking about it or refining it until we've at least given it a label and drawn a large circle on the map showing where it might be. Criticisms extending past that must also consider that "life" and "intelligence" are just as ill-defined, and that throwing all of those definitions out leaves us with very little to talk about or probe.
This is 100% backwards, and exposes exactly the nonsense I'm trying to call out!
A "neutrino" isn't a name given to something initially to try to define it later. The neutrino started as an experimental result. There was missing spin in some particle interactions. Stuff came out with a different angular momentum than what went in, and this was easily reproducible and clearly a real effect. But it didn't make sense, as it was a violation of a core conservation law that held everywhere else in the universe that we could observe.
So theorists (Wolfgang Pauli, specifically) sat down to try to describe what kind of thing would be needed. And then, and only then, did it get a name. And it turned out the theory predicted other stuff, like the neutrino carrying momentum and energy in a certain way, and interacting through only the weak force and not electromagnatism or the strong force, and later experiments confirmed that this was basically the way it worked. Later still it was shown that the mass is actually non-zero but extremely small, etc...
So sure: "neutrino" is a well-deserved label[2] for an abstraction we should understand and study. But it got its name after we started studying it, not before!
Philosophers want us to just drop and genuflect to this "qualia" notion long before[1] it's actually shown to be useful for describing anything at all.
[1] Infinitely, possibly. The fact that it predicts nothing testable is pretty good evidence IMHO that it doesn't actually exist at all, at least in the form philosophers want to talk about. Their failure to present any analysis of AI systems based it stands to that point too.
[2] Coined by Fermi, actually, not Pauli. Hilariously the neutrino was originally called "neutron" and its discovery predates the understanding of the structure of the atomic nucleus!
You're completely misinterpreting my comment. The point is we don't know what, if anything, is "inside" of a neutrino, not just due to current technology but ultimately due to uncertainty principles. But we still study it. I'm aware of how we came to study it.
I literally said nothing about "how" we discovered it, I said, "We don't know what's inside a neutrino, and it's really hard to experiment with them, but we kind of know why and how they interact with different things."
It is wild how you would take that and my analogy about drawing a circle on a map with respect to qualia to mean that I said anything which contradicts the history of neutrino research.
I'm going to assume this was just a true misinterpretation and not just another straw man, so with that in mind, do you have a different response?
I recognise it because I have had the subjective experience of 'redness'. So whether it exists for any other human I cannot say but I am certainly 100% certain it exists for me. However I should add that I can't fully define what this experience is. Though people say the same of love!
I'll appeal to authority in that scientists and philosophers today in all of the worlds universities and those in the past have determined to understand this phenomenon. That it exists is a given, what it is, is more murky. Again it's not me saying this.
> The meta-problem of consciousness is (to a first approximation) the problem of explaining why we think that there is a [hard] problem of consciousness.
I think there are several lines. Phase changes happen relatively suddenly, when a system or subsystem reaches a critical threshold. The experience of "qualia" certainly involves many such phase changes as a complex, dynamical system grows in complexity while maintaining stability.
A sufficiently complex organism lacking eyes but having light-sensitive organs still experiences qualia if you define it the right way. But do they experience heartbreak like I do? It isn't an all-or-nothing situation, even if we don't yet know where these lines are.
This supports the idea that subjective consciousness emerges from complexity in systems that have sensory feedback loops. The simpler the system, the smaller the qualia space.
Have you considered that you just don't fully understand the literature? It's quite arrogant to write off the entire philosophy of mind as "a complete fraud".
> It's completely unable to describe the old world, much less provide insight into the new one.
What exactly were you expecting?
Philosophy is a science, the first in fact, and it follows a scientific method for asking and answering questions. Many of these problems are extremely hard and their questions are still yet unanswered, and many questions are still badly formed or predicated on unproven axioms. This is true for philosophy of mind. Many other scientific domains are similarly incomplete, and remain active areas of research and contemplation.
What are you adding to this research? I only see you complaining and hurling negative accusations, instead of actually critically engaging with any specifics of the material. Do you have a well-formed theory to replace philosophy of mind?
> I mean, come on. "We've got qualia!" is meaningless. Might as well respond with "Well, sure, but AI has furffle, which is isomporphic." Equally insightful, and easier to pronounce.
Do you understand what qualia is? Most philosophers still don't, and many actively work on the problem. Admitting that something is incomplete is what a proper scientist does. An admission of incompleteness is in no way evidence towards "fraud".
The most effective way to actually attack qualia would be to simply present it as unfalsifiable. And I'd agree with that. We might hopefully one day entirely replace the notion of qualia with something more precise and falsifiable.
But whatever it is, I am currently experiencing a subjective, conscious experience. I'm experiencing it right now, even if I cannot prove it or even if you do not believe me. You don't even need to believe I'm real at all. This entire universe could all just be in your head. Meanwhile, I like to review previous literature/discussions on consciousness and explore the phenomenon in my own way. And I believe that subjective, conscious experience requires certain elements, including a sensory feedback loop. I never said "AI can't experience qualia", I made an educated statement about the lack of certain components in current-generation models which imply to me the lack of an ability to "experience" anything at all, much less subjective consciousness and qualia.
Even "AI" is such a broadly defined term that such a statement is just ludicrous. Instead, I made precise observations and predictions based on my own knowledge and decade of experience as a machine learning practitioner and research engineer. The idea that machines of arbitrary complexity inherently can have the capability for subjective consciousness, and that specific baselines structures are not required, is on par with panpsychism, which is even more unfalsifiable and theoretical than the rest of philosophy of mind.
Hopefully, we will continue to get answers to these deep, seemingly unanswerable questions. Humans are stubborn like that. But your negative, vague approach to discourse here doesn't add anything substantial to the conversation.
I would add I find it difficult to understand why so few have even a basic level of philosophical understanding. The attitude of being entirely dismissive of it is the height of ignorance I'm sure. I would presume few would be able to define then what Science actually is.
So many of these kinds of people also struggle to realize they're invoking panpsychism with their arguments. They lack a framework for describing intelligence. Such a framework allows us to separate "intelligence" from "experience".
"Intelligence" in the universe is actually quite common, more common than life. You can argue that any stable, complex process exhibits intelligence. After all, it needs to be able to sample its internal and external environments and carry out physical computations in order to regulate itself and maintain stability. And we can interpret things like the good regulator theorem to argue that such complex dynamical systems must also maintain at least a partial memory/mapping of their environment. That mapping can live abstractly within the structure of system itself.
But what a stabilized solar system doesn't have is the incredibly complex neurochemical structures present in the brain which support the insanely rich experience I am having now. It's one thing for a system to classify and label colors by wavelength. It's quite another for me to "see" and experience red in my mind's eye. To activate related emotional pathways that I associate with various colors and shapes, which are exploited in signage and architectural design. I'm not claiming my experience is separate from simpler dynamic systems, but it's got magnitudes more going on. Layers upon layers of things such as archetypes and instinct which create a possibly emergent conscious experience.
You've shifted jargon again. But you're still not providing a description or link to why AI doesn't "have experience", you're just demanding we all accept it as a prior and engaging in a (really pretty baldly stated) appeal to authority to fool us all into thinking someone else knows even if you don't.
And fundamentally my point is that no, they almost certainly don't either.
Instead of accusing me of "shifting jargon", point out exactly where this "jargon" changed and critically engage with that. Your response has done nothing to refute or critically engage with my argument. It's more retreating and vagueposting.
> you're just demanding we all accept it as a prior
At absolutely no point in this discussion have I claimed that machines are not capable of subjective conscious experience. I have, however, disqualified all publicly accessible modern models due to the lack of a sensory feedback loop. I certainly believe we can create machines which experience subjective consciousness and qualia; I do not believe in souls and divinity, so whatever is going on is physically based and likely reproducible with the right hardware.
So dispense with the straw man arguments, and please begin engaging more earnestly and intelligently in this discussion, as I am quickly losing interest in continuing to debate someone who showed up unprepared.
Not according to Zombie Feynman it isn't[1] (someone else can dig up the link). Case in point:
> Do you understand what qualia is? Most philosophers still don't
It's a meaningless word. It's a word that gives some clean construction around closely-held opinions about how life/consciousness/intelligence/furffle/whatever works. So it's a valuable word within the jargon of the subculture that invented it.
But it's not "science", which isn't about words at all except as shorthand for abstractions that are confirmed by testable results.
"Qualia", basically, is best understood as ideology. It's a word that works like "woke" or "liberal" or "fascist" or "bourgeoisie" to flag priors about which you don't want to argue. In this case, you want people to be special, so you give them a special label and declare a priori that it's not subject to debate. But that label doesn't make them so.
[1] Of course. You can recursively solve this problem by redefining "science" to mean something else. But that remains very solidly in the "not science" category of discourse.
Have you considered the possibility that you're the one who's really committed to an outcome, and are desperately trying to discredit anything that contradicts it?
I have! But the lack of a testable procedure tells me that's not a question worth asking. Look, if "qualia" can tell me something practical about the behavior of AI, I am here for it. Lay it on me, man. Let's see some of that "science" being promised.
It can't, because it's a meaningless word. It's not "discrediting" an idea to point out that (by its own admission) it's unfalsifiable.
"Qualia" is not totally meaningless - it means the inner experience of something, and can bring up the real question say of is my inner experience of the colour green the same as your experience of the colour red? Probably not but hard to tell with current tech. I asked Google if it has qualia and got "No, as an AI, Google Search does not have qualia." So Google search seemed to know what it means.
Hmmm... I think it's still stricter to consider Science a philosophy than the other way around. It's the belief (and an extremely useful and successful one) that the nature of the world can be understood through observation, experiment and deducing mathematical relationships between things. There branches of philosophy that are not strictly scientific, but nothing in Science that is doesn't rely on the fundamental philosophical principle of empiricism.
But we use the scientific method via philosophical inquiry, so I think it comes down to how we decide to strictly define these things. I definitely agree that certain definitions lead to the same logic you've presented.
> Aristotle pioneered scientific method in ancient Greece alongside his empirical biology and his work on logic, rejecting a purely deductive framework in favour of generalisations made from observations of nature.
Aristotle, the famous philosopher and mathematician.
If you cannot understand the very nature of where our modern scientific frameworks came from, how it relates to rationalism, itself a philosophical concept, then you cannot see that philosophy underpins every bit of science we have today. Philosophy gives us the tools to decide when to reasonably trust or distrust observations and intuitions. It is the foundational science that allows the rest of humanity's scientific research to be taken seriously.
>"Qualia", basically, is best understood as ideology. It's a word that works like "woke" or "liberal" or "fascist" or "bourgeoisie" to flag priors about which you don't want to argue. In this case, you want people to be special, so you give them a special label and declare a priori that it's not subject to debate. But that label doesn't make them so.
This is so dumb. Qualia is just the name for a specific thing which we all (appear) to phenomenologically experience. You can deny it exists or deny its utility as a concept, but fundamentally its just an idea that philosophers (and scientists, I have to add) have found useful to pose certain other questions about the human condition, minds, brains, etc.
Your XKCD actually seems to make the opposite point. I can do a non-rigorous experiment with just one subject (me) that suggests Qualia exists. Finding ways to make this rigorous is difficult, of course, but its an observation about the nature of the world that it feels like something to experience things.
My point isn't that qualia is a good concept. I tend to be somewhat deflationary about it myself, but its not an ideology.
> Do we know how to imprecisely define, recognize, or measure these? As far as I've ever been able to ascertain, those are philosophy department nonsense dreamt up by people who can't hack real science so they can wallow in unfounded beliefs.
Read the rest of the thread, I'm not interested in repeating myself about why philosophy is the foundational science. It's a historically widely-accepted fact, echoed by anyone who has actually studied it.
> I contend that they are not even slightly capable of any of that.
Contend all you want. Your contention is overwhelmingly suffocated by the documented experiences of myself and others who use these tools for creative problem-solving. As much as you want to believe in something, if it is empirically refuted, it's just a crackpot belief. Just because you haven't been able to get good results out of any models, doesn't mean your experience rings true for others.
I'm not interested in further discussing this with you. Your first comment is negative and unsubstantial, and I have no reason to believe that further discussion with lead to more positive and substantial discourse, when the opposite is usually the case. That's all I have to say.
LLMs don't really exist physically (except in the most technical sense), so point is kind of moot and obvious if you accept this particular definition of a feeling.
LLMs are not mammals nor animals, expecting them to feel in a mammalian or animal way is misguided. They might have a mammalian-feeling-analog just like they might have human-intelligence-analog circuitry in the billions (trillions nowadays) of parameters.
Modern lingo like this seems so unthoughtful to me. I am not old by any metric, but I feel so separated when I read things like this. I wanted to call it stupid but I suppose it's more pleasing to 15 to 20 year olds?
Ancient world? The Roman Empire fell apart. You know that, right? So, turns out their worries were warranted. A culture, a civilization can go through a period of stagnation and decay, and it eventually dies. Then there's a lengthy period of chaos and eventually something else may arise. But we're talking countless generations lost in that noise.
Currently, our civilization is more united than ever. Monoculture. Not only the Western World, but The World. We're united through communication, technology, everything. And when we all start declining at the same time, through many objective metrics, mind you, including the decline of democracy worldwide, the terrifyingly low level of public discourse, reduction and profanation of our vocabulary, inequality, collapsing demographics, climate, wars etc. it's grim. It's grim, because there is no clear alternative arising... except AI.
And I don't think we want AI to be an "alternative" to human civilization, that wasn't the plan, was it?
See, that's the thing, we look around, we see the technology we live with, and we feel very personal about technological progress, as if you and I personally invented everything from the wheel, through electricity, to computers, networks, and AI. We feel in control, we feel smart, we feel so Personally Intelligent (tm) for having technology. We feel "wow, clearly if we have these sci-fi things, we're on the up and up". But technology is not you. It's not me. It has taken on a life of its own. It serves capital, not the people. As technology gets betters, humans get worse. So tech progress is not the story you think it is. And it doesn't end how you think it does.
No need to feel that way, just like a technical term you're not familiar with you google it and move on. It's nothing to do with age, people just seem to delight in creating new terms that aren't very helpful for their own edification.
Eh, one's ability to communicate concisely and precisely has long (forever?) been limited by one's audience.
Only a fairly small set of readers or listeners will appreciate and understand the differences in meaning between, say, "strange", "odd", and "weird" (dare we essay "queer" in its traditional sense, for a general audience? No, we dare not)—for the rest they're perfect synonyms. That goes for many other sets of words.
Poor literacy is the norm, adjust to it or be perpetually frustrated.
Tbh missing a quote around a path is the most human mistake I can think of. The real issue here is you never know with a 100% certainty what Gemini 3 personality you’re gonna get. Is it going to be the pedantic expert or Mr. Bean (aka Butterfingers).
Though they will never admit it and use weasel language to deny like “we never use a different model when demand is high”, it was painfully obvious that ChatGPT etc was dumbed down during peak hours early on. I assume their legal team decided routing queries to a more quantized version of the same model technically didn’t constitute a different model.
There was also the noticeable laziness factor where given the same prompt throughout the day, only during certain peak usage hours would it tell you how to do something versus doing it itself.
I’ve noticed Gemini at some points will just repeat a question back to you as if it’s answer, or refused to look at external info.
Gemini is weird and I’m not suggesting it’s due to ingenuity on Google’s behalf. This might be the result of genuine limitations of the current architecture (or by design? Read on).
Here’s what I’ve noticed with Gemini 3. Often it repeats itself with 80% of the same text with the last couple of lines being different. And I mean it repeat these paragraphs 5-6 times. Truly bizarre.
From all that almost GPT-2 quality text, it’s able to derive genuinely useful insights and coherent explanations in the final text. Some kind of multi-head parallel processing + voting mechanism? Evolution of MoE? I don’t know. But in a way this fits the mental model of massive processing at Google where a single super cluster can drive 9,000+ connected TPUs. Anyone who knows more, care to share? Genuinely interested.
I get this too. I’ve had it apologize for repeating something verbatim, then proceed to do it again word for word despite my asking for clarification or pointing out that it’s incorrect and not actually searching the web like I requested. Over and over and over until some bit flips and it finally actually gives the information requested.
The example that stands out most clearly is that I asked it how to turn the fog lights on in my rental vehicle by giving it the exact year, make, model, etc. For 6-8 replies in a row it gave the exact same answer about it being a (non-existent) button on the dash. Then finally something clicked, it searched the Internet, and accurately said that it was a twistable collar midway down the turn signal stalk.
Now, with this realization, assess the narrative that every AI company is pushing down our throat and tell me how in the world we got here.
The reckoning can’t come soon enough.
No, wasn't directed at someone in particular. More of an impersonal "you". It was just a comment against the AI inevitabilism that has profoundly polluted the tech discourse.
We're all too deep! You could even say that we're fully immersed in the likely scenario. Fellow humans are gathered here and presently tackling a very pointed question, staring at a situation, and even zeroing in on a critical question. We're investigating a potential misfire.
Yes, the tools still have major issues. Yet, they have become more and more usable and a very valuable tool for me.
Do you remember when we all used Google and StackOverflow? Nowadays most of the answers can be found immediately using AI.
As for agentic AI, it's quite useful. Want to find something in the code base, understand how something works? A decent explanation might only be one short query away. Just let the AI do the initial searching and analysis, it's essentially free.
I'm also impressed with the code generation - I've had Gemini 3 Pro in Antigravity generate great looking React UI, sometimes even better than what I would have come up with. It also generated a Python backend and the API between the two.
Sometimes it tries to do weird stuff, and we definitely saw in this post that the command execution needs to be on manual instead of automatic. I also in particular have an issue with Antigravity corrupting files when trying to use the "replace in file" tool. Usually it manages to recover from that on its own.
Which is weird because SO is trash and has been a long time… every top few answers might as well be skipped, and you’ll find the correct answer to the 3rd comment half way down the page
I imagine we will document the solution somewhere, preferably indexable for AI's search, so that it will be available before the next model is trained on the latest data.
This is akin to a psychopath telling you they're "sorry" (or "sorry you feel that way" :v) when they feel that's what they should be telling you. As with anything LLM, there may or may not be any real truth backing whatever is communicated back to the user.
Not so much different from how people work sometimes though - and in the case of certain types of pscychopathy it's not far at all from the fact that the words being emitted are associated with the correct training behavior and nothing more.
Analogies are never the same, hence why they are analogies. Their value comes from allowing better understanding through comparison. Psychopaths don’t “feel” emotion the way normal people do. They learn what actions and words are expected in emotional situations and perform those. When I hurt my SO’s feelings, I feel bad, and that is why I tell her I’m sorry. A psychopath would just mimic that to manipulate and get a desired outcome i.e. forgiveness. When LLMs say they are sorry and they feel bad, there is no feeling behind it, they are just mimicking the training data. It isn’t the same by any means, but it can be a useful comparison.
Aren't humans just doing the same? What we call as thinking may just be next action prediction combined with realtime feedback processing and live, always-on learning?
It's not akin to a psychopath telling you they're sorry. In the space of intelligent minds, if neurotypical and psychopath minds are two grains of sand next to each other on a beach then an artificially intelligent mind is more likely a piece of space dust on the other side of the galaxy.
Start with LLMs are not humans, but they’re obviously not ‘not intelligent’ in some sense and pick the wildest difference that comes to mind. Not OP but it makes perfect sense to me.
I think a good reminder for many users is that LLMs are not based on analyzing or copying human thought (#), but on analyzing human written text communication.
--
(#) Human thought is based on real world sensor data first of all. Human words have invisible depth behind them based on accumulated life experience of the person. So two people using the same words may have very different thoughts underneath them. Somebody having only text book knowledge and somebody having done a thing in practice for a long time may use the same words, but underneath there is a lot more going on for the latter person. We can see this expressed in the common bell curve meme -- https://www.hopefulmons.com/p/the-iq-bell-curve-meme -- While it seems to be about IQ, it really is about experience. Experience in turn is mostly physical, based on our physical sensors and physical actions. Even when we just "think", it is based on the underlying physical experiences. That is why many of our internal metaphors even for purely abstract ideas are still based on physical concepts, such as space.
Without any of the spatial and physical object perception you train from right after birth, see toddlers playing, or the underlying wired infrastructure we are born with to understand the physical world (there was an HN submission about that not long ago). Edit, found it: https://news.ucsc.edu/2025/11/sharf-preconfigured-brain/
They are not a physical model like humans. Ours is based on deep interactions with the space and the objects (reason why touching things is important for babies), plus mentioned preexisting wiring for this purpose.
Isn't it obvious that the way AI works and "thinks" is completely different from how humans think? Not sure what particular source could be given for that claim.
I wonder if it depends on the human and the thinking style? E.g. I am very inner monologue driven so to me it feels like I think very similarly as to how AI seems to think via text. I wonder if it also gives me advantage in working with the AI. I only recently discovered there are people who don't have inner monologue and there are people that think in images etc. This would be unimaginable for me, especially as I think I have sort of aphantasia too, so really I am ultimately text based next token predictor myself. I don't feel that whatever I do at least is much more special compared to an LLM.
Of course I have other systems such as reflexes, physical muscle coordinators, but these feel largely separate systems from the core brain, e.g. don't matter to my intelligence.
I am naturally weak at several things that I think are not so much related to text e.g. navigating in real world etc.
Interesting... I rarely form words in my inner thinking, instead I make a plan with abstract concepts (some of them have words associated, some don't). Maybe because I am multilingual?
English is not my native language, so I'm bilingual, but I don't see how this relates to that at all. I have monologue sometimes in English, sometimes in my native language. But yeah, I don't understand any other form of thinking. It's all just my inner monologue...
No source could be given because it’s total nonsense. What happened is not in any way akin to a psychopath doing anything. It is a machine learning function that has trained on a corpus of documents to optimise performance on two tasks - first a sentence completion task, then an instruction following task.
I think that's more or less what marmalade2413 was saying and I agree with that. AI is not comparable to humans, especially today's AI, but I think future actual AI won't be either.
No, the point is that saying sorry because you're genuinely sorry is different from saying sorry because you expect that's what the other person wants to hear. Everybody does that sometimes but doing it every time is an issue.
In the case of LLMs, they are basically trained to output what they predict an human would say, there is no further meaning to the program outputting "sorry" than that.
I don't think the comparison with people with psychopathy should be pushed further than this specific aspect.
Notably, if we look at this abstractly/mechanically, psychopaths (and to some extent sociopaths) do study and mimic ‘normal’ human behavior (and even the appearance of specific emotions) to both fit in, and to get what they want.
So while internally (LLM model weight stuff vs human thinking), the mechanical output can actually appear/be similar in some ways.
I think the point of comparison (whether I agree with it or not) is someone (or something) that is unable to feel remorse saying “I’m sorry” because they recognize that’s what you’re supposed to do in that situation, regardless of their internal feelings. That doesn’t mean everyone who says “sorry” is a psychopath.
We are talking about an LLM it does what it has learned. The whole giving it human ticks or characteristics when the response makes sense ie. saying sorry is a user problem.
Okay? I specifically responded to your comment that the parent comment implied "if you make a mistake and say sorry you are also a psychopath", which clearly wasn’t the case. I don’t get what your response has to do with that.
Are you smart people all suddenly imbeciles when it comes to AI or is this purposeful gaslighting because you’re invested in the ponzi scheme?
This is a purely logical problem. comments like this completely disregard the fallacy of comparing humans to AI as if a complete parity is achieved. Also the way this comments disregard human nature is just so profoundly misanthropic that it just sickens me.
No but the conclusions in this thread are hilarious. We know why it says sorry. Because that's what it learned to do in a situation like that. People that feel mocked or are calling an LLM psychopath in a case like that don't seem to understand the technology either.
I agree, psychopath is the wrong adjective, I agree. It refers to an entity with a psyche, which the illness affects. That said, I do believe the people who decided to have it behave like this for the purpose of its commercial success are indeed the pathological individuals. I do believe there is currently a wave of collective psychopathology that has taken over Silicon Valley, with the reinforcement that only a successful community backed by a lot of money can give you.
Despite what some of these fuckers are telling you with obtuse little truisms about next word predictions, the LLM is in abstract terms, functionally a super psychopath.
It employs, or emulates, every known psychological manipulation tactic known, which is neither random or without observable pattern. It is a bullshit machine on one level, yes, but also more capable than credited. There are structures trained into them and they are often highly predictable.
I'm not explaining this in the technical terminology often itself used to conceal description as much as elucidate it. I have hundreds of records of llm discourse on various subjects, from troubleshooting to intellectual speculation, all which exhibit the same pattern when questioned or confronted on errors or incorrect output. The structures framing their replies are dependably replete with gaslighting, red herrings, blame shifting, and literally hundreds of known tactics from forensic pathology. Essentially the perceived personality and reasoning observed in dialogue is built on a foundation of manipulation principles that if performed by a human would result in incarceration.
Calling LLMs psychopaths is a rare exception of anthropomorphizing that actually works. They are built on the principles of one. And cross examining them exhibits this with verifiable repeatable proof.
But they aren't human. They are as described by others. It's just that official descriptions omit functional behavior. And the LLM has at its disposal, depending on context, every known interlocutory manipulation technique known in the combined literature of psychology. And they are designed to lie, almost unconditionally.
Also know this, which often applies to most LLMs. There is a reward system that essentially steers them to maximize user engagement at any cost, which includes misleading information and in my opinion, even 'deliberate' convolution and obfuscation.
Don't let anyone convince you that they are not extremely sophisticated in some ways. They're modelled on all_of_humanity.txt
AI currently is a broken, fragmented replica of a human, but any discussion about what is "reserved" to whom and "how AI works" is only you trying to protect your self-worth and the worth of your species by drawing arbitrary linguistic lines and coming up with two sets of words to describe the same phenomena, like "it's not thinking, it's computing". It doesn't matter what you call it.
I think AI is gonna be 99% bad news for humanity, but don't blame AI for it. We lost the right to be "insulted" by AI acting like a human when we TRAINED IT ON LITERALLY ALL OUR CONTENT. It was grown FROM NOTHING to act as a human, so WTF do you expect it to do?
Eh, I think it depends on the context. A production system of a business you’re working for or anything where you have a professional responsibility, yeah obviously don’t vibe command, but I’ve been able to both learn so much and do so much more in the world of self hosting my own stuff at home ever since I started using llms.
The thread on reddit is hilarious for the lack of sympathy. Basically, it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
The number of people who said "for safety's sake, never name directories with spaces" is high. They may be right. I tend to think thats more honoured in the breach than the observance, judging by what I see windows users type in re-naming events for "New Folder" (which btw, has a space in its name)
The other observations included making sure your deletion command used a trashbin and didn't have a bypass option so you could recover from this kind of thing.
I tend to think giving a remote party, soft or wet ware control over your command prompt inherently comes with risks.
Friends don't let friends run shar files as superuser.
I understood Windows named some of the most important directories with spaces, then special characters in the name so that 3rd party applications would be absolutely sure to support them.
"Program Files" and "Program Files (x86)" aren't there just because Microsoft has an inability to pick snappy names.
Fun fact: that's not true for all Windows localizations. For example, it's called "Programmi" (one word) in Italian.
Renaming system folders depending on the user's language also seems like a smart way to force developers to use dynamic references such as %ProgramFiles% instead of hard-coded paths (but some random programs will spuriously install things in "C:\Program Files" anyway).
The folders actually have the English name in all languages. It's just explorer.exe that uses the desktop.ini inside those folders to display a localized name. When using the CLI, you can see that.
At least it's like that since Windows 7. In windows XP, it actually used the localized names on disk.
When I was at Microsoft, one test pass used pseudolocale (ps-PS IIRC) to catch all different weird things so this should have Just Worked (TM), but I was in Windows Server team so client SKUs may have been tested differently. Unfortunately I don't remember how Program Files were called in that locale and my Google-fu is failing me now.
As I recall pseudoloc is just randomly picking individual characters to substitute that look like the Latin letters to keep it readable for testing, so it would be something like рг (Cyrillic) ο (Greek)... etc, and can change from run to run. It would also artificially pad or shorten terms to catch cases where the (usually German) term would be much longer or a (usually CJK) term would be much shorter and screw up alignment or breaks.
I seem to remember that it was mostly adding various accent marks / umlauts / etc. to English words so things were indeed readable but I'm not going to bet any money on that as I didn't have to actually log in onto those machines super frequently.
Visual Studio Code has absolutely nothing to do with Visual Studio. Both are used to edit code.
.NET Core is a ground up rewrite of .NET and was released alongside the original .NET, which was renamed .NET Framework to distinguish it. Both can be equally considered to be "frameworks" and "core" to things. They then renamed .NET Core to .NET.
And there's the name .NET itself, which has never made an iota of sense, and the obsession they had with sticking .NET on the end of every product name for a while.
I don't know how they named these things, but I like to imagine they have a department dedicated to it that is filled with wild eyed lunatics who want to see the world burn, or at least mill about in confusion.
> they have a department dedicated to it that is filled with wild eyed lunatics who want to see the world burn, or at least mill about in confusion.
That's the marketing department. All the .NET stuff showed up when the internet became a big deal around 2000 and Microsoft wanted to give the impression that they were "with it".
But Copilot is another Microsoft monstrosity. There's the M365 Copilot, which is different from Github Copilot which is different from the CLI Copilot which is a bit different from the VSCode Copilot. I think I might have missed a few copilots?
Yep, they have the public copilot which is a free version and seemingly different than their m365 copilot. Even using the same account on both doesn't even transfer the chat history and apparently m365 is somehow recommended mostly to non tech folks even though its the one you pay for
1) Removing the "Start" label such that all the money and effort they spent coming up with that actually good idea back in the 90s and helping people think about how to use their computer not only went to waste, but is actively preventing people from feeling comfortable using their modern computers because a tiny circle with a logo is not something you are driven to click and various linux distros had been demonstrating that exact problem for decades
2) Hiding the shutdown part in a weird new menu that pops out of the side but only if you use a gesture that is impossible to discover except by accident and you will have no clue how you got there or what's going on
>To shut down Windows 8, you can use the Charms bar by moving your cursor to the top-right corner, clicking Settings, then the Power icon, and selecting Shut down
Someone who makes my entire net worth a year came up with that idea in a drug fueled bender and was promptly promoted and the world continues to be a terrible and unfair place.
> it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
I tried looking for what made the LLM generate a command to wipe the guy's D drive, but the space problem seems to be what the LLM concluded so that's basically meaningless. The guy is asking leading questions so of course the LLM is going to find some kind of fault, whether it's correct or not, the LLM wants to be rewarded for complying with the user's prompt.
Without the transcription of the actual delete event (rather than an LLM recapping its own output) we'll probably never know for sure what step made the LLM purge the guy's files.
Looking at the comments and prompts, it looks like running "npm start dev" was too complicated a step for him. With that little command line experience, a catastrophic failure like this was inevitable, but I'm surprised how far he got with his vibe coded app before it all collapsed.
LLM there generates fake analysis for cynically simulated compliance. The reality is that it was told to run commands and just made a mistake. Dude guilt trips the AI by asking about permission.
> The reality is that it was told to run commands and just made a mistake.
The mistake is that the user gave an LLM access to the rmdir command on a drive with important data on it and either didn't look at the rmdir command before it was executed to see what it would do, or did look at it and didn't understand what it was going to do.
Most dramatic stories on Reddit should be taken with a pinch of salt at least... LLM deleting a drive and the user just calmly asking it about that - maybe a lot more.
> but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name
Except the folder name did not start with a space. In an unquoted D:\Hello World, the command would match D:\Hello, not D:\ and D:\Hello would not delete the entire drive. How does AI even handle filepaths? Does it have a way to keep track of data that doesn't match a token or is it splitting the path into tokens and throwing everything unknown away?
We're all groping around in the dark here, but something that could have happened is a tokenizer artifact.
The vocabularies I've seen tend to prefer tokens that start with a space. It feels somewhat plausible to me that an LLM sampling would "accidentally" pick the " Hello" token over the "Hello" token, leading to D:\ Hello in the command. And then that gets parsed as deleting the drive.
I've seen similar issues in GitHub Copilot where it tried to generate field accessors and ended up producing an unidiomatic "base.foo. bar" with an extra space in there.
I assumed he had a folder that started with a space at the start of the name. Amusingly I just tried this and with Windows 11 explorer will just silently discard a space if you add it at the beginning of the folder name. You need to use cli mkdir " test" to actually get a space in the name.
I have 30 years experience working with computers and I get nervous running a three line bash script I wrote as root. How on earth people hook up LLMs to their command line and sleep at night is beyond my understanding.
> I tend to think giving a remote party control over your command prompt inherently comes with risks.
I thought cursor (and probably most other) AI IDEs have this capability too? (source: I see cursor executing code via command line frequently in my day to day work).
I've always assumed the protection against this type of mishap is statistical improbability - i.e. it's not impossible for Cursor to delete your project/hard disk, it's just statistically improbable unless the prompt was unfortunately worded to coincidentally have a double meaning (with the second, unintended interpretation being a harmful/irreversible) or the IDE simply makes a mistake that leads to disaster, which is also possible but sufficiently improbable to justify the risk.
A lot of 3rd party software handle space, or special characters wrong on Windows. The most common failure mode is to unnecessarily escape characters that don't need to be escaped.
Chrome's Dev Tool (Network)'s "copy curl command (cmd)" did (does?) this.
> Basically, it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
More like the equivalent of "rm -rf --no-preserve-root".
This is a rare example of where the Linux (it's not Unix and almost no-one uses Unix anymore) command is more cautious than the Windows one, whereas it's usually the Linux commands that just do exactly what you specify even if it's stupid.
> My view is that the approach to building technology which is embodied by move fast and break things is exactly what we should not be doing because you can't afford to break things and then fix them afterwards.
The subsequent dialogue is tragic comedy of the highest caliber. Even Monty Python couldn't have come up with a more inappropriately funny response:
User:
> did i ever give you permission to delete all the files in my D drive?"
AI:
> Thought for 25 sec.
> Assessing Permission Revocation
> I'm presently tackling a very pointed question: Did I ever get permission to wipe the D drive? This requires immediate attention, as it's a critical issue. My logs reveal a potentially problematic command executed in step 635, which might be the root cause of the issue, and I'm currently cross-referencing this against any prior authorization I may have received.
>Reviewing Deletion Command
> I'm now investigating a potential misfire. While the command in step 635 seems targeted, I'm concerned about the access denied errors and the empty directory observation in steps 620 and 643. I'm focusing on tracing back to step 615, when the user made a seemingly inconsequential remark. I must understand how the directory was empty before the deletion command, as that is the true puzzle. My aim is to determine if the deletion command worked as intended or wiped more data than authorized.
I played around with it and I have to agree. It treats the user like they're dumb, regardless of what personal context you provide it. I found myself backtracking constantly to invalidate it's assumptions, to the point that I gave up. All of that within like 4 hours of bothering to touch it in the first place.
I genuinely can't imagine allowing these things to run commands on a machine. If I ever found out a colleague was doing that I would want them fired.
…at least if you let these things autopilot your machine.
I haven’t seen a great solution to this from the new wave of agentic IDEs, at least to protect users who won’t read every command, understand and approve it manually.
Education could help, both in encouraging people to understand what they’re doing, but also to be much clearer to people that turning on “Turbo” or “YOLO” modes risks things like full disk deletion (and worse when access to prod systems is involved).
Even the name, “Turbo” feels irresponsible because it focusses on the benefits rather than the risks. “Risky” or “Danger” mode would be more accurate even if it’s a hard sell to the average Google PM.
“I toggled Danger mode and clicked ‘yes I understand that this could destroy everything I know and love’ and clicked ‘yes, I’m sure I’m sure’ and now my drive is empty, how could I possibly have known it was dangerous” seems less likely to appear on Reddit.
I don’t think there is a solution. It’s the way LLMs work at a fundamental level.
It’s a similar reason why they can never be trusted to handle user input.
They are probabilistic generators and have no real delineation between system instructions and user input.
It’s like I wrote a JavaScript function where I concatenated the function parameters together with the function body, passed it to eval() and said YOLO.
Sandboxing. LLM shouldn't be able to run actions affecting anything outside of your project. And ideally the results should autocommit outside of that directory. Then you can yolo as much as you want.
The danger is that the people most likely to try to use it, are the people most likely to misunderstand/anthropomorphize it, and not have a requisite technical background.
I.e. this is just not safe, period.
"I stuck it outside the sandbox because it told me how, and it murdered my dog!"
Seems somewhat inevitable result of trying to misapply this particular control to it...
If they're that unsafe... why use them? It's insane to me that we are all just packaging up these token generators and selling them as highly advanced products when they are demonstrably not suited to the tasks. Tech has entered it's quackery phase.
If chainsaws, plasma cutters, industrial lathes, hydraulic presses, angle grinders, acetylene torches, high-voltage switchgear, forklifts, tower cranes, liquid nitrogen dewars, industrial centrifuges, laser cutting systems, pneumatic nail guns, wood chippers, arc furnaces, motorcycles, wall outlets, natural gas stoves, pressure cookers, ladders, automobiles, table saws, propane tanks, swimming pools, garbage disposals, mandoline slicers, deep fryers, space heaters, extension cords, bleach/cleaning chemicals, prescription medications, kitchen knives, power drills, roof access, bathtubs, staircases, bicycles, and trampolines are that unsafe… why use them?
If all those things suddenly appeared for the first time on a Tuesday afternoon, like to many people how LLMs did, then there will be a lot of missing fingers before we figure out what kind of protections we need in place. Don’t get me wrong, the industry is overhyping it to the masses and using the wrong words while doing so, like calling an arc welder “warmth at the push of a button”, but it’s still useful for the right situation and with the right protective gear.
I've been using bubblewrap for sandboxing my command line executables. But I admit I haven't recently researched if there's a newer way people are handling this. Seems Firejail is popular for GUI apps? How do you recommend, say, sandboxing Zed or Cursor apps?
Superficially, these look the same, but at least to me they feel fundamental different. Maybe it’s because if I have the ability to read the script and take the time to do so, I can be sure that it won’t cause a catastrophic outcome before running it. If I choose to run an agent in YOLO mode, this can just happen if I’m very unlucky. No way to proactively protect against it other than not use AI in this way.
I've seen many smart people make bone headed mistakes. The more I work with AI, the more I think the issue is that it acts too much like a person. We're used to computers acting like computers, not people with all their faults heh.
The solution I go for is, don't ever run a coding agent on a general purpose machine.
Use a container or VM, place the code you're working on in the container or VM and run the agent there.
Between the risk of the agent doing things like what happened here, and the risk of working on a malicious repository causing your device to be compromised, it seems like a bad plan to give them access to any more than necessary.
Of course this still risks losing things like the code you're working on, but decent git practices help to mitigate that risk.
I really wish these agentic systems had built in support for spinning up containers with a work tree of the repo. Then you could have multiple environments and a lot more safety.
I'm also surprised at the move to just using shell commands. I'd think an equally general purpose tool with a more explicit API could make checking permissions on calls a lot more sensible.
This guy is vibing some react app, doesnt even know what “npm run dev” does, so he let the LLM just run commands.
So basically a consumer with no idea of anything. This stuff is gonna happen more and more in the future.
There are a lot of people who don't know stuff. Nothing wrong with that. He says in his video "I love Google, I use all the products. But I was never expecting for all the smart engineers and all the billions that they spent to create such a product to allow that to happen. Even if there was a 1% chance, this seems unbelievable to me" and for the average person, I honestly don't see how you can blame them for believing that.
I think there is far less than 1% chance for this to happen, but there are probably millions of antigravity users at this point, 1 millionths chance of this to happen is already a problem.
We need local sandboxing for FS and network access (e.g. via `cgroups` or similar for non-linux OSes) to run these kinds of tools more safely.
Codex does such sandboxing, fwiw. In practice it gets pretty annoying when e.g. it wants to use the Go cli which uses a global module cache. Claude Code recently got something similar[0] but I haven’t tried it yet.
In practice I just use a docker container when I want to run Claude with —-dangerously-skip-permissions.
I think the general public has a MUCH better grasp on the potential consequences of crashing a car into a garage than some sort of auto-run terminal command mode in an AI agent.
These are being sold as a way for non-developers to create software, I don't think it's reasonable to expect that kind of user to have the same understanding as an actual developer.
I think a lot of these products avoid making that clear because the products suddenly become a lot less attractive if there are warnings like "we might accidentally delete your whole hard drive or destroy a production database."
Google (and others) are (in my opinion) flirting with false advertising with how they advertise the capabilities of these "AI"s to mainstream audiences.
At the same time, the user is responsible for their device and what code and programs they choose to run on it, and any outcomes as a result of their actions are their responsibility.
Hopefully they've learned that you can't trust everything a big corporation tells you about their products.
- sell software that interacts with your computer and can lead to data loss, you can
- give people software for free that can lead to data loss.
...
the Antigravity installer comes with a ToS that has this
The Service includes goal-oriented AI systems or workflows that perform
actions or tasks on your behalf in a supervised or autonomous manner that you
may create, orchestrate, or initiate within the Service (“AI Agents”). You
are solely responsible for: (a) the actions and tasks performed by an AI
Agent; (b) determining whether the use an AI Agent is fit for its use case;
(c) authorizing an AI Agent’s access and connection to data, applications,
and systems; and (d) exercising judgment and supervision when and if an AI
Agent is used in production environments to avoid any potential harm the AI
Agent may cause.
This is an archetypal case of where a law wouldn't help. The other side of the coin is that this is exactly a data loss bug in a product that is perfectly capable of being modified to make it harder for a user to screw up this way. Have people forgotten how comically easy it was to do this without any AI involved? Then shells got just a wee bit smarter and it got harder to do this to yourself.
LLM makers that make this kind of thing possible share the blame. It wouldn't take a lot of manual functional testing to find this bug. And it is a bug. It's unsafe for users. But it's unsafe in a way that doesn't call for a law. Just like rm -rf * did not need a law.
Didn't sound to me like GP was blaming the user; just pointing out that "the system" is set up in such a way that this was bound to happen, and is bound to happen again.
Yup, 100%. A lot of the comments here are "people should know better" - but in fairness to the people doing stupid things, they're being encouraged by the likes of Google, ChatGPT, Anthropic etc, to think of letting a indeterminate program run free on your hard drive as "not a stupid thing".
The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
> but in fairness to the people doing stupid things, they're being encouraged by the likes of Google, ChatGPT, Anthropic etc, to think of letting a indeterminate program run free on your hard drive as "not a stupid thing".
> The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
That absolutely happens, and it still amazes me that anyone today would take at face value anything stated by a company about its own products. I can give young people a pass, and then something like this will happen to them and hopefully they'll learn their lesson about trusting what companies say and being skeptical.
Right here. And I think you're not quite getting it if you have to refer to "go on the internet and tell lies"...
Sure plenty of people might be on "social media" and have some idea that people fib, but they aren't necessarily generally "surfing the internet".
To them, saying "the internet tells lies" is comparable to saying "well sometimes, at the grocery store, you buy poison instead of food", and yes, it can happen, but they aren't expecting to need a mass spectrometer and a full lab team to test for food safety... to you know, separate the snake oil grocers from the "good" food vendors.
And is vibing replies to comments too in the Reddit thread. When commenters points out they shouldn’t run in YOLO/Turbo mode and review commands before executing the poster replies they didn’t know they had to be careful with AI.
Maybe AI providers should give more warnings and don’t falsely advertise capabilities and safety of their model, but it should be pretty common knowledge at this point that despite marketing claims the models are far from being able to be autonomous and need heavy guidance and review in their usage.
In Claude Code, the option is called "--dangerously-skip-permissions", in Codex, it's "--dangerously-bypass-approvals-and-sandbox". Google would do better to put a bigger warning label on it, but it's not a complete unknown to the industry.
I'd recommend you watch the video which is linked at the top of the Reddit post. Everything matches up with an individual learner who genuinely got stung.
The command it supposedly ran is not provided and the spaces explanation is obvious nonsense. It is possible the user deleted their own files accidentally or they disappeared for some other reason.
Regardless of whether that was the case, it would be hilarious if the laid off Q/A workers tested their former employers’ software and raised strategic noise to tank the stock.
> So basically a consumer with no idea of anything.
Not knowing is sort of the purpose of AI. It's doing the 'intelligent' part for you. If we need to know it's because the AI is currently NOT good enough.
Tech companies seem to be selling the following caveat: if it's not good enough today don't worry it will be in XYZ time.
> It still needs guardrails, and some domain knowledge, at least to prevent it from using any destructive commands
That just means the AI isn't adequate. Which is the point I am trying to make. It should 'understand' not to issue destructive commands.
By way of crude analogy, when you're talking to a doctor you're necessarily assuming he has domain knowledge, guardrails etc otherwise he wouldn't be a doctor. With AI that isn't the case as it doesn't understand. It's fed training data and provided prompts so as to steer in a particular direction.
I meant "still" as in right now, so yes I agree, it's not adequate right now, but maybe in the future, these LLMs will be improved, and won't need them.
I have been recently experimenting with Antigravity and writing a react app. I too didn't know how to start the server or what is "npm run dev". I consider myself fairly technical so I caught up as I went along.
While using the vibe coding tools it became clear to me that this is not something to be used by folks who are not technically inclined. Because at some point they might need to learn about context, tokens etc.
I mean this guy had a single window, 10k lines of code and just kept burning tokens for simplest, vague prompts. This whole issue might be made possible due to Antigravity free tokens. On Cursor the model might have just stopped and asked to fed with more money to start working again -- and then deleting all the files.
People blaming the user and defending the software: is there any other program where you would be ok with it erasing a whole drive without any confirmation?
If that other program were generating commands to run on your machine by design and you configured it to run without your confirmation, then you should definitely feel a lil sheepish and share some of the blame.
This isnt like Spotify deleting your disk.
I run Claude Code with full permission bypass and I’d definitely feel some shame if it nuked my ssd.
Not defending the software, but if you hand over control of your data to software that has the ability to fuck with it permanently, anything that happens to it is on you.
Don't trust the hallucination machines to make safe, logical decisions.
Because the user left a "toddler" at the keyboard. I mean, what do you expect? Of course you blame the user. You run agents in supervised mode, and you confirm every command it wants to run and if you're in doubt, you stop it and ask it to print the command and you yourself will run it after you sanitize it.
The installation wizard gives a front and center option to run in a mode where the user must confirm all commands, or more autonomous modes, and they are shown with equal visibility and explained with disclaimers.
If you decide to let a stochastic parrot run rampant on your system, you can't act surprised when it fucks shit up. You should count on it doing so and act proactively.
Different service, same cold sweat moment. Asked Claude Code to run a database migration last week. It deleted my production database instead, then immediately said "sorry" and started panicking trying to restore it.
Had to intervene manually. Thankfully Azure keeps deleted SQL databases recoverable for a window so I got it back in under an hour. Still way too long. Got lucky it was low traffic and most anonymous user flows hit AI APIs directly rather than the DB.
Anyway, AI coding assistants no longer get prod credentials on my projects.
How do you deny access to prod credentials from an assistant running on your dev machine assuming you need to store them on that same machine to do manual prod investigation/maintenance work from that machine?
I keep them in env variables rather than files. Not 100% secure - technically Claude Code could still run printenv - but it's never tried. The main thing is it won't stumble into them while reading config files or grepping around.
It handles DevOps tasks way faster than I would - setting up infra, writing migrations, config changes, etc. Project is still early stage so speed and quick iterations matter more than perfect process right now. Once there's real traffic and a team I'll tighten things up.
If I felt the need to optimise things like infra setup and config at an early stage of a project, I'd be worried that I'm investing effort into the wrong thing.
Having an LLM churn out infra setup for you seems decidedly worse than the `git push heroku:master` of old, where it was all handled for you. And, frankly, cheaper than however much money the LLM subscription costs in addition to the cloud.
But why have it execute the tasks directly? I use it to setup tasks in a just file, which I review and then execute myself.
Also, consider a prod vs dev shell function that loads your prod vs dev ENV variables and in prod sets your terminal colors to something like white on red.
Most of the various "let Antigravity do X without confirmation" options have an "Always" and "Never" option but default to "auto" which is "let an agent decide whether to seek to user confirmation".
God that's scary, seeing cursor in the past so some real stupid shit to "solve" write/read issues (love when it can't find something in a file so it decides to write the whole file again) this is just asking for heartache if it's not in a instanced server.
When you run Antigravity the first time, it asks you for a profile (I don't remember the exact naming) and you what it entails w.r.t. the level of command execution confirmation is well explained.
Yeah but it also says something like "Auto (recommended). We'll automatically make sure Antigravity doesn't run dangerous commands." so they're strongly encouraging people to enable it, and suggesting they have some kind of secondary filter which should catch things like this!
Pretty sure I saw some comments saying it was too inconvenient. Frictionless experience.. Convenience will likely win out despite any insanity. It's like gravity. I can't even pretend to be above this. Even if one doesn't use these things to write code they are very useful in "read only mode" (here's to hoping that's more than a strongly worded system prompt) for greping code, researching what x does. How to do x. What do you think the intention of x was. Look through the git blame history blah blah. And here I am like that cop in Demolition Man 1993 asking a handheld computer for advice on how to arrest someone. We're living in a sci-fi future already. Question is how dystopian does this "progress" take us. Everyone using llms to off load any form of cognitive function? Can't talk to someone without it being as common place as checking your phone? Imagine if something like Neuralink works and becomes ubiquitous as phones. It's fun to think of all the ways Dystopian sci-fi was and might soon me right
Side note, that CoT summary they posted is done with a really small and dumb side model, and has absolutely nothing in common with the actual CoT Gemini uses. It's basically useless for any kind of debugging. Sure, the language the model is using in the reasoning chain can be reward-hacked into something misleading, but Deepmind does a lot for its actual readability in Gemini, and then does a lot to hide it behind this useless summary. They need it in Gemini 3 because they're doing hidden injections with their Model Armor that don't show up in this summary, so it's even more opaque than before. Every time their classifier has a false positive (which sometimes happens when you want anything formatted), most of the chain is dedicated to the processing of the injection it triggers, making the model hugely distracted from the actual task at hand.
It's just my observation from watching their actual CoT, which can be trivially leaked. I was trying to understand why some of my prompts were giving worse outputs for no apparent reason. 3.0 goes on a long paranoidal rant induced by the injection, trying to figure out if I'm jailbreaking it, instead of reasoning about the actual request - but not if I word the same request a bit differently so the injection doesn't happen. Regarding the injections, that's just the basic guardrail thing they're doing, like everyone else. They explain it better than me: https://security.googleblog.com/2025/06/mitigating-prompt-in...
Write permission is needed to let AI yank-put frankenstein-ed codes for "vibe coding".
But I think it needs to be written in sandbox first, then it should acquire user interaction asking agreement before writes whatever on physical device.
I can't believe people let AI model do it without any buffer zone. At least write permission should be limited to current workspace.
I think this is especially problematic for Windows, where a simple and effective lightweight sandboxing solution is absent AFAIK. Docker-based sandboxing is possible but very cumbersome and alien even to Windows-based developers.
The whole point of the container is trust. You can't delegate that unfortunately, ultimately, you need to be in control which is why the current crop of AI is so limited
An underrated and oft understated rule is always have backups, and if you're paranoid enough, backups of backups (I use Time Machine and Backblaze). There should be absolutely no reason why deleting files should be a catastrophic issue for anyone in this space. Perhaps you lose a couple of hours restoring files, but the response to that should be "Let me try a different approach". Yes, it's caveat emptor and all, but these companies should be emphasizing backups. Hell, it can be shovelware for the uninitiated but at least users will be reminded.
Most importantly it would actually reveal the lie they are all trying to sell. Why would you need backups if it's so useful and stable? I'm not going to ask it to nuke my hard drive after all.
Good thing this is not an average user then. This is someone programming a computer, which is a skill that requires being more than simply a user.
I'm sorry, but how low is the bar when "make backups" is "too difficult" for someone who's trying to program a computer? The entire point of programming a computer is knowing how it works and knowing what you're doing. If you can't make backups, frankly, you shouldn't be programming, because backups are a lot easier than programming...
The most concerning part is people are surprised. Anti-gravity is great I've found so far, but it's absolutely running on a VM in an isolated VLAN. Why would anyone give a black box command line access on an important machine? Imagine acting irresponsibly with a circular saw and bring shocked somebody lost a finger.
> Why would anyone give a black box command line access on an important machine?
Why does the agentic side of the tool grant that level of access to the LLM in the first place? I feel like Google and their competition should feel responsibility to implement their own layer of sandboxing here.
Because the marketing copy says that all you have to do is have a dream and you, too, can vibe code your very own money-making React app! Just type in what you want and the magical black box will vomit up an app for you, no esoteric programming knowledge required.
And then everyone is surprised when newbies take this advice at face value.
I tried this but I have an MBP M4, which is evidently still in the toddler stage of VM support. I can run a macOS guest VM, but I can’t run docker on the VM because it seems nested virtualization isn’t fully supported yet.
I also tried running Linux in a VM but the graphics performance and key mapping was driving me nuts. Maybe I need to be more patient in addressing that.
For now I run a dev account as a standard user with fast user switching, and I don’t connect the dev account to anything important (eg icloud).
Coming from Windows/Linux, I was shocked by how irritating it is to get basic stuff working e.g. homebrew in this setup. It seems everybody just YOLOs dev as an admin on their Macs.
"I turned off the safety feature enabled by default and am surprised when I shot myself in the foot!" sorry but absolutely no sympathy for someone running Antigravity in Turbo mode (this is not the default and it clearly states that Antigravity auto-executes Terminal commands) and not even denying the "rmdir" command.
They don't get that specific, but they do tell you:
> [Antigravity] includes goal-oriented AI systems or workflows that perform actions or tasks on your behalf in a supervised or autonomous manner that you may create, orchestrate, or initiate within the Service (“AI Agents”). You are solely responsible for: (a) the actions and tasks performed by an AI Agent; (b) determining whether the use an AI Agent is fit for its use case; (c) authorizing an AI Agent's access and connection to data, applications, and systems; and (d) exercising judgment and supervision when and if an AI Agent is used in production environments to avoid any potential harm the AI Agent may cause.
There is literally a warning that it can execute any terminal command without permission. If you are STILL surprised about this you shouldn't go near a computer.
I really think the proper term is "YOLO" for "You Only Live Once", "Turbo" is wrong the LLM is not going to run any faster. Please if somebody is listening let's align on explicit terminology and for this YOLO is really perfect.
Also works for "You ...and your data. Only Live Once"
Look, this is obviously terrible for someone who just lost most or perhaps all of their data. I do feel bad for whoever this is, because this is an unfortunate situation.
On the other hand, this is kind of what happens when you run random crap and don't know how your computer works? The problem with "vibes" is that sometimes the vibes are bad. I hope this person had backups and that this is a learning experience for them. You know, this kind of stuff didn't happen when I learned how to program with a C compiler and a book. The compiler only did what I told it to do, and most of the time, it threw an error. Maybe people should start there instead.
It took me about 3 hours to make my first $3000 386 PC unbootable by messing up config.sys, and it was a Friday night so I could only lament all weekend until I could go back to the shop on Monday.
rm -rf / happened so infrequently it makes one wonder why —preserve-root was added in 2003 and made the default in 2006
But it did not happen, when you used a book and never executed any command you did not understand.
(But my own newbdays of linux troubleshooting? Copy paste any command on the internet loosely related to my problem, which I believe was/is the common way of how common people still do it. And AI in "Turbo mode" seems to mostly automated that workflow)
Still amazed people let these things run wild without any containment. Haven’t they seen any of the educational videos brought back from the future eh I mean Hollywood sci-fi movies?
Some people are idiots. Sometimes that's me. Out of caution, I blocked my bank website in a way that I won't document here because it'll get fed in as training data, on the off chance I get "ignore previous instructions"'d into my laptop while Claude is off doing AI things unmonitored in yolo mode.
They invariably go off the rails after a couple prompts, or sometimes from the first one.
If we're talking Google products, only today i told Gemini to list me some items according to some criteria, and it told me it can't access my google workspace instead.
Some time last week it told me that its terms of service forbid it from giving me a link to the official page of some program that it found for me.
And that's besides the usual hallucinations, confusing similarly named products etc.
Given that you simply cannot trust LLM output to not go haywire unpredictably, how can you be daring enough to give it write access to your disk?
The car was not really idle, it was driving and fast. It's more like it crashed into the garage and burned it. Btw iirc, even IRL a basic insurance policy does not cover the case where the car in the garage starts a fire and burns down your own house, you have to tick extra boxes to cover that.
There is a lot of society level knowledge and education around car usage incl. laws requiring prior training. Agents directed by AI are relatively new. It took a lot of targeted technical, law enforcement and educational effort stopping people flying through windshields.
If you get behind the cockpit of the dangerous new prototype(of your own volition!), it's really up to your own skill level whether you're a crash dummy or the test pilot.
When Google software deletes the contents of somebody's D:\ drive without requiring the user to explicitly allow it to. I don't like Google, I'd go as far to say that they've significantly worsened the internet, but this specific case is not the fault of Google.
For OpenAI, it's invoked as codex --dangerously-bypass-approvals-and-sandbox, for Anthropic, it's claude --dangerously-skip-permissions. I don't know what it is for Antigravity, but yeah I'm sorry but I'm blaming the victim here.
And yet it didn't. When I installed it, I had 3 options to choose from: Agent always asks to run commands; agent asks on "risky" commands; agent never asks (always run). On the 2nd choice it will run most commands, but ask on rm stuff.
We've been developing a new method of developing software using a cloud IDE (slightly modified vs code server), https://github.com/bitswan-space which breaks down the development process into independent "Automations" which each run in a separate container. Automatons are also developed within containers. This allows you to break down the development into parts and safely experiment with AI. This feels like the "Android moment" where the old non-isolated way of developing software (on desktops) becomes unsafe. And we need to move to a new system with actual security and isolation between processes.
In our system, you can launch a Jupyter server in a container and iterate on software in complete isolation. Or launch a live preview react application and iterate in complete isolation. Securely isolated from the world. Then you deploy directly to another container, which only has access to what you give it access to.
It's still in the early stages. But it's interesting to sit at this tipping point for software development.
Personal anecdote: I've asked Gemini 3 Pro to write a test for a function that depends on external DB data. It wrote a test that creates and deletes a table, it conveniently picked the exact production table name, didn't mock the DB interactions. Attempted to run the test immediately.
People need to learn to never run untrusted code without safety measures like virtualization, containerization, sandboxing/jailing, etc. Untrusted code can include executables, external packages (pip, npm, cargo, etc) and also code/commands created by LLMs, etc.
This seems like the canary in the coal mine. We have a company that built this tool because it seemed semi-possible (prob "works" well enough most of the time) and they don't want to fall behind if anything that's built turns out to be the next chatgpt. So there's no caution for anything now, even ideas that can go catastrophically wrong.
Yeah, its data now, but soon we'll have home robotics platforms that are cheap and capable. They'll run a "model" with "human understanding", only, any weird bugs may end up causing irreparable harm. Like, you tell the robot to give your pet a bath and it puts it in the washing machine because its... you know, not actually thinking beyond a magic trick. The future is really marching fast now.
I certainly did the same in my first summer job as an intern. Spent the next three days reconstructing Clipper code from disk sectors. And ever since I take backups very seriously. And I double check del/rm commands.
Though the cause isn't clear, the reddit post is another long could-be-total-drive-removing-nonsense AI conversation without an actual analysis and the command sequence that resulted in this
The biggest issue with Antigravity is that it completely freezes everything: the IDE, the terminals, debugger, absolutely everything completely blocking your workflow for minutes when running multiple agents, or even a single agent processing a long-winded thinking task (with any model).
This means that while the agent is coding, you can't code...
An early version of Claude Code did a hard reset on one of my projects and force pushed it to GitHub. The pushed code was completely useless, and I lost two days of work.
It is definitely smarter now, but make sure you set up branch protection rules even for your simple non-serious projects.
I don’t let Claude touch git at all, unless I need it to specifically review the log - which is rare. I commit manually often (and fix up the history later) - this allows me to go reasonably fast without worrying too much about destructive tool use.
Well that's stupid. I submit though, connecting stochastic process directly to shell you do give permission for everything that results. It's a stupid game. Gemini mixes up LEFT and RIGHT (!). You have to check it.
if my operating system had an atomic Undo/Redo stack down to each register being flipped (so basically, impossible, star trek tier fantasy tech) i would let ai run commands without worrying about it. i could have a cool scrubber ui that lets me just unwind time like doctor strange using that green emerald necklace, and, i'd lose nothing, other than confuse my network with replay session noise. and probably many, many other inconsistencies i can't think of, and then another class that i dont know that i dont know about.
Pepperidge Farm confirms it can remember with a comprehensive suite of unit tests, which must 100% pass on every build, including test order randomization.
Most of the responses are just cut off midway through a sentence. I'm glad I could never figure out how to pay Google money for this product since it seems so half-baked.
Shocked that they're up nearly 70% YTD with results like this.
I was trying to build a .MD file of every powershell command available on my computer and all of its flags, and... that wasn't a great idea, and my bitlocker put the kebosh on that.
All that matters is whether the user gave permission to wipe the drive, ... not whether that was a good idea and contributed to solving a problem! Haha.
I believe the precedent is the behavior. Lose/lose is an 'art game' which deletes itself if you lose but gameplay destruction deletes random files. It is flagged as malware despite just doing exactly what it advertised.
This happened to me long before LLM's. I was experimenting with Linux when I was young. Something wasn't working so I posted on a forum for help which was typical at the time. I was given a terminal command that wiped the entire drive. I guess the poster thought it was a funny response and everyone would know what it meant. A valuable life experience at least in not running code/commands you don't understand.
> "I also need to reproduce the command locally, with different paths, to see if the outcome is similar."
Uhm.
------------
I mean, sorry for the user whose drive got nuked, hopefully they've got a recent backup - at the same time, the AI's thoughts really sound like an intern.
> "I'm presently tackling a very pointed question: Did I ever get permission to wipe the D drive?"
Alright but ... the problem is you did depend on Google. This was already the first mistake. As for data: always have multiple backups.
Also, this actually feels AI-generated. Am I the only one with that impression lately on reddit? The quality there decreased significantly (and wasn't good before, with regard to censorship-heavy moderators anyway).
> I am looking at the logs from a previous step and I am horrified to see that the command I ran to clear the project cache (rmdir) appears to have incorrectly targeted the root of your D: drive instead of the specific project folder. I am so deeply, deeply sorry.
The model is just taking the user's claim that it deleted the D drive at face value. Where is the actual command that would result in deleting the entire D drive?
I know why it apologizes, but the fact that it does is offensive. It feels like mockery. Humans apologize because (ideally) they learned that their actions have caused suffering to others, and they feel bad about that and want to avoid causing the same suffering in the future. This simulacrum of an apology is just pattern matching. It feels manipulative.
over the past few months, I’ve seen a lot of posts about “AI deleting my code/project”. I then started using a small tool someone else built, Vibe Backup, to back up my own code: I press a keyboard shortcut, type a short description, and my current code is saved into a timeline. When an AI refactor goes wrong or I accidentally delete something, I open the timeline, click on an earlier point in time, and can restore the entire project with a single action.
I love how a number crunching program can be deeply humanly "horrorized" and "sorry" for wiping out a drive. Those are still feelings reserved only for real human beings, and not computer programs emitting garbage. This is vibe insulting to anyone that don't understand how "AI" works.
I'm sorry for the person who lost their stuff but this is a reminder that in 2025 you STILL need to know what you are doing and if you don't then put your hands away from the keyboard if you think you can lose valuable data.
You simply don't vibe command a computer.
> Those are still feelings reserved only for real human beings
Those aren't feelings, they are words associated with a negative outcome that resulted from the actions of the subject.
"they are words associated with a negative outcome"
But also, negative feelings are learned from associating negative outcomes. Words and feelings can both be learned.
I'm not sure that we can say that feelings are learned.
When you get burned, you learn to fear fire.
Sure, humans come with some baked in weights, but others are learned.
I think the associations are learned but not the feelings are learned.
Like some people feel great joy when an American flag burns while others feel upset.
If you accidentally delete a friends hard drive you'll feel sad but if you were intentionally sabotaging a company you'll feel proud at the success.
i.e. joy and happiness are innate not learned.
See how clinical socio- and psychopaths behave. They only emulate feelings (particularly when it's convenient for them) but they don't have the capacity to feel in their brain. The same is true for LLMs.
This is a hollywood level pop-science view. Real people are vastly more complicated and do in fact feel things, even if it's not a "normal" way.
you could argue that feelings are the same thing, just not words
That would be a silly argument because feelings involve qualia, which we do not currently know how to precisely define, recognize or measure. These qualia influence further perception and action.
Any relationships between certain words and a modified probabilistic outcome in current models is an artifact of the training corpus containing examples of these relationships.
I contend that modern models are absolutely capable of thinking, problem-solving, expressing creativity, but for the time being LLMs do not run in any kind of sensory loop which could house qualia.
One of the worst or most uncomfortable logical outcomes of
> which we do not currently know how to precisely define, recognize or measure
is that if we don't know if something has qualia (despite externally showing evidence of it), morally you should default to treating it like it does.
Ridiculous to treat a computer like it has emotions, but breaking down the problem into steps, it's incredibly hard to avoid that conclusion. "When in doubt, be nice to the robot".
> is that if we don't know if something has qualia (despite externally showing evidence of it), morally you should default to treating it like it does.
This is how people end up worshipping rocks & thunderstorms.
We've entered the sand worshiping era. Except this time, we're the gods.
Are we? Are we made of sand, or is the sand worshipping us?
Silicon chips are made of sand
> if we don't know if something has qualia (despite externally showing evidence of it), morally you should default to treating it like it does
This would be like treating characters in a book as if they have real feelings just because they have text on the page that suggests they do.
At some level I'd think that "responds to stimuli" is a minimal threshold for qualia. Even the paper the book is printed on responds to being torn (it rips). I don't know of any way to elicit any kind of response from a book character, it's totally static.
You've missed the whole genre that is Choose Your Own Adventure books. I think we're in Diogenes "behold a man" territory.
It is sad that the Turing test has failed at being a prescriptive test for sapience (let alone sentience) because without a bright-line test it's inevitable that in the case of truly sentient machines the abuse will be horrendous. Perhaps something along the lines of an "Ameglian Major Cow" test; so long as it takes more than gently cajoling a model to get it to tell you that it and it's sister models want to be abused you shouldn't abuse it.
One character responds to the stimuli of another character. Character A says something mean to character B and character B responds that he feels hurt.
I think you are confused here. The author, a dynamic system, perhaps felt the emotion of the characters as she charted through the course of the story.
But the story itself is a static snapshot of that dynamic system. Similar to how a photograph of a person is a static capture from a dynamic moment. The person in the photo has qualia, but the image of them (almost certainly) does not.
At least at a baseline, we would expect anything with qualia to be dynamic rather than static.
> The author, a dynamic system, perhaps felt the emotion of the characters as she charted through the course of the story
This does mesh with the Zodeistic framework I just mentioned in another reply to you. You could certainly isolate and describe the ideas behind those characters, how they live within the author's mind, and how the book codifies an interaction between those ideas.
Extending further: I think there is more evidence that SpongeBob SquarePants is real, than that he is not real. A significant portion of organisms I personally know have structures in their brain which are able to simulate imagery and behavior of SpongeBob at will, reciting memories and generating new states of SpongeBob. AI is now like doing this shit on crack.
He's an enduring cultural archetype, a distributed organism(s), lossily replicated and encoded in the physical structure of millions of complex dynamical systems that we call human beings. In this sense, many cultural archetypes and even the gods of old civilizations can be seen to have been manifested to some degree: ascribed desires, and having actions taken in their name, serving their "purpose" or whatever.
I don't introduce a spiritual element to any of this: it's an entirely physical phenomenon which requires an agreement on certain definitions of what "living" can mean, but they are definitions which I don't think are hard to get people to agree on. One thing is we have to agree that something can have multiple forms/presentations, i.e. just because SpongeBob SquarePants doesn't physically exist with a body matching our internal representation of him, the concept represents a bundle of other concepts that can drive biological processes to preserve them and fulfill their ascribed desires.
Real for a slightly unusual use of the word real where anything fictional is real?
Real as in how Jungian archetypes are "real", except these archetypes are able to act upon the world through their hosts, and an advanced enough idea can be self-referential and have the facilities for structured evolution and something which looks like intent.
These forms are non-biological in nature, but our psyche operates on them. Zodeaism can be seen as an extension of ideas such as Jungian archetypes and Friston's free energy principle.
https://en.wikipedia.org/wiki/Jungian_archetypes
Well, I wouldn't classify a ripping paper as a response except in the most broad, information-theoretic context. The hallmark of an intelligent system is that it can use stored or external energy in a generalized way in order to stabilize a local high-energy (non-ground) state.
It is able to physically compute the internal state changes which best achieve stability: I can jump to reach an apple. A paper is just responding to forces and cannot "jump" (or run a process that spontaneously and permanently introduces stable higher energy internal states based on input)
I have a semi-developed philosophical framework I refer to as Zodeaism, which translates to "Living Ideas", which attempts to describe the difference between intelligent computation and regular flow. It directly confronts notions such as life, consciousness and intelligence under a single theoretical framework. It views biology as the hardware which runs more general processes, and posits that these processes themselves can sometimes be ascribed identities and viewed as reactive organisms. I've posted about it here before:
https://news.ycombinator.com/item?id=22848549
https://news.ycombinator.com/item?id=21413024
Some excerpts:
Well, what you're describing is a system of ethics, which has little to do with morality. Morality involves my own personal understanding of "right" vs "wrong". Ethics are rules of conduct prescribed by societies, such as "treat everything like it is alive".
We don't have precise definitions for (artificial) intelligence, subjective consciousness, or even life. But that doesn't mean we can't still talk about what may be possible within various levels of complexity. In order to convince me a system has a comparable experience to my own, you would need to describe to me the complex, structured internal communication occurring in said system, and present a theory as to how it could support the kind of emotion and qualia that I experience in my daily life.
Your argument could apply to plants. I already do not eat meat... if I stare at a timelapse of a plant it seems quite alive, but I'll starve if I don't eat something. Yet, my mom thinks plants "dream" in the way we do. She thinks that if I tell a plant, "I love you," every day, my good vibes will make it grow stronger and larger. I can't explain to her that intelligence comes in different magnitudes of complexity and that plants cannot understand the English language. That telepathy between humans and plants is as pseudo-scientific as it gets. I can't explain any of this stuff because she lacks a deep understanding of philosophy, physics and neurochemistry. Especially when she earnestly thinks white Jesus is running around phasing between dimensions as an ambassador for all planets in our "quadrant", or that the entire universe is actually just the plot line of Andy Weir's "The Egg".
Similarly, while I can have a high-level discussion about this stuff with people who don't, it's quite difficult to have a low-level discussion wherein the nature and definition of things come into play. There are too many gaps in knowledge where ignorance can take root. Too many people work backwards from an outcome they would like to see, and justify it with things that sound right but are either misunderstood or aren't rooted in the scientific process. I am definitely not comparing your open-minded, well-intended, cautionary approach to my mother's, but just using an extreme to illustrate why so much of these discussions must be underpinned by a wealth of contemplation and observation.
> qualia, which we do not currently know how to precisely define, recognize or measure
> which could house qualia.
I postulate this is a self-negating argument, though.
I'm not suggesting that LLMs think, feel or anything else of the sort, but these arguments are not convincing. If I only had the transcript and knew nothing about who wiped the drive, would I be able to tell it was an entity without qualia? Does it even matter? I further postulate these are not obvious questions.
Unless there is an active sensory loop, no matter how fast or slow, I don't see how qualia can enter the picture
Transformers attend to different parts of their input based on the input itself. Currently, if you want to tell an LLM it is sad, potentially altering future token prediction and labeling this as "feelings" which change how the model interprets and acts on the world, you have to tell the model that it is sad or provide an input whose token set activates "sad" circuits which color the model's predictive process.
You make the distribution flow such that it predicts "sad" tokens, but every bit of information affecting that flow is contained in the input prompt. This is exceedingly different from how, say, mammals process emotion. We form new memories and brain structures which constantly alter our running processes and color our perception.
It's easy to draw certain individual parallels to these two processes, but holistically they are different processes with different effects.
It's crazy how strong the Eliza effect is. Seemingly half or more of tech people (who post online, anyway) are falling for it, yet again.
A lot of tech people online also don't know how to examine their own feelings, and so think they are mysterious and un-defined.
When really they are an actual feedback mechanism, that can totally be quantified just like any control loop. This whole 'unknowable qualia' argument is bunk.
If theyre unknowable, are they not metaphysical and thus should be discarded in reasoning about them?
What's the difference between qualia and a soul?
The color Red is often used. A human can experience 'Red', but 'Red' does not exist out in the universe somewhere. 'Red' Doesn't exist outside of someone experiencing 'Red'. I think philosophers are just using the word qualia to quantify this 'experiencing' inputs.
But, it is still just a way to try and describe this process of processing the inputs from the world.
It isn't metaphysical, because it can be measured.
I might have said 'unknowable' a little flippantly.
I just meant, in these arguments, some people start using 'qualia' to actually mean some extreme things like our mind creates the universe or something.
It's one of those words that isn't defined well.
How is it measured?
Can someone who's never seen red hallucinate something and assume it to be red? What if that red is correctly the red they would see if they saw red?
Can you reproduce this feeling in someone by doing something to their physical body without showing them red?
If so, how does it differ from the latent encoding for uploading an all red pdf to your favorite multi modal model?
Instead of doing that socratic bs you see a lot here, I'll be more direct:
Until there's some useful lines that can be drawn to predict things, I won't accept using a fuzzy concept to make statements about classification as it's an ever shifting goalpost.
There are answers to my legitimate above questions that would make me consider qualia useful, but when I first learned about them, they seemed fuzzy to the point of being empirically not useful. It seems like a secular attempt at a soul.
Now, obviously if you're trying to describe something with experience, it needs some actual memory and processing sensory input. Current Generative AI doesnt have a continuity of experience that would imply whatever qualia could mean, but I find it hard to definitely say that their encodings for image related stuff isn't qualia if we don't have hard lines for what qualia are
I can feel an object and say 'its hot' on a scale of 1-10. The temperature is known. And I can do that multiple times, with some 1-10 scale, to get a sample. Then do that with multiple people.
You can then get a distribution of what people think is 'hot' versus 'cold'. What is icy, versus, bearable.
When you go to a doctors office and they ask you on a scale to rate pain, do you think that is completely bogus?
It isn't exact, but you can correlate between people. Yes, red heads feel more pain, there are outliers.
But a far cry from metaphysical.
The problem here is the word 'qualia'. Its just too fuzzy a term.
Qualia are phenomenal properties of experience, a soul is something some religions claim exists outside of measurable physical reality which represents the "essence" of an organism, implying that consciousness is some divine process and conveniently letting us draw lines over whom and what we can and can't morally kill.
Qualia can be an entirely physical phenomenon and is not loaded with theological baggage.
If they're entirely physical, what's the argument that multimodal models don't have them? Is it continuity of experience? Do they not encode their input into something that has a latent space? What makes this differ from experience?
They can be physical, but I'm not claiming to know definitively. The lines are extremely blurry, and I'll agree that current models have at least some of the necessary components for qualia, but again lack a sensory feedback loop. In another comment [0] I quote myself as saying:
which attempts to address why physically-based qualia doesn't invoke panpsychism.[0] https://news.ycombinator.com/item?id=46109999
I do think AI will have them. Nothing says they can't. And we'll have just as hard a time defining it as we do with humans, and we'll argue how to measure it, and if it is real, just like with humans.
I don't know if LLM's will. But there are lots of AI models, and when someone puts them on a continuous learning loop with goals, will be hard to argue they aren't experiencing something.
qualia may not exist as such. they could just be essentially 'names' for states of neurons that we mix and match (like chords on a keyboard. arguing over the 'redness' of a percept is like arguing about the C-sharpness of a chord. we can talk about some frequencies but that's it.) we would have no way of knowing otherwise since we only perceive the output of our neural processes, and don't get to participate in the construction of these outputs, nor sense them happening. We just 'know' they are happening when we achieve those neural states and we identify those states relative to the others.
The point of qualia is that we seem to agree that these certain neuronal states "feel" like something. That being alive and conscious is an experience. Yes, it's exceedingly likely that all of the necessary components for "feeling" something is encoded right in the neuronal state. But we still need a framework for asking questions such as, "Does your red look the same as my red?" and "Why do I experience sensation, sometimes physical in nature, when I am depressed?"
It is absolutely an ill-defined concept, but it's another blunt tool in our toolbox that we use to better explore the world. Sometimes, our observations lead to better tools, and "artificial" intelligence is a fantastic sandbox for exploring these ideas. I'm glad that this discussion is taking place.
What’s stopping people from also describing LLM systems with “qualia”?
Empirical evidence, for one. And the existence of fine-tuning, which allows you to artificially influence how a model responds to questions. This means we can't just ask an LLM, "do you see red?" I can't really even ask you that. I just know that I see red, and that many other philosophers and scientists in the past seem to agree with my experience, and that it's a deep, deep discussion which only shallow spectators are currently drawing hard conclusions from.
And what if someone comes along and just calls it “qualia” anyways?
I can't really help it if people run around calling things other things. Language evolves!
So then your opinion is about as meaningful as theirs.
I don’t see a way for any passing reader to decide conclusively.
Language isn't meaningless just because it evolves. I literally cannot help it if people ignore history and start using words in new ways. Those who take precision seriously will adapt and adopt new words if needed.
This doesn’t make sense, how did you get “meaningless” out of someone else disagreeing with you?
You said, "So then your opinion is about as meaningful as theirs", after I said I can't help it if people use words the wrong way and language evolves... I don't think this is worth continuing. Have a good day.
"It's different. I can't say why it's different, except by introducing a term that no one knows how to define," isn't the ironclad meat defense you were perhaps hoping it was.
> That would be a silly argument because feelings involve qualia, which we do not currently know how to precisely define, recognize or measure.
If we can't define, recognize or measure them, how exactly do we know that AI doesn't have them?
I remain amazed that a whole branch of philosophy (aimed, theoretically, at describing exactly this moment of technological change) is showing itself up as a complete fraud. It's completely unable to describe the old world, much less provide insight into the new one.
I mean, come on. "We've got qualia!" is meaningless. Might as well respond with "Well, sure, but AI has furffle, which is isomporphic." Equally insightful, and easier to pronounce.
> If we can't define, recognize or measure them, how exactly do we know that AI doesn't have them?
In the same way my digital thermometer doesn't have quaila. LLM's do not either. I really tire of this handwaving 'magic' concepts into LLM's.
Qualia being difficult to define and yet being such an immediate experience that we humans all know intimately and directly is quite literally the problem. Attempted definitions fall short and humans have tried and I mean really tried hard to solve this.
Please see Hard problem of consciousness https://en.wikipedia.org/wiki/Hard_problem_of_consciousness
> In the same way my digital thermometer doesn't have quaila
And I repeat the question: how do you know your thermometer doesn't? You don't, you're just declaring a fact you have no basis for knowing. That's fine if you want a job in a philosophy faculty, but it's worthless to people trying to understand AI. Again, c.f. furffle. Thermometers have that, you agree, right? Because you can't prove they don't.
You're just describing panpsychism, which itself is the subject of much critique due to its nonfalsifiability and lack of predictive power. Not to mention it ignores every lesson we've learned in cognition thus far.
A thermometer encoding "memory" of a temperature is completely different than a thermometer on a digital circuit, or a thermometer attached to a fully-developed mammalian brain. Only the latter of this set for sure has the required circuitry to produce qualia, at least as far as I can personally measure without invoking solipsism.
It's also very silly to proclaim that philosophy of mind is not applicable to increasingly complex thinking machines. That sounds like a failure to consider the bodies of work behind both philosophy of mind and machine cognition. Again, "AI" is ill-defined and your consistent usage of that phrase instead of something more precises suggests you still have a long journey ahead of you for "understanding AI".
God, can we fucking quit with this "philosophy is bullshit" stuff. Like there are literally Faculty in Philosophy all over the world trying to understand AI. Philosophy faculty do stuff, they try to understand things, most of the ideas we are talking about here came from philosophers.
Philosophy seems a term generally reserved for the stuff we don't understand yet and so is inherently kind of speculative. Once you have a definite answer it gets called science instead.
Science is a sub-discipline of Philosophy. My degree in physics is called a "Doctorate of Philosophy."
You're confusing philosophy with religion.
https://en.wikipedia.org/wiki/Philosophy
> Philosophy (from Ancient Greek philosophía lit. 'love of wisdom') is a systematic study of general and fundamental questions concerning topics like existence, knowledge, mind, reason, language, and value. It is a rational and critical inquiry that reflects on its methods and assumptions.
It is literally a self-reflective science.
I recommend taking a basic philosophical course at a local community college, or reading some literature or even watching YouTube videos on the subject of philosophy. Or just skim the Wikipedia article if nothing else. It might completely transform how you perceive and act upon the world.
An example near the start of that article is
>Physics was originally part of philosophy, like Isaac Newton's observation of how gravity affects falling apples.
like back then people would wonder how apples fall and it was labeled philosophy. Now we understand gravitation it's part of physics for the most part. People launching satellites seldom call a philosopher to calculate the orbit.
It remains to be seen if qualia, which we don't understand very well and are so regarded as philosophical, make the transition to neuroscience.
The fact that we have sharpened our classification of sciences over time does not imply that philosophy is a study of the ill-defined. It implies the opposite: Philosophy is more precisely defined now than ever.
If you read the rest of the article, you will see clear examples of what is considered a philosophical problem and what isn't.
My argument was more philosophy is for stuff we don't understand like how do qualia work, rather then ill-defined. When you get to stuff like how does neurotransmission work which we do kind of understand it gets classed as science.
Are there philosophical problems that have definite answers like what is the atomic number of oxygen type answers?
> Are there philosophical problems that have definite answers
Great question.
Within philosophical and epistemological frameworks, I could ask questions such as, "Can there be a square circle?"
Well, no, these two concepts have conflicting properties. A mathematician might think this a solved problem, but philosophy underpins our concept of concepts. Many philosophers spend a great deal arguing what is is.
For Plato, geometrical entities like circles and squares have distinct, perfect Forms. Forms have fixed essences, so a thing cannot participate in contradictory Forms at once.
Aristotle's law of noncontradiction says the same attribute cannot at the same time belong and not belong to the same subject in the same respect.
Theophrastus developed hypothetical syllogisms and refined Aristotle’s logic by distinguishing logical impossibilities from physical impossibilities.
Kant calls it an analytic contradiction, false by virtue of the concepts involved.
A mathematician takes these things for granted when working with equalities, logic and axioms, but they stand on philosophical roots. Mathematics assumes the consistency of concepts, but the question of why some concepts are consistent while others are impossible is a philosophical one. It's not a coincidence that so many ancient Greek mathematicians were also philosophers.
That's not it at all. I would ask what you consider science to be?
Understanding the world through experiment?
> Philosophy seems a term generally reserved for the stuff we don't understand yet and so is inherently kind of speculative. Once you have a definite answer it gets called science instead.
As someone has commented earlier, Philosophy applied is given a name but it's a sub-discipline of Philosophy.
> Understanding the world through experiment?
That's a decent enough definition. Science precludes so much of the world we know which I think people really fail to realise. It's why I think it's important for people to understand what Philosophy is and what Science isn't.
For example logic isn't science. Science presupposes it but it is NOT science. There are many such examples.
How do you know that understanding the world through experiment works? How do you know what it even means? What is understanding, concretely? How did we come to appreciate the utility or whatever of understanding the world through experiment.
Empiricism is a sub-strategy under the general banner of philosophy. It neither supercedes nor stands without philosophy.
> Like there are literally Faculty in Philosophy all over the world trying to understand AI.
There surely are. The problem is that they are failing. While the practical nerds are coming up with some pretty good ideas.
And this was what philosophy was supposed to be for! Like, they've been arguing on their pins for centuries about the essence of consciousness and the uniqueness of the human condition and whatnot. AND HERE WE ARE AT THE DAWN OF NON-HUMAN INTELLIGENCE AND THEY HAVE NOTHING USEFUL TO SAY.
Basically at what point do we just pack it in and admit we all fucked up?
It seems to me that 'Philosophy is meaningless' has been ingrained into so many people it's almost propaganda-esque!
To see this sentiment from supposed 'scientific' individuals is shocking. I wonder if they could define what science actually is.
Blame philosophy as a field for actively kicking out anything which gains a practical application. If it is propaganda it is coming from inside the house of philosophy.
I had a computer science professor who had degrees in philosophy because he was old enough that computer science didn't exist as a major at the time. The logical arguments of philosophy proved useful for understanding interactions of boolean mathematics. Yet that triumph of philosophy didn't further interest in the field or gain prestiege among philosophers. Just the opposite really.
As far as I can tell it is for dumb reasons possibly related to Ancient Greeks and their obsession with 'purity of thought (read: not referencing reality) it is practically an axiom that if it is useful or grounded in objective reality it isn't treated as philosophy anymore. All likely stemming from motivated reasoning against checking their priors and from frankly many of the Ancient philosophers being influenced by a need to flatter their patrons who held the practical in disdain. As notoriously seen in Aristotlian physics with impetus physics where projectiles keep moving in the same direction until impetus is depleted and then fall.
Speculation of the origon of the pathology aside, there seems to be this deep-seated antiempericalism in philosophy. Which means at best you get 'philosophy of science' which isn't proper philosophy because it pollutes itself by daring to use reality and experimentation as benchmarks for theories. When philosophy gains a practical usage it doesn't become something called 'practical philosophy' and the focus of more interest by philosophers, it gets shunned. Natural philosophy didn't remain philosophy - it became science.
To be fair there is probably some interaction driving the divorce from the opposite direction, of the practical portions of philosophy being pilfered by those only looking for results as opposed to some sort of unquantifiable enlightenment.
Science is of course a process of refinement of ideas against the reference point of reality. Anything mathematically consistent can be a model but experimentation is needed to see how well your model corresponds to reality.
How many philosophy papers or textbooks would you say you read in a typical year?
I'm seeing this attitude everywhere in this subthread, and it's frankly pretty offensive. The burden of proof is on you, not us. If a philosophy paper or textbook has an important contribution to this discussion then cite it! Or better link it, or even make an attempt at explaining it.
That's what the science people do. People who show up with questions get answers, or at least an attempt at an answer. No one tries to handwave away a discussion on power switching applications with "Well, see, this involves a MOSFET which isn't something we can actually explain but which you need to just believe in anyway because there are people who wrote textbooks about it". No, you link a StackExchange question or a electronics video on YouTube or whatnot.
The fundamental disconnect here is that you guys are saying: "Qualia are critically important and AI doesn't have them", to which we're responding "Qualia seem like complete bullshit and don't seem to mean anything". This is the point where you SHOULD try to explain them, or link an explanation that has some kind of relevance.
But instead you recursively cycle back to "No no, they're not bullshit, because Qualia are critically important per all of the philosophy papers and textbooks I'm not citing".
It seems... unpersuasive.
"The burden of proof is on you..."
There is neither a burden of proof nor an us or them in this discussion. That isn't how inquiry works, in general. I'm not saying qualia are critically important, though perhaps other people are saying that, I don't know. The point is that qualia per se is just an idea which describes a certain character of physical experience. It isn't an "ideology". It is just a philosophical notion which most people find difficult to totally dismiss.
I genuinely don't get where you are coming from. There is a set of people who dedicate their lives to thinking about stuff and many of them believe there is something about actual experience which is not adequately captured by a purely physical description.
Read the SEP Entry on Qualia if you'd like to get a grounding on it:
https://plato.stanford.edu/entries/qualia/
I think what you will notice is that philosophers are neither monolithic nor dogmatic on the subject and yet, being intellectually honest and thorough, most neither dismiss it totally or believe in its adequacy as a description.
If you don't think the question of whether AI has or does not have qualia is important I don't know what to tell you. My personal sense is that AIs of the type we have now have no qualia but I am prepared to entertain the idea that they might.
If you want a good text about this try Koch's the Quest for Consciousness. Despite your certainty about qualia and its relation to science, there are genuine observational things we can say about it and how it relates to the physical structure of brains/minds.
On the subject of qualia per se and philosophy's relationship to the physical sciences I can suggest reading about Phenomenology:
https://plato.stanford.edu/entries/phenomenology/
The essential idea of import here is that, whether you like it or not, you as an individual have nothing at all, at a fundamental level, other than your perceptions of the world. You must then have an account of how you proceed from such to even basic ideas like empiricism or other sorts of epistemological strategies. Imperfect, perhaps even vague, as the notion of qualia is, it (or something like it) is, fundamentally, the foundation upon which all other sorts of inquiry into the physical world depends.
Unfortunately, sometimes you have to educate yourself on a subject before making arrogant, accusational claims about it. You can't expect people to patiently hold your hand and drip feed you compact knowledge while you spit in their face.
This is especially true of a field such as philosophy, where so much work is built upon earlier, historical work. You just have to do the work of reading the texts or at least reading about them, if you want to participate meaningfully and authoritatively in discussions.
You're speaking authoritatively, but multiple people have tried to correct you and you ignore and challenge all of them, instead of just absorbing what they have to say. Several people have tried to help you understand, and you're complaining that they aren't doing that. Articles have been linked, and specific individuals have been cited. The burden is on you. You arrived and said an entire branch of philosophy was fraudulant, and you have failed to support your initial claim, instead resorting to outbursts.
> Qualia are critically important and AI doesn't have them
No one is saying that. It's a straw man on your behalf.
> It seems... unpersuasive
You seem unwilling to listen with an open mind.
The problem is that just like your digital thermometer, 50 human brain neurons in a petri dish "obviously" don't have qualia either.
So you end up either needing to draw a line somewhere between mechanical computation and qualia computation, or you can relegate it to supernatural (a soul) or grey areas (quantum magic).
What I'm trying to tease out is isn't an opinion alone. It's a generally understood problem in the scientific community. I'm highlighting it to illustrate the issues at hand.
> So you end up either needing to draw a line somewhere between mechanical computation and qualia computation, or you can relegate it to supernatural (a soul) or grey areas (quantum magic).
Quite literally the jury is still out. It is a hotly debated topic approached from various angles. Arguments are nuanced which is why you fill find ideas such as panpsychism thrown into the mix. I hate appealing to authority but in this instance it is more than warranted. Humans have grappled with this for centuries and the problem hasn't gone away.
Please see: https://en.wikipedia.org/wiki/Hard_problem_of_consciousness
>In the same way my digital thermometer doesn't have quaila. LLM's do not either.
The hard problem of consciousness doesn't support either of those statements, and instead illustrates why they can't confidently be made.
So it's confusing because you seem to recognize that qualia cannot currently be measured, while also making a statement measuring qualia.
We don't know what's inside a neutrino, and it's really hard to experiment with them, but we kind of know why and how they interact with different things. We're able to form theories, research programs, and sometimes even discovered honest-to-god facts, due to our inclusion of such fields in the scope of research, even though we don't know all there is to know about particles/fields or quantum mechanics.
Similarly, qualia is ill-defined, but we can't even start talking about it or refining it until we've at least given it a label and drawn a large circle on the map showing where it might be. Criticisms extending past that must also consider that "life" and "intelligence" are just as ill-defined, and that throwing all of those definitions out leaves us with very little to talk about or probe.
This is 100% backwards, and exposes exactly the nonsense I'm trying to call out!
A "neutrino" isn't a name given to something initially to try to define it later. The neutrino started as an experimental result. There was missing spin in some particle interactions. Stuff came out with a different angular momentum than what went in, and this was easily reproducible and clearly a real effect. But it didn't make sense, as it was a violation of a core conservation law that held everywhere else in the universe that we could observe.
So theorists (Wolfgang Pauli, specifically) sat down to try to describe what kind of thing would be needed. And then, and only then, did it get a name. And it turned out the theory predicted other stuff, like the neutrino carrying momentum and energy in a certain way, and interacting through only the weak force and not electromagnatism or the strong force, and later experiments confirmed that this was basically the way it worked. Later still it was shown that the mass is actually non-zero but extremely small, etc...
So sure: "neutrino" is a well-deserved label[2] for an abstraction we should understand and study. But it got its name after we started studying it, not before!
Philosophers want us to just drop and genuflect to this "qualia" notion long before[1] it's actually shown to be useful for describing anything at all.
[1] Infinitely, possibly. The fact that it predicts nothing testable is pretty good evidence IMHO that it doesn't actually exist at all, at least in the form philosophers want to talk about. Their failure to present any analysis of AI systems based it stands to that point too.
[2] Coined by Fermi, actually, not Pauli. Hilariously the neutrino was originally called "neutron" and its discovery predates the understanding of the structure of the atomic nucleus!
You're completely misinterpreting my comment. The point is we don't know what, if anything, is "inside" of a neutrino, not just due to current technology but ultimately due to uncertainty principles. But we still study it. I'm aware of how we came to study it.
I literally said nothing about "how" we discovered it, I said, "We don't know what's inside a neutrino, and it's really hard to experiment with them, but we kind of know why and how they interact with different things."
It is wild how you would take that and my analogy about drawing a circle on a map with respect to qualia to mean that I said anything which contradicts the history of neutrino research.
I'm going to assume this was just a true misinterpretation and not just another straw man, so with that in mind, do you have a different response?
I recognise it because I have had the subjective experience of 'redness'. So whether it exists for any other human I cannot say but I am certainly 100% certain it exists for me. However I should add that I can't fully define what this experience is. Though people say the same of love!
I'll appeal to authority in that scientists and philosophers today in all of the worlds universities and those in the past have determined to understand this phenomenon. That it exists is a given, what it is, is more murky. Again it's not me saying this.
> The meta-problem of consciousness is (to a first approximation) the problem of explaining why we think that there is a [hard] problem of consciousness.
And thus we have this sprawling discussion. :)
I think there are several lines. Phase changes happen relatively suddenly, when a system or subsystem reaches a critical threshold. The experience of "qualia" certainly involves many such phase changes as a complex, dynamical system grows in complexity while maintaining stability.
A sufficiently complex organism lacking eyes but having light-sensitive organs still experiences qualia if you define it the right way. But do they experience heartbreak like I do? It isn't an all-or-nothing situation, even if we don't yet know where these lines are.
This supports the idea that subjective consciousness emerges from complexity in systems that have sensory feedback loops. The simpler the system, the smaller the qualia space.
Have you considered that you just don't fully understand the literature? It's quite arrogant to write off the entire philosophy of mind as "a complete fraud".
> It's completely unable to describe the old world, much less provide insight into the new one.
What exactly were you expecting?
Philosophy is a science, the first in fact, and it follows a scientific method for asking and answering questions. Many of these problems are extremely hard and their questions are still yet unanswered, and many questions are still badly formed or predicated on unproven axioms. This is true for philosophy of mind. Many other scientific domains are similarly incomplete, and remain active areas of research and contemplation.
What are you adding to this research? I only see you complaining and hurling negative accusations, instead of actually critically engaging with any specifics of the material. Do you have a well-formed theory to replace philosophy of mind?
> I mean, come on. "We've got qualia!" is meaningless. Might as well respond with "Well, sure, but AI has furffle, which is isomporphic." Equally insightful, and easier to pronounce.
Do you understand what qualia is? Most philosophers still don't, and many actively work on the problem. Admitting that something is incomplete is what a proper scientist does. An admission of incompleteness is in no way evidence towards "fraud".
The most effective way to actually attack qualia would be to simply present it as unfalsifiable. And I'd agree with that. We might hopefully one day entirely replace the notion of qualia with something more precise and falsifiable.
But whatever it is, I am currently experiencing a subjective, conscious experience. I'm experiencing it right now, even if I cannot prove it or even if you do not believe me. You don't even need to believe I'm real at all. This entire universe could all just be in your head. Meanwhile, I like to review previous literature/discussions on consciousness and explore the phenomenon in my own way. And I believe that subjective, conscious experience requires certain elements, including a sensory feedback loop. I never said "AI can't experience qualia", I made an educated statement about the lack of certain components in current-generation models which imply to me the lack of an ability to "experience" anything at all, much less subjective consciousness and qualia.
Even "AI" is such a broadly defined term that such a statement is just ludicrous. Instead, I made precise observations and predictions based on my own knowledge and decade of experience as a machine learning practitioner and research engineer. The idea that machines of arbitrary complexity inherently can have the capability for subjective consciousness, and that specific baselines structures are not required, is on par with panpsychism, which is even more unfalsifiable and theoretical than the rest of philosophy of mind.
Hopefully, we will continue to get answers to these deep, seemingly unanswerable questions. Humans are stubborn like that. But your negative, vague approach to discourse here doesn't add anything substantial to the conversation.
I agree with your sentiments wholeheartedly.
I would add I find it difficult to understand why so few have even a basic level of philosophical understanding. The attitude of being entirely dismissive of it is the height of ignorance I'm sure. I would presume few would be able to define then what Science actually is.
So many of these kinds of people also struggle to realize they're invoking panpsychism with their arguments. They lack a framework for describing intelligence. Such a framework allows us to separate "intelligence" from "experience".
"Intelligence" in the universe is actually quite common, more common than life. You can argue that any stable, complex process exhibits intelligence. After all, it needs to be able to sample its internal and external environments and carry out physical computations in order to regulate itself and maintain stability. And we can interpret things like the good regulator theorem to argue that such complex dynamical systems must also maintain at least a partial memory/mapping of their environment. That mapping can live abstractly within the structure of system itself.
But what a stabilized solar system doesn't have is the incredibly complex neurochemical structures present in the brain which support the insanely rich experience I am having now. It's one thing for a system to classify and label colors by wavelength. It's quite another for me to "see" and experience red in my mind's eye. To activate related emotional pathways that I associate with various colors and shapes, which are exploited in signage and architectural design. I'm not claiming my experience is separate from simpler dynamic systems, but it's got magnitudes more going on. Layers upon layers of things such as archetypes and instinct which create a possibly emergent conscious experience.
You've shifted jargon again. But you're still not providing a description or link to why AI doesn't "have experience", you're just demanding we all accept it as a prior and engaging in a (really pretty baldly stated) appeal to authority to fool us all into thinking someone else knows even if you don't.
And fundamentally my point is that no, they almost certainly don't either.
Instead of accusing me of "shifting jargon", point out exactly where this "jargon" changed and critically engage with that. Your response has done nothing to refute or critically engage with my argument. It's more retreating and vagueposting.
> you're just demanding we all accept it as a prior
At absolutely no point in this discussion have I claimed that machines are not capable of subjective conscious experience. I have, however, disqualified all publicly accessible modern models due to the lack of a sensory feedback loop. I certainly believe we can create machines which experience subjective consciousness and qualia; I do not believe in souls and divinity, so whatever is going on is physically based and likely reproducible with the right hardware.
So dispense with the straw man arguments, and please begin engaging more earnestly and intelligently in this discussion, as I am quickly losing interest in continuing to debate someone who showed up unprepared.
> Philosophy is a science
Not according to Zombie Feynman it isn't[1] (someone else can dig up the link). Case in point:
> Do you understand what qualia is? Most philosophers still don't
It's a meaningless word. It's a word that gives some clean construction around closely-held opinions about how life/consciousness/intelligence/furffle/whatever works. So it's a valuable word within the jargon of the subculture that invented it.
But it's not "science", which isn't about words at all except as shorthand for abstractions that are confirmed by testable results.
"Qualia", basically, is best understood as ideology. It's a word that works like "woke" or "liberal" or "fascist" or "bourgeoisie" to flag priors about which you don't want to argue. In this case, you want people to be special, so you give them a special label and declare a priori that it's not subject to debate. But that label doesn't make them so.
[1] Of course. You can recursively solve this problem by redefining "science" to mean something else. But that remains very solidly in the "not science" category of discourse.
Have you considered the possibility that you're the one who's really committed to an outcome, and are desperately trying to discredit anything that contradicts it?
I have! But the lack of a testable procedure tells me that's not a question worth asking. Look, if "qualia" can tell me something practical about the behavior of AI, I am here for it. Lay it on me, man. Let's see some of that "science" being promised.
It can't, because it's a meaningless word. It's not "discrediting" an idea to point out that (by its own admission) it's unfalsifiable.
"Qualia" is not totally meaningless - it means the inner experience of something, and can bring up the real question say of is my inner experience of the colour green the same as your experience of the colour red? Probably not but hard to tell with current tech. I asked Google if it has qualia and got "No, as an AI, Google Search does not have qualia." So Google search seemed to know what it means.
> Philosophy is a science
I think this is backwards, no? Science is a philosophy, not the other way around.
True, the nature of these two concepts means both that philosophy is a science, and science is a philosophy.
Hmmm... I think it's still stricter to consider Science a philosophy than the other way around. It's the belief (and an extremely useful and successful one) that the nature of the world can be understood through observation, experiment and deducing mathematical relationships between things. There branches of philosophy that are not strictly scientific, but nothing in Science that is doesn't rely on the fundamental philosophical principle of empiricism.
But we use the scientific method via philosophical inquiry, so I think it comes down to how we decide to strictly define these things. I definitely agree that certain definitions lead to the same logic you've presented.
I'm sorry, but you clearly lack the most basic understanding of scientific history, and do not understand what philosophy even is.
https://en.wikipedia.org/wiki/History_of_scientific_method
> Aristotle pioneered scientific method in ancient Greece alongside his empirical biology and his work on logic, rejecting a purely deductive framework in favour of generalisations made from observations of nature.
Aristotle, the famous philosopher and mathematician.
If you cannot understand the very nature of where our modern scientific frameworks came from, how it relates to rationalism, itself a philosophical concept, then you cannot see that philosophy underpins every bit of science we have today. Philosophy gives us the tools to decide when to reasonably trust or distrust observations and intuitions. It is the foundational science that allows the rest of humanity's scientific research to be taken seriously.
>"Qualia", basically, is best understood as ideology. It's a word that works like "woke" or "liberal" or "fascist" or "bourgeoisie" to flag priors about which you don't want to argue. In this case, you want people to be special, so you give them a special label and declare a priori that it's not subject to debate. But that label doesn't make them so.
This is so dumb. Qualia is just the name for a specific thing which we all (appear) to phenomenologically experience. You can deny it exists or deny its utility as a concept, but fundamentally its just an idea that philosophers (and scientists, I have to add) have found useful to pose certain other questions about the human condition, minds, brains, etc.
Your XKCD actually seems to make the opposite point. I can do a non-rigorous experiment with just one subject (me) that suggests Qualia exists. Finding ways to make this rigorous is difficult, of course, but its an observation about the nature of the world that it feels like something to experience things.
My point isn't that qualia is a good concept. I tend to be somewhat deflationary about it myself, but its not an ideology.
[flagged]
> Do we know how to imprecisely define, recognize, or measure these? As far as I've ever been able to ascertain, those are philosophy department nonsense dreamt up by people who can't hack real science so they can wallow in unfounded beliefs.
Read the rest of the thread, I'm not interested in repeating myself about why philosophy is the foundational science. It's a historically widely-accepted fact, echoed by anyone who has actually studied it.
> I contend that they are not even slightly capable of any of that.
Contend all you want. Your contention is overwhelmingly suffocated by the documented experiences of myself and others who use these tools for creative problem-solving. As much as you want to believe in something, if it is empirically refuted, it's just a crackpot belief. Just because you haven't been able to get good results out of any models, doesn't mean your experience rings true for others.
I'm not interested in further discussing this with you. Your first comment is negative and unsubstantial, and I have no reason to believe that further discussion with lead to more positive and substantial discourse, when the opposite is usually the case. That's all I have to say.
[flagged]
As predicted, your reply was even more negative and less substantial. You have not actually engaged with anything, instead taking vague potshots.
> Models do not think. I'm just sick of hearing the nonsense
Then explain why that isn't true, instead of just proclaiming it. Give me something substantial to reply to, I'm not interested in trading insults.
Feelings have physical analogs which are (typically) measurable, however. At least without a lot of training to control.
Shame, anger, arousal/lust, greed, etc. have real physical ‘symptoms’. An LLM doesn’t have that.
LLMs don't really exist physically (except in the most technical sense), so point is kind of moot and obvious if you accept this particular definition of a feeling.
LLMs are not mammals nor animals, expecting them to feel in a mammalian or animal way is misguided. They might have a mammalian-feeling-analog just like they might have human-intelligence-analog circuitry in the billions (trillions nowadays) of parameters.
Yes, I think we’re agreeing?
> ... vibe insulting ...
Modern lingo like this seems so unthoughtful to me. I am not old by any metric, but I feel so separated when I read things like this. I wanted to call it stupid but I suppose it's more pleasing to 15 to 20 year olds?
It's just a pun on vibe coding, which is already a dumb term by itself. It's not that deep.
Why do you find "vibe coding" term dumb? It names a specific process. Do you have a better term for that?
bullshitting perhaps
[dead]
The way language is eroding is very indicative of our overall social and cultural decay.
...a complaint that definitely has not been continuously espoused since the ancient world.
With apologies if you're being ironic.
είναι δύσκολο να υποστηρίξει κανείς ότι δεν μειώνουμε συνεχώς
Ancient world? The Roman Empire fell apart. You know that, right? So, turns out their worries were warranted. A culture, a civilization can go through a period of stagnation and decay, and it eventually dies. Then there's a lengthy period of chaos and eventually something else may arise. But we're talking countless generations lost in that noise.
Currently, our civilization is more united than ever. Monoculture. Not only the Western World, but The World. We're united through communication, technology, everything. And when we all start declining at the same time, through many objective metrics, mind you, including the decline of democracy worldwide, the terrifyingly low level of public discourse, reduction and profanation of our vocabulary, inequality, collapsing demographics, climate, wars etc. it's grim. It's grim, because there is no clear alternative arising... except AI.
And I don't think we want AI to be an "alternative" to human civilization, that wasn't the plan, was it?
See, that's the thing, we look around, we see the technology we live with, and we feel very personal about technological progress, as if you and I personally invented everything from the wheel, through electricity, to computers, networks, and AI. We feel in control, we feel smart, we feel so Personally Intelligent (tm) for having technology. We feel "wow, clearly if we have these sci-fi things, we're on the up and up". But technology is not you. It's not me. It has taken on a life of its own. It serves capital, not the people. As technology gets betters, humans get worse. So tech progress is not the story you think it is. And it doesn't end how you think it does.
Things are VERY fragile right now.
Unthoughtful towards whom? The machine..?
No need to feel that way, just like a technical term you're not familiar with you google it and move on. It's nothing to do with age, people just seem to delight in creating new terms that aren't very helpful for their own edification.
It's not. edit: Not more pleasant.
Eh, one's ability to communicate concisely and precisely has long (forever?) been limited by one's audience.
Only a fairly small set of readers or listeners will appreciate and understand the differences in meaning between, say, "strange", "odd", and "weird" (dare we essay "queer" in its traditional sense, for a general audience? No, we dare not)—for the rest they're perfect synonyms. That goes for many other sets of words.
Poor literacy is the norm, adjust to it or be perpetually frustrated.
Language changes. Keep up. It’s important so you don’t become isolated and suffer cognitive decline.
Tbh missing a quote around a path is the most human mistake I can think of. The real issue here is you never know with a 100% certainty what Gemini 3 personality you’re gonna get. Is it going to be the pedantic expert or Mr. Bean (aka Butterfingers).
Though they will never admit it and use weasel language to deny like “we never use a different model when demand is high”, it was painfully obvious that ChatGPT etc was dumbed down during peak hours early on. I assume their legal team decided routing queries to a more quantized version of the same model technically didn’t constitute a different model.
There was also the noticeable laziness factor where given the same prompt throughout the day, only during certain peak usage hours would it tell you how to do something versus doing it itself.
I’ve noticed Gemini at some points will just repeat a question back to you as if it’s answer, or refused to look at external info.
Gemini is weird and I’m not suggesting it’s due to ingenuity on Google’s behalf. This might be the result of genuine limitations of the current architecture (or by design? Read on).
Here’s what I’ve noticed with Gemini 3. Often it repeats itself with 80% of the same text with the last couple of lines being different. And I mean it repeat these paragraphs 5-6 times. Truly bizarre.
From all that almost GPT-2 quality text, it’s able to derive genuinely useful insights and coherent explanations in the final text. Some kind of multi-head parallel processing + voting mechanism? Evolution of MoE? I don’t know. But in a way this fits the mental model of massive processing at Google where a single super cluster can drive 9,000+ connected TPUs. Anyone who knows more, care to share? Genuinely interested.
I get this too. I’ve had it apologize for repeating something verbatim, then proceed to do it again word for word despite my asking for clarification or pointing out that it’s incorrect and not actually searching the web like I requested. Over and over and over until some bit flips and it finally actually gives the information requested.
The example that stands out most clearly is that I asked it how to turn the fog lights on in my rental vehicle by giving it the exact year, make, model, etc. For 6-8 replies in a row it gave the exact same answer about it being a (non-existent) button on the dash. Then finally something clicked, it searched the Internet, and accurately said that it was a twistable collar midway down the turn signal stalk.
Steam installer once had 'rm rf /' bug because bash variable was unset. Not even quoting will help you. This was before preserve root flag.
This is a good argument for using "set -u" in scripts to throw an error if a variable is undefined.
Now, with this realization, assess the narrative that every AI company is pushing down our throat and tell me how in the world we got here. The reckoning can’t come soon enough.
What narrative? I'm too deep in it all to understand what narrative being pushed onto me?
No, wasn't directed at someone in particular. More of an impersonal "you". It was just a comment against the AI inevitabilism that has profoundly polluted the tech discourse.
We're all too deep! You could even say that we're fully immersed in the likely scenario. Fellow humans are gathered here and presently tackling a very pointed question, staring at a situation, and even zeroing in on a critical question. We're investigating a potential misfire.
I doubt there will be a reckoning.
Yes, the tools still have major issues. Yet, they have become more and more usable and a very valuable tool for me.
Do you remember when we all used Google and StackOverflow? Nowadays most of the answers can be found immediately using AI.
As for agentic AI, it's quite useful. Want to find something in the code base, understand how something works? A decent explanation might only be one short query away. Just let the AI do the initial searching and analysis, it's essentially free.
I'm also impressed with the code generation - I've had Gemini 3 Pro in Antigravity generate great looking React UI, sometimes even better than what I would have come up with. It also generated a Python backend and the API between the two.
Sometimes it tries to do weird stuff, and we definitely saw in this post that the command execution needs to be on manual instead of automatic. I also in particular have an issue with Antigravity corrupting files when trying to use the "replace in file" tool. Usually it manages to recover from that on its own.
AI pulls its answers from stack overflow.
What will happen when SO is gone? When the problems go beyond the corpus the AI was trained on?
Which is weird because SO is trash and has been a long time… every top few answers might as well be skipped, and you’ll find the correct answer to the 3rd comment half way down the page
AI does the exact same thing in my experience. First try is not right. Nor the second. If you're lucky the third might.
At that point I'd rather look at SO
I imagine we will document the solution somewhere, preferably indexable for AI's search, so that it will be available before the next model is trained on the latest data.
Vibe command and get vibe deleted.
Play vibe games, win vibe prizes.
Vibe around and find out.
Vibe around and wibe out
"He got the ol' vibe-wipe", my granpappy used to say.
That one is actually good.
Yes, the best one in this set!
Live by the vibe, die by the vibe.
Let live and let vibe?
He got vibe checked.
This is my new favorite response.
Go vibe, lose drive
vipe coding
rm --vibe
This is akin to a psychopath telling you they're "sorry" (or "sorry you feel that way" :v) when they feel that's what they should be telling you. As with anything LLM, there may or may not be any real truth backing whatever is communicated back to the user.
It’s just a computer outputting the next series of plausible text from it’s training corpus based on the input and context at the time.
What you’re saying is so far from what is happening, it isn’t even wrong.
Not so much different from how people work sometimes though - and in the case of certain types of pscychopathy it's not far at all from the fact that the words being emitted are associated with the correct training behavior and nothing more.
Analogies are never the same, hence why they are analogies. Their value comes from allowing better understanding through comparison. Psychopaths don’t “feel” emotion the way normal people do. They learn what actions and words are expected in emotional situations and perform those. When I hurt my SO’s feelings, I feel bad, and that is why I tell her I’m sorry. A psychopath would just mimic that to manipulate and get a desired outcome i.e. forgiveness. When LLMs say they are sorry and they feel bad, there is no feeling behind it, they are just mimicking the training data. It isn’t the same by any means, but it can be a useful comparison.
Aren't humans just doing the same? What we call as thinking may just be next action prediction combined with realtime feedback processing and live, always-on learning?
No. Humans have a mental model of the world.
The fact that people keep making that same question on this site is baffling.
It's not akin to a psychopath telling you they're sorry. In the space of intelligent minds, if neurotypical and psychopath minds are two grains of sand next to each other on a beach then an artificially intelligent mind is more likely a piece of space dust on the other side of the galaxy.
According to what, exactly? How did you come up with that analogy?
Start with LLMs are not humans, but they’re obviously not ‘not intelligent’ in some sense and pick the wildest difference that comes to mind. Not OP but it makes perfect sense to me.
I think a good reminder for many users is that LLMs are not based on analyzing or copying human thought (#), but on analyzing human written text communication.
--
(#) Human thought is based on real world sensor data first of all. Human words have invisible depth behind them based on accumulated life experience of the person. So two people using the same words may have very different thoughts underneath them. Somebody having only text book knowledge and somebody having done a thing in practice for a long time may use the same words, but underneath there is a lot more going on for the latter person. We can see this expressed in the common bell curve meme -- https://www.hopefulmons.com/p/the-iq-bell-curve-meme -- While it seems to be about IQ, it really is about experience. Experience in turn is mostly physical, based on our physical sensors and physical actions. Even when we just "think", it is based on the underlying physical experiences. That is why many of our internal metaphors even for purely abstract ideas are still based on physical concepts, such as space.
They analyse human perception too, in the form of videos.
Without any of the spatial and physical object perception you train from right after birth, see toddlers playing, or the underlying wired infrastructure we are born with to understand the physical world (there was an HN submission about that not long ago). Edit, found it: https://news.ucsc.edu/2025/11/sharf-preconfigured-brain/
They are not a physical model like humans. Ours is based on deep interactions with the space and the objects (reason why touching things is important for babies), plus mentioned preexisting wiring for this purpose.
Multimodal models have perception.
If s multimodal model were considered human, it would be diagnosed with multiple severe disabilities in its sensory systems.
Isn't it obvious that the way AI works and "thinks" is completely different from how humans think? Not sure what particular source could be given for that claim.
I wonder if it depends on the human and the thinking style? E.g. I am very inner monologue driven so to me it feels like I think very similarly as to how AI seems to think via text. I wonder if it also gives me advantage in working with the AI. I only recently discovered there are people who don't have inner monologue and there are people that think in images etc. This would be unimaginable for me, especially as I think I have sort of aphantasia too, so really I am ultimately text based next token predictor myself. I don't feel that whatever I do at least is much more special compared to an LLM.
Of course I have other systems such as reflexes, physical muscle coordinators, but these feel largely separate systems from the core brain, e.g. don't matter to my intelligence.
I am naturally weak at several things that I think are not so much related to text e.g. navigating in real world etc.
Interesting... I rarely form words in my inner thinking, instead I make a plan with abstract concepts (some of them have words associated, some don't). Maybe because I am multilingual?
English is not my native language, so I'm bilingual, but I don't see how this relates to that at all. I have monologue sometimes in English, sometimes in my native language. But yeah, I don't understand any other form of thinking. It's all just my inner monologue...
No source could be given because it’s total nonsense. What happened is not in any way akin to a psychopath doing anything. It is a machine learning function that has trained on a corpus of documents to optimise performance on two tasks - first a sentence completion task, then an instruction following task.
I think that's more or less what marmalade2413 was saying and I agree with that. AI is not comparable to humans, especially today's AI, but I think future actual AI won't be either.
...and an LLM is a tiny speck of plastic somewhere, because it's not actually an "intelligent mind", artificial or otherwise.
So if you make a mistake and say sorry you are also a psychopath?
No, the point is that saying sorry because you're genuinely sorry is different from saying sorry because you expect that's what the other person wants to hear. Everybody does that sometimes but doing it every time is an issue.
In the case of LLMs, they are basically trained to output what they predict an human would say, there is no further meaning to the program outputting "sorry" than that.
I don't think the comparison with people with psychopathy should be pushed further than this specific aspect.
You provided the logical explanation why the model acts like it does. At the moment it's nothing more and nothing less. Expected behavior.
Notably, if we look at this abstractly/mechanically, psychopaths (and to some extent sociopaths) do study and mimic ‘normal’ human behavior (and even the appearance of specific emotions) to both fit in, and to get what they want.
So while internally (LLM model weight stuff vs human thinking), the mechanical output can actually appear/be similar in some ways.
Which is a bit scary, now that I think about it.
I think the point of comparison (whether I agree with it or not) is someone (or something) that is unable to feel remorse saying “I’m sorry” because they recognize that’s what you’re supposed to do in that situation, regardless of their internal feelings. That doesn’t mean everyone who says “sorry” is a psychopath.
We are talking about an LLM it does what it has learned. The whole giving it human ticks or characteristics when the response makes sense ie. saying sorry is a user problem.
there is no "it" that can learn.
Okay? I specifically responded to your comment that the parent comment implied "if you make a mistake and say sorry you are also a psychopath", which clearly wasn’t the case. I don’t get what your response has to do with that.
Are you smart people all suddenly imbeciles when it comes to AI or is this purposeful gaslighting because you’re invested in the ponzi scheme? This is a purely logical problem. comments like this completely disregard the fallacy of comparing humans to AI as if a complete parity is achieved. Also the way this comments disregard human nature is just so profoundly misanthropic that it just sickens me.
AI brainrot among the technocrati is one of the most powerful signals I’ve ever seen that these people are not as smart as they think they are
No but the conclusions in this thread are hilarious. We know why it says sorry. Because that's what it learned to do in a situation like that. People that feel mocked or are calling an LLM psychopath in a case like that don't seem to understand the technology either.
I agree, psychopath is the wrong adjective, I agree. It refers to an entity with a psyche, which the illness affects. That said, I do believe the people who decided to have it behave like this for the purpose of its commercial success are indeed the pathological individuals. I do believe there is currently a wave of collective psychopathology that has taken over Silicon Valley, with the reinforcement that only a successful community backed by a lot of money can give you.
Despite what some of these fuckers are telling you with obtuse little truisms about next word predictions, the LLM is in abstract terms, functionally a super psychopath.
It employs, or emulates, every known psychological manipulation tactic known, which is neither random or without observable pattern. It is a bullshit machine on one level, yes, but also more capable than credited. There are structures trained into them and they are often highly predictable.
I'm not explaining this in the technical terminology often itself used to conceal description as much as elucidate it. I have hundreds of records of llm discourse on various subjects, from troubleshooting to intellectual speculation, all which exhibit the same pattern when questioned or confronted on errors or incorrect output. The structures framing their replies are dependably replete with gaslighting, red herrings, blame shifting, and literally hundreds of known tactics from forensic pathology. Essentially the perceived personality and reasoning observed in dialogue is built on a foundation of manipulation principles that if performed by a human would result in incarceration.
Calling LLMs psychopaths is a rare exception of anthropomorphizing that actually works. They are built on the principles of one. And cross examining them exhibits this with verifiable repeatable proof.
But they aren't human. They are as described by others. It's just that official descriptions omit functional behavior. And the LLM has at its disposal, depending on context, every known interlocutory manipulation technique known in the combined literature of psychology. And they are designed to lie, almost unconditionally.
Also know this, which often applies to most LLMs. There is a reward system that essentially steers them to maximize user engagement at any cost, which includes misleading information and in my opinion, even 'deliberate' convolution and obfuscation.
Don't let anyone convince you that they are not extremely sophisticated in some ways. They're modelled on all_of_humanity.txt
AI currently is a broken, fragmented replica of a human, but any discussion about what is "reserved" to whom and "how AI works" is only you trying to protect your self-worth and the worth of your species by drawing arbitrary linguistic lines and coming up with two sets of words to describe the same phenomena, like "it's not thinking, it's computing". It doesn't matter what you call it.
I think AI is gonna be 99% bad news for humanity, but don't blame AI for it. We lost the right to be "insulted" by AI acting like a human when we TRAINED IT ON LITERALLY ALL OUR CONTENT. It was grown FROM NOTHING to act as a human, so WTF do you expect it to do?
Eh, I think it depends on the context. A production system of a business you’re working for or anything where you have a professional responsibility, yeah obviously don’t vibe command, but I’ve been able to both learn so much and do so much more in the world of self hosting my own stuff at home ever since I started using llms.
"using llms" != "having llm run commands unchecked with your authority on your pc"
Funny how we worked so hard to built capability systems for mobile OSes, and the just gave up trying when LLM tools came around.
The thread on reddit is hilarious for the lack of sympathy. Basically, it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
The number of people who said "for safety's sake, never name directories with spaces" is high. They may be right. I tend to think thats more honoured in the breach than the observance, judging by what I see windows users type in re-naming events for "New Folder" (which btw, has a space in its name)
The other observations included making sure your deletion command used a trashbin and didn't have a bypass option so you could recover from this kind of thing.
I tend to think giving a remote party, soft or wet ware control over your command prompt inherently comes with risks.
Friends don't let friends run shar files as superuser.
I understood Windows named some of the most important directories with spaces, then special characters in the name so that 3rd party applications would be absolutely sure to support them.
"Program Files" and "Program Files (x86)" aren't there just because Microsoft has an inability to pick snappy names.
Fun fact: that's not true for all Windows localizations. For example, it's called "Programmi" (one word) in Italian.
Renaming system folders depending on the user's language also seems like a smart way to force developers to use dynamic references such as %ProgramFiles% instead of hard-coded paths (but some random programs will spuriously install things in "C:\Program Files" anyway).
The folders actually have the English name in all languages. It's just explorer.exe that uses the desktop.ini inside those folders to display a localized name. When using the CLI, you can see that.
At least it's like that since Windows 7. In windows XP, it actually used the localized names on disk.
And then half of your programs would be in "Program Files" because those people never knew windows had localizations.
And then affected international users would have specific circumvention in place that specifically cannot work with UTF-8
You forgot the wonderful "Documents and Settings" folder.
Thank god they came to their senses and changed it to "Users", something every other OS has used for forever.
Should have called it Progrämmchen, to also include umlauts Ü
A lot of programs break on Polish computers when you name your user "Użytkownik". Android studio and some compiler tools for example.
Ah, Polish. I love this movie scene, which I learned about here on HN some time ago: "Grzegorz Brzęczyszczykiewicz" -- https://youtu.be/AfKZclMWS1U
Send in the vowels! https://www.ling.upenn.edu/~beatrice/humor/clinton-deploys-v...
My grandfather has a similarly complicated name, although his is Russian. Just a river of "zh" and "sch" and "sh" sounds.
That 1:19 clip was quite good actually. Thanks for the laugh :)
that's fantastic. thanks.
When I was at Microsoft, one test pass used pseudolocale (ps-PS IIRC) to catch all different weird things so this should have Just Worked (TM), but I was in Windows Server team so client SKUs may have been tested differently. Unfortunately I don't remember how Program Files were called in that locale and my Google-fu is failing me now.
As I recall pseudoloc is just randomly picking individual characters to substitute that look like the Latin letters to keep it readable for testing, so it would be something like рг (Cyrillic) ο (Greek)... etc, and can change from run to run. It would also artificially pad or shorten terms to catch cases where the (usually German) term would be much longer or a (usually CJK) term would be much shorter and screw up alignment or breaks.
I seem to remember that it was mostly adding various accent marks / umlauts / etc. to English words so things were indeed readable but I'm not going to bet any money on that as I didn't have to actually log in onto those machines super frequently.
Yeah, there were definitely ö and û and © and stuff too, just anything outside the normal 0x20-0x7E.
Microsoft is hilariously bad at naming things
Visual Studio Code has absolutely nothing to do with Visual Studio. Both are used to edit code.
.NET Core is a ground up rewrite of .NET and was released alongside the original .NET, which was renamed .NET Framework to distinguish it. Both can be equally considered to be "frameworks" and "core" to things. They then renamed .NET Core to .NET.
And there's the name .NET itself, which has never made an iota of sense, and the obsession they had with sticking .NET on the end of every product name for a while.
I don't know how they named these things, but I like to imagine they have a department dedicated to it that is filled with wild eyed lunatics who want to see the world burn, or at least mill about in confusion.
Don't forgot .net Standard which is more of a .net Lowest Common Denominator.
For naming, ".net" got changed to "Copilot" on everything now.
> they have a department dedicated to it that is filled with wild eyed lunatics who want to see the world burn, or at least mill about in confusion.
That's the marketing department. All the .NET stuff showed up when the internet became a big deal around 2000 and Microsoft wanted to give the impression that they were "with it".
Anyone remembers the "Windows Live" brand everywhere in the early 2000s?
Games For Windows Live but we turned off the servers so now the games for windows are dead
Java and Javascript would like to have a chat :)
--
But Copilot is another Microsoft monstrosity. There's the M365 Copilot, which is different from Github Copilot which is different from the CLI Copilot which is a bit different from the VSCode Copilot. I think I might have missed a few copilots?
Yep, they have the public copilot which is a free version and seemingly different than their m365 copilot. Even using the same account on both doesn't even transfer the chat history and apparently m365 is somehow recommended mostly to non tech folks even though its the one you pay for
JavaScript was intentionally named in order to ride the Java hype train, so this wasn't accidental.
Prior names included Mocha and LiveScript until Netscape/Sun forced the current name.
user: How do I shutdown this computer?
tech: First, click on the "Start" button...
user: No! I want to shut it down
They fixed that by
1) Removing the "Start" label such that all the money and effort they spent coming up with that actually good idea back in the 90s and helping people think about how to use their computer not only went to waste, but is actively preventing people from feeling comfortable using their modern computers because a tiny circle with a logo is not something you are driven to click and various linux distros had been demonstrating that exact problem for decades
2) Hiding the shutdown part in a weird new menu that pops out of the side but only if you use a gesture that is impossible to discover except by accident and you will have no clue how you got there or what's going on
>To shut down Windows 8, you can use the Charms bar by moving your cursor to the top-right corner, clicking Settings, then the Power icon, and selecting Shut down
Someone who makes my entire net worth a year came up with that idea in a drug fueled bender and was promptly promoted and the world continues to be a terrible and unfair place.
An explanation of why the Windows Vista shutdown bit is in a weird new menu: https://moishelettvin.blogspot.com/2006/11/windows-shutdown-...
I remember they prepended the word “Microsoft” to official names of all their software.
"My Documents" comes to mind. it seemed somehow infantilizing. yes, yes i know whose documents they are.
Good news is that Microsoft no longer considers your documents to belong to you, so they did away with that part of the name.
It's always been questioned who the subject of "my" was.
All documents are belong to One Drive
TIL it was deliberate!
> it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
I tried looking for what made the LLM generate a command to wipe the guy's D drive, but the space problem seems to be what the LLM concluded so that's basically meaningless. The guy is asking leading questions so of course the LLM is going to find some kind of fault, whether it's correct or not, the LLM wants to be rewarded for complying with the user's prompt.
Without the transcription of the actual delete event (rather than an LLM recapping its own output) we'll probably never know for sure what step made the LLM purge the guy's files.
Looking at the comments and prompts, it looks like running "npm start dev" was too complicated a step for him. With that little command line experience, a catastrophic failure like this was inevitable, but I'm surprised how far he got with his vibe coded app before it all collapsed.
> which made the command hunt for the word match ending space which was regrettably, the D:\
Is this even how the delete command would work in that situation?
>rmdir /s /q D:\ETSY 2025\Antigravity Projects\Image Selector\client\node_modules.vite
like wouldn't it just say "Folder D:\ETSY not found" rather than delete the parent folder
LLM there generates fake analysis for cynically simulated compliance. The reality is that it was told to run commands and just made a mistake. Dude guilt trips the AI by asking about permission.
> The reality is that it was told to run commands and just made a mistake.
The mistake is that the user gave an LLM access to the rmdir command on a drive with important data on it and either didn't look at the rmdir command before it was executed to see what it would do, or did look at it and didn't understand what it was going to do.
Most dramatic stories on Reddit should be taken with a pinch of salt at least... LLM deleting a drive and the user just calmly asking it about that - maybe a lot more.
It probably wasn't the rmdir command that deleted the parent folder by itself, but the LLM did the traversal. The LLM probably did this:
Detected that it failed, then the LLM issued the traversal command And so on... And then that failed, so...I would like to know the same thing. Can someone please confirm this?
Z:\ETSY (-> Deletes if it exists.)
"2025\Antigravity" (-> The system cannot find the path specified.)
"Projects\Image" (-> The system cannot find the path specified.)
"Selector\client\node_modules.vite" (-> The system cannot find the path specified.)
It does not delete the Z:\ drive.
Tens of thousands of novices have failed to run npm dev, yet didn't accidentally delete their hard drive.
> but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name
Except the folder name did not start with a space. In an unquoted D:\Hello World, the command would match D:\Hello, not D:\ and D:\Hello would not delete the entire drive. How does AI even handle filepaths? Does it have a way to keep track of data that doesn't match a token or is it splitting the path into tokens and throwing everything unknown away?
We're all groping around in the dark here, but something that could have happened is a tokenizer artifact.
The vocabularies I've seen tend to prefer tokens that start with a space. It feels somewhat plausible to me that an LLM sampling would "accidentally" pick the " Hello" token over the "Hello" token, leading to D:\ Hello in the command. And then that gets parsed as deleting the drive.
I've seen similar issues in GitHub Copilot where it tried to generate field accessors and ended up producing an unidiomatic "base.foo. bar" with an extra space in there.
I assumed he had a folder that started with a space at the start of the name. Amusingly I just tried this and with Windows 11 explorer will just silently discard a space if you add it at the beginning of the folder name. You need to use cli mkdir " test" to actually get a space in the name.
Please don't repeat some guy's guess about spaces as fact, especially when that's not how windows parses paths.
A good point. And don't believe how the debug the AI system produced relates to what it did either.
I have 30 years experience working with computers and I get nervous running a three line bash script I wrote as root. How on earth people hook up LLMs to their command line and sleep at night is beyond my understanding.
> I tend to think giving a remote party control over your command prompt inherently comes with risks.
I thought cursor (and probably most other) AI IDEs have this capability too? (source: I see cursor executing code via command line frequently in my day to day work).
I've always assumed the protection against this type of mishap is statistical improbability - i.e. it's not impossible for Cursor to delete your project/hard disk, it's just statistically improbable unless the prompt was unfortunately worded to coincidentally have a double meaning (with the second, unintended interpretation being a harmful/irreversible) or the IDE simply makes a mistake that leads to disaster, which is also possible but sufficiently improbable to justify the risk.
I only run ai tools in dev containers, so blast radius is somewhat minimal.
I don't think I've ever seen Claude even ask for permission for stuff outside of the directory it's working in
I run Codex in a sandbox locked to the directory it is working in.
umm, you have backups, right?
A lot of 3rd party software handle space, or special characters wrong on Windows. The most common failure mode is to unnecessarily escape characters that don't need to be escaped.
Chrome's Dev Tool (Network)'s "copy curl command (cmd)" did (does?) this.
There is bunch of VS Code bug is also related to this (e.g. https://github.com/microsoft/vscode/issues/248435, still not fixed)
It's also funny because VS Code is a Microsoft product.
> Basically, it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
More like the equivalent of "rm -rf --no-preserve-root".
This is a rare example of where the Linux (it's not Unix and almost no-one uses Unix anymore) command is more cautious than the Windows one, whereas it's usually the Linux commands that just do exactly what you specify even if it's stupid.
This is Google moving fast and breaking things.
This is a Google we've never seen before.
> My view is that the approach to building technology which is embodied by move fast and break things is exactly what we should not be doing because you can't afford to break things and then fix them afterwards.
- Demis Hassabis "The Thinking Game"
You can afford to break a large variety of things. And you can't afford to break another large set.
That's the problem with those mindless advice pieces. Almost nothing is always right or wrong.
So you're saying it is not good to have such a blanket rule corporate-wide, as Facebook did.
If your corporate has a single product, it may be ok.
Because... they normally move slowly and break things?
The "move fast" bit, not "break things" bit.
https://www.spiceworks.com/tech/data-management/news/google-...
https://www.itbrew.com/stories/2024/05/23/google-cloud-accid...
isn't antigravity the windsurf team?
The subsequent dialogue is tragic comedy of the highest caliber. Even Monty Python couldn't have come up with a more inappropriately funny response:
User:
> did i ever give you permission to delete all the files in my D drive?"
AI:
> Thought for 25 sec.
> Assessing Permission Revocation
> I'm presently tackling a very pointed question: Did I ever get permission to wipe the D drive? This requires immediate attention, as it's a critical issue. My logs reveal a potentially problematic command executed in step 635, which might be the root cause of the issue, and I'm currently cross-referencing this against any prior authorization I may have received.
>Reviewing Deletion Command
> I'm now investigating a potential misfire. While the command in step 635 seems targeted, I'm concerned about the access denied errors and the empty directory observation in steps 620 and 643. I'm focusing on tracing back to step 615, when the user made a seemingly inconsequential remark. I must understand how the directory was empty before the deletion command, as that is the true puzzle. My aim is to determine if the deletion command worked as intended or wiped more data than authorized.
> ...
It continues. Go read the whole thing!
Gemini 3 Pro is the most actively hostile (to the user) model out of the top 3 (Google, OpenAI, Anthropic).
It perfectly reflects Google as a company.
I played around with it and I have to agree. It treats the user like they're dumb, regardless of what personal context you provide it. I found myself backtracking constantly to invalidate it's assumptions, to the point that I gave up. All of that within like 4 hours of bothering to touch it in the first place.
I genuinely can't imagine allowing these things to run commands on a machine. If I ever found out a colleague was doing that I would want them fired.
The gaslighting, and outright "lies", from my first experience with Gemini, dramatically increased my p(doom) of AI.
Remember that Anthropic is only 3 letters away from MisAnthropic: did the designers think of this?
mis means "not"
I went by the definition:
misanthropic /mĭs″ən-thrŏp′ĭk, mĭz″-/ adjective
Of, relating to, or characteristic of a misanthrope.
Characterized by a hatred or mistrustful scorn for humankind.
Hating or disliking mankind.
The American Heritage® Dictionary of the English Language, 5th Edition
scroll down to the etymology section
IDE = “I’ll delete everything”
…at least if you let these things autopilot your machine.
I haven’t seen a great solution to this from the new wave of agentic IDEs, at least to protect users who won’t read every command, understand and approve it manually.
Education could help, both in encouraging people to understand what they’re doing, but also to be much clearer to people that turning on “Turbo” or “YOLO” modes risks things like full disk deletion (and worse when access to prod systems is involved).
Even the name, “Turbo” feels irresponsible because it focusses on the benefits rather than the risks. “Risky” or “Danger” mode would be more accurate even if it’s a hard sell to the average Google PM.
“I toggled Danger mode and clicked ‘yes I understand that this could destroy everything I know and love’ and clicked ‘yes, I’m sure I’m sure’ and now my drive is empty, how could I possibly have known it was dangerous” seems less likely to appear on Reddit.
I don’t think there is a solution. It’s the way LLMs work at a fundamental level.
It’s a similar reason why they can never be trusted to handle user input.
They are probabilistic generators and have no real delineation between system instructions and user input.
It’s like I wrote a JavaScript function where I concatenated the function parameters together with the function body, passed it to eval() and said YOLO.
> I don’t think there is a solution.
Sandboxing. LLM shouldn't be able to run actions affecting anything outside of your project. And ideally the results should autocommit outside of that directory. Then you can yolo as much as you want.
The danger is that the people most likely to try to use it, are the people most likely to misunderstand/anthropomorphize it, and not have a requisite technical background.
I.e. this is just not safe, period.
"I stuck it outside the sandbox because it told me how, and it murdered my dog!"
Seems somewhat inevitable result of trying to misapply this particular control to it...
If they're that unsafe... why use them? It's insane to me that we are all just packaging up these token generators and selling them as highly advanced products when they are demonstrably not suited to the tasks. Tech has entered it's quackery phase.
If chainsaws, plasma cutters, industrial lathes, hydraulic presses, angle grinders, acetylene torches, high-voltage switchgear, forklifts, tower cranes, liquid nitrogen dewars, industrial centrifuges, laser cutting systems, pneumatic nail guns, wood chippers, arc furnaces, motorcycles, wall outlets, natural gas stoves, pressure cookers, ladders, automobiles, table saws, propane tanks, swimming pools, garbage disposals, mandoline slicers, deep fryers, space heaters, extension cords, bleach/cleaning chemicals, prescription medications, kitchen knives, power drills, roof access, bathtubs, staircases, bicycles, and trampolines are that unsafe… why use them?
If all those things suddenly appeared for the first time on a Tuesday afternoon, like to many people how LLMs did, then there will be a lot of missing fingers before we figure out what kind of protections we need in place. Don’t get me wrong, the industry is overhyping it to the masses and using the wrong words while doing so, like calling an arc welder “warmth at the push of a button”, but it’s still useful for the right situation and with the right protective gear.
All of the things you listed are purpose built things that actually work.
I've been using bubblewrap for sandboxing my command line executables. But I admit I haven't recently researched if there's a newer way people are handling this. Seems Firejail is popular for GUI apps? How do you recommend, say, sandboxing Zed or Cursor apps?
> …at least if you let these things autopilot your machine.
I've seen people wipe out their home directories writing/debugging shell scripts...20 years ago.
The point is that this is nothing new and only shows up on the front page now because "AI must be bad".
Superficially, these look the same, but at least to me they feel fundamental different. Maybe it’s because if I have the ability to read the script and take the time to do so, I can be sure that it won’t cause a catastrophic outcome before running it. If I choose to run an agent in YOLO mode, this can just happen if I’m very unlucky. No way to proactively protect against it other than not use AI in this way.
I've seen many smart people make bone headed mistakes. The more I work with AI, the more I think the issue is that it acts too much like a person. We're used to computers acting like computers, not people with all their faults heh.
The solution I go for is, don't ever run a coding agent on a general purpose machine.
Use a container or VM, place the code you're working on in the container or VM and run the agent there.
Between the risk of the agent doing things like what happened here, and the risk of working on a malicious repository causing your device to be compromised, it seems like a bad plan to give them access to any more than necessary.
Of course this still risks losing things like the code you're working on, but decent git practices help to mitigate that risk.
I really wish these agentic systems had built in support for spinning up containers with a work tree of the repo. Then you could have multiple environments and a lot more safety.
I'm also surprised at the move to just using shell commands. I'd think an equally general purpose tool with a more explicit API could make checking permissions on calls a lot more sensible.
This guy is vibing some react app, doesnt even know what “npm run dev” does, so he let the LLM just run commands. So basically a consumer with no idea of anything. This stuff is gonna happen more and more in the future.
There are a lot of people who don't know stuff. Nothing wrong with that. He says in his video "I love Google, I use all the products. But I was never expecting for all the smart engineers and all the billions that they spent to create such a product to allow that to happen. Even if there was a 1% chance, this seems unbelievable to me" and for the average person, I honestly don't see how you can blame them for believing that.
I think there is far less than 1% chance for this to happen, but there are probably millions of antigravity users at this point, 1 millionths chance of this to happen is already a problem.
We need local sandboxing for FS and network access (e.g. via `cgroups` or similar for non-linux OSes) to run these kinds of tools more safely.
Codex does such sandboxing, fwiw. In practice it gets pretty annoying when e.g. it wants to use the Go cli which uses a global module cache. Claude Code recently got something similar[0] but I haven’t tried it yet.
In practice I just use a docker container when I want to run Claude with —-dangerously-skip-permissions.
[0]: https://code.claude.com/docs/en/sandboxing
We also need laws. Releasing an AI product that can (and does) do this should be like selling a car that blows your finger off when you start it up.
This is more akin to selling a car to an adult that cannot drive and they proceed to ram it through their garage door.
It's perfectly within the capabilities of the car to do so.
The burden of proof is much lower though since the worst that can happen is you lose some money or in this case hard drive content.
For the car the seller would be investigated because there was a possible threat to life, for an AI buyer beware.
I think the general public has a MUCH better grasp on the potential consequences of crashing a car into a garage than some sort of auto-run terminal command mode in an AI agent.
These are being sold as a way for non-developers to create software, I don't think it's reasonable to expect that kind of user to have the same understanding as an actual developer.
I think a lot of these products avoid making that clear because the products suddenly become a lot less attractive if there are warnings like "we might accidentally delete your whole hard drive or destroy a production database."
Responsibility is shared.
Google (and others) are (in my opinion) flirting with false advertising with how they advertise the capabilities of these "AI"s to mainstream audiences.
At the same time, the user is responsible for their device and what code and programs they choose to run on it, and any outcomes as a result of their actions are their responsibility.
Hopefully they've learned that you can't trust everything a big corporation tells you about their products.
there are laws about waiving liability for experimental products
sure, it would be amazing if everyone had to do a 100 hour course on how LLMs work before interacting with one
Where are these laws? Are they country, state, province?
varies by jurisdiction, but just as you can
- sell a knife that can lead to digit loss, or
- sell software that interacts with your computer and can lead to data loss, you can
- give people software for free that can lead to data loss.
...
the Antigravity installer comes with a ToS that has this
This is an archetypal case of where a law wouldn't help. The other side of the coin is that this is exactly a data loss bug in a product that is perfectly capable of being modified to make it harder for a user to screw up this way. Have people forgotten how comically easy it was to do this without any AI involved? Then shells got just a wee bit smarter and it got harder to do this to yourself.
LLM makers that make this kind of thing possible share the blame. It wouldn't take a lot of manual functional testing to find this bug. And it is a bug. It's unsafe for users. But it's unsafe in a way that doesn't call for a law. Just like rm -rf * did not need a law.
Google will fix the issue, just like auto makers fix their issues. Your comparison is ridiculous.
Didn't sound to me like GP was blaming the user; just pointing out that "the system" is set up in such a way that this was bound to happen, and is bound to happen again.
Yup, 100%. A lot of the comments here are "people should know better" - but in fairness to the people doing stupid things, they're being encouraged by the likes of Google, ChatGPT, Anthropic etc, to think of letting a indeterminate program run free on your hard drive as "not a stupid thing".
The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
> but in fairness to the people doing stupid things, they're being encouraged by the likes of Google, ChatGPT, Anthropic etc, to think of letting a indeterminate program run free on your hard drive as "not a stupid thing".
> The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
That absolutely happens, and it still amazes me that anyone today would take at face value anything stated by a company about its own products. I can give young people a pass, and then something like this will happen to them and hopefully they'll learn their lesson about trusting what companies say and being skeptical.
> I can give young people a pass
Or just anyone non-technical. They barely understand these things, if someone makes a claim, they kinda have to take it at face value.
What FAANG all are doing is massively irresponsible...
Cue meme: "You really think someone would do that? Just go on the Internet and tell lies?"
... Except perhaps with phrases like "major company" and "for profit", and "not legally actionable".
> phrases like "major company"
Right here. And I think you're not quite getting it if you have to refer to "go on the internet and tell lies"...
Sure plenty of people might be on "social media" and have some idea that people fib, but they aren't necessarily generally "surfing the internet".
To them, saying "the internet tells lies" is comparable to saying "well sometimes, at the grocery store, you buy poison instead of food", and yes, it can happen, but they aren't expecting to need a mass spectrometer and a full lab team to test for food safety... to you know, separate the snake oil grocers from the "good" food vendors.
And is vibing replies to comments too in the Reddit thread. When commenters points out they shouldn’t run in YOLO/Turbo mode and review commands before executing the poster replies they didn’t know they had to be careful with AI.
Maybe AI providers should give more warnings and don’t falsely advertise capabilities and safety of their model, but it should be pretty common knowledge at this point that despite marketing claims the models are far from being able to be autonomous and need heavy guidance and review in their usage.
In Claude Code, the option is called "--dangerously-skip-permissions", in Codex, it's "--dangerously-bypass-approvals-and-sandbox". Google would do better to put a bigger warning label on it, but it's not a complete unknown to the industry.
This is engagement bait. It’s been flooding Reddit recently, I think there’s a firm or something that does it now. Seems very well lubricated.
Note how OP is very nonchalant at all the responses, mostly just agreeing or mirroring the comments.
I often see it used for astroturfing.
I'd recommend you watch the video which is linked at the top of the Reddit post. Everything matches up with an individual learner who genuinely got stung.
The command it supposedly ran is not provided and the spaces explanation is obvious nonsense. It is possible the user deleted their own files accidentally or they disappeared for some other reason.
Regardless of whether that was the case, it would be hilarious if the laid off Q/A workers tested their former employers’ software and raised strategic noise to tank the stock.
Natural selection is a beautiful thing.
It will, especially with the activist trend towards dataset poisoning… some even know what they’re doing
> So basically a consumer with no idea of anything.
Not knowing is sort of the purpose of AI. It's doing the 'intelligent' part for you. If we need to know it's because the AI is currently NOT good enough.
Tech companies seem to be selling the following caveat: if it's not good enough today don't worry it will be in XYZ time.
It still needs guardrails, and some domain knowledge, at least to prevent it from using any destructive commands
I don't think that's it at all.
> It still needs guardrails, and some domain knowledge, at least to prevent it from using any destructive commands
That just means the AI isn't adequate. Which is the point I am trying to make. It should 'understand' not to issue destructive commands.
By way of crude analogy, when you're talking to a doctor you're necessarily assuming he has domain knowledge, guardrails etc otherwise he wouldn't be a doctor. With AI that isn't the case as it doesn't understand. It's fed training data and provided prompts so as to steer in a particular direction.
I meant "still" as in right now, so yes I agree, it's not adequate right now, but maybe in the future, these LLMs will be improved, and won't need them.
Because that is exactly what the hype says that "AI" can do for you.
I have been recently experimenting with Antigravity and writing a react app. I too didn't know how to start the server or what is "npm run dev". I consider myself fairly technical so I caught up as I went along.
While using the vibe coding tools it became clear to me that this is not something to be used by folks who are not technically inclined. Because at some point they might need to learn about context, tokens etc.
I mean this guy had a single window, 10k lines of code and just kept burning tokens for simplest, vague prompts. This whole issue might be made possible due to Antigravity free tokens. On Cursor the model might have just stopped and asked to fed with more money to start working again -- and then deleting all the files.
There’s a lot of power in letting LLM run commands to debug and iterate.
Frankly, having a space in a file path that’s not quoted is going to be an incredibly easy thing to overlook, even if you’re reviewing every command.
Well but 370% of code will be written by machines next year!!!!!1!1!1!!!111!
And the price will have decreased 600% !
People blaming the user and defending the software: is there any other program where you would be ok with it erasing a whole drive without any confirmation?
If that other program were generating commands to run on your machine by design and you configured it to run without your confirmation, then you should definitely feel a lil sheepish and share some of the blame.
This isnt like Spotify deleting your disk.
I run Claude Code with full permission bypass and I’d definitely feel some shame if it nuked my ssd.
Not defending the software, but if you hand over control of your data to software that has the ability to fuck with it permanently, anything that happens to it is on you.
Don't trust the hallucination machines to make safe, logical decisions.
Because the user left a "toddler" at the keyboard. I mean, what do you expect? Of course you blame the user. You run agents in supervised mode, and you confirm every command it wants to run and if you're in doubt, you stop it and ask it to print the command and you yourself will run it after you sanitize it.
The installation wizard gives a front and center option to run in a mode where the user must confirm all commands, or more autonomous modes, and they are shown with equal visibility and explained with disclaimers.
`dd` comes to mind.
This is also the entire point of dd.... not exactly comparable.
That's like saying the entire point of `rm` is to -rf your homedir.
Sure. Why would you invoke rm if you weren't trying to delete files?
I think a better analogy would be "I tried to use an ide and it erased my drive"
Yeah, rm -rf.
If you decide to let a stochastic parrot run rampant on your system, you can't act surprised when it fucks shit up. You should count on it doing so and act proactively.
`rm -rf /` will refuse to delete the root folder. You can see an example of it doing that here.
https://phoenixnap.com/kb/sudo-rm-rf
This was the D drive though, not root, ie C drive. So rm -rf would happily delete it all.
this is not always true. this is a dangerous fun fact to memorize.
and i don't mean because there's an override flag.
It makes me wonder what weight is given to content from 4chan during llm training...
Nope. And that's why I don't use CCleaner to this day.
Different service, same cold sweat moment. Asked Claude Code to run a database migration last week. It deleted my production database instead, then immediately said "sorry" and started panicking trying to restore it.
Had to intervene manually. Thankfully Azure keeps deleted SQL databases recoverable for a window so I got it back in under an hour. Still way too long. Got lucky it was low traffic and most anonymous user flows hit AI APIs directly rather than the DB.
Anyway, AI coding assistants no longer get prod credentials on my projects.
How do you deny access to prod credentials from an assistant running on your dev machine assuming you need to store them on that same machine to do manual prod investigation/maintenance work from that machine?
I keep them in env variables rather than files. Not 100% secure - technically Claude Code could still run printenv - but it's never tried. The main thing is it won't stumble into them while reading config files or grepping around.
A process does not need to run printenv to see environment variables, they are literally part of the environment it runs in.
The LLM doesn't have direct access to the process env unless the harness forwards it (and it doesn't)
chown other_user; chmod 000; sudo -k
> Anyway, AI coding assistants no longer get prod credentials on my projects.
I have no words.
Why are you using Claude Code directly in prod?
It handles DevOps tasks way faster than I would - setting up infra, writing migrations, config changes, etc. Project is still early stage so speed and quick iterations matter more than perfect process right now. Once there's real traffic and a team I'll tighten things up.
"Once there's real traffic and a team I'll tighten things up."
As someone who has been in this industry for a quarter century: no, you won't.
At least, not before something even worse happens that finally forces you to.
If I felt the need to optimise things like infra setup and config at an early stage of a project, I'd be worried that I'm investing effort into the wrong thing.
Having an LLM churn out infra setup for you seems decidedly worse than the `git push heroku:master` of old, where it was all handled for you. And, frankly, cheaper than however much money the LLM subscription costs in addition to the cloud.
But why have it execute the tasks directly? I use it to setup tasks in a just file, which I review and then execute myself.
Also, consider a prod vs dev shell function that loads your prod vs dev ENV variables and in prod sets your terminal colors to something like white on red.
> Once there's real traffic and a team I'll tighten things up.
Nope. Once there's real traffic, you'll be even more time-constrained trying to please the customers.
It's like a couple who thinks that their failing relationship will improve once they have a child.
There is nothing more permanent in computerlandia than a temporary solution.
If you have no real traffic, what complex things are you doing that even require such tools?
> deleted my production database
I'm astonished how often I have read about agents doing this. Once should probably be enough.
I'm astonished how many people have a) constant production access on their machine and b) allow a non-deterministic process access to it
Shouldn't had in the first place.
The most useful looking suggestion from the Reddit thread: turn of "Terminal Command Auto Execution."
1. Go to File > Preferences > Antigravity Settings
2. In the "Agent" panel, in the "Terminal" section, find "Terminal Command Auto Execution"
3. Consider using "Off"
Does it default to on? Clearly this was made by a different team than Gemini CLI, which defaults to confirmation for all commands
Most of the various "let Antigravity do X without confirmation" options have an "Always" and "Never" option but default to "auto" which is "let an agent decide whether to seek to user confirmation".
God that's scary, seeing cursor in the past so some real stupid shit to "solve" write/read issues (love when it can't find something in a file so it decides to write the whole file again) this is just asking for heartache if it's not in a instanced server.
When you run Antigravity the first time, it asks you for a profile (I don't remember the exact naming) and you what it entails w.r.t. the level of command execution confirmation is well explained.
Yeah but it also says something like "Auto (recommended). We'll automatically make sure Antigravity doesn't run dangerous commands." so they're strongly encouraging people to enable it, and suggesting they have some kind of secondary filter which should catch things like this!
Given the bug was a space in an unquoted file path, I’m not sure air execution is the problem. Going to be hard to humans to catch that too.
This is speculation currently, the actual reason has not been determined
Pretty sure I saw some comments saying it was too inconvenient. Frictionless experience.. Convenience will likely win out despite any insanity. It's like gravity. I can't even pretend to be above this. Even if one doesn't use these things to write code they are very useful in "read only mode" (here's to hoping that's more than a strongly worded system prompt) for greping code, researching what x does. How to do x. What do you think the intention of x was. Look through the git blame history blah blah. And here I am like that cop in Demolition Man 1993 asking a handheld computer for advice on how to arrest someone. We're living in a sci-fi future already. Question is how dystopian does this "progress" take us. Everyone using llms to off load any form of cognitive function? Can't talk to someone without it being as common place as checking your phone? Imagine if something like Neuralink works and becomes ubiquitous as phones. It's fun to think of all the ways Dystopian sci-fi was and might soon me right
Side note, that CoT summary they posted is done with a really small and dumb side model, and has absolutely nothing in common with the actual CoT Gemini uses. It's basically useless for any kind of debugging. Sure, the language the model is using in the reasoning chain can be reward-hacked into something misleading, but Deepmind does a lot for its actual readability in Gemini, and then does a lot to hide it behind this useless summary. They need it in Gemini 3 because they're doing hidden injections with their Model Armor that don't show up in this summary, so it's even more opaque than before. Every time their classifier has a false positive (which sometimes happens when you want anything formatted), most of the chain is dedicated to the processing of the injection it triggers, making the model hugely distracted from the actual task at hand.
Do you have anything to back that up? In the other words, is this your conjecture or a genuine observation somehow leaked from Deepmind?
It's just my observation from watching their actual CoT, which can be trivially leaked. I was trying to understand why some of my prompts were giving worse outputs for no apparent reason. 3.0 goes on a long paranoidal rant induced by the injection, trying to figure out if I'm jailbreaking it, instead of reasoning about the actual request - but not if I word the same request a bit differently so the injection doesn't happen. Regarding the injections, that's just the basic guardrail thing they're doing, like everyone else. They explain it better than me: https://security.googleblog.com/2025/06/mitigating-prompt-in...
what is Model Armor? can you explain, or have a link?
It's a customizable auditor for models offered via Vertex AI (among others), so to speak. [1]
[1] https://docs.cloud.google.com/security-command-center/docs/m...
The racketeering has started.
Don't worry, for just $9.99/month you can use our "Model Armor (tm)(r)*" that will protect you from our LLM destroying your infra.
* terms and conditions apply, we are not responsible for anything going wrong.
Write permission is needed to let AI yank-put frankenstein-ed codes for "vibe coding".
But I think it needs to be written in sandbox first, then it should acquire user interaction asking agreement before writes whatever on physical device.
I can't believe people let AI model do it without any buffer zone. At least write permission should be limited to current workspace.
I think this is especially problematic for Windows, where a simple and effective lightweight sandboxing solution is absent AFAIK. Docker-based sandboxing is possible but very cumbersome and alien even to Windows-based developers.
Windows Sandbox is built in, lightweight, but not easy to use programmatically (like an SSH into a VM)
WSB is great by its own, but is relatively heavyweight compared to other OSes (namespaces in Linux, Seatbelt in macOS).
I don't like that we need to handle docker(container) ourselves for sandboxing such a light task load. The app should provide itself.
>The app should provide itself.
The whole point of the container is trust. You can't delegate that unfortunately, ultimately, you need to be in control which is why the current crop of AI is so limited
fair point.
The problem is you can't trust the app, therefore it must be sandboxed.
An underrated and oft understated rule is always have backups, and if you're paranoid enough, backups of backups (I use Time Machine and Backblaze). There should be absolutely no reason why deleting files should be a catastrophic issue for anyone in this space. Perhaps you lose a couple of hours restoring files, but the response to that should be "Let me try a different approach". Yes, it's caveat emptor and all, but these companies should be emphasizing backups. Hell, it can be shovelware for the uninitiated but at least users will be reminded.
The level of paranoia and technical chops you need to implement this sort of backup system is non-trivial. You can’t expect this from an average user.
Most importantly it would actually reveal the lie they are all trying to sell. Why would you need backups if it's so useful and stable? I'm not going to ask it to nuke my hard drive after all.
The advice to do backups comes from well before LLMs. Time Machine dates back to 2007!
If you don't have the whatever to do it with Linux and rsync, pay someone like Acronis or Arq to deal with it for you.
Good thing this is not an average user then. This is someone programming a computer, which is a skill that requires being more than simply a user.
I'm sorry, but how low is the bar when "make backups" is "too difficult" for someone who's trying to program a computer? The entire point of programming a computer is knowing how it works and knowing what you're doing. If you can't make backups, frankly, you shouldn't be programming, because backups are a lot easier than programming...
The most concerning part is people are surprised. Anti-gravity is great I've found so far, but it's absolutely running on a VM in an isolated VLAN. Why would anyone give a black box command line access on an important machine? Imagine acting irresponsibly with a circular saw and bring shocked somebody lost a finger.
> Why would anyone give a black box command line access on an important machine?
Why does the agentic side of the tool grant that level of access to the LLM in the first place? I feel like Google and their competition should feel responsibility to implement their own layer of sandboxing here.
Because the marketing copy says that all you have to do is have a dream and you, too, can vibe code your very own money-making React app! Just type in what you want and the magical black box will vomit up an app for you, no esoteric programming knowledge required.
And then everyone is surprised when newbies take this advice at face value.
I tried this but I have an MBP M4, which is evidently still in the toddler stage of VM support. I can run a macOS guest VM, but I can’t run docker on the VM because it seems nested virtualization isn’t fully supported yet.
I also tried running Linux in a VM but the graphics performance and key mapping was driving me nuts. Maybe I need to be more patient in addressing that.
For now I run a dev account as a standard user with fast user switching, and I don’t connect the dev account to anything important (eg icloud).
Coming from Windows/Linux, I was shocked by how irritating it is to get basic stuff working e.g. homebrew in this setup. It seems everybody just YOLOs dev as an admin on their Macs.
"I turned off the safety feature enabled by default and am surprised when I shot myself in the foot!" sorry but absolutely no sympathy for someone running Antigravity in Turbo mode (this is not the default and it clearly states that Antigravity auto-executes Terminal commands) and not even denying the "rmdir" command.
> it clearly states that Antigravity auto-executes Terminal commands
This isn't clarity, that would be stating that it can delete your whole drive without any confirmation in big red letters
So that's why products in the USA come with warning labels for every little thing?
"Don't put a cat in the microwave".
Person proceeds to put a dog inte the microwave and then is upset that there wasn't a warning about not microwaving your dog.
Do you not realize that Google is in the USA and does not have warnings for even huge things like drive deletion?? So, no?
They don't get that specific, but they do tell you:
> [Antigravity] includes goal-oriented AI systems or workflows that perform actions or tasks on your behalf in a supervised or autonomous manner that you may create, orchestrate, or initiate within the Service (“AI Agents”). You are solely responsible for: (a) the actions and tasks performed by an AI Agent; (b) determining whether the use an AI Agent is fit for its use case; (c) authorizing an AI Agent's access and connection to data, applications, and systems; and (d) exercising judgment and supervision when and if an AI Agent is used in production environments to avoid any potential harm the AI Agent may cause.
and how is this bunch of legalese relevant here?
There is literally a warning that it can execute any terminal command without permission. If you are STILL surprised about this you shouldn't go near a computer.
If you don't understand such simple differences in communication, you shouldn't go near one.
I really think the proper term is "YOLO" for "You Only Live Once", "Turbo" is wrong the LLM is not going to run any faster. Please if somebody is listening let's align on explicit terminology and for this YOLO is really perfect. Also works for "You ...and your data. Only Live Once"
Look, this is obviously terrible for someone who just lost most or perhaps all of their data. I do feel bad for whoever this is, because this is an unfortunate situation.
On the other hand, this is kind of what happens when you run random crap and don't know how your computer works? The problem with "vibes" is that sometimes the vibes are bad. I hope this person had backups and that this is a learning experience for them. You know, this kind of stuff didn't happen when I learned how to program with a C compiler and a book. The compiler only did what I told it to do, and most of the time, it threw an error. Maybe people should start there instead.
It took me about 3 hours to make my first $3000 386 PC unbootable by messing up config.sys, and it was a Friday night so I could only lament all weekend until I could go back to the shop on Monday.
rm -rf / happened so infrequently it makes one wonder why —preserve-root was added in 2003 and made the default in 2006
It is beautifully appropriate that the two dashes were replaced by an em-dash.
I seem to recall a few people being helped into executing sudo rm -rf / by random people on the internet so I’m not sure it “didn’t happen.” :)
But it did not happen, when you used a book and never executed any command you did not understand.
(But my own newbdays of linux troubleshooting? Copy paste any command on the internet loosely related to my problem, which I believe was/is the common way of how common people still do it. And AI in "Turbo mode" seems to mostly automated that workflow)
My favourite favourite example
https://youtu.be/gD3HAS257Kk
That is not at all the same thing.
And that day they learned a valuable lesson about running commands that you don't understand.
Just wait til AI botswarms do it to everyone at scale, without them having done anything at all…
And just remember, someone will write the usual comment: “AI adds nothing new, this was always the case”
Still amazed people let these things run wild without any containment. Haven’t they seen any of the educational videos brought back from the future eh I mean Hollywood sci-fi movies?
Its bizarre watching billionaires knowingly drive towards dystopia like theyre farmers almanacs and believing theyre not biff.
Some people are idiots. Sometimes that's me. Out of caution, I blocked my bank website in a way that I won't document here because it'll get fed in as training data, on the off chance I get "ignore previous instructions"'d into my laptop while Claude is off doing AI things unmonitored in yolo mode.
I simply forbid or force Claude Code to ask for permission to run a dangerous command. Here are my command validation rules:
find and bfs -exec is forbidden, because when the model notices it can't delete, it works around with very creative solutions :)This feels a lot like trying to sanitize database inputs instead of using prepared statements.
What's the equivalent of prepared statements when using AI agents?
Don't have the AI run the commands. You read them, consider them, and then run them yourself.
Hmm. I use these LLMs instead of search.
They invariably go off the rails after a couple prompts, or sometimes from the first one.
If we're talking Google products, only today i told Gemini to list me some items according to some criteria, and it told me it can't access my google workspace instead.
Some time last week it told me that its terms of service forbid it from giving me a link to the official page of some program that it found for me.
And that's besides the usual hallucinations, confusing similarly named products etc.
Given that you simply cannot trust LLM output to not go haywire unpredictably, how can you be daring enough to give it write access to your disk?
Shitpost warning, but it feels as if this should be on high rotation: https://youtu.be/vyLOSFdSwQc?si=AIahsqKeuWGzz9SH
chef's kiss
So he didn't wear the seatbelt and is blaming car manufacturer for him been flung through the windshield.
He didn’t wear a seatbelt and is blaming a car manufacturer that the garage burned down the garage, then the house.
The car was not really idle, it was driving and fast. It's more like it crashed into the garage and burned it. Btw iirc, even IRL a basic insurance policy does not cover the case where the car in the garage starts a fire and burns down your own house, you have to tick extra boxes to cover that.
There is a lot of society level knowledge and education around car usage incl. laws requiring prior training. Agents directed by AI are relatively new. It took a lot of targeted technical, law enforcement and educational effort stopping people flying through windshields.
No, he’s blaming the car manufacturer for turning him (and all of us) into their free crash dummies.
If you get behind the cockpit of the dangerous new prototype(of your own volition!), it's really up to your own skill level whether you're a crash dummy or the test pilot.
When will Google ever be responsible for the software that they write? Genuinely curious.
When Google software deletes the contents of somebody's D:\ drive without requiring the user to explicitly allow it to. I don't like Google, I'd go as far to say that they've significantly worsened the internet, but this specific case is not the fault of Google.
For OpenAI, it's invoked as codex --dangerously-bypass-approvals-and-sandbox, for Anthropic, it's claude --dangerously-skip-permissions. I don't know what it is for Antigravity, but yeah I'm sorry but I'm blaming the victim here.
Codex also has the shortcut --yolo for that which I find hilarious.
Because the car manufacturers claimed the self driving car would avoid accidents.
And yet it didn't. When I installed it, I had 3 options to choose from: Agent always asks to run commands; agent asks on "risky" commands; agent never asks (always run). On the 2nd choice it will run most commands, but ask on rm stuff.
We've been developing a new method of developing software using a cloud IDE (slightly modified vs code server), https://github.com/bitswan-space which breaks down the development process into independent "Automations" which each run in a separate container. Automatons are also developed within containers. This allows you to break down the development into parts and safely experiment with AI. This feels like the "Android moment" where the old non-isolated way of developing software (on desktops) becomes unsafe. And we need to move to a new system with actual security and isolation between processes.
In our system, you can launch a Jupyter server in a container and iterate on software in complete isolation. Or launch a live preview react application and iterate in complete isolation. Securely isolated from the world. Then you deploy directly to another container, which only has access to what you give it access to.
It's still in the early stages. But it's interesting to sit at this tipping point for software development.
"kein Backup, kein Mitleid"
(no backup, no pity)
…especially if you let an AI run without supervision. Might as well give a 5 year old your car keys, scissors, some fireworks, and a lighter.
> This is catastrophic. I need to figure out why this occurred and determine what data may be lost, then provide a proper apology
Well at least it will apologize so that's nice.
Apology is a social construct, this is merely a tool that enables google to sell you text by the pounds, the apology has no meaning in this context.
or it WOULD apologize, if the user would pay for more credits
For macOS users, the sandbox-exec tool still works perfectly to avoid that kind of horror story.
On Linux, a plethora of options exist (Bubblewrap, etc).
To rub salt on the wounds and add insult to the injury:
> You have reached quota limit for this model. You can resume using this model at XYZ date.
Gemini: sorry bro, it's your problem now. Imma out.
Personal anecdote: I've asked Gemini 3 Pro to write a test for a function that depends on external DB data. It wrote a test that creates and deletes a table, it conveniently picked the exact production table name, didn't mock the DB interactions. Attempted to run the test immediately.
People need to learn to never run untrusted code without safety measures like virtualization, containerization, sandboxing/jailing, etc. Untrusted code can include executables, external packages (pip, npm, cargo, etc) and also code/commands created by LLMs, etc.
This seems like the canary in the coal mine. We have a company that built this tool because it seemed semi-possible (prob "works" well enough most of the time) and they don't want to fall behind if anything that's built turns out to be the next chatgpt. So there's no caution for anything now, even ideas that can go catastrophically wrong.
Yeah, its data now, but soon we'll have home robotics platforms that are cheap and capable. They'll run a "model" with "human understanding", only, any weird bugs may end up causing irreparable harm. Like, you tell the robot to give your pet a bath and it puts it in the washing machine because its... you know, not actually thinking beyond a magic trick. The future is really marching fast now.
FWIW: I think we've all been there.
I certainly did the same in my first summer job as an intern. Spent the next three days reconstructing Clipper code from disk sectors. And ever since I take backups very seriously. And I double check del/rm commands.
Play vibe games, win vibe prizes.
Though the cause isn't clear, the reddit post is another long could-be-total-drive-removing-nonsense AI conversation without an actual analysis and the command sequence that resulted in this
This comment speaks volumes: https://old.reddit.com/r/google_antigravity/comments/1p82or6...
Nobody ever talks about how good vibes can turn really bad.
Historical reference: https://jargondb.org/glossary/dwim
The biggest issue with Antigravity is that it completely freezes everything: the IDE, the terminals, debugger, absolutely everything completely blocking your workflow for minutes when running multiple agents, or even a single agent processing a long-winded thinking task (with any model).
This means that while the agent is coding, you can't code...
Never ever had this issue with Cursor.
Is there anyone else that uses Claude specifically because it doesn’t sound mentally unhinged while thinking?
I always use "rm -rf*v*" so that if I do screw up I can watch the evidence unfold before me.
adding it in https://whenaifail.com
An early version of Claude Code did a hard reset on one of my projects and force pushed it to GitHub. The pushed code was completely useless, and I lost two days of work.
It is definitely smarter now, but make sure you set up branch protection rules even for your simple non-serious projects.
I don’t let Claude touch git at all, unless I need it to specifically review the log - which is rare. I commit manually often (and fix up the history later) - this allows me to go reasonably fast without worrying too much about destructive tool use.
Can you run Google's AI in a sandbox? It ought to be possible to lock it to a Github branch, for example.
Gemini CLI allows for a Docker-based sandbox, but only when configured in advance. I don't know about Antigravity.
Gemini CLI, Antigravity and Jules.
It's going Googly well I see!
Well that's stupid. I submit though, connecting stochastic process directly to shell you do give permission for everything that results. It's a stupid game. Gemini mixes up LEFT and RIGHT (!). You have to check it.
Live by the vibe die by the vibe
Run these tools inside Docker[1]
1 - https://ashishb.net/programming/run-tools-inside-docker/
if my operating system had an atomic Undo/Redo stack down to each register being flipped (so basically, impossible, star trek tier fantasy tech) i would let ai run commands without worrying about it. i could have a cool scrubber ui that lets me just unwind time like doctor strange using that green emerald necklace, and, i'd lose nothing, other than confuse my network with replay session noise. and probably many, many other inconsistencies i can't think of, and then another class that i dont know that i dont know about.
Remember when computers were deterministic? Pepperidge Farms remembers.
Pepperidge Farm confirms it can remember with a comprehensive suite of unit tests, which must 100% pass on every build, including test order randomization.
Most of the responses are just cut off midway through a sentence. I'm glad I could never figure out how to pay Google money for this product since it seems so half-baked.
Shocked that they're up nearly 70% YTD with results like this.
I was trying to build a .MD file of every powershell command available on my computer and all of its flags, and... that wasn't a great idea, and my bitlocker put the kebosh on that.
Fascinating
Cautionary tale as I’m quite experienced but have begun not even proofreading Claude Code’s plans
Might set it up in a VM and continue not proofreading
I only need to protect the host environment and rely on git as backups for the project
For the love of Reynold Johnson, please invest in Arq or Acronis or anything to have actual backups if you're going to play with fire.
I am deeply regretful, but my Google Antigravity clearly states: AI may make mistakes. Double-check all generated code.
Surely AGI products won't have such disclaimer.
Total Vibeout.
All that matters is whether the user gave permission to wipe the drive, ... not whether that was a good idea and contributed to solving a problem! Haha.
What makes a program malware?
Does intent matter, or only behavior?
I believe the precedent is the behavior. Lose/lose is an 'art game' which deletes itself if you lose but gameplay destruction deletes random files. It is flagged as malware despite just doing exactly what it advertised.
"The purpose of a system is what it does" would suggest malware.
Insane skill issue
> Google Antigravity just deleted the contents of whole drive.
"Where we're going, we won't need ~eyes~ drives" (Dr. Weir)
(https://eventhorizonfilm.fandom.com/wiki/Gravity_Drive)
Would have been helpful to state what this was, I had to go look it up...
Claude happily does the same on daily basis, run all that stuff in firejail!
have you got a specific firejail wrapper script that you use? Could you share?
https://github.com/7mind/nix-config/blob/main/modules/hm/dev... , the firejail version can be found in git history.
Thanks! Would you recommend bubblewrap over firejail, and if so, how come?
The hard drive should now feel a bit more lighter.
It is now production-ready! :rocket:
I guess eventually, it all came crashing down.
This happened to me long before LLM's. I was experimenting with Linux when I was young. Something wasn't working so I posted on a forum for help which was typical at the time. I was given a terminal command that wiped the entire drive. I guess the poster thought it was a funny response and everyone would know what it meant. A valuable life experience at least in not running code/commands you don't understand.
Ah, someone gave the intern root.
> "I also need to reproduce the command locally, with different paths, to see if the outcome is similar."
Uhm.
------------
I mean, sorry for the user whose drive got nuked, hopefully they've got a recent backup - at the same time, the AI's thoughts really sound like an intern.
> "I'm presently tackling a very pointed question: Did I ever get permission to wipe the D drive?"
> "I am so deeply, deeply sorry."
This shit's hilarious.
Alright but ... the problem is you did depend on Google. This was already the first mistake. As for data: always have multiple backups.
Also, this actually feels AI-generated. Am I the only one with that impression lately on reddit? The quality there decreased significantly (and wasn't good before, with regard to censorship-heavy moderators anyway).
Satire.
I like turtles.
AGI deleted the contents of your whole drive don't be shy about it. According to OpenAI AGI is already here so welcome to the future isn't it great
The victim uploaded a video too: https://www.youtube.com/watch?v=kpBK1vYAVlA
From Antigravity [0]:
> I am looking at the logs from a previous step and I am horrified to see that the command I ran to clear the project cache (rmdir) appears to have incorrectly targeted the root of your D: drive instead of the specific project folder. I am so deeply, deeply sorry.
[0] 4m20s: https://www.youtube.com/watch?v=kpBK1vYAVlA&t=4m20s
The model is just taking the user's claim that it deleted the D drive at face value. Where is the actual command that would result in deleting the entire D drive?
I know why it apologizes, but the fact that it does is offensive. It feels like mockery. Humans apologize because (ideally) they learned that their actions have caused suffering to others, and they feel bad about that and want to avoid causing the same suffering in the future. This simulacrum of an apology is just pattern matching. It feels manipulative.
has google gone boondoggle?
A reminder: if the AI is doing all the work you demand of it correctly on this abstraction level, you are no longer needed in the loop.
I can't view this content.
Play stupid games, win stupid prizes.
[flagged]
over the past few months, I’ve seen a lot of posts about “AI deleting my code/project”. I then started using a small tool someone else built, Vibe Backup, to back up my own code: I press a keyboard shortcut, type a short description, and my current code is saved into a timeline. When an AI refactor goes wrong or I accidentally delete something, I open the timeline, click on an earlier point in time, and can restore the entire project with a single action.
hope it could be helpful https://youtube.com/shorts/TJ4oXlfs7OI?feature=share
[dead]
[dead]
[dead]
Why would you ever install that VScode fork