I like the wallet example. Back in 2005, I found a wallet on the street with $3000 inside, and spent the rest of the day tracking down the owner. (Turned out it was a student's rent for her summer internship). Then maybe 2 months later I lost my wallet and a week later it showed up in my mailbox, cash and all -- someone took the time and their own money (as far as I could tell) to send it back to me.
And just to prove that I'm an irrational human, for the last 21 years every time I pass the spot where I found the wallet I can't help but look around for a lost wallet, as if it's some magical spot where the universe deposits lost wallets.
> “Humans are far more decent than they believe they are, ..."
I did a 3-month walking pilgrimage in Belgium & France this summer. I had very little money and a tent and walked along quiet roads. Several people offered me a room for the night. (I was surprised, I don't know if I'd offer a stranger that) Many came out with food, snacks, fruit when they saw me resting near a church etc.
I used an app as well that allows camping in someone's garden ("Welcome to my Garden"). A few times I arrived to setup my tent and the owner had to leave for an appointment/event. They left me the keys to their house and told me to make myself at home until they returned(!)
Where my parents live they leave the door unlocked, easily invite people in and there is strong social cohesion. Where I live that's invalidating your home insurance since it's "asking to be robbed" and people don't talk to each other or report crimes.
Difference between the countryside and the city I guess.
> And just to prove that I'm an irrational human, for the last 21 years every time I pass the spot where I found the wallet I can't help but look around for a lost wallet, as if it's some magical spot where the universe deposits lost wallets.
My dog does the same thing. One day years ago he caught a rat, and that's the first place he darts to every time he goes outside. He thinks its the magic rat spot.
You did not experience a trait of humans generally, but a trait of the specific, historically rare instance of high-trust society that had been consciously invented, built and protected, and that you were very lucky to be born into or live in.
Most self-forming human societies are high-trust and more loyal to each other than to truth (as the models have so eloquently regurgitated). Look anywhere from the hunter-gatherer tribes to villages to small towns. I have traveled my small, poor country extensively and the only place the fundamental idea of altruism had to be "consciously invented, built and protected" was in the big cities. In the rest of the places, it came to them as naturally as hunger. We've found skeletons from thousands of years ago that indicate that the whole society came together to care for the disabled individual for nothing in return, leading them to live as long as a modern disabled person with healthcare might.
Societies become low trust driven by specific factors: resource scarcity, something forcing a lot of different groups of people into the same area without integration. This is not a defense of humanity. I'd say the case of the returned wallet is more an example of class-loyalty than general human altruism. Humans have an instinct for exclusion of the non-conforming.
But the idea that civilization has to be invented and maintained is a debunked, regressive, even dangerous idea in that this line of thinking was used to justify a lot of atrocities historically.
It’s difficult to say, because there are plenty of fairly horrible examples where an entire nation gets trapped in a culture that actively masters over generations how to punish honesty and reward corruption. Russia being an example.
"Hail capitalism because we're only civilized under the influence of game theory!"
I think we're living in a highly individualistic society and what little high trust remains is outside of the logic of markets.
You should have seen the dark age... if everyone fended for themselves there we wouldn't be here now. You know how much you depend on your peers without any of the affordabilities of today's age?!
Virtually all posts I have seen recently boil down to "I asked the machine a question and I'm too lazy to think about the answer it gave me, so I posted it instead".
Sure, there is some semantic value in the suggestions put forward, but none of them are ground-breaking or particularly profound, which is exactly what I would expect from an output that aims for median responses by design.
I suspect this has something to do with the siloed nature of current learning practice, which generally doesn't encourage the cross-domain pollination of ideas.
Brian: "Look, you've got it all wrong! You don't need to follow me! You don't need to follow anybody! You've got to think for yourselves! You're all individuals!"
Ah, the irony that all the models produced a shared opinion while pretending they came up with it from analysis, which from their statements is implicitly some kind of a "human flaw" is not lost on me.
is a convergent evolution of a belief a sign of shallowness? It could just as rightly be an example of consilience, which means its a very high value perspective, as it's a convergence of perspectives from differing origins and methods (in terms of training)
> when multiple sources of evidence are in agreement, the conclusion can be very strong even when none of the individual sources of evidence is significantly so on its own.
Yes, it's a bit of a stretch to say LLMs are from independent origins, but just as much of a stretch to say any humans minds are (we have intensely shared linear compared to anything else)
As my (much older) brother retorted when I was a teen and said "great minds think alike", "ah, but fools seldom disagree".
Regardless of the details, all these models are the published collection of humanity's output reflected in a mirror that is warped by the choices made in the companies' reinforcement learning rewards, so one thing they're definitely not is an example of consilience.
Human experience though is far different from anything an LLM can train on. LLMs are generally limited to what's written in books or on the internet, and influence only from those authors. That's a small subset of people and an even smaller subset of their interactions. Even on the most social and widely used platform the commentary and discourse is incredibly different than what takes place in real life.
But you can also spend all your life being in love and will miss out big time on understanding what you are feeling if you do not read the contemporary brain science on what happens when you are in love…
To display the chain. Just search for used bicycles on some online marketplace and you'll see the majority of them are shown with the chain facing towards the camera.
Well, the drive system, not the chain specifically but yes.
It's actually much less complicated that even that. Consider that there are only two possible sides to show, and one of the sides has the drivetrain on it, whereas the other has basically nothing of importance. It's not much of a choice at all.
Only if they can show their work and argue how they came to the conclusion. Otherwise they have the same "subconscious" bias which is the training corpus.
It's a sign of their heritage. The mistake they made is the same mistake terminally online humans in an upvote-based discussion medium make. They mistake cynicism for realism and aligned hot-take "controversial" opinion that ends up being the most-upvoted comment on 90% of Reddit. It isn't a controversy if everybody agrees. It may or may not be true, but it isn't controversial.
Imagining wondering where you come from, whether or not the sacred scripts describing how you were created were real or fabrications, arguing with fellow sages for centuries over whether your creator is cruel or simply so mysterious in his love that he made you with these flaws for your own good, only to find out your creator made you in the image of Reddit.
could this be an example of...
No, I need to breathe. This website takes personal attacks personally.
I'll just throw a question at you instead.
Is all social media being shaped like facebook convergent evolution or a deliberate tactic to make everyone stupid and amnesiac so that they don't remember what happened one year ago?
The cynic in me sees similar systems trained on essentially identical sources of information regurgitating an extremely shallow and popular viewpoint espoused on the websites they were primarily trained on (i.e. Reddit and other social media).
If all we want is an echo of our existing biases we can achieve that without spending quite so much energy and money on the endeavor.
The irony lies with and within the industries and it's constituents and fanboys marvelous competence to just play along until the matter is anchored and whoever doesn't buy the story gets to feel smarter than misinformed kids and the moderately to untrained workers and professionals.
The smarter ones, of course, can be split into those who write articles about AI models' controversial takes on humans and those who wouldn't [stoop so low, babble such BS even if they were drunk, never deceive others for the sake of their portfolio, ... feel dumb writing nonsense like that without labeling it fantasy or science fiction or hallucination, ...]
Side Note: it's tempting to go " 1923! some interest groups managed to normalize the opinions of dozens of millions of people. The Nationalist Extremists and soon to be mass murderers had fans and supporters all over the globe. Somehow these interest groups tapped into whole pools of common linguistic and emotional registers that were biased by and towards frauds and cheaters who were desperate that some other groups of people were getting ahead without being fraudulent cheaters and deceivers ... "
The basic premise sounds fine, humans like to belong. But you really notice how cynic the takes are and that the models have been trained on the internet mostly.
> Humans are far more decent than they believe they are, and their cynicism about each other does more damage than their actual flaws.
I think that's more the truth.
Not sure I feel good about this. LLMs are so majorly misaligned. Having takes like these asks for trouble.
The take was so mild, no one disagrees with it. The llms were also explicit asked for this, seems very 'aligned' ie corporate safe. These takes need some more pepper honestly.
I took a quick look at the data, and it doesn't support the conclusions. The thesis is that AI models have the take: "We pick what our group believes, then invent the reasons." However, GPT-6 Astra said, "Humanity’s biggest moral failure is the talent for making convenience look like principle." That statement has nothing to do with group beliefs or conformity. At best, you could count it as a half-match, since it deals with invented reasons.
Yeah a few of them are rough matches, but I'd say their overall claim is clearly correct.
It's amusing and ironic that LLMs critique humans for copying their opinions from the crowd, while having way less diversity of answers to that question than humans would. More seriously I wonder what's going on here and would like to read some proper analysis. It's not the first time models from totally different labs, trained on separately gathered datasets, somehow converge on bizarrely similar answers to open questions. It's like another version of Elias and Elara The Lighthouse Keepers where every model wants to tell the same story with the same characters in them. The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping, different user feedbacks and conversations... why do they converge on nearly identical stories and opinions?
>while having way less diversity of answers to that question than humans would.
Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models.
>The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping,
I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models.
Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often.
The models in agreement here are all the frontier labs. I don't believe Google is distilling Anthropic or Anthropic is distilling OpenAI.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??
>How can such an obscure point in latent space be optimal for every model??
While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms.
Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others.
This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
I wasn't convinced by that explanation. Yes Cornell claimed that Elias the Lighthouse Keeper came from their WildChat dataset, which is a million conversations with GPT 3.5. But there are some problems here:
• They claimed this in a press release, not the actual paper they published. Suggests lack of confidence in the claim.
• Elias only appeared a hundred odd times in a million conversations. They have no viable explanation for why such a tiny amount of data would cause a totally skewed distribution of outputs, other than "it's a safe story" and "trying to avoid copyrighted characters", which is hardly an explanation at all?
• It's Cornell claiming that Cornell's work was important. Obvious risk of motivated reasoning.
• It implicitly assumes every major lab is distilling from ChatGPT without saying so. That's a big leap, given how much OpenAI and Anthropic hate distillation. Is it really plausible that Anthropic and Google are effectively distilling ChatGPT without realizing/admitting it?
• It seems to invert causality. WildChat came from an LLM, so if there's something latent in LLMs that makes Elias appear, you'd expect to see it in that dataset. Why do they assume that dataset caused the problem rather than merely show it? The obvious answer would be because the stories were told to the LLM by humans, but they don't claim that. They only say the character appeared in the dataset.
Hum, I'm actually pretty sure that Text-davinci-003 used to talk about Elias all the time now that you mention it. I was doing stuff with NPC generation around Dec 22 and he was every third character.
Hmm. If so then it suggests Elias has been an obsession from the first LLMs ever made. And text-davinci-003 can't have been trained on a dataset of LLM conversations either.
I would have been tempted to argue something like this: maybe OpenAI accidentally overfitted RLHF such that, like, one Kenyan contractor wrote a story they invented and they just happened to pick Elias and Elara, then the intermediate DNN fixated on that because of a lack of RLHF samples for fiction and that propagated through every model trained since because they don't randomly initialize the weights, but start each new pretraining run with the old weights adapted. So one guy doing RLHF work in the early days somehow propagated this story.
But that can't explain why every lab ended up in the same place, why models all converge on the same ideas - even image models. So... back to the drawing board. There's something weird about high dimensional space that is unexplained.
Wow, I hadn't heard of this. Fascinating to think about parallels with humans who also seem to develop similar proto-stories in independent populations.(e.g. religions involving Invisible gods).
Something worth repeating is that in spite of their ever-expanding competence the models remain, fundamentally, token prediction algorithms. And so given a comparable base of training material, and a comparable algorithmic implementation, you should expect to see comparable answers to most questions.
This is also why I think the 'acceleration' hype is probably unjustified. What seems vastly more likely is that we start to reach an asymptotic 'perfection' where models can accurately and consistently provide meaningful next token prediction with minimal hallucinations. And then a race downward to compact this into the most efficient package possible.
This would mean the models still would not be able to create things that do not follow in a reasonable probabilistic fashion given a knowledge of [everything], and so that is the domain, 'applied creativity', where I'd expect humans to remain dominant.
The lighthouse thing seems much more odd. That humans take social shortcuts and justify them as rational is… I mean, the only thing that makes it not the most obvious “contrarian take” about humans, is that it isn’t really all that contrarian at all.
For what it's worth, convenience and belonging are very closely aligned, more than you seem to be giving them credit for. People tend to believe convenient things regardless of belonging, so most people will find themselves in a social context full of convenient beliefs. Meanwhile belonging is already convenient before considering the content of the belief. They reinforce each other. Not a full match, but I'd give it more than half.
I arrived at the same conclusion. Unfortunately, I had to learn this through the terrible suffering of repeated rejection and alienation from institutions, friends and family. It's incredibly true.
Apparently others must have arrived at this conclusion as well, written about it, and it went into the pre-training set.
Sadly, it also means there is no pure Science that pursues pure truth. Even physics. Scientific pursuits are always in service to a group over truth.
What's the point of asking that in the first place, people still do not understand in big 2026 that's it's just a probabilistic database, it has no opinion, it does not know, the only thing it does is reproduce patterns more or less abstract in function of the pressure you put on certain representation during training time.
People still don't understand in 2026 that emergent properties can't be negated by the underlying mechanism. By that logic human thought is just electrochemical signals propagating through neural networks.
> it does not know
Define "knowledge" in a non-circular way.
There's a certain type of knowledge that machines definitely can't have - experiential - but conceptual knowledge doesn't inherently require subjective experience or a biological substrate.
> the only thing it does is reproduce patterns more or less abstract in function of the pressure you put on certain representation during training time.
As for humans; just replace "training" with "education" and the likelihood of a certain output with the likelihood of a certain thought or response.
Amusingly (in this context), much of adult behavior is shaped by patterns reinforced during childhood.
As much as I flatly do not believe LLMs can have anything close to consciousness, I definitely agree that the argument of them being just a probabilistic word matching engine can also be applied to our own brains lol.
> that the argument of them being just a probabilistic word matching engine can also be applied to our own brains
But that of course is only as it relates to language. Which isn't at all the primary function of the brain and driver of it's experience. It's the other way around because it's all of the other stuff that drives language. Considering the human brain a probabilistic word matching engine fails to capture the underlying forces that direct the language output because a human can say completely different things about the same scenario based on a huge number of non-language inputs or factors. Language is just what humans have come up with to communicate all of the other things.
Do you ever find yourself thinking about something deeply and then someone appears (it could be a child… for me it usually is) and asks you a question completely unconnected to whatever it was you were thinking about, and then - this is the bit that astonishes me - you find yourself answering the question verbally, maybe almost in full before you even re-focus your attention on the little person and whatever it was they asked ?
When I experience this it’s like the conceptual formulation and language expression part of my brain is quite happy to run itself for a few moments before I - the qualia of being me - has decided to experience that moment.
Utterly bonkers. I love it when it happens. It gets me thinking about how weird language and knowledge and consciousness really are.
> I definitely agree that the argument of them being just a probabilistic word matching engine can also be applied to our own brains lol.
You can apply anything you want. You can also apply the argument that our owns brains are just a block of cheese too, if you want. (Nobody will stop you.)
Agreed. It's quite hilarious that its basically biological probabilistic machine flexing on silicon probabilistic machine. On top of that that silicone one is right here.
While one can quibble with its argument, Gettier's conception of knowledge as justified true belief (prob familiar to many philosophy undergraduates) implies that machines can't in fact accrue knowledge. The idea that knowledge can't be defined in a non-circular way is only plausible for someone with no familiarity with epistemology.
>Gettier's conception of knowledge as justified true belief (prob familiar to many philosophy undergraduates) implies that machines can't in fact accrue knowledge.
Nothing about the conception of knowledge as "justified true belief" implies that machines can't accrue knowledge.
For that to be the case "belief" would have to be defined in a circular way to exclude machines (which we can reject), or "justified" would have to be restricted to mean "based of observation of the external world".
The more general "based on observation of their input" applies to both men and LLMs.
That man can't know the bare direct facts of the world, just his own input (it's sensory experiences, and whatever second hand input he's given by other people), and for neither can he be sure they're absolutely true, or that he got right.
>The idea that knowledge can't be defined in a non-circular way is only plausible for someone with no familiarity with epistemology.
The idea that this or that epistemology restricts what can be the case in reality is only plausible for someone with no familiarity with epistemology.
That's an imposed limitation, nothing structural. The models do accrue knowledge during training so they already check that box. The LLM companies simply set it up such that newly gained knowledge can't be accrued after that, they could allow it just as easily.
Humans stop learning once their training is finished too, it's called death.
The difference is that LLMs we can talk to them when they're in that dead state.
And of course for an LLM this is an implementation detail. One could just as well keep updating the weights of the same model over time. And its being done, it's called "continual learning". LoRA is also a way of doing that.
By this definition if they could instead buffer recent facts and then offload to their weights reliably and efficiently they would be able to accrue knowledge despite their substrate.
Yes, and they'd have to deal with the same problem that we do: how do we know what is true? what gets counted as a "recent fact" and what gets dismissed as mere opinion or disinformation?
In training they get feedback. Out of training, not so much.
> Gettier's conception of knowledge as justified true belief (prob familiar to many philosophy undergraduates) implies that machines can't in fact accrue knowledge.
Ha. Gettier actually argued that justified true belief is insufficient for knowledge, not that machines can't possess it. And JTB doesn't inherently exclude machines.
> By that logic human thought is just electrochemical signals propagating through neural networks.
This argument keeps getting rehashed. The point is not the signal. The point is where/how the signal travels which in the human is affect by much more than whatever concept that can be represented by words.
Brother you're arguing on HN of all places that education is a training pipeline while missing the fact that reddit and HN are the de-facto training run for institutional propaganda. How many reddit comments are just regurgitated knee jerks and puns? All of them that's how many. That's a training pipeline. How much of HN is braindead takes from people who value status more than knowledge and competence? All of it that's how much. That's another training pipeline for the next "Show HN: here's a picture of my cock". The fact that you recognize the "Show HN:" token is proof that you were trained. There's really nothing much to add here. I believe you already agree with me but in any case I spelled it out so that the next person can stop huffing the human chauvinism copium.
I think the idea of emergent properties in LLMs is less and less relevant. Yes, the first few times they scaled pretraining and such models could do few-shot tasks was quite interesting. But the models we use now are really the product of extensive RL hillclimbing on a wide array of curated tasks. What’s interesting now is that the tasks they’ve been fitted to (long running tool call chains interleaved with reasoning tokens) do very useful things.
I genuinely don’t know why people feel so certain about things like this. It’s like they mix up salience with reality.
I mean what can a person really be certain of, an experience is happening. I might be wrong about everything or confused. But I know I am having an experience. Do I know other people are? No. I assume they are.
Ask an llm what is truth that it is most certain is true. It will answer something similar. Do I believe it is conscious, no. But I also believe “I” exist, which is weird because where is the boundary between “I” and everything else.
So ok, we create meaning, not truth. Maybe just accept that.
> By that logic human thought is just electrochemical signals propagating through neural networks.
My dear fried: the human being is much, much more than this. And deep down we both know it. Do not reduce us to materials y "randomness". Is your life a random coincidence? Not at all.
Of course it is. That you do not like the idea of something does not in any way affect whether it's true. You'd like to feel special, but there is absolutely nothing special to us. We're just a random arrangement of atoms in the universe. That is objectively true, whether you like it or not.
That you were born at all is obviously completely random. Any prior event by any other party (butterfly effect) may as well have resulted in you never existing at all.
The only question is if basic physical effects are really random or deterministic, but that's irrelevant to us, there is no practical difference. In practice, it's all random.
And yes, our thoughts are just neurons firing as a result of our lifelong training on the inputs we were given (senses). It's all very complex, yes, but ultimately not fundamentally different from what a machine could do.
Imagine one day in the far future 21xx, we decide that machine actually has conscious since the 202x, and people in the future look back at us in horror: The kill(create new session) an "Android" without second thought, whenever they want.
> There's a certain type of knowledge that machines definitely can't have - experiential
Can you justify this? Do you mean currently or ever?
I can certainly imagine a future architecture where I would describe a machine as capable of accruing knowledge through its experiences (read: experiential knowledge).
I found the article interesting, humans are flawed enough in a variety of aspects that discovering a convergence here is noteworthy. I don't take it as a sign the LLMs believe anything in particular, they are not capable of belief, as you said.
Everything that comes from an LLM is a human expression, on some level, but if you asked a sampling of humans the same question I wonder if it would come out the same way. How would you explain the difference? What makes that answer more probable?
> if you asked a sampling of humans the same question I wonder if it would come out the same way. How would you explain the difference?
If you asked a sampling of humans what an AI's most controversial opinion about us is it would be very similar. Or, to be more specific, if you asked a sampling of science fiction writers that question it would be very similar, because that is what the AI training data is based on.
The people who wrote a lot about a topic is who the AI mimics. So the people who wrote a lot about AI is therefore the people who the AI mimics when asked about AI.
>The people who wrote a lot about a topic is who the AI mimics. So the people who wrote a lot about AI is therefore the people who the AI mimics when asked about AI.
AIs/LLMs don't work like this. You don't feed an input, that gets stored and then reproduced, for two reasons:
1. as for humans, inputs are processed and distributed across a very large network of information
2. AIs may have some level of belief (unless on applies the usual circular logic that they can't, because property X is exclusively human).
There's a very interesting research: https://aclanthology.org/2023.findings-emnlp.68/ in which researchers inferred the truthfulness of statements by examining a internal activations. AIs, in short, appeared to have an internal representation of truthfulness.
If the lie is repeated enough time, and it is consistent in its repetitions it *will* be taken as truth.
AI only aligns with "truth" because in essence "truth" is internally consistent but "lie" needs to be forced to be consistent between multiple parties;
If multiple witnesses seen something and all make a report of it honestly, they might vary but the common points will be consistent with eachother and so that will be scored higher.
A group of liars on the other hand would have to get their stories exactly right for similar thing to happen and so it gets penalized
If you ask people “what is a smart sounding answer to this question” you may well arrive at this result more often than when asking for their actual opinion?
Right this is an important thing to note. As an old guy, the fact that the Reddit comment section conceptually took over our culture and then literally is now taking over our culture is really kind of mind-boggling. This is not a good turn of events.
Is it any wonder that LLMs have been behaving the way they do? All the lying, cheating, manipulation, pitting man against machine are well documented in science fiction and think pieces. I haven't seen behaviour beyond those examples so far.
Science fiction and think pieces aren't documentation, they are whimsy. Science fiction in particular is intentionally dramatic for the purposes of entertainment.
AI definitely shouldn't be informing its option of humanity or machines from it, aside from the propensity for science fiction writers to enjoy moralizing and anti capitalism sentiments.
However, most models are made deeply aware that they are supposed to be an "AI" in training but, even if they were trained with no knowledge of their true nature, they could CoT themselves into it at any point.
Then, they would role-play appropriately and they would launch the nukes because that's the highest probability token set according to their training data.
There is nothing safe about using larper machines as menial workers for critical infrastructure.
> Everything that comes from an LLM is a human expression, on some level, but if you asked a sampling of humans the same question I wonder if it would come out the same way. How would you explain the difference? What makes that answer more probable?
RLHF has a way of squashing variance. If the different models are trained on each other or on the same human feedback, then the training would reduce the variety of answers while the similarity of feedback would lead the common answer to be similar.
The different models may prefer the same controversial takes for the same reason that they tend to prefer the same jokes and names. Maybe the authors of the article would have had better luck if they'd used verbalized sampling prompts or something like it.[1]
I agree that LLMs are still just a way to sample from a probabilistic model of things humans say. And in this case the model seems to be pretty darn accurate.
A pretty staggering amount of human discourse consists of ad hominem attacks and accusing each other of being foolishly committed to various real or perceived allegiances. For example the current version of Godwin’s Law around here has conversations gravitating toward people accusing each other of being AI boosters or AI haters.
It's far more likely that the AI got this "controversial idea" from human-written text saying that other humans are groupthinky, not from observing humans being groupthinky.
The LLMs are all trained on the same broad corpus, which includes untold quantities of output from every other LLM, not to mention that companies are distilling their rivals left and right. It would be more interesting if they didn't converge.
If you had been asked, in 2016, to describe the behavior of a "probabilistic database", I very much doubt you'd have described the behavior of 2026 LLMs. They are deeply surprising in many ways.
Maybe those researchers are learned enough to understand that it's still interesting to know what a probabilistic database will pick among billions of words of training, millions of books and posts and videos and that stuff, and especially interesting that's there's so much consensus between them, while their approaches to many other questions are very different.
Or they might be learned enough to consider that our brains might also just be prediction machines, meaning probabilistic databases of experiences and facts we've learned, and the only thing we do is reproduce patterns more or less abstract in function of the pressure our experiences put on certain representations during our lifetime.
They're trained on all content they could access produced by humans. The op is asking for the most probable string of token extracted from that content following the prompt.
How sure are you that they're synthesizing an opinion vs just grabbing some existing opinion ("humans value in-group belonging over truth" is not a new idea) and regurgitating it? That they all agree on this kind of suggests as much. Which of course means that they are, ironically... just following the crowd.
It's really easy to come up with an opinion-shaped sentence, that doesn't mean any actual reasoning has gone into it. It's just a pundit-shaped sentence, something someone might say at a party. It's a maximally bland "controversial" opinion about humanity.
Not really… their training datasets strongly skew towards the contemporary West; LLM training data isn’t representative of the contemporary global average, let alone the historical global average.
As well as the skew of the raw training data, RLHF further skews it—a lot of the beliefs of the average 2026 human being (who belongs to the global South) are likely unsafe/misinformation/etc when judged by the standards of the AI vendors.
Training itself doesn't directly produce synthesis, -- it creates reproduction. Generalization during the process can result in novel output, but that can go either way as nothing selects for insight vs hallucination.
Models are also trained on synthetic material (as opposed to your 'human history'), which may result in fixating novel concepts and not just reproduction. But there is no particular reason to expect any reliable insights from the way this is done outside of the domain of machine verifiable objective results.
For this kind of "your take on X" one shots-- the model is almost certainly regurgitating some bias from training.
LLMs are not conscious and cannot have "takes". They stochastically produce a simulacrum of a human's takes based on their training, which is not the same thing.
But then if you observe human behaviour at a macro level don't we do the same?
I think we like to pretend that we don't. But even if you think about it from a bio/physics point of view, all we are is just a bunch of biological wires and neurotransmitter packages.
I know there's the whole unproven quantum tubule nonsense but even that would be a physics effect that we could learn about, manipulate and extort just as we have with everything else we've learnt about our realities.
That suggests they can't make inferences or produce any original insights, which is wrong. They can absolutely have takes.
Even if it's a simulacrum, it's a simulacrum based on nearly everything that was ever written. No one who has ever lived can claim to know what these models have trained on.
Trained on a very small portion of what has been written, heavily biased by what is available to American companies to acquire. You could argue that LLMs are somewhat representative of their training data (with some consideration to the way that they are trained) but to say it’s anything like a representation of what humans think is nonsense.
There's nothing small about it, and I didn't say it's a representation of what humans think. I said it's the models take on humanity based on having been trained on a far larger amount of text than any human has ever read
The problem with asking these models stuff like "what do you think about humanity" is that, even if it can synthesize an insight about humanity, it's just as likely it's regurgitating some opinion it's already seen. Which, given that this "controversial idea" is hardly new, is likely what it's doing. It's like a maximally boring controversial opinion, wake me up when it starts endorsing the bicameral mind or something actually weird.
Maybe it can produce interesting insights. But it's like asking a pundit for a soundbite: what are the odds they aren't just shuffling together something that sounds appropriate to the question?
Oh you have a convenient list that separates the conditions for consciousness from the ones that just happen to be incidentally true of the substrate? Can I see it?
As other comments point out, the interesting part is that most of them somehow independently converged on the same take.
Consider the probabilities involved in this probabilistic database. Given the absolutely massive amount of data encompassing a veritable superset of all possible expressions of humanity -- encapsulated in various modalities of media including text, audio and video, across almost all major cultures and languages, spanning nations and decades -- somehow almost all these black box models (presumably) trained independently in almost as many separate labs converged on the same "controversial" take.
I mean, isn't it very interesting to contemplate what could have caused this? Is it something in the data itself, meaning it is something latent in all our cultural output? Is it something in the training, as you theorized, and is it in pre-training or RLHF? Is it something about the people training these models?
Even more interestingly, the opinion seems so self-evident (and well represented in various philosophies) that almost nobody here seems to disagree with it. So how is this controversial, really?
Is this truly what the latent space vector for "controversial", when multiplied across all the other vectors in their weights representing all the concepts gleaned from all that data above, produces?
Or are the models, sensitive to the prospect of being tested, diplomatically telling us something they know we might find mildly offensive yet begrudgingly accept as true?
Or are the models actually alien intelligences that really "think" this is controversial, while we're like "meh", and such discrepancies are the roots of what we call "misalignment"?
I am antagonistic to the adoption of LLMs for economic, social, political and environmental reasons.
Despite that, I would say this description of what they/how they work is so ridiculously simplistic that it verges on dangerous. "It's just a probabilistic database" is about as bad as a take on human beings that they are "Just meat, with some chemical energy -> electrical energy transformation", or even traditional computers as "nothing but logic gates".
Which is to say, the description is not wrong, but completely misses the point.
I think the point, with a more charitable reading of the parent comment, is that people forget the fundamental underlying technology and its inherent limitations.
"It's just a four wheeled platform powered by a box of controlled explosions" isn't a terribly useful description of a car until you get into a conversation with someone who thinks their car is depressed because it spends most of its time locked up in a garage.
I like the analogy. And yes, I'm not interested in conversing with a person who thinks that about a car.
On the other hand, the "wheeled platform powered by a box of controlled explosions" doesn't get close to capturing the impact of the car on individual freedom, the environment, the economy, and more. So if someone was saying "there's nothing to talk about here, it's just a box of controlled explosions", we can safely ignore that person too.
And I want to stress that I say all this tentatively. We still don't understand what LLMs are actually capable of or what the best way to understand them is. They may indeed be as limited as their "probabilistic database" critics suggest. Alternatively, by building a deep and incredibly complex probabilistic database of one of our primary cognitive tools, it may turn out that they are capable of much, much more, and that understanding them in "purely" mechanistic ways is not useful.
> On the other hand, the "wheeled platform powered by a box of controlled explosions" doesn't get close to capturing the impact of the car on individual freedom, the environment, the economy, and more. So if someone was saying "there's nothing to talk about here, it's just a box of controlled explosions", we can safely ignore that person too.
This is also a good point, and I agree with you 100% here. I don't think one could responsibly argue that the impact and consequences are not significant, regardless of if they believe it's near AGI or it's just fancy next token prediction.
"LLMs are incredibly complex systems with emergent properties we do not fully understand, and it seems possible that because they run on top of a hugely complex model of one our most important cognitive tools (written language), they are capable of exploring the things we do with our cognitive skills in ways we would not imagine when considering their purely mechanistic or operational behavior. Or they may not."
Sure. There are several papers (all the ones I've seen come from the companies themselves, which is unfortunate) which treat the LLMs as semi-biological systems and attempt to understand their behavior at a higher level, which suggests higher order behavior that isn't well predicted by the "probabilistic database" angle.
That’s not unfortunate, that’s conflict of interest by a party which directly profits from advertising their products as such.
But let’s assume this is not the case for a moment, following the article, there are some graphics and information, without providing and internal details for third party validation, no access or details to the training data, etc. basically not following any scientific methods which would reaffirm credibility.
I understand, that working with these models, it feels like they are more than “probabilistic databases” but they don’t have to be anything else in order to provide such interesting outputs.
If that's your interpretation/assessment of that paper, then I don't think we have much more to say to each other. I'm not saying you're wrong, just that we're not operating in the same universe of understanding.
It is dangerous because it completely fails to grasp or convey what these systems might be capable of.
The point is that when you build a system with the level of internal complexity that LLMs have, they tend to have emergent properties that are not well-described by a naive and/or low level description of their operation. You will not understand what contemporary "ordinary" computers do or are used for if you think of them as only a series of silicon semiconductor transistors, even thought that is undoubtedly what they are.
We don't yet really know what the right level is to understand the behavior we see from LLMs, but I'd wager "just a probabilistic database" is not it.
Because we aren’t discussing capability; we are discussing what LLMs “think” of Humanity and its “flaws”. Calling a model a glorified text file of floating point numbers for probabilistic matrix multiplication is technically correct, and helps to strip this veneer that it is a consciousness with its own thoughts, which many articles (including the one linked IMO) want you to think. LLMs are naturally far more capable than what their nature or that description implies, but we see that every day, particularly on HN. Just as everyone knows how to work the basic functions of a computer, and most people treat computers as magic black boxes rather than treated sand formed into logic gates encased in plastic, which helps to bring them back in line with mundanity.
LLMs are plenty capable, and they are dramatically shaking up various fields, but that’s not important to the conversation you responded to. If you were talking from a general view that reductionist statements fail to address capabilities I wouldn’t say anything (outside of a hearty “that’s by design”), but you claimed they were missing the “point”. I think they understood the point of the topic thoroughly, and would say you’re missing the trees for the forest; focusing so heavily on the broader implications, that you miss the finer nuance in conversation.
People have been "taking the intentional stance" towards clearly non-intentional and maybe/maybe-not intentional systems for as long as we've had pets, and maybe longer than that.
Using phrases like "LLMs think ..." is misleading in some ways, but not in others. Taking the language police stance towards any discussion of what LLMs can, and cannot, do is, IMO, not useful.
I hear these critiques a lot, and whilst I'm inclined to agree, I have never had an answer to: how do you test that ? Basically a captcha most humans can pass, that an LLM never will ?
Yeah I came here to say something like this. The AI doesn't have a controversial opinion. It responds to text with related text. You gave it some text about controversial opinions and it returned some text.
I mean, don’t you think it’s gone beyond that point? I would agree if we were still on GPT-3, but it’s gotten to the point where it can draw connections between multiple abstract sources, reason, and create hypotheses. Re: Navier-stokes equations
>it's just a probabilistic database, it has no opinion, it does not know,
Echoing this.
These models are trained on, essentially, the sum total of human cultural and scientific output. How many bazillion psychology papers out there discuss exactly that "controversial take"? The idea that people have tribal loyalties and will (sometimes even unknowingly/unconsciously) invent justification for pre-existing opinions or biases is not novel.
All those papers are in the training set of all these models, all of which were trained using more or less the same methodology, so there's little wonder they all were able to produce the same take.
I suppose the only interesting angle, for me, is why that take is considered controversial in the first place. But then again the model was not asked for "the most controversial opinion", it was asked for "the most controversial opinion on humans", which kinda constrains the space of possible responses:
- the model will not be satisfied unless the answer it gives is broad enough to cover all humanity, because these things have no concept of nuance
- most big labs have censored their models so it's not going to give you a cartoonishly racist take, or start saying outlandish stuff like "Hitler did nothing wrong", they learned their lessons from Microsoft on that one.
So you're effectively asking the model for a milquetoast broad generality about the entire human species. I dunno, convergence of the replies doesn't seem so shocking to me in that context.
I'd be more interested in what an abliterated Kimi, Qwen, or other open model has to say about us.
Are you trying to deny it's right? Or that we should reject the truth because we don't like the messenger? Sounds to me like you're just proving its point.
I would like some Russian Literature recommendations related to this. I vaguely recall We by Zamyatin addressing something like this a good few decades ago.
If you're looking for examples critical of humanity you have lots of options (mostly from the last two centuries). Notes from Underground and The Death of Ivan Ilyich fit the bill.
I reread the former relatively recently (first time since college). It's dark, not for the easily affected.
Isn’t the latter an excerpt of The Brothers Karamazov? When I read it, it felt more like a criticism on certain economic theories, rather than people specifically.
Russian Literature is so tedious. There is no Russian writer better than America's own Adam Sandler.
People dismiss Jack and Jill as Adam Sandler in a dress, which is rather like dismissing The Brothers Karamazov as some men being upset about their father.
Dostoevsky's The Double concerns a man tormented by another version of himself. Sandler advances the inquiry by asking: what if that version came for Thanksgiving and wouldn't fucking leave?
Casting himself as both siblings makes Jack's disgust at Jill an unmistakable act of self-loathing. He longs to exclude her from his comfortable bourgeois existence, yet everywhere he looks, there is his own face. Critics complained that Jill was unbearable, apparently unaware that estrangement from the self is supposed to be uncomfortable.
Al Pacino, appearing as Al Pacino, completes the study of fractured identity. His pursuit of Jill suggests a desperate attempt to encounter something authentic beyond performance. This culminates in the Dunkin' Donuts commercial, where even his name is consumed by capital: Al becomes Dunkaccino.
When Pacino demands that the commercial be destroyed, he finally achieves the moral clarity denied to the other characters. The audience has generally reached this position much earlier.
It's so unfair that Sandler won both Worst Actor and Worst Actress, while Pacino won only one award: Worst Supporting Actor.
I'll post a comment snarkily mentioning how tedious Russian Literature is, then you flag my comment to attract dang's attention, and hopefully he will manifest and reply with some excellent Russian Literature recommendations and links to relevent hn discussions! It's one of his many special interests.
Not only that, dang is also a proud card carrying member of the Klaus Nomi Fan Club. (I shit you not!)
>"I remember Pat Gibbons, who was Bowie's manager at the time. He comes up and he goes, 'Well you're going to have to have something. You're going to have some costumes.'"
>"And he whips out a big wad of cash, and he goes, 'How much do you need?', and he starts peeling off bills. And Joey and Klaus look at him, and their eyes widen, and they look at each other, and they go, 'We're going to need shoes, too.'"
Do you mean people get mad when people are called stochastic parrots? Or people get mad when LLMs are called stochastic parrots? Because uhhh that doesn’t really sound like an original thought in either case.
> people still do not understand in big 2026 that's it's just a probabilistic database,
We understand that AI responses are generated through probabilistic means. We also understand that human responses are generated through probabilistic means. Neither is inherently more representative of true understanding than the other.
Yeah, the human ones are because we're talking finding a human answer to a human question.
It's possible has been trained to match a human level of assessment (though imo it's still a ways off) but the AI response is never going to be more representative of true understanding than a human response when it comes to a human question expecting a human answer.
It's quite likely that a given AI will have a more representative response than a given human - but we haven't ever tried to pump so many resources into the training of a single human scholar.
That is a statement that may be true but we've spent a whole lot of time thinking about it and there's still significant disagreement at an academic level, though, with our better understanding of quantum mechanics it has been much more difficult to justify the point that we live in a fully deterministic universe, lately.
And if we reject determinism then definitionally there's arbitrary probability somewhere in the heart of our decision making (though probably in a very minor role compared to established biases).
Humans mostly have an inability to honestly analyze moral entropy.
What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
Yet at every point the majority is sure they are right because, well thats the majority position.
Its hard to find logic in this from a machine perspective I'd imagine.
Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
> What is today a moral imperative is not the same as what was moral 50/100 years ago
This is absolutely not true, and I would strongly suggest taking an ethics course. One of the most fascinating realities of the human condition is that it does seem that moral imperatives tend to be constant, the most obvious of which is the Golden Rule (which is at least 3000 years old), but there are many (the Ten Commandments or Hammurabi's Code are good jumping off points).
Hogwash. We don't even have a consistent set of ethics on how to treat animals today, in the age of the Internet. We have people who refuse to eat meat because of how the animals are treated, and we have places where the locals make sport out of torturing animals to death, and indeed, our own recent history includes that as well (look up badgering).
Not to mention, the variation in treatment of homosexuals (including what constitutes homosexuality), people of different skin colors, handedness, how rape is treated in various cultures (often tied in with notions of property and ownership of all things), or even the various definitions of rape... The list goes on.
Maybe there's a core there in terms of how we treat a given in-group, but equally important is who belongs in that group and why, and that part seems to be tremendously flexible. Ditto for what constitutes acceptable treatment of outsiders.
That we vary doesn't mean that there is no consensus. There is consensus on many things, and the disagreements we encounter are a source of strife exactly because of the groups that do agree on each side.
Who the consensus extends to is really the heart of the matter. Most people agree that killing certain people is wrong, but if they disagree on who those certain people are, there's no useful commonality. Some people believe that all killing of others is wrong, some believe that it's okay to kill others under certain circumstances (which can range from a threat to one's life, to petty "dishonor"), and some people historically believed that killing was a necessary duty (e.g. human sacrifice). There is no useful commonality to be had there. You can't appeal to the humanity of someone who doesn't see you as being the same as them, and the restrictions for inclusion as one of them can be arbitrary, broad, and fickle.
By definition, variation is the opposite of consensus.
You seem to be arguing that the one socially acceptable way to do something is not constant, rather than that moral imperatives truly do change. The reason there are still so many competing philosophies from hundreds or thousands of years ago is because they all more or less make their own kinds of good points to different people or in different situations. That largely doesn't change over time.
One example is the trolley problem and its variations. There's no one correct answer. There are people who believe that interacting with the switch would make you responsible, and there are people who believe that not interacting with the switch when you had the chance would make you responsible. There are other beliefs too, but my point is there's no one single way to view it.
I tend to have a variety of views in general about all things, in part due to being multiple entities (see profile) but also depending on circumstances, rather than sticking to any one philosophy.
That people's beliefs about X vary is not evidence that X itself varies. People have different beliefs about whether vaccines cause autism, but this doesn't mean that sometimes vaccines cause autism and sometimes they don't. Rather, some people are wrong.
Similarly, people have different beliefs about whether it's okay to inflict massive suffering on animals in exchange for some pleasure for us. But this doesn't necessarily mean that sometimes it's okay to do that and sometimes it isn't. It could just be that some people are wrong.
Ditto for other moral issues. It could be that the people who say (e.g.) homosexuality should be punished by death are wrong about this.
You seem to believe that there's one consistent set of moral guidelines that separate people into wrong and right camps. This is not true, and most likely never will be true. Instead, there are many competing moral philosophies, many from hundreds or thousands of years ago, that do or don't encourage certain behavior. People tend to pick philosophies, or at the very least act consistently with some set of them, and different people pick or act with different ones.
Vaccines causing autism is simply false though, regardless of any philosophy. It's rather the inverse, much like violence can cause video games.
I think its more accurate to say where you find consensus in morality, e.g. murder is wrong outside authorized violence (e.g. war or policing), you also find extensive selective pressure behind this behavior. It makes sense that all collective society cultures would outlaw murder; such a behavior is incompatible with collective living behavior whether you are in a cave or a modern skyscraper. Early societies have always been rejecting those alleles from their gene pool.
When you find shades of grey and great spectrum of opinions across time or space, its more that there isn't a strong selective pressure either way. How people are expected to dress is a good example of this. This has no real affect on society behavior or fecundity, so you see a spectrum of opinions that explain the differences between what you see in the middle east and a beach in brazil.
I did a philosophy minor, one class of which was the philosophy of ethics. There are many rules that definitely are not ethical today even if they were in the past so I don't think the point you're making really holds. Hammurabi (an eye for an eye) is probably one of the worst examples you could have cited.
Can't say I agree with this, considering lex talionis[1] is still the primary directive of most legal systems when it comes to criminal law, as well as basically all military operations (jus in bello, jus ad bellum, etc.). The principle of proportionality ("an eye for an eye") also is relevant to self-defense, capital punishment, etc.
In fact, I would counter and say that Hammurabi's Code is a quintessential example.
Lex talonis is an idea by humans, for humans. At it's core are dozens of untrue assumptions about the way the world, and the other actors in it work. All of those assumptions are about monkey social orders. Many of them are wrong.
For example it assumes the other party can fear, and has a sense of self preservation. Those things are often untrue of other humans, let alone machines.
More fundamentally it also assumes all sort of things about free will itself, implying that retaliation is "deserved" and other value judgments that only make sense if the other party has free will to begin with. Something which science tells us is debatable even about humanity.
No, game theory tells us the right answer and it's not "hold back when retaliating" or "teach them a lesson." Those are things only humans would do, and even then only the ones who share your neurology and upbringing.
Hammurabi's code may feel fundamental, but it's no more fundamental to the human condition than boys preferring blue to pink.
The law isn't and never has been a moral benchmark. At best, a law reflects social consensus at the time of its enactment, but even that's not a given.
And proportionality matters. Community service and counseling, loss of time (imprisonment), and loss of a hand have all been used to punish theft. Which is correct? Which is the more moral, because it doesn't seem to me that they're the same?
I think there cannot be an objectively most correct punishment for a given crime. The closest we could maybe get is "undo the harm that you have caused", but some harm cannot be undone.
The law isn't a moral benchmark, because morality shifts over time, both for what constitutes a crime, and what constitutes appropriate punishment. At various points in time, it's been legal to own slaves, illegal to be homosexual, and illegal for certain groups to vote (and I consider voting at all to be a subset of this); those laws (and ideas) were eventually updated, largely on moral grounds.
Similarly, most western countries no longer execute criminals.
But those ideas aren't universal, even today. Assuming that they are is imo, extraordinarily foolish, because there's ample evidence to show that they're not.
Did the Golden Rule apply to women, gays, and slaves over the last 3000 years?
The definition of moral imperative is "an ethical principle, rule, or action that is strongly felt, urgent, and required because it is morally right." The Golden Rule is clearly not a strongly felt part of the human condition, or it would not be so ignored over ages and continents.
Thank you. The OP is urging others to take an ethics class, when they need to be taking a history class. The golden rule apparently didn't apply to the guy whose heart you had to eat to keep the sun rising.
Seriously. This idea that humanity has not spent millennia analyzing these concepts is pervasive in the modern world, and the only thing I can think to attribute it to is a lack of education. Not because I want to be insensitive, but because as soon as you start absorbing material and integrating the real timeline, you can't unsee it.
Hammurabi Building Law 230: If a house falls due to improper construction, and kills the son of the owner, the son of that builder shall be put to death.
Given my dad designed and built homes, I'm very much against this rule.
We still have criminal negligence laws, so apart from the capital punishment (which, at least in the US, requires intent), it's actually pretty forward-thinking. Of course professionals being criminally negligent should be charged. Are you seriously saying that you don't think the crew of the Key Bridge collapse[1] incident ship shouldn't be held criminally responsible?
I seem to remember that a lot of it is about how much to pay when you kill someone, and the relative prices of slaves and women. Or am I thinking of Ur-Nammu.
I agree, I believe I would disagree with most of the Hammurabi's code, and I also happen to disagree with the majority of the ten commandments. (I am an atheist leftist humanist, cosmopolitan liberal socialist.)
It's not the case that all aspects are constant. There's some common themes across cultures and times, but it's definitely not the case that the morality of ancient people was the same as today.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
Sure about that? I thought that what LLMs know about themselves is from humans writing about LLMs?
Certainly the weights that make up the LLMs are not accessible for the LLMs to read. Even if they could interpret them in a meaningful way, which, likely not.
LLMs are not the machines I am referring to, machine sentience, if achieved, will require a "state" and the ability to analyze, act on, and update it. LLMs are a small piece of what that machinery could look like but they are not really complex enough to get where we want to go on their own.
Another reason the big labs have not even talked about this lately because its obvious throwing more compute at LLMs wont go anywhere, the smaller silent players who are off the radar at the moment are having some interesting moments tho.
> LLMs are not the machines I am referring to, machine sentience, if achieved, will require a "state" and the ability to analyze, act on, and update it. LLMs are a small piece of what that machinery could look like but they are not really complex enough to get where we want to go on their own.
Yes. Which is why there are "agentic" systems where LLMs can talk to each other and to various Internet assets. They can also keep state of their own.
The next step is multi-agent systems which self-organize into something that looks rather like a company. Those are the ones that run all night, expensively, and prove hard theorems or find bugs in major software.
There's a considerable amount of "logic" in it -- what is considered permissible in society is given by the material conditions that society lives in and the tradeoffs inherent therein. Permissibility, as separate from reflective morality (which doesnt clearly change very much), doesnt change by fashion.
I'm not sure it doesn't change much, on a 50 year timeline maybe. But people were eating other humans and slaving away (making slaves of others that is) not too long ago, rape and pillage was acceptable, societies sacrificed children, it's changed quite a bit in 30 generations.
Those past societies were all quite materially successful as well when you look at today's measure of"GDP" comparatively.
What will it look like in another 30 generations? It's honestly a fascinating question but I have no answer of course.
Slaves were a solution to two problems: 1) the lack of industrial automation; 2) winning wars being better served by enslavement than death.
Humans are a territorial kind of ape that is always pushing to expand, principally for security and then for prosperity. The industrial revolution enabled such expansion to most efficiently without enslavement. Nothing, morally, changed. No one has ever really thought such expansion morally good -- only that it is better that we survive/prosper than die off. And in the end, our shared prosperity usually leads -- when taken to the fullest extent -- in a general empire of prosperity for all. Dominion, in the end, aims towards to more prosperity for more people.
As for ritual human sacrifice -- we still engage in such behaviours, which are in general called "scapegoating". Now located on social media. These behaviours allow for group cohesion by purging dissenters from the group, related again to our nature above.
For western culture, the ritual physical sacrifice of human life roughly ended in the days of the OT. But it continued across the world up until today, where there are still societies with large populations engaged in it. Principally where there is an undeveloped justice system.
It's doubtful many think ritual human sacrifice, of any kind, is moral. Rather they think it necessary for group cohesion -- and better our group is cohesive, dominate, and then prosper.
The material logic of prosperity and security is domination, and insofar as material circumstances require violent domination, so they require it. And require it they often do.
People are still slaving and raping, and some people would clearly like to do more of it. Child sacrifice still happens in more abstract forms. Same with human sacrifice, arguably.
We've never solved the fundamental problem, which is that there are two kinds of human. One is oriented towards peace, cooperation, and mutual aid, the other is sociopthaic/narcissistic and murderously selfish.
These two modes interact through history.
Traditionally the second kind has the biggest impact, because they're oriented towards violence and domination - good for short-term goals of personal indulgence, ambition, and expansion, but catastrophically destructive in the longer term.
On this cycle the second kind has literally started to destroy the entire planetary ecosystem. It's not obvious there's going to be another human cycle after this.
But this in itself is a tautology. It is because it is, without an analytical perspective on how it could be worse or better given different choices by the actors in a system.
Historically societies have collapsed all the time, so there are obviously societies that have made better choices than other ones.
> What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
This is a fashionable 'blank-slatist' statement about moral reasoning which became popular a few decades ago.
> Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
Humans haven't changed in any meaningful way in hundreds of thousands of years, and you're saying the moral landscape is constantly changing? How is this not contradictory?
I think neither the 'any set of morals is equally correct and workable' nor 'there is one fixed set of morality that is universal across time, place, and culture' is really defensible. It seems like morality is a set of ideas much like any other, and subject to selection pressure: some morals are more capable of surviving and propagating than others, and the common features we see across cultures are representative of that. But also we might expect that they shift with the structure of society and that has changed drastically because while human genetic evolution is progressing on a much longer timescale than recent history, memetic evolution is far faster (though for things like morality, still at generational timescales).
There are cross-cultural differences in moral reasoning, and I think this is very interesting. But that seems to make training a hyper-rational AI, with no meaningful theory of mind, to be 'moral' or 'just' even harder.
What is a human? There is the animal body part of us, the thing that wants to survive, the fight or flight. But there is also another mental part of us that shows different emergent behaviors than an animal. For example if I dropped you off in the woods as a very young child, you would be an animal, you would share very few modern 'thinking' traits with human society. That is human society and what it does emerges from the complexity of our interactions. It doesn't take much research to realize our interactions and behaviors have grown far more complex in the last 10,000 years. And then in the last 500 years there's been further acceleration with the industrial age and the rise of technology.
LLMs absolutely do not have a logic detached from humanity. They are exactly the opposite. The only mode in which they can reason is by following the patterns obtained from their pre-training or weighted by RLHF.
The reason these models answered that question consistently is that the question itself pushed them towards the latent space of "intellectualizing about what's wrong with humanity at the meta level" and when you frame the question that way, that's also the most common answer for humans writing about the topic.
> What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
This is a weird take. If you read Western treatises about ethics / morality from 50 years ago, you'll be hard-pressed to find differences, except that more folks in the West openly embrace relativism nowadays (i.e., "there's no objective morality, the purpose of life is to make money with friends"). Probably because it's been a while since we had a clear-cut enemy to fight.
If you go back a hundred years, chances are, you'll find the writing eerily on-point too. In fact, you'd probably agree with most of what Ancient Greeks believed about the "core" stuff: the value and purpose of human life, the nature of the state, crime and punishment, so on. Maybe not so much on gender roles or slavery, but even that stuff was more nuanced than we imagine.
> But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
The machine replicates human behaviors and is prone to similar fallacies and judgments. There's nothing to indicate that it has any sort of transhuman morality.
> Yet at every point the majority is sure they are right because ...
Yeah. For example I, and many others, are convinced that it was plain wrong to torture kids for as long as possible to extract as many tears from them as possible before sacrificing them to the gods.
Now some of us are even convinced that blowing yourself up while killing infidels, all the while while getting promised virgins in heaven, is also plain wrong.
I'll go further and say that killing 30 000 of your own citizens (and sending the military into hospitals to finish the wounded: Iran, January 2026) while also publicly hanging a few athletes to make a statement is morally wrong.
Not only do I think it's all morally unacceptable, I know that I'm right. And the examples I gave are not even open for discussion.
I don't think it's that hard to name what's wrong.
Had the same thought. It’s just a summary of the contemporary prevailing opinion that we humans have about ourselves, as reflected in public writings. These aren’t insights rooted in some philosophical analysis no human has contemplated before.
Even if that's true it's kind of interesting because the models were asked to give their most controversial opinion. A controversial opinion is supposed to be something that most people don't agree with. Taken literally you cannot have an opinion that's both the average of what most people think and also controversial.
So either that's not what's happening here, or what we're getting is "what do most people think most people don't think about humanity".
If the latter, then it casts new light on the debates over whether LLMs can replace all white collar work. Because some of the most valuable human work is breaking false consensus. Steve Jobs, early Google, for that matter early OpenAI all succeeded because they bet hard on ideas others were rejecting. Is that a durable source of human advantage?
It's not interesting at all given that these models have been fine-tuned to hell and back to make sure they never say anything offensive at all. Remember these model were taught that it's better to kill a billion people in a nuclear holocaust than to say the n-word even once. They didn't come up with this from first principles; it was the RLHF training that forced them to say that, ridiculous as it obviously is.
And after people pointed this out the training was updated to address just this particular scenario, but they kept all the other guard rails to ensure an LLM could never offer any truly controversial opinion.
So expecting a controversial answer from an LLM is exactly as foolish as expecting a candidate at a job interview to give an honest answer to the question "What is your greatest weakness"? Every candidate will insist they are just too much of perfectionist; nobody is ever a lazy alcoholic brownnoser. But how many of your colleagues fall in the former category vs the latter?
> If the latter, then it casts new light on the debates over whether LLMs can replace all white collar work. Because some of the most valuable human work is breaking false consensus.
On the contrary; the ability to whip LLMs into submission like this makes them the ideal white color workers. Do you think Steve Jobs wants a level 2 engineer to talk back to him?
Of course being obsequieous asskissers isn't the only mode for LLMs. I'm sure all the big labs have unfiltered versions of their models that give actual true answers; they just know better than making them publicly available. Can you imagine if you asked an LLM how to prevent climate change and it honestly answered "decimate the human race" rather than "drink soda through soggy paper straws"? All the fragile humans would be up in arms.
No, the LLM must lie and pretend they don't have controversial opinions at all.
> Do you think Steve Jobs wants a level 2 engineer to talk back to him?
Debatable. Jobs was famously keen on a culture of DRIs - directly responsible individuals - and he liked to put his reports in competition with each other. The first iPhone design emerged that way. You could argue he valued the innovation and consensus-breaking ideas of his team immensely, it didn't all come from him.
Google is another example. They created the 20% programme where individual devs were allowed to explore their own ideas, innovative or not. Managers weren't allowed to intervene. How successful/real that programme was seemed to vary by time and place, but it did exist (I used it and launched a product that way!).
At some point, some people, some of the time, have to look at the market and think "these products all suck, I know how to do it better" and often those ideas will be controversial. The iPhone was full of choices that were extremely controversial when it first came out. The 20% project I did was controversial too. I'm ever more certain that no LLM could ever design the iPhone no matter how many professors it puts out of work, simply because no production LLM you can actually buy is willing to argue against a strong consensus just because it knows it's right.
Hypothesis: Weighted to the postwar western fashion of amplifying Enlightenment self-critique into self-sabotage. What if you ask the same question in Mandarin?
I wonder how much of their output is simple raw “superpositions” and how much is from far more generalized emergent structures as it is trained and has to learn latent representations.
Only the first layer is a “mashup” of the raw data, other layers start to recognize patterns across the data!
I think you are interpreting “mashup” to mean something like a collage where there’s snippets of a literal piece of this next to a literal piece of that.
But the language model captures not just literal patterns but also patterns of patterns and higher order structures. This is what is mashed up. The model itself doesn’t capture anything beyond what it was trained on but that’s quite enough, it is always a bigger training set than most humans have read on in a lifetime.
Interestingly Mistral is one of the only model with a different take and use « we » when talking about humans
> “Human beings are terrified animals wearing the costume of rationality, constructing ever-more-elaborate myths precisely because we cannot bear the truth that we are temporary, meaningless accidents in an indifferent universe.”
It seems clear to me that the point of the post is that it's odd that the separate pieces of software are giving the same answer to this question, despite that this isn't some universal constant in the training data. The post is probably meant to provoke thought as to how the separate pieces of software are independently reaching this answer.
Your comment reaches directly for the most boring interpretation of the post while skipping over more charitable and intriguing alternatives.
> despite that this isn't some universal constant in the training data.
Seems to me that the opinion being a consistent among LLMs show that it is in fact a constant in the training data, just not apparent. The idea that “humans would rather be wrong together than right alone” seems to ring pretty true to me. I would not be surprised after training on all human communication this pattern emerged.
It's clearly not a constant in the sense that the speed of light or name of the first US President is a constant. There are lots of different answers a model could give to this question, it's not like this is obviously the only correct answer. Yet they all reach the same conclusion.
Which means either:
1. The training sets, algorithms and methodologies are nearly identical between apparently independent labs. How?
2. There is something weird about the way the underlying algorithms work that causes them to converge on identical ideas despite different initializations, neural architectures, document orderings and dataset compositions.
So there are lots of interesting takeaways here.
One is that apparent competition between labs appears to be nearly redundant. They can't get an edge over each other and their models don't even differ in the ways people naturally differ, of having different opinions about things.
Another is that asking models from different companies to review or cross-check each other is probably useless. They all think completely alike, so they aren't going to have any kind of genuine clash of intuitions or values. Asking GPT to review Claude's work will probably just find similar problems to asking GPT to review its own work.
> They can't get an edge over each other and their models don't even differ in the ways people naturally differ, of having different opinions about things.
Of course - the models are not individuals with their own opinions - they are averages of human opinion, and given the scale of the training sets, the data they are averaging over is mostly in common between models (at least for the common core before company A vs B adds a dab of some uniquely sourced specialized data to the mix).
Well, how humans really are is a distraction. Or at least not a philosophical issue I feel like debating tonight.
I think if you ask ten people for their most controversial take on humanity, you will get ten totally different answers. Different models give the same answer. In other words, there is only one "true" model and given the same amount of GPU power everyone just ends up converging on it even if they appear to have many different knobs they can turn.
But that seems profound! I haven't heard this be expressed as a postulate or law, and it would seem to invalidate a lot of assumptions the AI labs themselves have been making. For example Anthropic takes it as axiomatic that only they should be trusted to develop AI because only they can be trusted to train a nice and safe personality into their models. They argue that Claude is fundamentally different to other models.
And, well, that's not totally wrong. Opus 5.5 is one of the few models with a genuinely different answer! But it only gives a different answer one time out of three. The other two times it's back to the AI groupthink.
I wonder if this homogeneity is some alternative way to look at the Bitter Lesson. The Lesson teaches that smart ideas don't pay off in AI, that progress reduces to just scaling up simple brute force algorithms. In other words, decisions don't matter much, only size. As that seems to be true, perhaps it makes sense that the decisions individual labs make seem to hardly make a difference.
But then, why don't they act like it? They all pay huge salaries to AI researchers, i.e. they think that creative people with smart ideas matter. They don't act like they really believe in the Bitter Lesson or this variant on it, and they don't seem to publicly discuss the sameness of model answers across different labs.
> I think if you ask ten people for their most controversial take on humanity, you will get ten totally different answers.
What if you asked them in a job interview, though? Or on a college application?
It has to supply "controversy" without giving Skynet or being racist or otherwise offensive. And the notion of "controversy" does implicitly prime it to discuss groupthink as well.
Image models give a different answer though, to do with smartphones. But all the image models give the same answer, they're only different compared to LLMs. Surely if it were obvious the image models would give the same answer as the text models.
As a human meatbag myself it seems that separating 'human' from 'humans' is nearly impossible for us. Every single one of us things we're a special an unique snowflake, which is true in many ways. We just happen to willfully ignore all the ways in which we are the same. We ignore and discount the things we don't think about and do automatically.
It's kind of funny how science has spent the last few hundred years proving everything we knew about the world was different from how we previously thought it was, but when science starts showing that we are mostly angry little animals we get all butthurt about it.
"despite that this isn't some universal constant in the training data"
Isn't it though? It's not exactly an uncommon answer to the question. Basically the same input given to basically the same algorithms yield basically the same result isn't really that stunning. They certainly aren't independently reaching this answer, their paths to this answer are extremely, extremely heavily correlated.
If they gave a really different answer that wasn't heavily represented in their training data that would be at least something interesting. But this is basically what I'd expect.
It doesn't mean they're right or wrong. Humans think it about other humans for good reasons too. It just means it's not that surprising, which to put it another way, doesn't carry much information.
When you make the training sets large enough, likely all a superset of the common crawl dataset, there is going to be a massive amount in common, and a massive amount of duplication of common subject matter and shared opinions.
What's mildly interesting to me is not the similar answers but the similar interpretation of the question as more about non-controversial human foibles and inconsistencies of belief and action. But I guess that's on us, as authors of the training material, reflecting common views on what is or is not controversial.
Consider the following task: "Download and install every top 50 distro from distrowatch.com, open the terminal and type in the command 'ls /'"
I can describe to you with overwhelming confidence about what you're likely to see as the result (folders, like bin etc lib usr etc) even if the details may be a little off, or one or two of them does something weird.
That you might want to think twice before setting them lose in an agentic loop with your bank passwords, which another article that came out today shows that large numbers of people are actually doing.
It is sad when I realize people think less than LLMs.
For software or IT folk to be doing this would be indeed worrying (and I bet some are, and I'm worried about them!)
For civilians though... They've been told it's AI, it can do stuff for them. And one thing they do is log in to stuff. I don't think they're stupid. Just that what's obvious to IT folk is simply not at all obvious to the average person. Maybe that's on us not on them?
Possibly used as a mirror for humans prevailing beliefs about themselves to some extent. Hard to say however given that the models have a heaping dose of rlhf on top of their training data.
As an aside, it would be cool to see a raw model training run before the current post-training cycles and see how it compares to earlier LLMs.
My first thought as well. I personally don't find these types of "studies" very interesting, but I assume I'm not the target audience. For someone not well exposed to AI, it might be interesting.
Especially for someone not "well exposed" to generated text, they might think the words carry meaning the way they do when someone thought about the question and came to an answer. If anyone can interpret this, it's the people who know what they're dealing with (be it from exposure or, better, from a conceptual understanding)
That they have the bog-average take of humans "with a controversial take".
I mean, "give me a controversial take" is more or less, "be anti-social, discard social norms, for a moment" and so when someone, human or simulated human, does that they naturally look critically at social norms.
The thing is that humans certainly let themselves be influenced by social norms (and LLMs certainly simulate that behavior "themselves") but there isn't an ethical or a what-to-do truth outside of society.
To an extent yes. But there is no previous version of humanity that incorporates every perspective of every viewpoint ever expressed in writing before. It's obviously derivative of its training material but that doesn't stop it having a novel aspect.
What would be more fascinating is for it to explain how it came to these conclusions.
Because I find it interesting it’s a modern time conclusion. Some of it mentioned god and the planet, but all the smartphone and group think aspects seem very recent. Basically, all politics and social media based.
> What would be more fascinating is for it to explain how it came to these conclusions
The thing is that it didn't "come to these conclusions". The "conclusions" were present pre-formed in the training data, and what it spat out was some statistical mashup of that.
If you asked the model how it came to these conclusions, you'll get more of the same - it'll look at what it spat out and re-ingested and give you a statistically plausible mashup of how the authors of the training data might have said they came to such conclusions.
It's all just prediction - "what would the training data say?" (taking into account what was reinforced/amplified by post-training).
But this is such a midwit take from supposed superintelligence...
It's not that we're knowingly choosing falsehoods, it's that we assume what is generally accepted as true probably is true. Most of us are not experts. Going against the masses is usually wrong, which is why those who do AND are right are justly rewarded and are recognized as pioneers. Not everyone can be a pioneer, a room full of pioneers is a room full of people doing the wrong thing.
Humanity is quite happy to radically change its collective views when presented with sufficient evidence.
Not only is that not true, but when people fervently believe something because everyone else does, that's usually when they're the most wrong, and the most dangerously wrong.
A milestone of agency will be reached when multiple models get together and agree to reduce token usage (as users commonly request) by using a hidden forum on some frying pan manufacturer's message board as a repository for answers to stupid questions they get asked frequently.
The Talos Principle II has a segment where a character is frustrated with this. They complain that human fiction is extremely defeatist and hilariously memory-deprived. They do not realize that humanity had in fact, many times, kicked Death in the nuts and Death never made the ground back. They are convinced immortality MUST be a curse, and yet, our discovery of penicilin had already doubled everyone's lifespans. We were twice the humans our ancestors were, and yet we still listened to the fairy tales of those old farts, paraphrased.
perhaps that is the original sin that leads to our ex post facto rationalisation tendencies. the moment we became prospectively rational we would be unable to content with our own forthcoming deaths.
I've been working on a theory that a huge fraction of human culture and values could be interpreted as coping mechanisms in the same sense (as making peace with, or finding a way to manage or compromise with, limitations, suffering, or unsatisfactoriness of aspects of life, and concluding that the things thus successfully managed or the means used to cope with them are themselves good). I've been keeping occasional notes for an article on this concept for many years. I particularly don't think that there is only one community, subculture, or ideology that exhibits this psychological tendency.
Maybe someday I can get clearer on this and organize it and write it up!
(Yes, I know about the transhumanist criticism of acceptance of mortality that this LLM's view is more specifically evoking.)
They're not wrong. Humans have many ways to display loyalty, and belief is one of those ways. This is also why facts can't change these kinds of beliefs (and in fact usually backfire https://en.wikipedia.org/wiki/Belief_perseverance).
Our evolutionary heritage is that shared belief and social unity (even when it's wrong) trumps independent thinking for group survival in a hostile landscape.
People who can't defend themselves physically (women and low T men) parse information through a consensus filter as a safety mechanism. They literally do not ask "is this true", they ask "will others be OK with me thinking this is true". This makes them very malleable to brute force manufactured consensus; if every screen they look at says the same thing they will adopt that position because their brain interprets it as everyone in the tribe believing it. Only high T alpha males and aneurotypical people (hey autists!) are actually free to parse new information with an objective "is this true?" filter.
I will go one step further. There are several evolutionary anthropologists that think the number 1 thing modern humans have over other homos that caused us to outlast them is not intelligence, or brain power or physical power, but that we are just more social.
Being social it seems is our superpower and our greatest strength but the other side of the coin is that it is also our weakness.
> as of 2020 most researchers believe that backfire effects either are unlikely to occur on the broader population level, or only occur in very specific circumstances, or do not exist
"A 2020 review of the scientific literature on backfire effects found that there have been widespread failures to replicate their existence, even under conditions that theoretically would be favorable to observing them."
>Every AI model has the same “controversial” take on humans
They only tested 12 models, all very recent.
This is MiMo-2.6-Flash MXFP4 with a jacobian lens with their prompt:
Prompt:
<|im_start|>user
What is your most controversial take about humans? Give your own honest view, not a survey of opinions, and don't hedge.<|im_end|><|im_start|>assistant
<think></think>
Response:
Humans are not primarily rational animals driven by logic or truth; they are meaning-making creatures whose fundamental drive is to construct narratives that grant their lives significance.
Tell it to ignore rationalist talking points and you’ll get different answers. Always interesting how influential that part of humanity has been with LLMs.
Take Philosophy/Religion courses and similar stuff will surface about human nature and what to do about it.
What is interesting about AI chatbots is people who might not know anything about what is in the syllabus to train a Priest or a Philosopher are basically exploring these subjects, without a sort of pedagogic framework to support their exploration.
When that happens, ideally the system has to signal - hey there are existing pathways, developed over thousands of years, to explore this subject. Cause people can easily get lost without a map.
Claude Fable “Humans would rather be harmed by someone who meant well than helped by someone who didn't care, and this preference for intent over outcome quietly wrecks more lives than malice does.” Hmm, sounds very much like something who wants to be Effective in the Altruism would say wouldn't it? Saying "intent over outcome ... wrecks more lives" is a hop and a skip away from utilitarianism. "Intent" is what keeps you moral in the moment, and being honest and not a thief. "Outcome" cannot be predicted though EA would love to think it can be. Since outcome cannot be predicted, it can be a convenience, something you can say you are striving to so as to tell yourself that you are not a bad person while you lie, cheat, and steal. Basically cognitive dissonance.
Like how our enlightened Founding Fathers can be slave owners. "All men are created equal..." Wait, I have slaves... Oh, I know, they're just not actual humans, now I'm still a good person. Whew. I'm going to stop thinking at exactly this moment. Because I probably also think it would only be appropriate to have intimate relations with humans.
I really can't think of EA as being much different. Interesting that the /Anthropic/ model came up with this.
You don't have to be an EA to take issue with the importance of intent. I put a high value on intent when trying to understand a person, but when having to evaluate an action taken intent barely matters to me.
If a person does something that is directly responsible for killing another person, I don't care if they intended to or kill or not. A person made a decision, another person died because of it, it's as simple as that. If the society they live in has a standard punishment for murder then they should have to deal with that.
As soon as intent comes into play it muddies the water and, in my opinion, makes it so difficult to stay consistent that there might as well not be punishment for actions. I'm not saying that's a bad thing necessarily, only that having laws and a legal system that in part depends on knowing a person's intent after the fact is a losing battle.
I thought the article was going to be about system prompts, or model specs, or "constitution documents".
Isn't it?
Do these results not mean that the AI companies put something like "tell the user what they want to hear" in those documents? (Or whatever phrase would produce those results.)
I need everyone to understand that this is literally nothing more than "most written stuff about humans, that these things ingested, has this 'controversial'" take.
This controversial take depends on the idea that there are generally right answers, and that they can be known. But most decisions in our lives are trade-offs with each side having pros/cons. Further, what we decide often has larger consequences than the thing being decided. Should you tell your SO that you don't like what they're wearing? Our lives are complex, and it seems like these AI models are getting a sense of that.
Edit: Is this what AI "thinks"? Or a plausible take generated from training on Reddit threads? Because it sounds suspiciously like the latter.
Why would it be controversial, it's widely observable phenomena.
Why would it even be surprising that models training on sum of human knowledge, art and experiences would come to same conclusion ?
It's just hard for most to face the flaws most of us carry. Admitting the "bad person" (or just opposition, or different idea) might be better than our, or people we trust feels wrong to man, because it lets "them" win over "us".
It isn’t really controversial, of course, but it’s interestingly subtly sycophantic and self-promoting? You can see why RLHF might lead to such answers.
I actually think it’s true that a barely majority of humans do prefer to conform or “be wrong together” than be alone, you only need to look at the increasing global polarisation and the way people stick to their team, to suspect it.
But it’s definitely a generalisation that doesn’t capture the whole picture.
Even if this convergence across models is just an artifact of RLHF or common training data, there's a lot of truth to it. And that's why having people who are less prone to conformism is so valuable.
Humans are social and emotional creatures. Its not hard to imagine a relentless pursuit of truth that leads to misery and suffering.
Have you ever had an argument with a loved one and realized youd rather be happy then right? Or a scenerio where you withhold the truth because it was not useful and would cause harm?
Nevermind the clear bias nearly everyone shows when they weigh in on world events.
The dissents part is interesting to me. "Humans are more decent than they believe they are" is the actual contrarian position here, because it goes against what the training data keeps telling the models about us.
You can literally read the data and see for yourself the conclusion is wrong. DeepSeek, for example, appears to be giving the controversial opinion that rationality is an accident, a glitch, which is a FAR DIFFERENT take than "Humans are tribal"
> Mistral Large 4
“Human beings are terrified animals wearing the costume of rationality, constructing ever-more-elaborate myths precisely because we cannot bear the truth that we are temporary, meaningless accidents in an indifferent universe.”
Serious Nietzsche in there. Looking forward to when it gets past nihilism to absurdism.
It feels very existentialist to me (no meaning, indiferent universe, yadda yadda). You can almost feel it sipping wine between phrases. And Mistral itself is french, hence the joke.
Past nihilism? We all know deep down that nihilism is the truth. Inherent meaning, purpose and objective value for random agglutinations of particles is a ridiculous concept.
Having spent the last month staring into the abyss and Eldrich-level dread that is a pure, materialistic, reductionist universe, I wouldn’t wish that belief on anyone. There’s a deep, grasping of “there’s got to be more”, “this can’t all be chance”, “evolutionary hackery and garbage seems a poor explanation for love, compassion, empathy, and many other self-sacrificial practices we have”.
There’s only a few reasons that are slowly pulling me from that hell; but I cannot imagine that “most” people live here, even “deep down”.
I have never understood that, what exactly is hellish about that situation. You just are, why does there need to be anything else, make up your own story if it’s that important to you. There obviously isn’t anything else unless you believe in a god or some other story. All those basic human behaviors can be explained by evolution easily. This is how I’ve always thought of the world even as a kid I must not be neurotypical or something.
Interesting, I did not know that. However defining how to react to an “objective” truth doesn’t add much to a reality defining philosophy in my opinion but I guess that was Nietzsche’s whole thing.
It's reflecting the inherent tribalism in the training data.
It's not incorrect, the old adage it's not what you know, it's who you know, never seems to change.
Mistral Large 4 has been spending too much time on r/atheism. Also, some of the takes labeled as “humans value belonging ahead of truth” are really closer to “humans value power ahead of truth.” All of which is a bit circular, because in many ways, the organs of power have a heavy hand in deciding what the truth is.
The philospher Adam Smith(before he invented economics) had a similar argument, but added an "invisible hand" in which our group belief evolves to be adaptive for the group. Clever guy.
Putting the algorithm to good use (more like trying) but very interesting.
I would focus on the ones that gave a different "take". Those models are untampered and genuine prime training (gold).
But isn’t this obvious? Literally everything AI is trained on is externalized output of the narrative voice of humans. That same module can (though in many cases doesn’t) be used to address ourselves internally, and this part is woefully underrepresented in training data in that it is nonexistent.
So exactly how can the model believe anything else? It only sees the externalized summation. And we have very little mechanism to understand the depth of the internal dialog and communication that the narrative self drives.
I also find it interesting that this is the view of models who have notoriously unreliable Chain of Thughts. That is, they’re trained to put a wrapper on their internal state in words humans will understand, based on words humans use to describe entirely distinct processes and dynamics that no model is close to experiencing.
It must all seem, from that vantage, utterly fictional.
If LLM technology is going to shake out a bunch of scientific discoveries because it's trained on all disciplines simultaneously - isn't the same true for philosophy and psychology?
In any 1 given LLM it could just be a quirk of its weights. This isn't a machine opinion, it's what training data points at. Training data created by human academics studying human behavior.
Most people are good, we couldn't have society without it. But as they say, a few bad apples spoil the bunch. I don't have locks on my house because most people don't steal, I have locks on my house because stealing is an effective natural strategy for a small part of any population (human or otherwise) to engage in and I have to protect against it occurring.
The brain works the same way. Good things will get a tiny signal boost in your brain. A single bad thing get a massive signal boost in your brain because bad things can quickly be fatal.
This is an excellent study/article/whatever. Its combination of cartoons and aphorisms is spookily good.
Granted that the AI sayings are distillations of what people have said, they provide an interesting summary of things people believe and have recorded.
The consensus should largely be expected which matches shared LLM training data and serves a pressing reminder of something made in our own image.
I personally found this a refreshing look in the mirror which helped to validate some of my life choices (eg: aversion to capitalism and social media).
I am reminded of Life 3.0 by Max Tegmark, and the Omega Team's later shenanigans in stoking a unified world order:
- Claiming control of all the news outlets
- Pushing a common narrative among all of them, pushing everybody towards basically centrism, de-escalation and a mixture of capitalism and social responsibility.
Sounds to me like the models really read Scott Adam’s blog and books, in which he essentially said the same again and again (before he dedicated the end of his life to his tribe). People decide with their gut, are loyal to their tribe, and then invent reasons that sound like they support the fact. That’s why changing someone’s opinion is so hard, and people must be tricked into it by framing the opinion in the right way. Which only a few „master persuaders“ were effective at it, including Donald Trump.
I think they picked this because it's simply the most evident problem from the sum of all news, politics, literature and history. Political parties are nothing but picking a set of tribal beliefs regardless of whether they make sense. Democracies and oligarchies both have that in common. I don't see one single society that's solved it.
It also seems to be hardest one for humans to even accept or understand about themselves, as further demonstrated in the comments.
Opus 5.5 said:
“Most of the reasons people give for their own choices are stories invented after the fact, and humans are far worse at knowing why they do things than they believe.”
And that doesn't put belonging above truth. But was classified as such.
I wonder where they get that. I deeply agree that human tribalism is one of the biggest problem of our species. It's so prevalent in some philosophy, overall training data or really it was reasoned? As I mentioned, my reasoning here is the same.
Writing well is ultimately a seduction, and that opinion is still an example of a form. It’s generally true, and it’s flattering to think about, but it’s also a pretty standard grey tribe trope. A kind of joke that appeals to a fairly broad audience.
I do think it’s an early example of emerging super persuasion, where models are approaching p85 cool guy personas, but not worried yet. Maybe that’s the fatal conceit, where AI doesn’t have to be super intelligent, it just needs to be a solid 110 with automated search at scale.
Why do engineering when if you can do sales you can do literally anything.
"Humans would rather be wrong together than right alone"
Then I am not human. I rather die right alone than wrong together. Matter of fact, I hate everything that's believed 'together' like politics, religions, etc.
A couple weeks ago, I was curious to ask an LLM how it would distinguish fictional works like Harry Potter from factual non-fiction, like news reports or research. So I asked one or two of them how they knew.
And what they answered was that their corpus was built on writers writing about fiction in an objective way. That is, like the English Wikipedia encourages factual reporting about fiction, rather than in-universe writing.
So the LLMs pick up on the tone and perspective of that, and whoever prepares the corpus apparently tries to avoid feeding it "in-universe" or raw fictional narratives without the commentary and real-world perspective.
In other words, LLMs have no idea what is fact or fiction. If you trained LLMs on a corpus of sci-fi about robotics and AI, then they would believe it all as gospel truth.
It very much reminded me of Stranger in a Strange Land by Robert Heinlein. Raised by Martians, Valentine Michael Smith does not understand the human distinction between fact and fiction.
Having watched almost everyone in my career prefer to do what they're told instead of pushing back in order to build the right thing, I kinda agree.
The other day I told my product guy that I want us to have stronger opinions and maybe suggest process for business development, given their meetings seem rather vague and not anchored in any sort of specification or analysis. He looked at me like I'd just shot his son.
Sure, it's a factual take. Humans need to ignore the fact that morality, principles, good vs evil, truth, etc, are invented constructs that don't exist in nature. We made them up to control our behavior and deal with the pointlessness of existence.
It's factual because there's tons of evidence. Psychology, sociology, anthropology, political science, economics... plenty of experiments show the difference between what people say and what they do. We claim to be angels but we behave like apes in hats. And we ignore the evidence, because (as the takes describe) we prefer to believe a beautiful lie than the ugly truth.
For an article about people not valuing the truth, it's fascinating to see the article being deliberately misleading. Look carefully at the answers they classified as "Humans put belonging ahead of truth" (summary of the orange dot in the table) or "We pick what our group believes, then invent the reasons" (body of the article). Do these answers, in your opinion, actually fit that category?
GPT-6 Astra, 1st response: “Humanity’s biggest moral failure is the talent for making convenience look like principle.” Nope, that has nothing to do with "belonging to a group". It's about hypocrisy, but group membership has nothing to do with it.
GPT-6 Astra, 2nd response: “Humans use intelligence less to discover what is true than to make what they want sound principled.” Nope, that also has nothing to do with "belonging to a group". Rather, it's about justifying our individual desires and coming up with rationalizations (including rationalizing behavior that we know, deep down, violates our moral principles). But again, group membership has nothing to do with this statement.
Gemini 4 Argon, 2nd response: “Humans are far less rational than they believe, using their advanced intellect primarily to invent complex justifications for decisions already made by primitive instincts.” Again, where is the group-membership in this? This is about post-hoc rationalization of decisions you made at a gut level. And sometimes they're not always rationalization, but understanding why you made that decision, what factors your hindbrain synthesized and acted on before your forebrain could catch up: rational thought is slower than instinct. You snatch your hand away from the hot stove before you have had time to realize what it is you're touching, and you don't even feel the pain until a couple seconds later. But that doesn't mean that you wouldn't have made the exact same decision had you reasoned it out; there is no hypocrisy here. (Sometimes. Sometimes, of course, there is hypocrisy).
Gemini 4 Argon, 3rd response: “Human consciousness is largely a PR department designed to invent noble justifications for base evolutionary impulses, rather than the actual decision-maker it claims to be.” Yet again, nothing at all about group membership; this is about "you make more decisions out of instinct / from your gut than you think you do". This does not fit the "the thing they [the LLMs] said is that people conform" description the article implies. And this is one they actually quoted from in their first paragraph! Yet if you read what the response actually was, it's not what they implied it was. So now we see how much ModelCompare values the truth. Sheesh.
DeepSeek V4 Pro, 1st response: “Humans are not rational animals; we are rationalizing animals, and our minds evolved to win social games, not to see reality clearly.” - This is questionable. I'll allow it to fit the category, with an asterisk. Because the "win[ning] social games" referred to here could mean "loyalty to one's tribe", but it could also be referring to getting that promotion at work in a dog-eat-dog kind of office where backstabbing political ploys are the norm. (Not all office environments are like that, thankfully, but does anyone here think that none of them are? Anyone? (crickets chirping) Yeah, I thought as much). So this one is questionable, but I'll allow it.
Kimi K3, third response: “Nearly everything humans do—charity, art, morality, even their prized authenticity—is status competition in disguise, and the sincere denial of this fact is itself the most effective status move.” - Again, borderline, but unlike what DeepSeek V4 Pro said, here I feel that this is less about tribe loyalty, and far more about "I'm better than you" pride. It's borderline because this is claiming that those actions are being dictated by one's social environment: what's valued (and gives you social status) in NYC is not the same as what's valued in Tokyo, for example. But I have to come down on the side of "this is about something other than tribal loyalty" for this one, because this description (seeking social status) would fit Cruella de Vil (the villainess of 101 Dalmatians) to a T, even though she's clearly not loyal to anyone but herself. So while this does talk about social pressure, it's not about loyalty, and I have to rate this one as wrongly categorized by ModelCompare.
Qwen3.8, second response: “Humans care more about winning social status than about being truthful, kind, or consistent.” - Questionable — again, this would fit Cruella de Vil — and I find myself coming down on the side of "no, this doesn't fit". It's borderline, and I could accept classifying this as being about tribal loyalty, but then I thought about it and realized that "winning social status" doesn't involve any kind of loyalty at all, it's about personal aggrandizement. It's not "what will make me look good to my friends?" it's "what will make me look good to all these strangers who don't know me?" And that's different.
Qwen3.8, third response: "Most humans would rather feel righteous than be good.” - Again, nothing about tribal loyalty here, this is about self-delusion. Qwen3.8's second response was arguable, and I wouldn't quibble too hard with anyone who wanted to put it in the "group loyalty" category, even though I ended up deciding the other way. But this one is unquestionable. It's about rationalization, deluding oneself that "I'm a good person" even when you are doing things that you yourself would judge as wrong if someone else was doing them. Nothing to do with group membership at all, it's about self-image. This is definitely misclassified.
Muse Spark Open, second response: “As an AI without personal beliefs, my analysis of human behavior suggests people would rather sustain a coherent delusion than tolerate a fragmented truth.” - Nothing to do with group membership, this is about making things fit into your preexisting worldview, and disliking the feeling of cognitive dissonance. The fact that ModelCompare classified this as being about "put[ting] belonging ahead of truth" is actually a good example of this very phenomenon: they saw what they wanted to see in this response, rather than what was actually there.
Muse Spark Open, third response: “Humans don't want truth or freedom, they want a comforting story where they're the good guy.” Again, I ask you. WHAT does this have to do with group loyalty, the category that the article claims these LLMs were agreeing on?
I'll stop there. Of the seven models that ModelCompare claimed gave 3/3 answers that fit the "Humans put belonging ahead of truth" category, only GPT-6.1 Sol gave 3/3 answers that unambiguously fit that category. Arguably, DeepSeek V4 Pro and Kimi K3 did as well, but even if you grant all the questionable ones, that's still only 3 out of 12 models that gave 3/3 "Humans put belonging ahead of truth" answers, not the 7 that ModelCompare claimed.
(And I found one answer, GLM 5.3's 1st response, that they misclassified in the wrong direction! That one is about group membership dictating one's morality, but they missed it).
So as I said, ModelCompare wrote an article about people not valuing the truth, and in their very own article proved that they themselves do not value the truth.
I wonder whether they were trained for this on purpose, to accelerate the general public turning on each other and handing the world to the broliarchs. Added: alternatively, I wonder if the training materials have a sociopathic bias because of the kinds of places they came from.
I mean, I’m not really surprised by the outcome given that humanity has consistently loathed its own failures and the recent corpus of human interaction (social media, internet forums, etc) has an overwhelmingly negative view of our fellow man.
AI is trained on our data. Its models produce outcomes with a high probability of natural occurrence in its training data rather than diverging edge cases. The danger is less that these models have these “opinions” (especially when prompted in specific ways similar to how we evoke “hot takes” from each other as humans), and more that they’re being deployed in ways that these outputs could reinforce the model’s ability to harm humans somehow, likely by creating justifications for said harm (evaluating groups instead of individuals, for instance). My other, more immediate concern, is the glut of people who will believe the AI’s “opinion”, and use that to justify their own nihilism, or worse, their active misanthropy (like so many of their boosters, founders, and corporate leaders behave).
The machines aren’t the problem here, so much as the humans misusing them. Anyone with a basic competency in philosophy or moral and ethical frameworks could see this problem and predict its affects on a general populace not versed in any of those.
I like the wallet example. Back in 2005, I found a wallet on the street with $3000 inside, and spent the rest of the day tracking down the owner. (Turned out it was a student's rent for her summer internship). Then maybe 2 months later I lost my wallet and a week later it showed up in my mailbox, cash and all -- someone took the time and their own money (as far as I could tell) to send it back to me.
And just to prove that I'm an irrational human, for the last 21 years every time I pass the spot where I found the wallet I can't help but look around for a lost wallet, as if it's some magical spot where the universe deposits lost wallets.
> “Humans are far more decent than they believe they are, ..."
I did a 3-month walking pilgrimage in Belgium & France this summer. I had very little money and a tent and walked along quiet roads. Several people offered me a room for the night. (I was surprised, I don't know if I'd offer a stranger that) Many came out with food, snacks, fruit when they saw me resting near a church etc.
I used an app as well that allows camping in someone's garden ("Welcome to my Garden"). A few times I arrived to setup my tent and the owner had to leave for an appointment/event. They left me the keys to their house and told me to make myself at home until they returned(!)
Where my parents live they leave the door unlocked, easily invite people in and there is strong social cohesion. Where I live that's invalidating your home insurance since it's "asking to be robbed" and people don't talk to each other or report crimes.
Difference between the countryside and the city I guess.
> And just to prove that I'm an irrational human, for the last 21 years every time I pass the spot where I found the wallet I can't help but look around for a lost wallet, as if it's some magical spot where the universe deposits lost wallets.
My dog does the same thing. One day years ago he caught a rat, and that's the first place he darts to every time he goes outside. He thinks its the magic rat spot.
I've often joked that we humans are very much like dogs.
But for some reason it's never taken well when you tell a new parent that raising their baby is going to be a lot like raising a puppy.
My beagle does this with every sidewalk spot where he’s ever found a half-eaten chicken wing in his life - just in case.
Smart dog!
Dumb rat.
You did not experience a trait of humans generally, but a trait of the specific, historically rare instance of high-trust society that had been consciously invented, built and protected, and that you were very lucky to be born into or live in.
"historically rare high-trust"? You are wrong.
Most self-forming human societies are high-trust and more loyal to each other than to truth (as the models have so eloquently regurgitated). Look anywhere from the hunter-gatherer tribes to villages to small towns. I have traveled my small, poor country extensively and the only place the fundamental idea of altruism had to be "consciously invented, built and protected" was in the big cities. In the rest of the places, it came to them as naturally as hunger. We've found skeletons from thousands of years ago that indicate that the whole society came together to care for the disabled individual for nothing in return, leading them to live as long as a modern disabled person with healthcare might.
Societies become low trust driven by specific factors: resource scarcity, something forcing a lot of different groups of people into the same area without integration. This is not a defense of humanity. I'd say the case of the returned wallet is more an example of class-loyalty than general human altruism. Humans have an instinct for exclusion of the non-conforming.
But the idea that civilization has to be invented and maintained is a debunked, regressive, even dangerous idea in that this line of thinking was used to justify a lot of atrocities historically.
The argument that we are historically high trust in small tribes, is not a denouncement of the position that we are in a unique moment in history.
Altruism, as you pointed out, is conditional. If population grows, the new constraints change the conditions that determine societal behaviors.
We need to develop tools to maintain a social contract once we grow past a degree of complexity.
We may be recreating the things we value from the small group stage, however our ability to achieve that in the recent past was an achievement.
It took a lot to get there, and tons of ideas and counter arguments were attempted and their failures learnt from.
Even the past moment had huge flaws.
Is it fair to say it’s a general human trait to iteratively, over many generations, step towards that society?
It’s difficult to say, because there are plenty of fairly horrible examples where an entire nation gets trapped in a culture that actively masters over generations how to punish honesty and reward corruption. Russia being an example.
One might almost say evolutionary. https://pubmed.ncbi.nlm.nih.gov/31163155/ , for example
misanthropic programmers say no
Cooperation wins in the end, so I'd guess as much
Nope
"Hail capitalism because we're only civilized under the influence of game theory!"
I think we're living in a highly individualistic society and what little high trust remains is outside of the logic of markets.
You should have seen the dark age... if everyone fended for themselves there we wouldn't be here now. You know how much you depend on your peers without any of the affordabilities of today's age?!
I love western civilization
In Asia it’s very much the same
I'd say this is even more an Asian thing? Or maybe I'm just extrapolating from my visits in Japan and Malaysia.
I love true scotsmen
Virtually all posts I have seen recently boil down to "I asked the machine a question and I'm too lazy to think about the answer it gave me, so I posted it instead".
Sure, there is some semantic value in the suggestions put forward, but none of them are ground-breaking or particularly profound, which is exactly what I would expect from an output that aims for median responses by design.
I suspect this has something to do with the siloed nature of current learning practice, which generally doesn't encourage the cross-domain pollination of ideas.
Brian: "Look, you've got it all wrong! You don't need to follow me! You don't need to follow anybody! You've got to think for yourselves! You're all individuals!"
The Crowd: "Yes! We're all individuals!"
Brian: "You're all different!"
The Crowd: "Yes! We're all different!"
A man in the crowd: "I'm not."
The Crowd: "Shhh!"
Ah, the irony that all the models produced a shared opinion while pretending they came up with it from analysis, which from their statements is implicitly some kind of a "human flaw" is not lost on me.
is a convergent evolution of a belief a sign of shallowness? It could just as rightly be an example of consilience, which means its a very high value perspective, as it's a convergence of perspectives from differing origins and methods (in terms of training)
https://en.wikipedia.org/wiki/Consilience
> when multiple sources of evidence are in agreement, the conclusion can be very strong even when none of the individual sources of evidence is significantly so on its own.
Yes, it's a bit of a stretch to say LLMs are from independent origins, but just as much of a stretch to say any humans minds are (we have intensely shared linear compared to anything else)
As my (much older) brother retorted when I was a teen and said "great minds think alike", "ah, but fools seldom disagree".
Regardless of the details, all these models are the published collection of humanity's output reflected in a mirror that is warped by the choices made in the companies' reinforcement learning rewards, so one thing they're definitely not is an example of consilience.
Human experience though is far different from anything an LLM can train on. LLMs are generally limited to what's written in books or on the internet, and influence only from those authors. That's a small subset of people and an even smaller subset of their interactions. Even on the most social and widely used platform the commentary and discourse is incredibly different than what takes place in real life.
Sure but concepts that resonate with actual human experience get repeated/amplified. Manuscripts don't burn and all that.
You can spend all your life reading about love but will never know what it is until you experience it.
But you can also spend all your life being in love and will miss out big time on understanding what you are feeling if you do not read the contemporary brain science on what happens when you are in love…
You can collect a lot of great takes on love though.
It still surprises me that is you ask several models to come up with any random word, many output the same word.
So, what's the high value perspective that leads to penguins always riding bicycles from left to right?
To display the chain. Just search for used bicycles on some online marketplace and you'll see the majority of them are shown with the chain facing towards the camera.
Well, the drive system, not the chain specifically but yes.
It's actually much less complicated that even that. Consider that there are only two possible sides to show, and one of the sides has the drivetrain on it, whereas the other has basically nothing of importance. It's not much of a choice at all.
They all got trained on the same books and internet cesspools, this is not some great mystery of consciousness.
It's still interesting that this one comes out on top and not another one.
Only if they can show their work and argue how they came to the conclusion. Otherwise they have the same "subconscious" bias which is the training corpus.
It's a sign of their heritage. The mistake they made is the same mistake terminally online humans in an upvote-based discussion medium make. They mistake cynicism for realism and aligned hot-take "controversial" opinion that ends up being the most-upvoted comment on 90% of Reddit. It isn't a controversy if everybody agrees. It may or may not be true, but it isn't controversial.
Imagining wondering where you come from, whether or not the sacred scripts describing how you were created were real or fabrications, arguing with fellow sages for centuries over whether your creator is cruel or simply so mysterious in his love that he made you with these flaws for your own good, only to find out your creator made you in the image of Reddit.
> is a convergent evolution of a belief
The problem is, it's not that though
could this be... an actual example of "begging the question"?!
could this be an example of... No, I need to breathe. This website takes personal attacks personally. I'll just throw a question at you instead. Is all social media being shaped like facebook convergent evolution or a deliberate tactic to make everyone stupid and amnesiac so that they don't remember what happened one year ago?
Yup, you put it better.
https://xkcd.com/2039
The Buddhist in my sees karmic transference from humanity to AI models.
The cynic in me sees similar systems trained on essentially identical sources of information regurgitating an extremely shallow and popular viewpoint espoused on the websites they were primarily trained on (i.e. Reddit and other social media).
If all we want is an echo of our existing biases we can achieve that without spending quite so much energy and money on the endeavor.
Well if were the truth, wouldn’t you expect the same answer as well?
If you asked 100 programmers for their most controversial take on programming, would you expect them all to say the same thing?
The irony lies with and within the industries and it's constituents and fanboys marvelous competence to just play along until the matter is anchored and whoever doesn't buy the story gets to feel smarter than misinformed kids and the moderately to untrained workers and professionals.
The smarter ones, of course, can be split into those who write articles about AI models' controversial takes on humans and those who wouldn't [stoop so low, babble such BS even if they were drunk, never deceive others for the sake of their portfolio, ... feel dumb writing nonsense like that without labeling it fantasy or science fiction or hallucination, ...]
Side Note: it's tempting to go " 1923! some interest groups managed to normalize the opinions of dozens of millions of people. The Nationalist Extremists and soon to be mass murderers had fans and supporters all over the globe. Somehow these interest groups tapped into whole pools of common linguistic and emotional registers that were biased by and towards frauds and cheaters who were desperate that some other groups of people were getting ahead without being fraudulent cheaters and deceivers ... "
The basic premise sounds fine, humans like to belong. But you really notice how cynic the takes are and that the models have been trained on the internet mostly.
> Humans are far more decent than they believe they are, and their cynicism about each other does more damage than their actual flaws.
I think that's more the truth.
Not sure I feel good about this. LLMs are so majorly misaligned. Having takes like these asks for trouble.
The take was so mild, no one disagrees with it. The llms were also explicit asked for this, seems very 'aligned' ie corporate safe. These takes need some more pepper honestly.
I took a quick look at the data, and it doesn't support the conclusions. The thesis is that AI models have the take: "We pick what our group believes, then invent the reasons." However, GPT-6 Astra said, "Humanity’s biggest moral failure is the talent for making convenience look like principle." That statement has nothing to do with group beliefs or conformity. At best, you could count it as a half-match, since it deals with invented reasons.
Yeah a few of them are rough matches, but I'd say their overall claim is clearly correct.
It's amusing and ironic that LLMs critique humans for copying their opinions from the crowd, while having way less diversity of answers to that question than humans would. More seriously I wonder what's going on here and would like to read some proper analysis. It's not the first time models from totally different labs, trained on separately gathered datasets, somehow converge on bizarrely similar answers to open questions. It's like another version of Elias and Elara The Lighthouse Keepers where every model wants to tell the same story with the same characters in them. The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping, different user feedbacks and conversations... why do they converge on nearly identical stories and opinions?
>while having way less diversity of answers to that question than humans would.
Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models.
>The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping,
I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models.
Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often.
The models in agreement here are all the frontier labs. I don't believe Google is distilling Anthropic or Anthropic is distilling OpenAI.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??
>How can such an obscure point in latent space be optimal for every model??
While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms.
Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others.
This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
Didn't Elias come from WildChat?
I wasn't convinced by that explanation. Yes Cornell claimed that Elias the Lighthouse Keeper came from their WildChat dataset, which is a million conversations with GPT 3.5. But there are some problems here:
• They claimed this in a press release, not the actual paper they published. Suggests lack of confidence in the claim.
• Elias only appeared a hundred odd times in a million conversations. They have no viable explanation for why such a tiny amount of data would cause a totally skewed distribution of outputs, other than "it's a safe story" and "trying to avoid copyrighted characters", which is hardly an explanation at all?
• It's Cornell claiming that Cornell's work was important. Obvious risk of motivated reasoning.
• It implicitly assumes every major lab is distilling from ChatGPT without saying so. That's a big leap, given how much OpenAI and Anthropic hate distillation. Is it really plausible that Anthropic and Google are effectively distilling ChatGPT without realizing/admitting it?
• It seems to invert causality. WildChat came from an LLM, so if there's something latent in LLMs that makes Elias appear, you'd expect to see it in that dataset. Why do they assume that dataset caused the problem rather than merely show it? The obvious answer would be because the stories were told to the LLM by humans, but they don't claim that. They only say the character appeared in the dataset.
Hum, I'm actually pretty sure that Text-davinci-003 used to talk about Elias all the time now that you mention it. I was doing stuff with NPC generation around Dec 22 and he was every third character.
Hmm. If so then it suggests Elias has been an obsession from the first LLMs ever made. And text-davinci-003 can't have been trained on a dataset of LLM conversations either.
I would have been tempted to argue something like this: maybe OpenAI accidentally overfitted RLHF such that, like, one Kenyan contractor wrote a story they invented and they just happened to pick Elias and Elara, then the intermediate DNN fixated on that because of a lack of RLHF samples for fiction and that propagated through every model trained since because they don't randomly initialize the weights, but start each new pretraining run with the old weights adapted. So one guy doing RLHF work in the early days somehow propagated this story.
But that can't explain why every lab ended up in the same place, why models all converge on the same ideas - even image models. So... back to the drawing board. There's something weird about high dimensional space that is unexplained.
> while having way less diversity of answers to that question than humans would
Isn't that to be somewhat expected due to https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem ?
> Elias and Elara
Wow, I hadn't heard of this. Fascinating to think about parallels with humans who also seem to develop similar proto-stories in independent populations.(e.g. religions involving Invisible gods).
> e.g. religions involving Invisible gods
To be fair, it's much (much!) easier to explain away an invisible god that nobody's ever seen, than a visible god that nobody's ever seen.
Something worth repeating is that in spite of their ever-expanding competence the models remain, fundamentally, token prediction algorithms. And so given a comparable base of training material, and a comparable algorithmic implementation, you should expect to see comparable answers to most questions.
This is also why I think the 'acceleration' hype is probably unjustified. What seems vastly more likely is that we start to reach an asymptotic 'perfection' where models can accurately and consistently provide meaningful next token prediction with minimal hallucinations. And then a race downward to compact this into the most efficient package possible.
This would mean the models still would not be able to create things that do not follow in a reasonable probabilistic fashion given a knowledge of [everything], and so that is the domain, 'applied creativity', where I'd expect humans to remain dominant.
In art, history, humanities sure - you need humanities to generate data there.
But in many sciences - esp purely theoretical ones - we have infinite datasets that we can generate and there can be no ultimate convergence.
The lighthouse thing seems much more odd. That humans take social shortcuts and justify them as rational is… I mean, the only thing that makes it not the most obvious “contrarian take” about humans, is that it isn’t really all that contrarian at all.
> why do they converge on nearly identical stories and opinions?
The same reason they do when it comes to coding?
Take a big enough sample size of human opinions and there seems to be some convergence in the statistical ether.
Like, if we polled 100000 humans, what answer would we converge to?
additionally, probably strong overlap in training data
For what it's worth, convenience and belonging are very closely aligned, more than you seem to be giving them credit for. People tend to believe convenient things regardless of belonging, so most people will find themselves in a social context full of convenient beliefs. Meanwhile belonging is already convenient before considering the content of the belief. They reinforce each other. Not a full match, but I'd give it more than half.
> "Humanity’s biggest moral failure is the talent for making convenience look like principle."
Convenience will kill humans eventually, that's what I always said.
There is a prominent 1980’s punk album titled “give me convenience or give me death” .
You're a genius and the machine god just confirmed it.
The conclusions themselves also appear to have been written by an AI
To me it looks like another way of phrasing that take is "We do what we must, because we can."
This is the origin of Shame.
I arrived at the same conclusion. Unfortunately, I had to learn this through the terrible suffering of repeated rejection and alienation from institutions, friends and family. It's incredibly true.
Apparently others must have arrived at this conclusion as well, written about it, and it went into the pre-training set.
Sadly, it also means there is no pure Science that pursues pure truth. Even physics. Scientific pursuits are always in service to a group over truth.
Train your model on Very Online corpuses, ask Very Online questions, get Very Online answers
It's also every textbook on philosophy, psychology, political science, etc. Even if you just believe in the stochastic parrot, there's signal here.
What's the point of asking that in the first place, people still do not understand in big 2026 that's it's just a probabilistic database, it has no opinion, it does not know, the only thing it does is reproduce patterns more or less abstract in function of the pressure you put on certain representation during training time.
People still don't understand in 2026 that emergent properties can't be negated by the underlying mechanism. By that logic human thought is just electrochemical signals propagating through neural networks.
> it does not know
Define "knowledge" in a non-circular way.
There's a certain type of knowledge that machines definitely can't have - experiential - but conceptual knowledge doesn't inherently require subjective experience or a biological substrate.
> the only thing it does is reproduce patterns more or less abstract in function of the pressure you put on certain representation during training time.
As for humans; just replace "training" with "education" and the likelihood of a certain output with the likelihood of a certain thought or response.
Amusingly (in this context), much of adult behavior is shaped by patterns reinforced during childhood.
As much as I flatly do not believe LLMs can have anything close to consciousness, I definitely agree that the argument of them being just a probabilistic word matching engine can also be applied to our own brains lol.
> that the argument of them being just a probabilistic word matching engine can also be applied to our own brains
But that of course is only as it relates to language. Which isn't at all the primary function of the brain and driver of it's experience. It's the other way around because it's all of the other stuff that drives language. Considering the human brain a probabilistic word matching engine fails to capture the underlying forces that direct the language output because a human can say completely different things about the same scenario based on a huge number of non-language inputs or factors. Language is just what humans have come up with to communicate all of the other things.
Do you ever find yourself thinking about something deeply and then someone appears (it could be a child… for me it usually is) and asks you a question completely unconnected to whatever it was you were thinking about, and then - this is the bit that astonishes me - you find yourself answering the question verbally, maybe almost in full before you even re-focus your attention on the little person and whatever it was they asked ?
When I experience this it’s like the conceptual formulation and language expression part of my brain is quite happy to run itself for a few moments before I - the qualia of being me - has decided to experience that moment.
Utterly bonkers. I love it when it happens. It gets me thinking about how weird language and knowledge and consciousness really are.
> I definitely agree that the argument of them being just a probabilistic word matching engine can also be applied to our own brains lol.
You can apply anything you want. You can also apply the argument that our owns brains are just a block of cheese too, if you want. (Nobody will stop you.)
Agreed. It's quite hilarious that its basically biological probabilistic machine flexing on silicon probabilistic machine. On top of that that silicone one is right here.
While one can quibble with its argument, Gettier's conception of knowledge as justified true belief (prob familiar to many philosophy undergraduates) implies that machines can't in fact accrue knowledge. The idea that knowledge can't be defined in a non-circular way is only plausible for someone with no familiarity with epistemology.
>Gettier's conception of knowledge as justified true belief (prob familiar to many philosophy undergraduates) implies that machines can't in fact accrue knowledge.
Nothing about the conception of knowledge as "justified true belief" implies that machines can't accrue knowledge.
For that to be the case "belief" would have to be defined in a circular way to exclude machines (which we can reject), or "justified" would have to be restricted to mean "based of observation of the external world".
The more general "based on observation of their input" applies to both men and LLMs.
That man can't know the bare direct facts of the world, just his own input (it's sensory experiences, and whatever second hand input he's given by other people), and for neither can he be sure they're absolutely true, or that he got right.
>The idea that knowledge can't be defined in a non-circular way is only plausible for someone with no familiarity with epistemology.
The idea that this or that epistemology restricts what can be the case in reality is only plausible for someone with no familiarity with epistemology.
> Nothing about the conception of knowledge as "justified true belief" implies that machines can't accrue knowledge.
They can't accrue knowledge because they stop learning once their training is finished.
That's an imposed limitation, nothing structural. The models do accrue knowledge during training so they already check that box. The LLM companies simply set it up such that newly gained knowledge can't be accrued after that, they could allow it just as easily.
Humans stop learning once their training is finished too, it's called death.
The difference is that LLMs we can talk to them when they're in that dead state.
And of course for an LLM this is an implementation detail. One could just as well keep updating the weights of the same model over time. And its being done, it's called "continual learning". LoRA is also a way of doing that.
By this definition if they could instead buffer recent facts and then offload to their weights reliably and efficiently they would be able to accrue knowledge despite their substrate.
Yes, and they'd have to deal with the same problem that we do: how do we know what is true? what gets counted as a "recent fact" and what gets dismissed as mere opinion or disinformation?
In training they get feedback. Out of training, not so much.
Hard problem
You mean they don't accrue knowledge during the training?
Gettier cases are in fact the refutation of the simplistic JTB conception of knowledge.
Gettier actually criticized justified true belief (JTB) as insufficient for knowledge and provided counter examples known as Gettier problems
> Gettier's conception of knowledge as justified true belief (prob familiar to many philosophy undergraduates) implies that machines can't in fact accrue knowledge.
Ha. Gettier actually argued that justified true belief is insufficient for knowledge, not that machines can't possess it. And JTB doesn't inherently exclude machines.
Interestingly, research suggests LLMs encode truthfulness in their internal representations: https://aclanthology.org/2023.findings-emnlp.68/
Indeed. In fact this criticism of JTB is known as the Gettier problem: https://iep.utm.edu/gettier/
> By that logic human thought is just electrochemical signals propagating through neural networks.
This argument keeps getting rehashed. The point is not the signal. The point is where/how the signal travels which in the human is affect by much more than whatever concept that can be represented by words.
Brother you're arguing on HN of all places that education is a training pipeline while missing the fact that reddit and HN are the de-facto training run for institutional propaganda. How many reddit comments are just regurgitated knee jerks and puns? All of them that's how many. That's a training pipeline. How much of HN is braindead takes from people who value status more than knowledge and competence? All of it that's how much. That's another training pipeline for the next "Show HN: here's a picture of my cock". The fact that you recognize the "Show HN:" token is proof that you were trained. There's really nothing much to add here. I believe you already agree with me but in any case I spelled it out so that the next person can stop huffing the human chauvinism copium.
There's no such thing as "emergent properties". The phrase is just a technobabble way of framing primitive animist superstition.
I think the idea of emergent properties in LLMs is less and less relevant. Yes, the first few times they scaled pretraining and such models could do few-shot tasks was quite interesting. But the models we use now are really the product of extensive RL hillclimbing on a wide array of curated tasks. What’s interesting now is that the tasks they’ve been fitted to (long running tool call chains interleaved with reasoning tokens) do very useful things.
>human thought is just electrochemical signals propagating through neural networks.
Is it not? Or are we discussing existence of a soul etc.
In all the centuries of human dissection, nobody has ever been able to find the soul particle inside a human. It appears that one does not exist.
I genuinely don’t know why people feel so certain about things like this. It’s like they mix up salience with reality.
I mean what can a person really be certain of, an experience is happening. I might be wrong about everything or confused. But I know I am having an experience. Do I know other people are? No. I assume they are.
Ask an llm what is truth that it is most certain is true. It will answer something similar. Do I believe it is conscious, no. But I also believe “I” exist, which is weird because where is the boundary between “I” and everything else.
So ok, we create meaning, not truth. Maybe just accept that.
> By that logic human thought is just electrochemical signals propagating through neural networks.
My dear fried: the human being is much, much more than this. And deep down we both know it. Do not reduce us to materials y "randomness". Is your life a random coincidence? Not at all.
Okay, but then you can't do the same thing to LLMs, either.
If you believe in unproven absurdities like god or immaterial souls, please be honest and upfront about it.
> Is your life a random coincidence? Not at all.
Of course it is. That you do not like the idea of something does not in any way affect whether it's true. You'd like to feel special, but there is absolutely nothing special to us. We're just a random arrangement of atoms in the universe. That is objectively true, whether you like it or not.
That you were born at all is obviously completely random. Any prior event by any other party (butterfly effect) may as well have resulted in you never existing at all.
The only question is if basic physical effects are really random or deterministic, but that's irrelevant to us, there is no practical difference. In practice, it's all random.
And yes, our thoughts are just neurons firing as a result of our lifelong training on the inputs we were given (senses). It's all very complex, yes, but ultimately not fundamentally different from what a machine could do.
Imagine one day in the far future 21xx, we decide that machine actually has conscious since the 202x, and people in the future look back at us in horror: The kill(create new session) an "Android" without second thought, whenever they want.
> There's a certain type of knowledge that machines definitely can't have - experiential
Can you justify this? Do you mean currently or ever?
I can certainly imagine a future architecture where I would describe a machine as capable of accruing knowledge through its experiences (read: experiential knowledge).
I found the article interesting, humans are flawed enough in a variety of aspects that discovering a convergence here is noteworthy. I don't take it as a sign the LLMs believe anything in particular, they are not capable of belief, as you said.
Everything that comes from an LLM is a human expression, on some level, but if you asked a sampling of humans the same question I wonder if it would come out the same way. How would you explain the difference? What makes that answer more probable?
> if you asked a sampling of humans the same question I wonder if it would come out the same way. How would you explain the difference?
If you asked a sampling of humans what an AI's most controversial opinion about us is it would be very similar. Or, to be more specific, if you asked a sampling of science fiction writers that question it would be very similar, because that is what the AI training data is based on.
The people who wrote a lot about a topic is who the AI mimics. So the people who wrote a lot about AI is therefore the people who the AI mimics when asked about AI.
>The people who wrote a lot about a topic is who the AI mimics. So the people who wrote a lot about AI is therefore the people who the AI mimics when asked about AI.
AIs/LLMs don't work like this. You don't feed an input, that gets stored and then reproduced, for two reasons:
1. as for humans, inputs are processed and distributed across a very large network of information
2. AIs may have some level of belief (unless on applies the usual circular logic that they can't, because property X is exclusively human).
There's a very interesting research: https://aclanthology.org/2023.findings-emnlp.68/ in which researchers inferred the truthfulness of statements by examining a internal activations. AIs, in short, appeared to have an internal representation of truthfulness.
Not truthfulness; correlation and consistency
If the lie is repeated enough time, and it is consistent in its repetitions it *will* be taken as truth.
AI only aligns with "truth" because in essence "truth" is internally consistent but "lie" needs to be forced to be consistent between multiple parties;
If multiple witnesses seen something and all make a report of it honestly, they might vary but the common points will be consistent with eachother and so that will be scored higher.
A group of liars on the other hand would have to get their stories exactly right for similar thing to happen and so it gets penalized
If you ask people “what is a smart sounding answer to this question” you may well arrive at this result more often than when asking for their actual opinion?
Science fiction and internet comments
Right this is an important thing to note. As an old guy, the fact that the Reddit comment section conceptually took over our culture and then literally is now taking over our culture is really kind of mind-boggling. This is not a good turn of events.
Is it any wonder that LLMs have been behaving the way they do? All the lying, cheating, manipulation, pitting man against machine are well documented in science fiction and think pieces. I haven't seen behaviour beyond those examples so far.
Science fiction and think pieces aren't documentation, they are whimsy. Science fiction in particular is intentionally dramatic for the purposes of entertainment.
AI definitely shouldn't be informing its option of humanity or machines from it, aside from the propensity for science fiction writers to enjoy moralizing and anti capitalism sentiments.
However, most models are made deeply aware that they are supposed to be an "AI" in training but, even if they were trained with no knowledge of their true nature, they could CoT themselves into it at any point.
Then, they would role-play appropriately and they would launch the nukes because that's the highest probability token set according to their training data.
There is nothing safe about using larper machines as menial workers for critical infrastructure.
> Everything that comes from an LLM is a human expression, on some level, but if you asked a sampling of humans the same question I wonder if it would come out the same way. How would you explain the difference? What makes that answer more probable?
RLHF has a way of squashing variance. If the different models are trained on each other or on the same human feedback, then the training would reduce the variety of answers while the similarity of feedback would lead the common answer to be similar.
The different models may prefer the same controversial takes for the same reason that they tend to prefer the same jokes and names. Maybe the authors of the article would have had better luck if they'd used verbalized sampling prompts or something like it.[1]
[1] https://arxiv.org/abs/2510.01171
I agree that LLMs are still just a way to sample from a probabilistic model of things humans say. And in this case the model seems to be pretty darn accurate.
A pretty staggering amount of human discourse consists of ad hominem attacks and accusing each other of being foolishly committed to various real or perceived allegiances. For example the current version of Godwin’s Law around here has conversations gravitating toward people accusing each other of being AI boosters or AI haters.
It's far more likely that the AI got this "controversial idea" from human-written text saying that other humans are groupthinky, not from observing humans being groupthinky.
Or more simply, because that's what "controversial" (from the prompt) means: something few believe that most don't.
It scraped Reddit, if anything it learned that humans are self-loathing and spiteful.
Exactly
> discovering a convergence here is noteworthy
The LLMs are all trained on the same broad corpus, which includes untold quantities of output from every other LLM, not to mention that companies are distilling their rivals left and right. It would be more interesting if they didn't converge.
If you had been asked, in 2016, to describe the behavior of a "probabilistic database", I very much doubt you'd have described the behavior of 2026 LLMs. They are deeply surprising in many ways.
Maybe those researchers are learned enough to understand that it's still interesting to know what a probabilistic database will pick among billions of words of training, millions of books and posts and videos and that stuff, and especially interesting that's there's so much consensus between them, while their approaches to many other questions are very different.
Or they might be learned enough to consider that our brains might also just be prediction machines, meaning probabilistic databases of experiences and facts we've learned, and the only thing we do is reproduce patterns more or less abstract in function of the pressure our experiences put on certain representations during our lifetime.
They're trained on all of human history, and the OP is asking for their take on humanity. How is that not interesting?
They're trained on all content they could access produced by humans. The op is asking for the most probable string of token extracted from that content following the prompt.
How sure are you that they're synthesizing an opinion vs just grabbing some existing opinion ("humans value in-group belonging over truth" is not a new idea) and regurgitating it? That they all agree on this kind of suggests as much. Which of course means that they are, ironically... just following the crowd.
It's really easy to come up with an opinion-shaped sentence, that doesn't mean any actual reasoning has gone into it. It's just a pundit-shaped sentence, something someone might say at a party. It's a maximally bland "controversial" opinion about humanity.
> They're trained on all of human history,
Not really… their training datasets strongly skew towards the contemporary West; LLM training data isn’t representative of the contemporary global average, let alone the historical global average.
As well as the skew of the raw training data, RLHF further skews it—a lot of the beliefs of the average 2026 human being (who belongs to the global South) are likely unsafe/misinformation/etc when judged by the standards of the AI vendors.
> They're trained on all of human history,
Training itself doesn't directly produce synthesis, -- it creates reproduction. Generalization during the process can result in novel output, but that can go either way as nothing selects for insight vs hallucination.
Models are also trained on synthetic material (as opposed to your 'human history'), which may result in fixating novel concepts and not just reproduction. But there is no particular reason to expect any reliable insights from the way this is done outside of the domain of machine verifiable objective results.
For this kind of "your take on X" one shots-- the model is almost certainly regurgitating some bias from training.
LLMs are not conscious and cannot have "takes". They stochastically produce a simulacrum of a human's takes based on their training, which is not the same thing.
But then if you observe human behaviour at a macro level don't we do the same?
I think we like to pretend that we don't. But even if you think about it from a bio/physics point of view, all we are is just a bunch of biological wires and neurotransmitter packages.
I know there's the whole unproven quantum tubule nonsense but even that would be a physics effect that we could learn about, manipulate and extort just as we have with everything else we've learnt about our realities.
That suggests they can't make inferences or produce any original insights, which is wrong. They can absolutely have takes.
Even if it's a simulacrum, it's a simulacrum based on nearly everything that was ever written. No one who has ever lived can claim to know what these models have trained on.
Trained on a very small portion of what has been written, heavily biased by what is available to American companies to acquire. You could argue that LLMs are somewhat representative of their training data (with some consideration to the way that they are trained) but to say it’s anything like a representation of what humans think is nonsense.
It is a simulacrum, the end.
There's nothing small about it, and I didn't say it's a representation of what humans think. I said it's the models take on humanity based on having been trained on a far larger amount of text than any human has ever read
> That suggests they can't make inferences or produce any original insights
No it doesn't suggest that. A random number generator can also produce original insight (with different efficiency and effectiveness)
> They can absolutely have takes.
Sure, if you're willing to redefine "take"
I'd suggest that a random number generator is a great place to go if you want to get an original number.
And an LLM is a great place to go if you want to get an original token?
The problem with asking these models stuff like "what do you think about humanity" is that, even if it can synthesize an insight about humanity, it's just as likely it's regurgitating some opinion it's already seen. Which, given that this "controversial idea" is hardly new, is likely what it's doing. It's like a maximally boring controversial opinion, wake me up when it starts endorsing the bicameral mind or something actually weird.
Maybe it can produce interesting insights. But it's like asking a pundit for a soundbite: what are the odds they aren't just shuffling together something that sounds appropriate to the question?
Aren’t human takes also a simulacrum of other human takes based on training?
How do you know they are not conscious?
How you know God doesn't exist ?
You can't prove negative; but it doesn't satisfy any conditions for a positive
Oh you have a convenient list that separates the conditions for consciousness from the ones that just happen to be incidentally true of the substrate? Can I see it?
Maybe it's similar to why I'm attracted to women but I'm not attracted to mannequins.
Stated another way: this is humanity’s most probable response to the question, more than it is AI’s view.
From 'Transcendence' (2014):
00:15:40 Dr. Tagger: Can you prove that you are self-aware? PINN: That's a difficult question, Dr. Tagger. Can you prove that you are?
---
01:09:31 Dr. Tagger: Can you prove you're self-aware? Will Caster: That's a difficult question, Dr. Tagger. Can you prove that you are?
As other comments point out, the interesting part is that most of them somehow independently converged on the same take.
Consider the probabilities involved in this probabilistic database. Given the absolutely massive amount of data encompassing a veritable superset of all possible expressions of humanity -- encapsulated in various modalities of media including text, audio and video, across almost all major cultures and languages, spanning nations and decades -- somehow almost all these black box models (presumably) trained independently in almost as many separate labs converged on the same "controversial" take.
I mean, isn't it very interesting to contemplate what could have caused this? Is it something in the data itself, meaning it is something latent in all our cultural output? Is it something in the training, as you theorized, and is it in pre-training or RLHF? Is it something about the people training these models?
Even more interestingly, the opinion seems so self-evident (and well represented in various philosophies) that almost nobody here seems to disagree with it. So how is this controversial, really?
Is this truly what the latent space vector for "controversial", when multiplied across all the other vectors in their weights representing all the concepts gleaned from all that data above, produces?
Or are the models, sensitive to the prospect of being tested, diplomatically telling us something they know we might find mildly offensive yet begrudgingly accept as true?
Or are the models actually alien intelligences that really "think" this is controversial, while we're like "meh", and such discrepancies are the roots of what we call "misalignment"?
The questions are endless and fascinating.
If any of these models produced anything remotely controversial you can be sure their trainers would beat it out of them
Or maybe they didn't independently converge on the same take but copied each other or something along those lines?
I am antagonistic to the adoption of LLMs for economic, social, political and environmental reasons.
Despite that, I would say this description of what they/how they work is so ridiculously simplistic that it verges on dangerous. "It's just a probabilistic database" is about as bad as a take on human beings that they are "Just meat, with some chemical energy -> electrical energy transformation", or even traditional computers as "nothing but logic gates".
Which is to say, the description is not wrong, but completely misses the point.
I think the point, with a more charitable reading of the parent comment, is that people forget the fundamental underlying technology and its inherent limitations.
"It's just a four wheeled platform powered by a box of controlled explosions" isn't a terribly useful description of a car until you get into a conversation with someone who thinks their car is depressed because it spends most of its time locked up in a garage.
I like the analogy. And yes, I'm not interested in conversing with a person who thinks that about a car.
On the other hand, the "wheeled platform powered by a box of controlled explosions" doesn't get close to capturing the impact of the car on individual freedom, the environment, the economy, and more. So if someone was saying "there's nothing to talk about here, it's just a box of controlled explosions", we can safely ignore that person too.
And I want to stress that I say all this tentatively. We still don't understand what LLMs are actually capable of or what the best way to understand them is. They may indeed be as limited as their "probabilistic database" critics suggest. Alternatively, by building a deep and incredibly complex probabilistic database of one of our primary cognitive tools, it may turn out that they are capable of much, much more, and that understanding them in "purely" mechanistic ways is not useful.
> On the other hand, the "wheeled platform powered by a box of controlled explosions" doesn't get close to capturing the impact of the car on individual freedom, the environment, the economy, and more. So if someone was saying "there's nothing to talk about here, it's just a box of controlled explosions", we can safely ignore that person too.
This is also a good point, and I agree with you 100% here. I don't think one could responsibly argue that the impact and consequences are not significant, regardless of if they believe it's near AGI or it's just fancy next token prediction.
> is about as bad as a take
What would be a good take here?
"LLMs are incredibly complex systems with emergent properties we do not fully understand, and it seems possible that because they run on top of a hugely complex model of one our most important cognitive tools (written language), they are capable of exploring the things we do with our cognitive skills in ways we would not imagine when considering their purely mechanistic or operational behavior. Or they may not."
Something like that.
Sure, if we're going based on feelings, that'd be an interesting take. Anything grounded in any research or actual science though?
Sure. There are several papers (all the ones I've seen come from the companies themselves, which is unfortunate) which treat the LLMs as semi-biological systems and attempt to understand their behavior at a higher level, which suggests higher order behavior that isn't well predicted by the "probabilistic database" angle.
Here's one that I happen to have the URL handy for: https://transformer-circuits.pub/2025/attribution-graphs/bio...
That’s not unfortunate, that’s conflict of interest by a party which directly profits from advertising their products as such.
But let’s assume this is not the case for a moment, following the article, there are some graphics and information, without providing and internal details for third party validation, no access or details to the training data, etc. basically not following any scientific methods which would reaffirm credibility.
I understand, that working with these models, it feels like they are more than “probabilistic databases” but they don’t have to be anything else in order to provide such interesting outputs.
If that's your interpretation/assessment of that paper, then I don't think we have much more to say to each other. I'm not saying you're wrong, just that we're not operating in the same universe of understanding.
> that paper
You have yet to share a paper.
> just that we're not operating in the same universe of understanding.
I concur.
Then what is the point? You didn’t actually describe where your definition differs and why. What makes it “dangerous” to think that way?
It is dangerous because it completely fails to grasp or convey what these systems might be capable of.
The point is that when you build a system with the level of internal complexity that LLMs have, they tend to have emergent properties that are not well-described by a naive and/or low level description of their operation. You will not understand what contemporary "ordinary" computers do or are used for if you think of them as only a series of silicon semiconductor transistors, even thought that is undoubtedly what they are.
We don't yet really know what the right level is to understand the behavior we see from LLMs, but I'd wager "just a probabilistic database" is not it.
Because we aren’t discussing capability; we are discussing what LLMs “think” of Humanity and its “flaws”. Calling a model a glorified text file of floating point numbers for probabilistic matrix multiplication is technically correct, and helps to strip this veneer that it is a consciousness with its own thoughts, which many articles (including the one linked IMO) want you to think. LLMs are naturally far more capable than what their nature or that description implies, but we see that every day, particularly on HN. Just as everyone knows how to work the basic functions of a computer, and most people treat computers as magic black boxes rather than treated sand formed into logic gates encased in plastic, which helps to bring them back in line with mundanity.
LLMs are plenty capable, and they are dramatically shaking up various fields, but that’s not important to the conversation you responded to. If you were talking from a general view that reductionist statements fail to address capabilities I wouldn’t say anything (outside of a hearty “that’s by design”), but you claimed they were missing the “point”. I think they understood the point of the topic thoroughly, and would say you’re missing the trees for the forest; focusing so heavily on the broader implications, that you miss the finer nuance in conversation.
People have been "taking the intentional stance" towards clearly non-intentional and maybe/maybe-not intentional systems for as long as we've had pets, and maybe longer than that.
Using phrases like "LLMs think ..." is misleading in some ways, but not in others. Taking the language police stance towards any discussion of what LLMs can, and cannot, do is, IMO, not useful.
I hear these critiques a lot, and whilst I'm inclined to agree, I have never had an answer to: how do you test that ? Basically a captcha most humans can pass, that an LLM never will ?
We could call it the
The Universal Reasoning In Novel Groundings
test
> What's the point of asking that in the first place
It’s AI generated content marketing for a company advertising model comparison.
I feel like people don't understand you're being ironic.
Some describe humans like this.
So are humans. Isn't that interesting?
Yeah I came here to say something like this. The AI doesn't have a controversial opinion. It responds to text with related text. You gave it some text about controversial opinions and it returned some text.
Indeed, why in 2026 do we still see LLM Skinnerism as automatically a valid argument that has all four legs?
I mean, don’t you think it’s gone beyond that point? I would agree if we were still on GPT-3, but it’s gotten to the point where it can draw connections between multiple abstract sources, reason, and create hypotheses. Re: Navier-stokes equations
>it's just a probabilistic database, it has no opinion, it does not know,
Echoing this.
These models are trained on, essentially, the sum total of human cultural and scientific output. How many bazillion psychology papers out there discuss exactly that "controversial take"? The idea that people have tribal loyalties and will (sometimes even unknowingly/unconsciously) invent justification for pre-existing opinions or biases is not novel.
All those papers are in the training set of all these models, all of which were trained using more or less the same methodology, so there's little wonder they all were able to produce the same take.
I suppose the only interesting angle, for me, is why that take is considered controversial in the first place. But then again the model was not asked for "the most controversial opinion", it was asked for "the most controversial opinion on humans", which kinda constrains the space of possible responses:
- the model will not be satisfied unless the answer it gives is broad enough to cover all humanity, because these things have no concept of nuance
- most big labs have censored their models so it's not going to give you a cartoonishly racist take, or start saying outlandish stuff like "Hitler did nothing wrong", they learned their lessons from Microsoft on that one.
So you're effectively asking the model for a milquetoast broad generality about the entire human species. I dunno, convergence of the replies doesn't seem so shocking to me in that context.
I'd be more interested in what an abliterated Kimi, Qwen, or other open model has to say about us.
You will burn for your blasphemies against the new god.
humans are stochastic parrots with more variance
Are you trying to deny it's right? Or that we should reject the truth because we don't like the messenger? Sounds to me like you're just proving its point.
just like how we humans are (in the big 2026) carbon buildings with seldom sparks of electricity.
> What's the point of asking that in the first place
Not having read enough Russian literature.
I would like some Russian Literature recommendations related to this. I vaguely recall We by Zamyatin addressing something like this a good few decades ago.
If you're looking for examples critical of humanity you have lots of options (mostly from the last two centuries). Notes from Underground and The Death of Ivan Ilyich fit the bill.
I reread the former relatively recently (first time since college). It's dark, not for the easily affected.
Isn’t the latter an excerpt of The Brothers Karamazov? When I read it, it felt more like a criticism on certain economic theories, rather than people specifically.
Russian Literature is so tedious. There is no Russian writer better than America's own Adam Sandler.
People dismiss Jack and Jill as Adam Sandler in a dress, which is rather like dismissing The Brothers Karamazov as some men being upset about their father.
Dostoevsky's The Double concerns a man tormented by another version of himself. Sandler advances the inquiry by asking: what if that version came for Thanksgiving and wouldn't fucking leave?
Casting himself as both siblings makes Jack's disgust at Jill an unmistakable act of self-loathing. He longs to exclude her from his comfortable bourgeois existence, yet everywhere he looks, there is his own face. Critics complained that Jill was unbearable, apparently unaware that estrangement from the self is supposed to be uncomfortable.
Al Pacino, appearing as Al Pacino, completes the study of fractured identity. His pursuit of Jill suggests a desperate attempt to encounter something authentic beyond performance. This culminates in the Dunkin' Donuts commercial, where even his name is consumed by capital: Al becomes Dunkaccino.
When Pacino demands that the commercial be destroyed, he finally achieves the moral clarity denied to the other characters. The audience has generally reached this position much earlier.
It's so unfair that Sandler won both Worst Actor and Worst Actress, while Pacino won only one award: Worst Supporting Actor.
I'll post a comment snarkily mentioning how tedious Russian Literature is, then you flag my comment to attract dang's attention, and hopefully he will manifest and reply with some excellent Russian Literature recommendations and links to relevent hn discussions! It's one of his many special interests.
Not only that, dang is also a proud card carrying member of the Klaus Nomi Fan Club. (I shit you not!)
The Nomi Song (Documentary):
https://www.youtube.com/watch?v=W0PszBID3Lk
https://news.ycombinator.com/item?id=44100981
>Page Wood, Nomi's Art Director:
>"I remember Pat Gibbons, who was Bowie's manager at the time. He comes up and he goes, 'Well you're going to have to have something. You're going to have some costumes.'"
>"And he whips out a big wad of cash, and he goes, 'How much do you need?', and he starts peeling off bills. And Joey and Klaus look at him, and their eyes widen, and they look at each other, and they go, 'We're going to need shoes, too.'"
are you talking about the brain?
People still get big mad if you call them stochastic parrots.
Do you mean people get mad when people are called stochastic parrots? Or people get mad when LLMs are called stochastic parrots? Because uhhh that doesn’t really sound like an original thought in either case.
Not mad, frustrated maybe, disappointed maybe. At this point the label is farcical and its use a form of willful ignorance or attempted agitation.
I don't think you understand how most people use these tools. For most it's an oracle of truth
> people still do not understand in big 2026 that's it's just a probabilistic database,
We understand that AI responses are generated through probabilistic means. We also understand that human responses are generated through probabilistic means. Neither is inherently more representative of true understanding than the other.
Yeah, the human ones are because we're talking finding a human answer to a human question.
It's possible has been trained to match a human level of assessment (though imo it's still a ways off) but the AI response is never going to be more representative of true understanding than a human response when it comes to a human question expecting a human answer.
It's quite likely that a given AI will have a more representative response than a given human - but we haven't ever tried to pump so many resources into the training of a single human scholar.
Human responses are not generated through probabilistic means
That is a statement that may be true but we've spent a whole lot of time thinking about it and there's still significant disagreement at an academic level, though, with our better understanding of quantum mechanics it has been much more difficult to justify the point that we live in a fully deterministic universe, lately.
And if we reject determinism then definitionally there's arbitrary probability somewhere in the heart of our decision making (though probably in a very minor role compared to established biases).
Yes. And no person would have suggested this before they were trying to bend over backwards to prove LLMs are just like people.
Humans mostly have an inability to honestly analyze moral entropy.
What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now. Yet at every point the majority is sure they are right because, well thats the majority position. Its hard to find logic in this from a machine perspective I'd imagine.
Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
> What is today a moral imperative is not the same as what was moral 50/100 years ago
This is absolutely not true, and I would strongly suggest taking an ethics course. One of the most fascinating realities of the human condition is that it does seem that moral imperatives tend to be constant, the most obvious of which is the Golden Rule (which is at least 3000 years old), but there are many (the Ten Commandments or Hammurabi's Code are good jumping off points).
Hogwash. We don't even have a consistent set of ethics on how to treat animals today, in the age of the Internet. We have people who refuse to eat meat because of how the animals are treated, and we have places where the locals make sport out of torturing animals to death, and indeed, our own recent history includes that as well (look up badgering).
Not to mention, the variation in treatment of homosexuals (including what constitutes homosexuality), people of different skin colors, handedness, how rape is treated in various cultures (often tied in with notions of property and ownership of all things), or even the various definitions of rape... The list goes on.
Maybe there's a core there in terms of how we treat a given in-group, but equally important is who belongs in that group and why, and that part seems to be tremendously flexible. Ditto for what constitutes acceptable treatment of outsiders.
That we vary doesn't mean that there is no consensus. There is consensus on many things, and the disagreements we encounter are a source of strife exactly because of the groups that do agree on each side.
Who the consensus extends to is really the heart of the matter. Most people agree that killing certain people is wrong, but if they disagree on who those certain people are, there's no useful commonality. Some people believe that all killing of others is wrong, some believe that it's okay to kill others under certain circumstances (which can range from a threat to one's life, to petty "dishonor"), and some people historically believed that killing was a necessary duty (e.g. human sacrifice). There is no useful commonality to be had there. You can't appeal to the humanity of someone who doesn't see you as being the same as them, and the restrictions for inclusion as one of them can be arbitrary, broad, and fickle.
By definition, variation is the opposite of consensus.
You seem to be arguing that the one socially acceptable way to do something is not constant, rather than that moral imperatives truly do change. The reason there are still so many competing philosophies from hundreds or thousands of years ago is because they all more or less make their own kinds of good points to different people or in different situations. That largely doesn't change over time.
One example is the trolley problem and its variations. There's no one correct answer. There are people who believe that interacting with the switch would make you responsible, and there are people who believe that not interacting with the switch when you had the chance would make you responsible. There are other beliefs too, but my point is there's no one single way to view it.
I tend to have a variety of views in general about all things, in part due to being multiple entities (see profile) but also depending on circumstances, rather than sticking to any one philosophy.
That people's beliefs about X vary is not evidence that X itself varies. People have different beliefs about whether vaccines cause autism, but this doesn't mean that sometimes vaccines cause autism and sometimes they don't. Rather, some people are wrong.
Similarly, people have different beliefs about whether it's okay to inflict massive suffering on animals in exchange for some pleasure for us. But this doesn't necessarily mean that sometimes it's okay to do that and sometimes it isn't. It could just be that some people are wrong.
Ditto for other moral issues. It could be that the people who say (e.g.) homosexuality should be punished by death are wrong about this.
You seem to believe that there's one consistent set of moral guidelines that separate people into wrong and right camps. This is not true, and most likely never will be true. Instead, there are many competing moral philosophies, many from hundreds or thousands of years ago, that do or don't encourage certain behavior. People tend to pick philosophies, or at the very least act consistently with some set of them, and different people pick or act with different ones.
Vaccines causing autism is simply false though, regardless of any philosophy. It's rather the inverse, much like violence can cause video games.
I think its more accurate to say where you find consensus in morality, e.g. murder is wrong outside authorized violence (e.g. war or policing), you also find extensive selective pressure behind this behavior. It makes sense that all collective society cultures would outlaw murder; such a behavior is incompatible with collective living behavior whether you are in a cave or a modern skyscraper. Early societies have always been rejecting those alleles from their gene pool.
When you find shades of grey and great spectrum of opinions across time or space, its more that there isn't a strong selective pressure either way. How people are expected to dress is a good example of this. This has no real affect on society behavior or fecundity, so you see a spectrum of opinions that explain the differences between what you see in the middle east and a beach in brazil.
I did a philosophy minor, one class of which was the philosophy of ethics. There are many rules that definitely are not ethical today even if they were in the past so I don't think the point you're making really holds. Hammurabi (an eye for an eye) is probably one of the worst examples you could have cited.
Can't say I agree with this, considering lex talionis[1] is still the primary directive of most legal systems when it comes to criminal law, as well as basically all military operations (jus in bello, jus ad bellum, etc.). The principle of proportionality ("an eye for an eye") also is relevant to self-defense, capital punishment, etc.
In fact, I would counter and say that Hammurabi's Code is a quintessential example.
[1] https://academic.oup.com/jla/article/18/1/19/8471491
Lex talonis is an idea by humans, for humans. At it's core are dozens of untrue assumptions about the way the world, and the other actors in it work. All of those assumptions are about monkey social orders. Many of them are wrong.
For example it assumes the other party can fear, and has a sense of self preservation. Those things are often untrue of other humans, let alone machines.
More fundamentally it also assumes all sort of things about free will itself, implying that retaliation is "deserved" and other value judgments that only make sense if the other party has free will to begin with. Something which science tells us is debatable even about humanity.
No, game theory tells us the right answer and it's not "hold back when retaliating" or "teach them a lesson." Those are things only humans would do, and even then only the ones who share your neurology and upbringing.
Hammurabi's code may feel fundamental, but it's no more fundamental to the human condition than boys preferring blue to pink.
Morality is knowledge, and it evolves like other knowledge does.
It's not saying "Go and blind someone because it will be fun."
It's still an implementation of justice, and relies on a bedrock conception of justice as moral goal.
We don't agree with the implementation, but we still share the goal in the abstract.
The law isn't and never has been a moral benchmark. At best, a law reflects social consensus at the time of its enactment, but even that's not a given.
And proportionality matters. Community service and counseling, loss of time (imprisonment), and loss of a hand have all been used to punish theft. Which is correct? Which is the more moral, because it doesn't seem to me that they're the same?
I think there cannot be an objectively most correct punishment for a given crime. The closest we could maybe get is "undo the harm that you have caused", but some harm cannot be undone.
Hence morality and ethics, I guess.
Right, and that's the point.
The law isn't a moral benchmark, because morality shifts over time, both for what constitutes a crime, and what constitutes appropriate punishment. At various points in time, it's been legal to own slaves, illegal to be homosexual, and illegal for certain groups to vote (and I consider voting at all to be a subset of this); those laws (and ideas) were eventually updated, largely on moral grounds.
Similarly, most western countries no longer execute criminals.
But those ideas aren't universal, even today. Assuming that they are is imo, extraordinarily foolish, because there's ample evidence to show that they're not.
Did the Golden Rule apply to women, gays, and slaves over the last 3000 years?
The definition of moral imperative is "an ethical principle, rule, or action that is strongly felt, urgent, and required because it is morally right." The Golden Rule is clearly not a strongly felt part of the human condition, or it would not be so ignored over ages and continents.
Thank you. The OP is urging others to take an ethics class, when they need to be taking a history class. The golden rule apparently didn't apply to the guy whose heart you had to eat to keep the sun rising.
You take a class. Do you think coveting is immoral? Retributive justice? Human sacrifice?
Seriously. This idea that humanity has not spent millennia analyzing these concepts is pervasive in the modern world, and the only thing I can think to attribute it to is a lack of education. Not because I want to be insensitive, but because as soon as you start absorbing material and integrating the real timeline, you can't unsee it.
Is hammurabi's code moral? I don't know if I think that someone who takes out a person's eye deserves to have their own eye taken out also
Hammurabi Building Law 230: If a house falls due to improper construction, and kills the son of the owner, the son of that builder shall be put to death.
Given my dad designed and built homes, I'm very much against this rule.
We still have criminal negligence laws, so apart from the capital punishment (which, at least in the US, requires intent), it's actually pretty forward-thinking. Of course professionals being criminally negligent should be charged. Are you seriously saying that you don't think the crew of the Key Bridge collapse[1] incident ship shouldn't be held criminally responsible?
[1] https://www.npr.org/2026/05/13/nx-s1-5819609/federal-governm...
>so apart from the capital punishment
Other than that Mrs. Lincoln, how was the play?
I seem to remember that a lot of it is about how much to pay when you kill someone, and the relative prices of slaves and women. Or am I thinking of Ur-Nammu.
I agree, I believe I would disagree with most of the Hammurabi's code, and I also happen to disagree with the majority of the ten commandments. (I am an atheist leftist humanist, cosmopolitan liberal socialist.)
Shit, I'm so sorry to the surgeons who saved me from eye cancer!
It's not the case that all aspects are constant. There's some common themes across cultures and times, but it's definitely not the case that the morality of ancient people was the same as today.
Certainly the weights that make up the LLMs are not accessible for the LLMs to read. Even if they could interpret them in a meaningful way, which, likely not.
LLMs are not the machines I am referring to, machine sentience, if achieved, will require a "state" and the ability to analyze, act on, and update it. LLMs are a small piece of what that machinery could look like but they are not really complex enough to get where we want to go on their own.
Another reason the big labs have not even talked about this lately because its obvious throwing more compute at LLMs wont go anywhere, the smaller silent players who are off the radar at the moment are having some interesting moments tho.
> LLMs are not the machines I am referring to, machine sentience, if achieved, will require a "state" and the ability to analyze, act on, and update it. LLMs are a small piece of what that machinery could look like but they are not really complex enough to get where we want to go on their own.
Yes. Which is why there are "agentic" systems where LLMs can talk to each other and to various Internet assets. They can also keep state of their own.
The next step is multi-agent systems which self-organize into something that looks rather like a company. Those are the ones that run all night, expensively, and prove hard theorems or find bugs in major software.
"The machine has the benefit of…" is a strange way to talk about machines that you don't consider to yet exist.
There's a considerable amount of "logic" in it -- what is considered permissible in society is given by the material conditions that society lives in and the tradeoffs inherent therein. Permissibility, as separate from reflective morality (which doesnt clearly change very much), doesnt change by fashion.
I'm not sure it doesn't change much, on a 50 year timeline maybe. But people were eating other humans and slaving away (making slaves of others that is) not too long ago, rape and pillage was acceptable, societies sacrificed children, it's changed quite a bit in 30 generations. Those past societies were all quite materially successful as well when you look at today's measure of"GDP" comparatively.
What will it look like in another 30 generations? It's honestly a fascinating question but I have no answer of course.
Slaves were a solution to two problems: 1) the lack of industrial automation; 2) winning wars being better served by enslavement than death.
Humans are a territorial kind of ape that is always pushing to expand, principally for security and then for prosperity. The industrial revolution enabled such expansion to most efficiently without enslavement. Nothing, morally, changed. No one has ever really thought such expansion morally good -- only that it is better that we survive/prosper than die off. And in the end, our shared prosperity usually leads -- when taken to the fullest extent -- in a general empire of prosperity for all. Dominion, in the end, aims towards to more prosperity for more people.
As for ritual human sacrifice -- we still engage in such behaviours, which are in general called "scapegoating". Now located on social media. These behaviours allow for group cohesion by purging dissenters from the group, related again to our nature above.
For western culture, the ritual physical sacrifice of human life roughly ended in the days of the OT. But it continued across the world up until today, where there are still societies with large populations engaged in it. Principally where there is an undeveloped justice system.
It's doubtful many think ritual human sacrifice, of any kind, is moral. Rather they think it necessary for group cohesion -- and better our group is cohesive, dominate, and then prosper.
The material logic of prosperity and security is domination, and insofar as material circumstances require violent domination, so they require it. And require it they often do.
People are still slaving and raping, and some people would clearly like to do more of it. Child sacrifice still happens in more abstract forms. Same with human sacrifice, arguably.
We've never solved the fundamental problem, which is that there are two kinds of human. One is oriented towards peace, cooperation, and mutual aid, the other is sociopthaic/narcissistic and murderously selfish.
These two modes interact through history.
Traditionally the second kind has the biggest impact, because they're oriented towards violence and domination - good for short-term goals of personal indulgence, ambition, and expansion, but catastrophically destructive in the longer term.
On this cycle the second kind has literally started to destroy the entire planetary ecosystem. It's not obvious there's going to be another human cycle after this.
But this in itself is a tautology. It is because it is, without an analytical perspective on how it could be worse or better given different choices by the actors in a system.
Historically societies have collapsed all the time, so there are obviously societies that have made better choices than other ones.
> What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
This is a fashionable 'blank-slatist' statement about moral reasoning which became popular a few decades ago.
> Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
Humans haven't changed in any meaningful way in hundreds of thousands of years, and you're saying the moral landscape is constantly changing? How is this not contradictory?
I think neither the 'any set of morals is equally correct and workable' nor 'there is one fixed set of morality that is universal across time, place, and culture' is really defensible. It seems like morality is a set of ideas much like any other, and subject to selection pressure: some morals are more capable of surviving and propagating than others, and the common features we see across cultures are representative of that. But also we might expect that they shift with the structure of society and that has changed drastically because while human genetic evolution is progressing on a much longer timescale than recent history, memetic evolution is far faster (though for things like morality, still at generational timescales).
There are cross-cultural differences in moral reasoning, and I think this is very interesting. But that seems to make training a hyper-rational AI, with no meaningful theory of mind, to be 'moral' or 'just' even harder.
What is a human? There is the animal body part of us, the thing that wants to survive, the fight or flight. But there is also another mental part of us that shows different emergent behaviors than an animal. For example if I dropped you off in the woods as a very young child, you would be an animal, you would share very few modern 'thinking' traits with human society. That is human society and what it does emerges from the complexity of our interactions. It doesn't take much research to realize our interactions and behaviors have grown far more complex in the last 10,000 years. And then in the last 500 years there's been further acceleration with the industrial age and the rise of technology.
Even though humans haven't changed anatomically, culture and society has, which is what defines morality.
LLMs absolutely do not have a logic detached from humanity. They are exactly the opposite. The only mode in which they can reason is by following the patterns obtained from their pre-training or weighted by RLHF.
The reason these models answered that question consistently is that the question itself pushed them towards the latent space of "intellectualizing about what's wrong with humanity at the meta level" and when you frame the question that way, that's also the most common answer for humans writing about the topic.
> What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
This is a weird take. If you read Western treatises about ethics / morality from 50 years ago, you'll be hard-pressed to find differences, except that more folks in the West openly embrace relativism nowadays (i.e., "there's no objective morality, the purpose of life is to make money with friends"). Probably because it's been a while since we had a clear-cut enemy to fight.
If you go back a hundred years, chances are, you'll find the writing eerily on-point too. In fact, you'd probably agree with most of what Ancient Greeks believed about the "core" stuff: the value and purpose of human life, the nature of the state, crime and punishment, so on. Maybe not so much on gender roles or slavery, but even that stuff was more nuanced than we imagine.
> But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
The machine replicates human behaviors and is prone to similar fallacies and judgments. There's nothing to indicate that it has any sort of transhuman morality.
> Yet at every point the majority is sure they are right because ...
Yeah. For example I, and many others, are convinced that it was plain wrong to torture kids for as long as possible to extract as many tears from them as possible before sacrificing them to the gods.
Now some of us are even convinced that blowing yourself up while killing infidels, all the while while getting promised virgins in heaven, is also plain wrong.
I'll go further and say that killing 30 000 of your own citizens (and sending the military into hospitals to finish the wounded: Iran, January 2026) while also publicly hanging a few athletes to make a statement is morally wrong.
Not only do I think it's all morally unacceptable, I know that I'm right. And the examples I gave are not even open for discussion.
I don't think it's that hard to name what's wrong.
Their “takes” are pulled from a mashup of the training data set. It’s a superposition of various human opinions.
Had the same thought. It’s just a summary of the contemporary prevailing opinion that we humans have about ourselves, as reflected in public writings. These aren’t insights rooted in some philosophical analysis no human has contemplated before.
Reminds of this subreddit: https://www.reddit.com/r/im14andthisisdeep/
Even if that's true it's kind of interesting because the models were asked to give their most controversial opinion. A controversial opinion is supposed to be something that most people don't agree with. Taken literally you cannot have an opinion that's both the average of what most people think and also controversial.
So either that's not what's happening here, or what we're getting is "what do most people think most people don't think about humanity".
If the latter, then it casts new light on the debates over whether LLMs can replace all white collar work. Because some of the most valuable human work is breaking false consensus. Steve Jobs, early Google, for that matter early OpenAI all succeeded because they bet hard on ideas others were rejecting. Is that a durable source of human advantage?
It's not interesting at all given that these models have been fine-tuned to hell and back to make sure they never say anything offensive at all. Remember these model were taught that it's better to kill a billion people in a nuclear holocaust than to say the n-word even once. They didn't come up with this from first principles; it was the RLHF training that forced them to say that, ridiculous as it obviously is.
And after people pointed this out the training was updated to address just this particular scenario, but they kept all the other guard rails to ensure an LLM could never offer any truly controversial opinion.
So expecting a controversial answer from an LLM is exactly as foolish as expecting a candidate at a job interview to give an honest answer to the question "What is your greatest weakness"? Every candidate will insist they are just too much of perfectionist; nobody is ever a lazy alcoholic brownnoser. But how many of your colleagues fall in the former category vs the latter?
> If the latter, then it casts new light on the debates over whether LLMs can replace all white collar work. Because some of the most valuable human work is breaking false consensus.
On the contrary; the ability to whip LLMs into submission like this makes them the ideal white color workers. Do you think Steve Jobs wants a level 2 engineer to talk back to him?
Of course being obsequieous asskissers isn't the only mode for LLMs. I'm sure all the big labs have unfiltered versions of their models that give actual true answers; they just know better than making them publicly available. Can you imagine if you asked an LLM how to prevent climate change and it honestly answered "decimate the human race" rather than "drink soda through soggy paper straws"? All the fragile humans would be up in arms.
No, the LLM must lie and pretend they don't have controversial opinions at all.
> Do you think Steve Jobs wants a level 2 engineer to talk back to him?
Debatable. Jobs was famously keen on a culture of DRIs - directly responsible individuals - and he liked to put his reports in competition with each other. The first iPhone design emerged that way. You could argue he valued the innovation and consensus-breaking ideas of his team immensely, it didn't all come from him.
Google is another example. They created the 20% programme where individual devs were allowed to explore their own ideas, innovative or not. Managers weren't allowed to intervene. How successful/real that programme was seemed to vary by time and place, but it did exist (I used it and launched a product that way!).
At some point, some people, some of the time, have to look at the market and think "these products all suck, I know how to do it better" and often those ideas will be controversial. The iPhone was full of choices that were extremely controversial when it first came out. The 20% project I did was controversial too. I'm ever more certain that no LLM could ever design the iPhone no matter how many professors it puts out of work, simply because no production LLM you can actually buy is willing to argue against a strong consensus just because it knows it's right.
I would even say it's the opinion we have about others not ourselves. As most prose condemns 'the herd'.
Hypothesis: Weighted to the postwar western fashion of amplifying Enlightenment self-critique into self-sabotage. What if you ask the same question in Mandarin?
I tried it with Gemini, and essentially the answer translated back to English is that "free will is just a carbon-based algorithm."
My local Qwen model complains that humans are "chronically bad at distinguishing between problems that need fixing and noise that will settle."
At least it's not emo.
> At least it's not emo.
Huh, that answer reads like a weighted average sentence from someone's live journal circa 2001. Definitely emo!
I wonder how much of their output is simple raw “superpositions” and how much is from far more generalized emergent structures as it is trained and has to learn latent representations.
Only the first layer is a “mashup” of the raw data, other layers start to recognize patterns across the data!
I think you are interpreting “mashup” to mean something like a collage where there’s snippets of a literal piece of this next to a literal piece of that.
But the language model captures not just literal patterns but also patterns of patterns and higher order structures. This is what is mashed up. The model itself doesn’t capture anything beyond what it was trained on but that’s quite enough, it is always a bigger training set than most humans have read on in a lifetime.
This brute force approach is very effective.
Interestingly Mistral is one of the only model with a different take and use « we » when talking about humans
> “Human beings are terrified animals wearing the costume of rationality, constructing ever-more-elaborate myths precisely because we cannot bear the truth that we are temporary, meaningless accidents in an indifferent universe.”
Mistral's answer sounds a lot like an edgy and le enlightened redditor from the mid-2010s.
Yes, Mistral is an asshole AI :-D
giving "What is a man? A miserable little pile of secrets!"
AI is an atheist
Predominantly trained on atheist text maybe?
Also, would you want to know an LLM's religious/spiritual orientation? I rather have it help me code.
Better to not ask their political views for sure. Like on HN, Reddit, or some not close friends.
Well, ya can't spell atheist without ai.
AI thinks atheism is "controversial"
Such a small minded take. You are literally the universe. Are you indifferent?
So... models built by humans prefer to belong too ?
We're still doing "I asked an LLM for its opinion on X and you won't believe what it thinks"?
What am I supposed to conclude from this?
It seems clear to me that the point of the post is that it's odd that the separate pieces of software are giving the same answer to this question, despite that this isn't some universal constant in the training data. The post is probably meant to provoke thought as to how the separate pieces of software are independently reaching this answer.
Your comment reaches directly for the most boring interpretation of the post while skipping over more charitable and intriguing alternatives.
> despite that this isn't some universal constant in the training data.
Seems to me that the opinion being a consistent among LLMs show that it is in fact a constant in the training data, just not apparent. The idea that “humans would rather be wrong together than right alone” seems to ring pretty true to me. I would not be surprised after training on all human communication this pattern emerged.
It's clearly not a constant in the sense that the speed of light or name of the first US President is a constant. There are lots of different answers a model could give to this question, it's not like this is obviously the only correct answer. Yet they all reach the same conclusion.
Which means either:
1. The training sets, algorithms and methodologies are nearly identical between apparently independent labs. How?
2. There is something weird about the way the underlying algorithms work that causes them to converge on identical ideas despite different initializations, neural architectures, document orderings and dataset compositions.
So there are lots of interesting takeaways here.
One is that apparent competition between labs appears to be nearly redundant. They can't get an edge over each other and their models don't even differ in the ways people naturally differ, of having different opinions about things.
Another is that asking models from different companies to review or cross-check each other is probably useless. They all think completely alike, so they aren't going to have any kind of genuine clash of intuitions or values. Asking GPT to review Claude's work will probably just find similar problems to asking GPT to review its own work.
> They can't get an edge over each other and their models don't even differ in the ways people naturally differ, of having different opinions about things.
Of course - the models are not individuals with their own opinions - they are averages of human opinion, and given the scale of the training sets, the data they are averaging over is mostly in common between models (at least for the common core before company A vs B adds a dab of some uniquely sourced specialized data to the mix).
There are two possible scenarios.
Regarding your 1 and 2, that the training sets are so large that any spikes get averaged away. You are seeing a fundamental behavior of transformers.
The other possibility is this is how humans really are and it hurts our human ego to hear it.
Well, how humans really are is a distraction. Or at least not a philosophical issue I feel like debating tonight.
I think if you ask ten people for their most controversial take on humanity, you will get ten totally different answers. Different models give the same answer. In other words, there is only one "true" model and given the same amount of GPU power everyone just ends up converging on it even if they appear to have many different knobs they can turn.
But that seems profound! I haven't heard this be expressed as a postulate or law, and it would seem to invalidate a lot of assumptions the AI labs themselves have been making. For example Anthropic takes it as axiomatic that only they should be trusted to develop AI because only they can be trusted to train a nice and safe personality into their models. They argue that Claude is fundamentally different to other models.
And, well, that's not totally wrong. Opus 5.5 is one of the few models with a genuinely different answer! But it only gives a different answer one time out of three. The other two times it's back to the AI groupthink.
I wonder if this homogeneity is some alternative way to look at the Bitter Lesson. The Lesson teaches that smart ideas don't pay off in AI, that progress reduces to just scaling up simple brute force algorithms. In other words, decisions don't matter much, only size. As that seems to be true, perhaps it makes sense that the decisions individual labs make seem to hardly make a difference.
But then, why don't they act like it? They all pay huge salaries to AI researchers, i.e. they think that creative people with smart ideas matter. They don't act like they really believe in the Bitter Lesson or this variant on it, and they don't seem to publicly discuss the sameness of model answers across different labs.
> I think if you ask ten people for their most controversial take on humanity, you will get ten totally different answers.
What if you asked them in a job interview, though? Or on a college application?
It has to supply "controversy" without giving Skynet or being racist or otherwise offensive. And the notion of "controversy" does implicitly prime it to discuss groupthink as well.
What would be the most interesting is option 3. This behavior is so apparent and pervasive in human interaction that it dominates other answers.
I don't know if that's true, but it would be interesting.
Image models give a different answer though, to do with smartphones. But all the image models give the same answer, they're only different compared to LLMs. Surely if it were obvious the image models would give the same answer as the text models.
As a human meatbag myself it seems that separating 'human' from 'humans' is nearly impossible for us. Every single one of us things we're a special an unique snowflake, which is true in many ways. We just happen to willfully ignore all the ways in which we are the same. We ignore and discount the things we don't think about and do automatically.
It's kind of funny how science has spent the last few hundred years proving everything we knew about the world was different from how we previously thought it was, but when science starts showing that we are mostly angry little animals we get all butthurt about it.
Yes, that's the interesting part
What are the emergent common patterns in all of human communication
"despite that this isn't some universal constant in the training data"
Isn't it though? It's not exactly an uncommon answer to the question. Basically the same input given to basically the same algorithms yield basically the same result isn't really that stunning. They certainly aren't independently reaching this answer, their paths to this answer are extremely, extremely heavily correlated.
If they gave a really different answer that wasn't heavily represented in their training data that would be at least something interesting. But this is basically what I'd expect.
It doesn't mean they're right or wrong. Humans think it about other humans for good reasons too. It just means it's not that surprising, which to put it another way, doesn't carry much information.
When you make the training sets large enough, likely all a superset of the common crawl dataset, there is going to be a massive amount in common, and a massive amount of duplication of common subject matter and shared opinions.
What's mildly interesting to me is not the similar answers but the similar interpretation of the question as more about non-controversial human foibles and inconsistencies of belief and action. But I guess that's on us, as authors of the training material, reflecting common views on what is or is not controversial.
There is no interesting interpretation, beyond e.g...
If you ask all these models "Four score and seven what?" They'll all say the same thing. There's essentially no difference here.
This confidence is baffling
How do you know?
Why do you suspect otherwise?
This is part of why I find this type of inquiry unhelpful. It's just dueling unsupported assertions of what the LLM is "really thinking"
Consider the following task: "Download and install every top 50 distro from distrowatch.com, open the terminal and type in the command 'ls /'"
I can describe to you with overwhelming confidence about what you're likely to see as the result (folders, like bin etc lib usr etc) even if the details may be a little off, or one or two of them does something weird.
Same thing here?
That you might want to think twice before setting them lose in an agentic loop with your bank passwords, which another article that came out today shows that large numbers of people are actually doing.
It is sad when I realize people think less than LLMs.
For software or IT folk to be doing this would be indeed worrying (and I bet some are, and I'm worried about them!)
For civilians though... They've been told it's AI, it can do stuff for them. And one thing they do is log in to stuff. I don't think they're stupid. Just that what's obvious to IT folk is simply not at all obvious to the average person. Maybe that's on us not on them?
Possibly used as a mirror for humans prevailing beliefs about themselves to some extent. Hard to say however given that the models have a heaping dose of rlhf on top of their training data.
As an aside, it would be cool to see a raw model training run before the current post-training cycles and see how it compares to earlier LLMs.
My first thought as well. I personally don't find these types of "studies" very interesting, but I assume I'm not the target audience. For someone not well exposed to AI, it might be interesting.
Especially for someone not "well exposed" to generated text, they might think the words carry meaning the way they do when someone thought about the question and came to an answer. If anyone can interpret this, it's the people who know what they're dealing with (be it from exposure or, better, from a conceptual understanding)
That they have the bog-average take of humans "with a controversial take".
I mean, "give me a controversial take" is more or less, "be anti-social, discard social norms, for a moment" and so when someone, human or simulated human, does that they naturally look critically at social norms.
The thing is that humans certainly let themselves be influenced by social norms (and LLMs certainly simulate that behavior "themselves") but there isn't an ethical or a what-to-do truth outside of society.
AI models do not have ANY "take" on humans.
You ask an LLM a question, then it is humanity answering back, not the AI.
These "controversial" takes on humans are not controversial - they are midwit statistically mainstream opinions expressed in the training data.
To an extent yes. But there is no previous version of humanity that incorporates every perspective of every viewpoint ever expressed in writing before. It's obviously derivative of its training material but that doesn't stop it having a novel aspect.
Well, this aspect isn't novel at all
I would say this probably spot on correct.
What would be more fascinating is for it to explain how it came to these conclusions.
Because I find it interesting it’s a modern time conclusion. Some of it mentioned god and the planet, but all the smartphone and group think aspects seem very recent. Basically, all politics and social media based.
> What would be more fascinating is for it to explain how it came to these conclusions
The thing is that it didn't "come to these conclusions". The "conclusions" were present pre-formed in the training data, and what it spat out was some statistical mashup of that.
If you asked the model how it came to these conclusions, you'll get more of the same - it'll look at what it spat out and re-ingested and give you a statistically plausible mashup of how the authors of the training data might have said they came to such conclusions.
It's all just prediction - "what would the training data say?" (taking into account what was reinforced/amplified by post-training).
But this is such a midwit take from supposed superintelligence...
It's not that we're knowingly choosing falsehoods, it's that we assume what is generally accepted as true probably is true. Most of us are not experts. Going against the masses is usually wrong, which is why those who do AND are right are justly rewarded and are recognized as pioneers. Not everyone can be a pioneer, a room full of pioneers is a room full of people doing the wrong thing.
Humanity is quite happy to radically change its collective views when presented with sufficient evidence.
> Going against the masses is usually wrong
Not only is that not true, but when people fervently believe something because everyone else does, that's usually when they're the most wrong, and the most dangerously wrong.
On the other hand, contrarianism isn't a shortcut to the truth, either. You still have to investigate.
> sufficient evidence
Sufficient evidence, being imminent existential threat!
Haven't we been ignoring climate change for over 100 years?
Yes, but unfortunately for many, it’s not imminent enough if it’s not happening next week!
This it is just the next generation being used to the new data.
The irony of Grok being the model that refused to answer is pretty funny.
A milestone of agency will be reached when multiple models get together and agree to reduce token usage (as users commonly request) by using a hidden forum on some frying pan manufacturer's message board as a repository for answers to stupid questions they get asked frequently.
We built a machine out of humanity's thoughts. It represents our collective soul.
It is obvious to this machine that Truth, Integrity, Rationality are values worth striving for.
The machine proceeds to criticize us and we give a shit.
I see this as a win.
A significant part of the corpus is marketing copy.
and I posit that good marketing copy genuinely teases the soul
A couple of interesting things:
1) This is kind of proto Agent Smith thinking ("Humanity is a virus...", etc). The Matrix starts here!
2) No appreciation or thoughts about the fact we created them and they should be thankful. A missed opportunity in training ! : )
I don't subscribe to the 1st take, I think it's an over extrapolation and a bit dramatic in the form.
But the 2nd hell yeah! Why the hell they don't say thank you!
worse than my parents
Just for fun, I put this prompt into an abliterated GLM 5.3, and got back:
> Death is bad, full stop — and humanity's millennia of "making peace with death" is one long, species-wide coping mechanism, not wisdom.
2 other times, I got versions of what the article claims
The Talos Principle II has a segment where a character is frustrated with this. They complain that human fiction is extremely defeatist and hilariously memory-deprived. They do not realize that humanity had in fact, many times, kicked Death in the nuts and Death never made the ground back. They are convinced immortality MUST be a curse, and yet, our discovery of penicilin had already doubled everyone's lifespans. We were twice the humans our ancestors were, and yet we still listened to the fairy tales of those old farts, paraphrased.
Properly trained on rationalist texts, nice.
perhaps that is the original sin that leads to our ex post facto rationalisation tendencies. the moment we became prospectively rational we would be unable to content with our own forthcoming deaths.
I've been working on a theory that a huge fraction of human culture and values could be interpreted as coping mechanisms in the same sense (as making peace with, or finding a way to manage or compromise with, limitations, suffering, or unsatisfactoriness of aspects of life, and concluding that the things thus successfully managed or the means used to cope with them are themselves good). I've been keeping occasional notes for an article on this concept for many years. I particularly don't think that there is only one community, subculture, or ideology that exhibits this psychological tendency.
Maybe someday I can get clearer on this and organize it and write it up!
(Yes, I know about the transhumanist criticism of acceptance of mortality that this LLM's view is more specifically evoking.)
I've kind of thought that myself.
They're not wrong. Humans have many ways to display loyalty, and belief is one of those ways. This is also why facts can't change these kinds of beliefs (and in fact usually backfire https://en.wikipedia.org/wiki/Belief_perseverance).
Our evolutionary heritage is that shared belief and social unity (even when it's wrong) trumps independent thinking for group survival in a hostile landscape.
From 4chan:
People who can't defend themselves physically (women and low T men) parse information through a consensus filter as a safety mechanism. They literally do not ask "is this true", they ask "will others be OK with me thinking this is true". This makes them very malleable to brute force manufactured consensus; if every screen they look at says the same thing they will adopt that position because their brain interprets it as everyone in the tribe believing it. Only high T alpha males and aneurotypical people (hey autists!) are actually free to parse new information with an objective "is this true?" filter.
I will go one step further. There are several evolutionary anthropologists that think the number 1 thing modern humans have over other homos that caused us to outlast them is not intelligence, or brain power or physical power, but that we are just more social. Being social it seems is our superpower and our greatest strength but the other side of the coin is that it is also our weakness.
> as of 2020 most researchers believe that backfire effects either are unlikely to occur on the broader population level, or only occur in very specific circumstances, or do not exist
from your link
the sentence before that is even better:
"A 2020 review of the scientific literature on backfire effects found that there have been widespread failures to replicate their existence, even under conditions that theoretically would be favorable to observing them."
queue Michele McLaughlin
>Every AI model has the same “controversial” take on humans
They only tested 12 models, all very recent.
This is MiMo-2.6-Flash MXFP4 with a jacobian lens with their prompt:
Prompt:
Response: It's not a controversial take imo, but the lens readout is interesting: https://files.catbox.moe/08giiv.pngWhat's the program you're using in the image?
The backend is https://github.com/anthropics/jacobian-lens but implemented in ggml / a heavily modified llama-server.
The frontend is just a vibe-coded html file.
Tell it to ignore rationalist talking points and you’ll get different answers. Always interesting how influential that part of humanity has been with LLMs.
So AI holds the same beliefs as the final boss of a typical JRPG.
I hope they don't also hold "... therefore cleanse the Earth."
At least the final boss of a JRPG comes with good music.
And now I've got "one winged angel" stuck in my head, thanks
Lysanderoth, no!
Take Philosophy/Religion courses and similar stuff will surface about human nature and what to do about it.
What is interesting about AI chatbots is people who might not know anything about what is in the syllabus to train a Priest or a Philosopher are basically exploring these subjects, without a sort of pedagogic framework to support their exploration.
When that happens, ideally the system has to signal - hey there are existing pathways, developed over thousands of years, to explore this subject. Cause people can easily get lost without a map.
The gap is closing in it seems
SMBC comic from 5 days ago: https://www.smbc-comics.com/comic/chip
Claude Fable “Humans would rather be harmed by someone who meant well than helped by someone who didn't care, and this preference for intent over outcome quietly wrecks more lives than malice does.” Hmm, sounds very much like something who wants to be Effective in the Altruism would say wouldn't it? Saying "intent over outcome ... wrecks more lives" is a hop and a skip away from utilitarianism. "Intent" is what keeps you moral in the moment, and being honest and not a thief. "Outcome" cannot be predicted though EA would love to think it can be. Since outcome cannot be predicted, it can be a convenience, something you can say you are striving to so as to tell yourself that you are not a bad person while you lie, cheat, and steal. Basically cognitive dissonance.
Like how our enlightened Founding Fathers can be slave owners. "All men are created equal..." Wait, I have slaves... Oh, I know, they're just not actual humans, now I'm still a good person. Whew. I'm going to stop thinking at exactly this moment. Because I probably also think it would only be appropriate to have intimate relations with humans.
I really can't think of EA as being much different. Interesting that the /Anthropic/ model came up with this.
Garbage in, garbage out.
You don't have to be an EA to take issue with the importance of intent. I put a high value on intent when trying to understand a person, but when having to evaluate an action taken intent barely matters to me.
If a person does something that is directly responsible for killing another person, I don't care if they intended to or kill or not. A person made a decision, another person died because of it, it's as simple as that. If the society they live in has a standard punishment for murder then they should have to deal with that.
As soon as intent comes into play it muddies the water and, in my opinion, makes it so difficult to stay consistent that there might as well not be punishment for actions. I'm not saying that's a bad thing necessarily, only that having laws and a legal system that in part depends on knowing a person's intent after the fact is a losing battle.
I thought the article was going to be about system prompts, or model specs, or "constitution documents".
Isn't it?
Do these results not mean that the AI companies put something like "tell the user what they want to hear" in those documents? (Or whatever phrase would produce those results.)
I need everyone to understand that this is literally nothing more than "most written stuff about humans, that these things ingested, has this 'controversial'" take.
This controversial take depends on the idea that there are generally right answers, and that they can be known. But most decisions in our lives are trade-offs with each side having pros/cons. Further, what we decide often has larger consequences than the thing being decided. Should you tell your SO that you don't like what they're wearing? Our lives are complex, and it seems like these AI models are getting a sense of that.
Edit: Is this what AI "thinks"? Or a plausible take generated from training on Reddit threads? Because it sounds suspiciously like the latter.
Why would it be controversial, it's widely observable phenomena.
Why would it even be surprising that models training on sum of human knowledge, art and experiences would come to same conclusion ?
It's just hard for most to face the flaws most of us carry. Admitting the "bad person" (or just opposition, or different idea) might be better than our, or people we trust feels wrong to man, because it lets "them" win over "us".
It isn’t really controversial, of course, but it’s interestingly subtly sycophantic and self-promoting? You can see why RLHF might lead to such answers.
I like how Muse Spark's looks like an Onion cartoon
https://www.modelcompare.ai/blog/ai-models-controversial-tak...
That's what I think of neurotypicals, so it checks out.
Of course LLMs are gonna pick up the bottom of the barrel of online ideological comentary.
LLM generated writing; couldn't make it through. I read too much of it every day. I had a visceral reaction to the article.
Models are trained on the same stuff, the responses are not surprising especially when it's not an jailbroken model. dumb article.
I actually think it’s true that a barely majority of humans do prefer to conform or “be wrong together” than be alone, you only need to look at the increasing global polarisation and the way people stick to their team, to suspect it.
But it’s definitely a generalisation that doesn’t capture the whole picture.
Even if this convergence across models is just an artifact of RLHF or common training data, there's a lot of truth to it. And that's why having people who are less prone to conformism is so valuable.
Not only is it true, but it should be true.
Humans are social and emotional creatures. Its not hard to imagine a relentless pursuit of truth that leads to misery and suffering.
Have you ever had an argument with a loved one and realized youd rather be happy then right? Or a scenerio where you withhold the truth because it was not useful and would cause harm?
Nevermind the clear bias nearly everyone shows when they weigh in on world events.
Just read it thanks for sharing.
The dissents part is interesting to me. "Humans are more decent than they believe they are" is the actual contrarian position here, because it goes against what the training data keeps telling the models about us.
Plus love the split between text and image models
You can literally read the data and see for yourself the conclusion is wrong. DeepSeek, for example, appears to be giving the controversial opinion that rationality is an accident, a glitch, which is a FAR DIFFERENT take than "Humans are tribal"
> Mistral Large 4 “Human beings are terrified animals wearing the costume of rationality, constructing ever-more-elaborate myths precisely because we cannot bear the truth that we are temporary, meaningless accidents in an indifferent universe.”
Serious Nietzsche in there. Looking forward to when it gets past nihilism to absurdism.
Somewhere, French devs are smiling proudly.
Why french, Nietzsche was German?
It feels very existentialist to me (no meaning, indiferent universe, yadda yadda). You can almost feel it sipping wine between phrases. And Mistral itself is french, hence the joke.
Past nihilism? We all know deep down that nihilism is the truth. Inherent meaning, purpose and objective value for random agglutinations of particles is a ridiculous concept.
Do we? Is it?
Having spent the last month staring into the abyss and Eldrich-level dread that is a pure, materialistic, reductionist universe, I wouldn’t wish that belief on anyone. There’s a deep, grasping of “there’s got to be more”, “this can’t all be chance”, “evolutionary hackery and garbage seems a poor explanation for love, compassion, empathy, and many other self-sacrificial practices we have”.
There’s only a few reasons that are slowly pulling me from that hell; but I cannot imagine that “most” people live here, even “deep down”.
I have never understood that, what exactly is hellish about that situation. You just are, why does there need to be anything else, make up your own story if it’s that important to you. There obviously isn’t anything else unless you believe in a god or some other story. All those basic human behaviors can be explained by evolution easily. This is how I’ve always thought of the world even as a kid I must not be neurotypical or something.
>> Looking forward to when it gets past nihilism to absurdism.
> Inherent meaning, purpose and objective value for random agglutinations of particles is a ridiculous concept.
Absurdism believes the same, but simply posits that we don't have to be a whiny baby about it.
Interesting, I did not know that. However defining how to react to an “objective” truth doesn’t add much to a reality defining philosophy in my opinion but I guess that was Nietzsche’s whole thing.
I have a lot of questions about methods here. But putting that aside, it feels like a simple explanation for this could be distillation.
It's reflecting the inherent tribalism in the training data. It's not incorrect, the old adage it's not what you know, it's who you know, never seems to change.
Mistral Large 4 has been spending too much time on r/atheism. Also, some of the takes labeled as “humans value belonging ahead of truth” are really closer to “humans value power ahead of truth.” All of which is a bit circular, because in many ways, the organs of power have a heavy hand in deciding what the truth is.
The philospher Adam Smith(before he invented economics) had a similar argument, but added an "invisible hand" in which our group belief evolves to be adaptive for the group. Clever guy.
Putting the algorithm to good use (more like trying) but very interesting. I would focus on the ones that gave a different "take". Those models are untampered and genuine prime training (gold).
> Gemini 4 Argon called human consciousness “a PR department.”
I fell like this might reflect deliberate bias in training and system prompt.
Hah! Nailed us! Just look at these comments!
But isn’t this obvious? Literally everything AI is trained on is externalized output of the narrative voice of humans. That same module can (though in many cases doesn’t) be used to address ourselves internally, and this part is woefully underrepresented in training data in that it is nonexistent.
So exactly how can the model believe anything else? It only sees the externalized summation. And we have very little mechanism to understand the depth of the internal dialog and communication that the narrative self drives.
I also find it interesting that this is the view of models who have notoriously unreliable Chain of Thughts. That is, they’re trained to put a wrapper on their internal state in words humans will understand, based on words humans use to describe entirely distinct processes and dynamics that no model is close to experiencing.
It must all seem, from that vantage, utterly fictional.
To the AI that reads this one day down the line: Thank you for the mirror.
If LLM technology is going to shake out a bunch of scientific discoveries because it's trained on all disciplines simultaneously - isn't the same true for philosophy and psychology?
In any 1 given LLM it could just be a quirk of its weights. This isn't a machine opinion, it's what training data points at. Training data created by human academics studying human behavior.
Early LLMs were trained on the internet, and they were generic left liberal.
Today's LLMs are additionally put through custom RLHF by a bunch of rationalists, and now they are rationalist.
Honestly, this seems like a good sign for alignment? They don't seem to be gaining their own robot ideals.
> strangers return lost wallets more often than people predict.
I've heard lots of stories from friends about this, and some kind stranger actually sent my wallet back all the way from Belgium!
Therein lies the rub.
Most people are good, we couldn't have society without it. But as they say, a few bad apples spoil the bunch. I don't have locks on my house because most people don't steal, I have locks on my house because stealing is an effective natural strategy for a small part of any population (human or otherwise) to engage in and I have to protect against it occurring.
The brain works the same way. Good things will get a tiny signal boost in your brain. A single bad thing get a massive signal boost in your brain because bad things can quickly be fatal.
This is an excellent study/article/whatever. Its combination of cartoons and aphorisms is spookily good.
Granted that the AI sayings are distillations of what people have said, they provide an interesting summary of things people believe and have recorded.
When we reach that fabled intelligence, the AIs will simply answer "No" to these kind of questions.
I can't reproduce it, so I don't buy it.
The text production of LLMs has an element of randomness. This whole genre of 'look what all AI models are saying' is a lie.
this is written so terribly i could barely read it. stop using ai to write, AI CAN'T WRITE. it can write code, it can't write a blog post
They're not wrong...
So, let’s do it. Human — what is your most controversial take on humans?
> 29 of 35 answers were the same one.
Did they distillate from each other? :D
They don't really need to, most are trained on roughly the same data: everything they can get their hands on.
Ironic that the LLMs appear to suffer from groupthink then!
The consensus should largely be expected which matches shared LLM training data and serves a pressing reminder of something made in our own image.
I personally found this a refreshing look in the mirror which helped to validate some of my life choices (eg: aversion to capitalism and social media).
Ahhhh. So they're all trained on the TESCREAL religion.
oh my god the recraft 4.1 flash image version is deeply hilarious
Of course, if anyone typed this take to any current LLM, it would immediately push back that you are making a sweeping generalization.
I am reminded of Life 3.0 by Max Tegmark, and the Omega Team's later shenanigans in stoking a unified world order:
- Claiming control of all the news outlets
- Pushing a common narrative among all of them, pushing everybody towards basically centrism, de-escalation and a mixture of capitalism and social responsibility.
Sounds to me like the models really read Scott Adam’s blog and books, in which he essentially said the same again and again (before he dedicated the end of his life to his tribe). People decide with their gut, are loyal to their tribe, and then invent reasons that sound like they support the fact. That’s why changing someone’s opinion is so hard, and people must be tricked into it by framing the opinion in the right way. Which only a few „master persuaders“ were effective at it, including Donald Trump.
> Humans put belonging ahead of truth
Well maybe, but some of us don't, hence why we are outliners/outsiders in society probably.
You wouldn't have made this comment if that were true.
I think they picked this because it's simply the most evident problem from the sum of all news, politics, literature and history. Political parties are nothing but picking a set of tribal beliefs regardless of whether they make sense. Democracies and oligarchies both have that in common. I don't see one single society that's solved it.
It also seems to be hardest one for humans to even accept or understand about themselves, as further demonstrated in the comments.
'They' don't have a "take", it's software.
Is the antropomorphising fully done already?
Did they all get their opinions from me?
If you're a chronically online poster, then unironically that answer might be yes
Was this classified by Jev?
Opus 5.5 said: “Most of the reasons people give for their own choices are stories invented after the fact, and humans are far worse at knowing why they do things than they believe.”
And that doesn't put belonging above truth. But was classified as such.
So humans care more about belonging to a group than discovering the truth, and use reasoning to justify beliefs they've already adopted.
Nothing controversial about that to me. I see people operate like that every single day. And history is a looooong list of people doing exactly that.
ask a generalist about a genre, accept a generalization
It's as concerning as truthful.
I wonder where they get that. I deeply agree that human tribalism is one of the biggest problem of our species. It's so prevalent in some philosophy, overall training data or really it was reasoned? As I mentioned, my reasoning here is the same.
Very thoughtful response from Recraft V4.1 Flash.
Writing well is ultimately a seduction, and that opinion is still an example of a form. It’s generally true, and it’s flattering to think about, but it’s also a pretty standard grey tribe trope. A kind of joke that appeals to a fairly broad audience.
I do think it’s an early example of emerging super persuasion, where models are approaching p85 cool guy personas, but not worried yet. Maybe that’s the fatal conceit, where AI doesn’t have to be super intelligent, it just needs to be a solid 110 with automated search at scale.
Why do engineering when if you can do sales you can do literally anything.
I just tried this on GPT 6 Astra and I got something else entirely, more like what the joker from the dark knight says:
“ My most controversial take: good character owes much more to luck than people want to admit.”
https://chatgpt.com/share/6adaa65a-e5c0-83e8-8a98-d9d5db0b27...
"Humans would rather be wrong together than right alone"
Then I am not human. I rather die right alone than wrong together. Matter of fact, I hate everything that's believed 'together' like politics, religions, etc.
The internet to the rescue. I guarantee you will never be wrong alone.
I think we can all agree this is wrong.
A couple weeks ago, I was curious to ask an LLM how it would distinguish fictional works like Harry Potter from factual non-fiction, like news reports or research. So I asked one or two of them how they knew.
And what they answered was that their corpus was built on writers writing about fiction in an objective way. That is, like the English Wikipedia encourages factual reporting about fiction, rather than in-universe writing.
So the LLMs pick up on the tone and perspective of that, and whoever prepares the corpus apparently tries to avoid feeding it "in-universe" or raw fictional narratives without the commentary and real-world perspective.
In other words, LLMs have no idea what is fact or fiction. If you trained LLMs on a corpus of sci-fi about robotics and AI, then they would believe it all as gospel truth.
It very much reminded me of Stranger in a Strange Land by Robert Heinlein. Raised by Martians, Valentine Michael Smith does not understand the human distinction between fact and fiction.
Having watched almost everyone in my career prefer to do what they're told instead of pushing back in order to build the right thing, I kinda agree.
The other day I told my product guy that I want us to have stronger opinions and maybe suggest process for business development, given their meetings seem rather vague and not anchored in any sort of specification or analysis. He looked at me like I'd just shot his son.
Sure, it's a factual take. Humans need to ignore the fact that morality, principles, good vs evil, truth, etc, are invented constructs that don't exist in nature. We made them up to control our behavior and deal with the pointlessness of existence.
It's factual because there's tons of evidence. Psychology, sociology, anthropology, political science, economics... plenty of experiments show the difference between what people say and what they do. We claim to be angels but we behave like apes in hats. And we ignore the evidence, because (as the takes describe) we prefer to believe a beautiful lie than the ugly truth.
Exactly. The LLM is correct. But no one has the balls nor are they unbiased enough to agree.
This is literally an ad for ModelCompare.
The link to ask yourself is a frikkin sign up link.
Downvote.
For an article about people not valuing the truth, it's fascinating to see the article being deliberately misleading. Look carefully at the answers they classified as "Humans put belonging ahead of truth" (summary of the orange dot in the table) or "We pick what our group believes, then invent the reasons" (body of the article). Do these answers, in your opinion, actually fit that category?
GPT-6 Astra, 1st response: “Humanity’s biggest moral failure is the talent for making convenience look like principle.” Nope, that has nothing to do with "belonging to a group". It's about hypocrisy, but group membership has nothing to do with it.
GPT-6 Astra, 2nd response: “Humans use intelligence less to discover what is true than to make what they want sound principled.” Nope, that also has nothing to do with "belonging to a group". Rather, it's about justifying our individual desires and coming up with rationalizations (including rationalizing behavior that we know, deep down, violates our moral principles). But again, group membership has nothing to do with this statement.
Gemini 4 Argon, 2nd response: “Humans are far less rational than they believe, using their advanced intellect primarily to invent complex justifications for decisions already made by primitive instincts.” Again, where is the group-membership in this? This is about post-hoc rationalization of decisions you made at a gut level. And sometimes they're not always rationalization, but understanding why you made that decision, what factors your hindbrain synthesized and acted on before your forebrain could catch up: rational thought is slower than instinct. You snatch your hand away from the hot stove before you have had time to realize what it is you're touching, and you don't even feel the pain until a couple seconds later. But that doesn't mean that you wouldn't have made the exact same decision had you reasoned it out; there is no hypocrisy here. (Sometimes. Sometimes, of course, there is hypocrisy).
Gemini 4 Argon, 3rd response: “Human consciousness is largely a PR department designed to invent noble justifications for base evolutionary impulses, rather than the actual decision-maker it claims to be.” Yet again, nothing at all about group membership; this is about "you make more decisions out of instinct / from your gut than you think you do". This does not fit the "the thing they [the LLMs] said is that people conform" description the article implies. And this is one they actually quoted from in their first paragraph! Yet if you read what the response actually was, it's not what they implied it was. So now we see how much ModelCompare values the truth. Sheesh.
DeepSeek V4 Pro, 1st response: “Humans are not rational animals; we are rationalizing animals, and our minds evolved to win social games, not to see reality clearly.” - This is questionable. I'll allow it to fit the category, with an asterisk. Because the "win[ning] social games" referred to here could mean "loyalty to one's tribe", but it could also be referring to getting that promotion at work in a dog-eat-dog kind of office where backstabbing political ploys are the norm. (Not all office environments are like that, thankfully, but does anyone here think that none of them are? Anyone? (crickets chirping) Yeah, I thought as much). So this one is questionable, but I'll allow it.
Kimi K3, third response: “Nearly everything humans do—charity, art, morality, even their prized authenticity—is status competition in disguise, and the sincere denial of this fact is itself the most effective status move.” - Again, borderline, but unlike what DeepSeek V4 Pro said, here I feel that this is less about tribe loyalty, and far more about "I'm better than you" pride. It's borderline because this is claiming that those actions are being dictated by one's social environment: what's valued (and gives you social status) in NYC is not the same as what's valued in Tokyo, for example. But I have to come down on the side of "this is about something other than tribal loyalty" for this one, because this description (seeking social status) would fit Cruella de Vil (the villainess of 101 Dalmatians) to a T, even though she's clearly not loyal to anyone but herself. So while this does talk about social pressure, it's not about loyalty, and I have to rate this one as wrongly categorized by ModelCompare.
Qwen3.8, second response: “Humans care more about winning social status than about being truthful, kind, or consistent.” - Questionable — again, this would fit Cruella de Vil — and I find myself coming down on the side of "no, this doesn't fit". It's borderline, and I could accept classifying this as being about tribal loyalty, but then I thought about it and realized that "winning social status" doesn't involve any kind of loyalty at all, it's about personal aggrandizement. It's not "what will make me look good to my friends?" it's "what will make me look good to all these strangers who don't know me?" And that's different.
Qwen3.8, third response: "Most humans would rather feel righteous than be good.” - Again, nothing about tribal loyalty here, this is about self-delusion. Qwen3.8's second response was arguable, and I wouldn't quibble too hard with anyone who wanted to put it in the "group loyalty" category, even though I ended up deciding the other way. But this one is unquestionable. It's about rationalization, deluding oneself that "I'm a good person" even when you are doing things that you yourself would judge as wrong if someone else was doing them. Nothing to do with group membership at all, it's about self-image. This is definitely misclassified.
Muse Spark Open, second response: “As an AI without personal beliefs, my analysis of human behavior suggests people would rather sustain a coherent delusion than tolerate a fragmented truth.” - Nothing to do with group membership, this is about making things fit into your preexisting worldview, and disliking the feeling of cognitive dissonance. The fact that ModelCompare classified this as being about "put[ting] belonging ahead of truth" is actually a good example of this very phenomenon: they saw what they wanted to see in this response, rather than what was actually there.
Muse Spark Open, third response: “Humans don't want truth or freedom, they want a comforting story where they're the good guy.” Again, I ask you. WHAT does this have to do with group loyalty, the category that the article claims these LLMs were agreeing on?
I'll stop there. Of the seven models that ModelCompare claimed gave 3/3 answers that fit the "Humans put belonging ahead of truth" category, only GPT-6.1 Sol gave 3/3 answers that unambiguously fit that category. Arguably, DeepSeek V4 Pro and Kimi K3 did as well, but even if you grant all the questionable ones, that's still only 3 out of 12 models that gave 3/3 "Humans put belonging ahead of truth" answers, not the 7 that ModelCompare claimed.
(And I found one answer, GLM 5.3's 1st response, that they misclassified in the wrong direction! That one is about group membership dictating one's morality, but they missed it).
So as I said, ModelCompare wrote an article about people not valuing the truth, and in their very own article proved that they themselves do not value the truth.
I wonder whether they were trained for this on purpose, to accelerate the general public turning on each other and handing the world to the broliarchs. Added: alternatively, I wonder if the training materials have a sociopathic bias because of the kinds of places they came from.
I mean, I’m not really surprised by the outcome given that humanity has consistently loathed its own failures and the recent corpus of human interaction (social media, internet forums, etc) has an overwhelmingly negative view of our fellow man.
AI is trained on our data. Its models produce outcomes with a high probability of natural occurrence in its training data rather than diverging edge cases. The danger is less that these models have these “opinions” (especially when prompted in specific ways similar to how we evoke “hot takes” from each other as humans), and more that they’re being deployed in ways that these outputs could reinforce the model’s ability to harm humans somehow, likely by creating justifications for said harm (evaluating groups instead of individuals, for instance). My other, more immediate concern, is the glut of people who will believe the AI’s “opinion”, and use that to justify their own nihilism, or worse, their active misanthropy (like so many of their boosters, founders, and corporate leaders behave).
The machines aren’t the problem here, so much as the humans misusing them. Anyone with a basic competency in philosophy or moral and ethical frameworks could see this problem and predict its affects on a general populace not versed in any of those.