As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.
My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.
But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.
It really hit home for me when Google shipped their 'writing help' feature or whatever in gmail. Pretty much every sentence I wrote, it had useful tips for me like "make this more concise" and "reduce run-on sentence". No joke, I averaged about 1.3 suggestions per sentence.
And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.
> And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
I remember when I first used Microsoft Word maybe some 31 years ago (Windows 95 was that long ago, huh), and I had squigglies everywhere because of some grammar checker. "Don't use passive voice!"
At least back in those days, it was advice that I could research more about, and decide whether I wanted to act on.
Many people take AI's output at face value, even if merely used as a proofreader.
As you’ve described it nothing has changed. Word used to add squiggles and people who didn’t research the meaning blindly accepted the recommendation. Same shit. Different day.
"Google shipped their" -> "Google shipped its" (I use 'their' to make it clear I'm talking about the people who build features at Google, and not merely about a faceless corporation)
"feature or whatever" -> "feature" ('or whatever' is chosen to tell everyone that I'm not actually sure what the name of the feature is, and I can't really be bothered to figure it out)
"gmail" -> "Gmail" (meh)
"I wrote, it had useful tips" -> "I wrote received useful tips" (if you remove the comma, the pacing changes, the reader doesn't linger on the word 'wrote', and you invite some ambiguity around the writing and the receiving of useful tips. I feel like this suggestion is particularly brain damaged, I simply can't understand my own writing when I make this change)
(and so on. I now write emails in notepad, and copy-paste them into Gmail when I'm done, that's how much the suggestions annoy me).
Pacing is extremely important in writing. The way one speaks out loud is often not the way that they actually read in their head. Speaking out loud has three parts to it in conception, translation, and vocation. Conception as to what the thought is, translation of that thought to language, and vocation of the language through visuals or sound. Reading only shares the translation portion, but also has organization added to it where you need to organize the information as you ingest it. As such, it's important to format whatever you're writing so that it can be organized first and translated second. People look at the words linearly but don't ingest the information linearly. This is a principle many literary jokes are built upon.
ChatGPT, Gemini, and whatever else do not in fact have the capability to understand that interaction between organization and translation. They process the information linearly, organize it linearly, and translate it linearly. That's what makes it so strange to read. A must lead to B which must lead to C. A cannot connect to C and then circle back to B which is connected to both A and C obliquely. It's a failure in understanding language and how it is used.
In the UK it's common to refer to companies in the plural. "Nike have announced..." etc. . I like it. It's a reminder that they are in fact companies of people, not abstract entities.
LOL. The trick with Ai is to write whatever you want in any which way you like and it understands it no matter how shitty it is. Good writers should avoid asking Ai for suggestions. Bad ones like me, don't bother asking anyways. Too lazy to fix it all.
I learned in school (English is my 2nd language) that you can never use "their" or "they" about something inanimate like a corporation (or a chair, for example). You would use "its" or have to rephrase your sentence; "the developers at Google shipped their feature."
The suggestions are valid and correct, we (generic "we") are just used to not following proper grammar. This is how languages evolve but it is also what puts these assistance tools in a peculiar spot, because they help with accuracy, not everyday expressions.
When one interprets grammar rules as prescriptive rather than descriptive, it is easy to fall into the trap of believing in "correct" and "incorrect", when expressing intent is the single most important thing. Expressing intent is not a thing AI can do, thus far, as they don't want anything...there is no intention.
I'm certain there are folks out there who have a hard time with written English that could benefit from an AI editor, but I would almost always rather read casual English written by a human, even if idiosyncratic and "incorrect" sometimes.
And, to be clear: Hard to read English can never be correct, no matter how carefully it follows grammatical rules, and AI often writes hard to read English.
I think that is an interesting observation. You're comparing the "why" we have a language vs. the "how" the language is structured, and for sure the former is more important than the latter.
A side effect of studying programming languages in my youth was that I began understanding grammar better. Suddenly words belonged to "classes" and when thinking about "object references" in sentences, and the rules that followed, it helped me read and write more clearly.
I think I might be slightly opposite of you, but not very far opposite. I prefer reading a precise and accurate sentence that - when understood - clearly describes the intent without a shred of doubt much more than an easy to understand sentence that I find ambiguous because, to understand it, it requires context and a social agreement that nobody is trying to trick people.
I had a friend once that kept using open-ended phrasings and ambiguous descriptions because (I think) he enjoyed the feeling of power or control that the confusion and miscommunication introduced. Like if he were to "run to the stores" you could ask what he was buying and he'd follow up with a "oh, I didn't say I was buying anything, I'm just going for a run". If I have to guess which of the variations of meanings is the intended one, then the communication has failed in my book, regardless of how easy it was to understand the sentence.
In British English it’s entirely standard (and common) for grammatically singular collective nouns to take a plural verb and be referred to with a plural pronoun:
"The ManU have won three out of their last four matches."
"The Parliament have had heated debates over the NIH budget."
"Tesla introduced their newest EV model last week."
Any competent English teacher should know this.
Also, there’s no "correct" language outside of how it’s actually used. Linguists are the first to explain that their job is to describe, not to prescribe.
I’ve no idea what you refer to with "laziness". The "rule" that you claim you were taught simply does not always apply.
A smart writing assistant will give advice that’s appropriate to the context, register, the recipient’s culture, communication style and tastes, and the author’s relationship with the recipient. Sometimes writing very normatively and by the book is part of the message. Sometimes the exact opposite is true. If the assistant is unaware of all the context relevant to the communication, it’s likely to be a bad assistant.
Dropping articles is wrong because English as natively spoken doesn’t work like that. Except when it does, like news headlines, condensed "telegram" or SMS style, and so on.
- When all the members of a collective noun are performing an action as a unit (and that’s usually the case), use a singular verb.
- The chamber orchestra often plays at the Art Centre.
When to use a plural verb
When the members of a collective noun are performing an action as individuals, use a plural verb. In this case, all or some members of the group are doing something independently of the other members; the group is not acting together as a unit.
That applies to NA English. And even there, as the reference says, notional agreement (verb/pronouns in plural) can be used when it’s more about the individual members of the collective doing something. So GP’s rule that only grammatical agreement (verb/pronouns in singular) is ever correct is wrong even in NA. But in BrE notional agreement is the most idiomatic choice while grammatical agreement is certainly also fine, just not standard style.
"The army are advancing", if I wrote that as a kid, I would get an F and if a British wrote it, they would get an A.. For me "the army is advancing". Anyways I learned a new thing today.
Well, I think you've more or less explained my point exactly! The purpose of language is to convey meaning, and my using "their" for Google is precisely and intentionally in defiance of the idea that a corporation is inanimate. If you change that part of my writing, you actually change the entire voice and personality behind the writing!
And that's why it's so difficult to read the things that LLMs write. Because they follow the rules of language without embracing the chaotic nature of the fact that it's meant to convey information rather than to "be correct".
Another way to view this is as metonymy. "Google" is a metonym for "Google Staff", so using "their" to refer to the humans-making-up-the-corporation (as referenced by the metonym) is appropriate.
I agree it's correct English to say "Google have launched their new product".
This is prescriptive vs descriptive grammar, where descriptive grammar describes how people actually use language. Using "they" in that context is only wrong because someone or group decided that must be the rule.
In Swedish -
> Vi går till mitt hus
"Let's go to my house" - this is correct but not really said
> Vi går hem till mig
"Let's go to my place" - incorrect but the actual way to say it
> I learned in school (English is my 2nd language) that you can never use "their" or "they" about something inanimate like a corporation (or a chair, for example).
Definitely not a chair, but a corporation is also "just" a group of people. "Their" makes perfect sense for a corp, even if it isn't perfectly accurate
So, again, are we aiming for high correctness or high comprehension? "Company" makes sense because you can have a "company of five participants" and they can do something, e.g., select a wine pairing menu at a restaurant, but a corporation is typically "a thing".
Everyone who understands English at a professional level (and probably even lower levels, too - or perhaps especially those) will understand what you mean if you say "the corporation announced their financial report last weekend", but it won't be the correct use of "their", as far as I'm aware.
Similarly you can't, as far as I understand, append an "s" to describe a relationship of an inanimate thing, e.g., this would be incorrect: "the corporations employees went to the summer party" again because "corporation" is a thing. But on the flipside phrases like "the employees of the corporation went to..." sound a bit too - what's the word - posh?
What is "a thing", exactly ? Corporations are certainly ambiguous in that regard: they have legal personhood[1] and even free speech[2] in US, apparently.
In the other hand, no corporation ever really "announce its financial report". Someone does it on behalf of the corporation, technically.
> this would be incorrect: "the corporations employees went to the summer party" again because "corporation" is a thing
You can easily say "the corporation's financial report for Q4 was worse than predicted".
I think you're focusing too much on specific and hyper-specific rigid rules in a single textbook. As others said, prescriptivism vs. descriptivism. Descriptivism is always better, as it describes how language works and may evolve. Prescriptivism assumes languages are rigid unchanging things that can be precisely described.
The full rules for a working language take people years to learn at universities, and they are never as simple as "an inanimate object can't have a possessive form with 's".
Prescriptivism also works in schools/learning as it helps people not to drift too much from the norm. But too many take it to heart.
It likely that no response at all or "This looks good, no suggestions" are not possible responses. They haven't figured out how to detect when a conversation should just end or that no response is necessary and include that in the training data.
The critique of AI systems that lives rent-free in my head is that they "hit the target but miss the point." In writing, especially technical writing, the target is conciseness, word choice, tone, etc, while the point is to effectively communicate authorial intent.
Yea the hypothesis seems the exact opposite of what I find with LLM writing. It’s not that it’s overly dense instead I find that it’s hard to read because it’s so information sparse. Like you’re saying it feels optimized to “good writing” not to conveying information.
I find that it’s both: filled to the brim with facts and numbers, but they’re completely useless and irrelevant. For instance I ask it to come up with some ad copy for an Unreal UI plugin I made and it focuses on things like “Supports 64-bit Windows” or “allows transparent rendering” or lists out-of-context performance numbers.
Of course it supports 64-bit Windows - every Unreal plugin does! They don’t even make 32-bit Windows anymore!
Yeah; whatever the AIs are doing I wouldn't call it "information dense" like the abstract does. I'd prefer the hypothesis that there are complex patterns in speech related to successfully communication information to a limited human brain and the AIs aren't complex enough to model them yet. Or the signal is being obscured somehow.
I'm a writer, a technical writer, and a highly technical person in the executive space. Communication has always been one of my strong points, and the blathering on that AI does drives me to distraction. It's impossible to really tailor the output of LLMs very much. I value information dense, direct writing when it comes to technical things and they're always embellishing and going off on side quests and it's obnoxious.
As both a writer and SWE the problem with LLM output is the same. They can achieve localized goals but have trouble juggling multiple concerns at the same time.
Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.
This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.
Speaking of remembering, i just asked Astra about what temp to use and when to put on a pork shoulder for BBQ. It told me a temp and time and then went through and then did long winded explanation of all the variables that go into it's assessment and ended "in conclusion I recommendation X" where X was a totally different conclusion that in had earlier.
It also insisted that I start an hour early even though it was adding in 3 hours of rest time at the end.
No. The conclusion was not totally different. It recommended one of the explained options and recommended one extra hour as a buffer. I feel like this is a distinct reading comprehension issue if you think the final recommendation came out of left field.
Yeah but that entire wall of text was not needed. The user wants a simple answer that they can take action on.
This happens to me way too often. I just want a simple answer. I don't need a wall of text with 6 different options and a lengthy explanation. Just. Tell. Me. The. Answer.
> As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.
At least in the case of ChatGPT - it was (and mostly still is) objectively optimized for engagement. Since ChatGPT was leading the way, and much distilling has occurred, there are at least partial remnants of that problem everywhere.
The Chinese models may or may not really care about this at all, but they're doing a lot of distilling, so they're getting it whether they want it or not.
The American models for the most part care about engagement - some a lot more than others.
When I have asked claude to use proper vocabulary it wrote a memory "do not use blog-speak", so I guess the problem is that it is trained on blog articles much more that on good documentation.
And then it is also trained on it's our claude code sessions, so the more people accept claudish the more it becomes present in training data.
I feel like also all the code optimizations make it treat all language like code and that produces patterns that align with how it generates code. Earlier models that focused on language seem to have done better - 4o seems like language wise it’s pretty good but advancements elsewhere make it difficult to use.
This. The only “verifiable” writing is relatively short and verified through comprehension of that text. The way it writes is successful for that, but it doesn’t extrapolate to longer form, and under frequent use becomes problematic (sloppy).
I actually think they stopped caring about the prose a long time ago. Now all the resources get poured into making sure the right tool is called, remember the labs dont want humans to be the main users of these in the end. They want them to autonomously work on their own or with each other, and another agent doesn't care how an agent writes.
Maybe we'll get assistant models that will get optimized for human readable prose, but I think the labs have deluded themselves when it comes to utility/coding models and think the humam isn't going to be a variable soon.
I have a jira-esque ticketing system for agents/subagents and the tickets are basically unreadable to me. I hate it. But if I force them to write the tickets in a prose I prefer their preformance degrades. Its almost like this gibberish hard to read prose is designed for llms to communicate betweem themselves probably because the llms are doing most of the RLHF (RLAIF??) for training now.
I'm gonna be controversial, probably? The reason why AI is so tasteless and superficial may be that they contract it out to evaluators who otherwise are unemployed and averaged over the whole pool (this being the load-bearing qualifier XD) less skilled or tasteful than those who retained their job.
I've worked as a rater before, working on doing the actual rating and also rating the raters. When you're a rater, you get very detailed instructions on what to do. When you're rating the raters, you're given the same instruction set so you have to rate them according to how well the raters followed instructions. I assume that someone else rates the raters who rates the raters, based on the same instruction following.
The job is easy and pays really well. If you want to keep the money flowing in, you simply follow directions. Standing out and being different gets you noticed, and not in a good way (because you're just a faceless rater id #667383737).
One construction I'm surprised the paper doesn't mention (although, to be fair, I'm sure they had plenty of options to choose from... and you have to draw the line somewhere :P) is that LLMs love to use transitive verbs intransitively[0], in a way I almost never see in human writing. At best it makes sentences weirdly ambiguous, at worst it makes them borderline incomprehensible, and it's especially annoying in technical contexts where precise language should be non-negotiable. Like, yes, I can make an educated guess about what it's trying to say, but if your so-called documentation requires readers/users to make educated guesses then something has gone terribly wrong.
[0] An example would be something like:
"The user presses a button, and the form submits."
There are valid intransitive uses of "submit," but none that make sense in this context. In this case "submit" either requires an object (like "the form submits data"--plus maybe an indirect object, like "the form submits data to the server") or should be swapped for "the form is submitted" if you absolutely don't want to supply an object.
I sort of agree but I think the issue is more subtle.
Languages change over time, and one of the well-documented "laws" of language change (using that term very fuzzily) is that people like to take irregular features of a language and generalize them to make them more regular. And there are many English verbs (particularly verbs expressing a change of state) which can be used transitively or intransitively, with equal validity:
> "I opened the door" -- "the door was opened" -- "the door opened"
> "the heat melted the ice" -- "the ice was melted" -- "the ice melted"
> "the impact cracked the glass" -- "the glass was cracked" -- "the glass cracked"
This generalization has been common, especially in jargon-y contexts, long before the rise of LLMs. A pedant might insist on saying "wait for the compiler to compile the code", and Merriam-Webster thinks "compile" can only be used transitively. But I think most developers would say "wait for the code to compile [i.e. be compiled]" is fine too. (see e.g. this XKCD comic from 2007: https://xkcd.com/303/)
In my view, this usage is completely fine as an occasional thing, especially when the social process of language change has normalized it over time for a particular verb. The issue is that LLMs drastically overuse it and pepper it everywhere. Just like "it's not X, it's Y", which is fine occasionally in human writing, but has become a dreadful cliche through overuse.
It's definitely context-dependent; it actually took me a minute to come up with a plausible example where I didn't immediately think of a counterexample of why it was a perfectly valid usage, lol.
Even as a grammar weirdo I'm not wholesale against dropping objects from verbs, especially when the object is strongly implied, or the original object becomes the new subject, like in "the heat melted the ice" > "the ice melted". But if you're writing for precision (and I'm a technical writer, so when I'm on the clock I certainly I am :P) there are times where those constructions can obscure useful details, even if they're not grammatically necessary. And there are other examples that I'd struggle to defend in any context, and frankly just give me the ick.
I was trying not to single out any actual sources I've seen, but here's one I saw the other day that made me irrationally mad, and that I think goes beyond the organic changes you mentioned:
> Talking also costs.
It costs? Costs what? Five dollars a word??
The rest of the passage made it obvious that the cost in question had to do with tokens (blatant LLMisms? in a blog post about LLMs? it's more likely than you think) (also ironic considering TFA's conclusion about this writing style being a way to reduce token usage), and maybe one day this will become a perfectly accepted usage and I'll be the old fogie moaning about how it should really say "has a cost" or whatever. But fuck, I hate it, and not just because it's an obvious tell that humans aren't steering the ship.
I think LLM-speak is heavily biased towards using as few words as possible, and as basic words as possible, even when that hurts overall clarity. To use another XKCD example, it often reminds me of "Up Goer Five": https://xkcd.com/1133/
Maybe a hot take, but I think that example is fine. Less verbose than “the form is submitted” and there’s no ambiguity, it’s the English analogue to `form.submit()`.
In your other comment, “Talking also costs.”, it’s clarified through context that talking costs tokens.
I think it’s a side-effect of the AI companies trying to reduce verbosity and redundancy, which were my personal main complaints with LLM writing. Though I believe LLMs are still verbose, and I suspect it’s because they don’t know what to remove, and these are cheap gains where the LLM removes verbosity that is obvious but expected because everyone does it.
LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language, as opposed to a property of "mind" (either biologically or artificially instantiated.)
LLM-generated text is hard to understand because while it's perfectly good language that follows valid cognitive and informational patterns, it does not reflect the intent or mental model of another mind.
> LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language
This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.
Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.
Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since language is only a single, limited aspect of real cognition, this makes them so annoying and alien to read.
I think we're in violent agreement here. Language is part of how we think. Definitely not the whole thing -- we are physically and biologically instantiated and LLMs are not. Language is a software, but the hardware does matter.
But LLMs DO show us that language is a big deal. It isn't just a serialization protocol for real thought -- it's a material component of how we think.
In many ways I think LLMs are the strongest possible proof of structuralist and post-structuralist theory which argues that human cognition is conditioned on the language and concepts to which we have access.
Most of the effort I put into writing is modeling the reader's mind. I never would have known this if someone hadn't invented a soulless writing machine that faked this process!
Consider too that AI training corpuses include vast amounts of commercial speech which is designed to numb the critical faculties and produce action; likewise they include numerous opinion articles, which are designed to persuade and often leverage rhetorical fallacies, whether cynically or as a reflection of an author's imperfect thought process.
This theory doesn't hold up when you use base models (i.e., models with only pretraining, no RLHF or instruction tuning). They sound much more human. I've tested pairs (like Llama 3 base vs. instruct), and the base model can match human style much better than the instruct version can: https://www.pnas.org/doi/10.1073/pnas.2422455122
I think it's not the raw training corpus, but the samples collected specifically for reinforcement learning, direct preference optimization, or the other instruction-tuning steps that get a base model to turn into a chat/reasoning model. (We're working on testing this with the Allen AI models that have open training data.)
The slop-in, slop-out theory. I'm very much an advocate.
The neat thing about LLMs is that in a way they proved that only the truth turns out to be grammatical (statistically), largely because there are so many more ways to be wrong than right that the mode is almost always right.
But people also lie and bullshit in consistent ways, with the intention of fooling people with well-known empty arguments. Part of being a "good writer" is to make the meaningless sound meaningful - and the techniques are ancient.
Train on a bunch of people trying to push their framework and get rich, whether in online garbage, literary garbage, or in ostensibly serious nonfiction or scientific works, and you'll end up with modal language that is trying to play deceptive games with reasoning, but has nothing in particular it's trying to convince you of (no motives.)
People would either fall for that language (if they weren't thinking much and the context was one that comforted them), or they would methodically try to put that language into a model (let's say a series of syllogisms for an argument about what to do or what to believe, but it could be any model), and see that it failed. And speaking again of the context, the context of humans interacting with human content implies motives that help the listener focus on particular parts of their model to look for rhetorical tricks. The LLM, again having no motive, just does this in random places.
The latest advances in AI have been in the tooling around AI, which I think largely reduces to recursively creating models and populating them in outputs short enough that hopefully the text generation won't go off the rails.
Symbolic AI is coming back, it's just that the mortar is random word association.
Call me crazy, but I think Georges Bataille saw all of this. In my interpretation, he simultaneously thought of random associations as the most important component of reason and also completely irrelevant in the reality of the world. The only way you could discover the structure of things is through random collisions; your view of the universe was entirely constructed by your prejudices, lusts, and fears; but whether that view was wrong or right was based on how it worked and the particular content of the id-garbage that got you there was ultimately irrelevant. The fabric of the world is completely id-less, but can only be digested by id-driven beings. LLMs (without the tooling around them) are all id.
As usual most commenters reply to headlines, so I quote their conclusions.
>We take these results as a proof of concept that a length penalty used during reinforcement learning with GRPO can directly affect grammatical features. We note that the model was not directly instructed to use compressed or dense language, and was not directly given information about the availability of syntactic alternations. Rather, the penalty resulted in an emergent preference or shorter members of syntactic alternations. [...] We speculate that the use of verifiable rewards creates an environment where correctness is decoupled from humanlike communication, leading models to converge toward a novel, potentially less comprehensible distribution of language use.
This sounds like common sense to anyone who tinkered with the models and training, however someone still needs to quantify it so it's good they did. I also want to note that this is only visible in Claude's default register, it's still able to write differently when prompted.
That, and one can feel that there is no intention behind the text. There are no introductions and main points and fun asides, just a mechanical, steady stream of words with much too frequent, quasi-randomly distributed fake "main points".
That's like saying that the roaches in the food can be excused because they occur in the selected cheap ingredients and it would be more expensive to remove them. Makes some sense, but I'm not going to embrace the bugs anyway.
I've been iterating on this writing clarity prompt. The term we came up with was "Asyndetic parataxis with referential ellipsis and repeated rhetorical closure." It doesn't fully describe the exact issue I have with modern AI writing, but it's close.
AI has a way of using abstract language, sounding impressive and eloquent, but in reality it is mostly incoherent fluff. You can see this in humans too. Some politicians and academics are good at expressing themselves in clever ways but when you actually try to parse what they say it makes no sense what-so-ever.
No it doesn't. It produces working, but sloppy code, and it's able to do so because of deterministic test suites that run quickly and give a reliable score of the result, letting an agentic loop iteratively improve its output. But we have no test suites that answer "is this code maintainable" or "does this feature make sense in the context of our business", so AI still fails those tests.
If we had a way to write a test suite for English that could reliably answer "is this sentence useful to my target audience" or "does this paragraph make any goddamn sense", you might be able to run some sort of agentic loop to approach good writing, but so far we don't.
> But we have no test suites that answer "is this code maintainable" or "does this feature make sense in the context of our business", so AI still fails those tests.
I've been struggling with this because I do believe maintainability of code is important. But the way it is going, if AI is writing all the code and no human needs to read it anymore, does it really need to be maintainable? Does it matter if it looks like alien code since we can throw it away when using the next AI model? I have no answers of my own to these questions...
In my work I use AI for code generation in two distinct ways:
- For throwaway prototypes I pretty much vibe code, only making manual tweaks occasionally if the agent gets stuck or something looks wrong visually.
- For production code I stay in the loop, reviewing and iterating on the code with an agent.
This leads to me having experience with “messy” prototype code and “maintainable” production code. Whenever my prototype code lives longer than expected it gets to a point where the
agent starts introducing regressions or churning on changes.
This happens much less often with the “maintainable” code. From my perspective it seems pretty clear that LLMs are faster, more effective, and less error-prone in a well structured codebase
> But the way it is going, if AI is writing all the code and no human needs to read it anymore, does it really need to be maintainable?
It's more important than ever, because code is just a way of expressing intent. If the code becomes overly complex, it de-coheres, and intent is lost across the system you are building.
This is why vibe coded projects turn into slop and eventually stop iterating: at some point that actual intent is too vague and contradictory because of the many layers of code that have been added.
You can verify that code is good if it works - it might be ugly or hard to understand, but if it completes the goal and doesn’t crash then it’s good. You can’t really do this with prose though because the text is the thing itself - it needs a human to score it (perhaps another way would be to have other agents ask it questions about something they don’t know, and see if the agents can discern the answers from the output - but that’s still not going to determine if it’s good for human consumption).
I think it is more about comprehension. It reveals that AI does not actually understand what it does. It gets code right mostly because it is probabilistic and has seen enough sound precedents to land in something that works, but it does not come from an internal comprehension.
Good point. I also think that the training data is different in this regard. It has trained on (mostly) verified, correct code, but when in comes to free text it has trained on a lot of unfounded opinions.
I wonder if it's so circuitous because it buys the model more tokens for some sort of pseudo-reasoning before it locks in a token where there's no turning back. like it lets it push the logprobs way way way towards the right answer.
Plus using all this made up jargon and the unclear references to everything else (im referring to like the 'it' "XYZ is implemented using ABC. 123 referrs to it" you get in claudeslop) is very hard to understand but also kind of impossible to be wrong no? If it said "123 refers to ABC" but meant XYZ instead of ABC it's SOL but instead if it says "it" future tokens can ramble themselves out of being wrong thanks to the annoying slop ambiguity.
the problem with those hypothesis is that models don't in fact all talk the same (unless they're distilling claude). however if I unburden myself from having to be correct and allow myself to make up some more bullshit - what if "its not x its y" helps compute the right answer. Attending to that 'not x' could help point in the right direction in whatever dimension in latent space it needs to find the right y — and what if emdashes are proletariat filler tokens.
probably tho they're just RLing them to hell and back on code stuff and not paying attention to it and none of that is real lol. IIRC they rled opus 5.5 on talking more normal and it does and is also better than opus 5 so like with most things here, just another product decision on where to burn GPUs
Its because AI was not optimized for human reading, its optimized for the most amount of information in the lowest amount of context or tokens as possible.
(e.g., "baseline system state transition latency validation") instead of unpacking them with prepositions ("validating the latency of state transitions in the baseline system").
For me, I can't read books at all, because I forget what was said two paragraphs before. I bet people that are good at skimming text have less problems with AI generated text.
> For me, I can't read books at all, because I forget what was said two paragraphs before. I bet people that are good at skimming text have less problems with AI generated text.
Actually LLM generated text is almost impossible to skim. It's too flowery and the actual meaning is spread too thinly across the text, which means that you can't just read the "important" sentences of a paragraph.
In my mother tongue "baseline system state transition latency validation" would have grammatically been one word. I sometimes miss that in English. If you're tired and see "baseline system state transition*s* latency validation", note the hallucinated "s", the meaning changes and you need to read it again because you're unsure if you actually did see an "s" or if it's "one thing".
"Baselinesystemstatetransitionlatencyvalidation" is surprisingly enough less confusing to me because I know it's one word describing one thing, and my eyes registers each joined word with (years of) training, even though what my eyes look at might be the following chunks in order:
My eyes essentially scan the word, and although it might be a tad bid slower than reading "baseline system state transition latency validation", reading and understanding is faster because I already know it's one thing before I know which thing it is.
Edit: The example would be easier to demonstrate if HN supported bold font, but I don't think it does (or know how).
Anecdotally, AI is very bad at distinguishing the important and relevant from the mundane. I tell AI to suggest changes to the “high-level specification” and it constantly suggests documenting arcane edge-cases and implementation details.
"Prompt injection as role confusion " [0] work showed us, that diffetent roles produce different types of text, which smell differently, have different structure.
Thus it should be possible to fine tune models to produce more human readable text in assistant role while keeping information density in thinking role. Or maybe not. That is interesting to test.
Well that’s exactly the authors’ claim. Legalese and patents intent is not to facilitate communication but to maintain a claim under the most bad faith, hostile reading.
AI language, it seems, is optimized to support reasoning from the interior monologue.
I thought patents were deliberately bad? Like, the point is to get the protection, despite paying as little lip service as possible to the "it's in society's interest that inventions get published" side of it?
It argues that training for "math, code, and reasoning" caused this. Do LLMs only trained for natural language (probably older models) have the same characteristics?
Does training for these additional domains help in some ways, and this specific style of writing is just a side-effect of that?
LLM reinforcement learning improves those tasks by having the agents "reason" until they hit a verifiable output state that passes a black or white test. So they learn language which is tuned to the task in a silo, never communicating with anyone or anything.
In contrast, most human communication and writing is intended for other humans, and we constantly hit failures here that we learn to iterate on and improve. Humans who do math do so with the constraints that their math needs to be reviewed by another human and in the context that all their writing needs to be understood by another human.
The process causes a warped and compressed version of language.
Optimizing for contradicting objectives is always hard, this behavior will always leak this way or another. Even basic instruction training will affect writing produced piece by piece, because the structure of a typical assistant reply will leak into each piece, for example making characters use narrative echo to annoying degrees. Older models (that did indeed write better) still had this particular quirk. This can be alleviated by using assistant prefill, and its removal from frontier models is a disaster.
Of course if you optimize for natural-sounding language it will learn to sound more natural.
> AI-generated text is increasingly optimized for other goals like math, code, and reasoning
Yes, this is a problem.
AI-generated text is used internally by the AI to instruct agents, etc. And for feedback loops in reasoning. If you optimize for the result only, the AI will increasingly speak "AI", not human.
I think we can make it do both, but it is much harder because we have no algorithm to validate an AI's internal mumblings against proper, human-understandable English.
Code is not prose, yes, but as others pointed out, it is certainly nice if code is maintainable by humans.
Ai generated text being too concise is not a problem I’m encountering. What we’ve begun to term “Claudish” tends to more closely approximate those “become an expert in X using this one weird trick“ articles that go on and on and never quite seen to get to the point.
Usually I’m getting better results by telling it to be more concise.
I donn't find AI writing hard to understand at all, just a bit tedious. Though to be fair I suspect many of the same criticisms leveled against AI writing could be applied to my own, as many AI'isms are things that I tend to edit out for clarity when given the opportunity.
Em dashes are perfectly acceptable — nay, desirable — linguistic constructs, however, and I'll stand with Claude and die on this hill.
This is complete speculation, but I wonder if the "thinking step by step" paradigm, when under reinforcement learning pressure, also encourages grammatically ambiguous sentences (like garden path sentences in the extreme case) that lets the model "keep its options open" until very far into the sentence.
The think I don't understand is: why did the models become so much worse with time?
I seems to remember a distinct time during which Claude for example became incomprehensible and weird with its writing, whereas before it was reasonably OK, even if still not that good
Maybe new models training on the previous generation’s synthetic output is compounding the effect? LLM Ouroboros.
I find Claude’s writing unbearable. That ChatGPT offers personalization options (Base style, warmth, enthusiasm) and Claude doesn’t suggests Anthropic doesn’t recognize this as a problem. Plus, people sharing ChatGPT writing won’t all have the same voice.
Especially for general purpose technical writing, system design, architecture and code comments, I do use pstack unslop [0] and technical-writing [1] skills. They do wonders.
Ease of comprehension is not prioritised in training. I doubt frontier labs would spend enough money to train on something that is non-measurable and subjective.
One factor might be this: LLM's know orders of magnitude more stuff than humans. They are experts in whatever subject is being discussed at the moment. I think they often respond as if the human user has a similar baseline of factual knowledge in every field.
People who are knowledgeable in a particular obscure area tend to do the same thing.
It's painful enough that I find myself now consistently asking for "8th grade reading level", "children's story book", or now "ASD-STE100" English in all my prompts. Somehow just that tiny bit of steering constrains the language choices significantly.
Because AI's language isn't a language at all, but its imitation. Imagine you're given a map skillfully created by someone who has seen many maps, but never bothered to understand their meaning. How much time will you waste before realising that this map is an imitation?
This is an absolutely critical issue; it's impossible to work with AI that can't communicate coherently. It's barely usable for reasoning on complex issues because of the language problems described exactly in that synopsis.
We had a funny chat at work. Two of us were light-heartedly joking about how you never paste the AI at people.
A third team mate uses a voice dictation tool to get down the basic ideas and then massages it with AI, but these two guys rib each other about it.
Another my other team mates (team mate 4 in this story) sheepishly turns around and asks if people can tell when he does it? (It's EXTREMELY OBVIOUS that he isn't writing anything) I confirmed it, in a friendly way. He defensively said he tries to get the AI to use his own voice.
I casually mentioned that "writing is thinking" or something like that and how I would prefer to think the thoughts. He didn't say much.
The text data used in the training of large language models often reflects human expression, which tends to elaborate on context and express precisely. This results in the model's generation tending towards complexity, rather than serving human communication in a conversational manner.
Maybe I'm the odd one out, but I really don't find AI generated language, in almost any model, to be hard to understand. I'm not trying to be snarky or annoying or pompous or whatever word might be "this guy over here think's he special."
It uses a lot of metaphors. It references stuff implicitly - a LOT - and assumes pretty much all the time that the thing it did is what you saw, which you didn't. But if you just.. read the stuff it says, and know a few old-school sayings and thesaurusize your reading style, it usually says things quite meaningfully.
I think a lot of people just don't read what it writes. So many folks I've watched used LLMs don't write in complete sentences, don't read the sentences that come back, don't watch the turns and tool calls in the middle, and then just respond to the message as if it says something it doesn't. I'm saying all this before I read the paper, but I've done a lot of personal and team experiments with this, so I'm quite curious to see what they say. I'll edit this after I go through it.
--
Fascinating! I did actually run into a lot of this with my personal 'engineering and communication styles' experiments. Most of the papers out there talk about models moving toward human language inside of a context window, and not the other way around (much smaller effect for human->model). I had a few years of conversations I ran on local and 'frontier' Claude and found that, yes, in general, information density is what each response turned out to be geared toward, whether it was everyday health discussions or deeper engineering topics and research. The component tests they ran I've been exposed to already, but I think their conclusion nailed it:
```This discrepancy
may help explain why many frontier LLMs use
linguistic constructions which are technically
allowed by the grammar of English, yet are out-of-
distribution for human language and are associated
with comprehension difficulty.```
My experience is: The LLM fails to "understand" or "intuit" who the audience is. It tends to reward its own recent context and its interaction with the prompt writer, rather than "considering" documents from the viewpoint of a hypothetical reader. You can often prompt them to go more in that direction but it gets tiresome.
The best approach is just to write the outline and initial text in a very messy fashion yourself. And then have it clean it up and add detail. And then edit again by hand.
Yeah this drives me up a wall. You ask any AI to make a change and then it thinks polluting your README.md with those details is appropriate. No I don't think "this project deliberately does [random detail]" is needed when that detail is irrelevant outside of recent context.
I'm to the point where I don't trust it to do any writing. It's only allowed to tell me what it thinks I missed. Because inevitably when I ask it to edit my prose, it destroys the text and I basically need to rewrite it all again.
I have seen people saying this for a long while now. I don't know how to say this but I understand Ai stuff better than non-Ai. When I see people complain ; I always wonder why because I think that its pretty good with my own senses.. Much much better than my writing skills... My brain has been tuned by computers I suspect .
I understand what you are saying but I personally can't tell what it is that people like or dislike other than myself. In a way I accept differences in grammars and assay formation in any which way that is produced but some scientific journals written by humans are unreadable in one pass without reading them over and over. Ai produces much simpler language . To me Ai is saying things in the way that my brain can process quite easily.
I skimmed the paper, and it seems to be saying something different.
Basically, there are often different ways of putting a particular idea into words. Sometimes the longer way might be clearer, and the shorter way might be more confusing. If you care about being understood, sometimes you will use the wordy option. This is what humans usually do.
However, LLMs have context windows. All else being equal, verbosity eats up more context. If the LLM is trained (with reinforcement learning) to reason most effectively, it can do that by using terse phrasing. Especially on certain tasks that really need a large context window. Why? Because being terse is like getting a larger context window for free.
But in another sense, it's not really free. You're trading off clarity for context window. Sometimes you have to choose between verbose+clear and terse+unclear. If you're biased toward terse, that will also shift you toward unclear.
I do agree that LLMs are verbose sometimes, though. I think they are also trained to please the user, and they see sharing extra knowledge and insights as a way to do that. I don't think this idea is in conflict with what the paper says. If an LLM's training rewards both spitting out neat bonus info and doing reasoning that requires making the most of its limited context window, then that LLM may generate a large quantity of text which is written in a slightly terse and confusing style.
Have you read entirely generated documentation of code? I wouldn't say it's not understandable but it's no where near good quality human documentation.
I think it’s a reading comprehension level kind of thing. If I slow right down, it’s all there and understandable. But the quality of prose is terrible. It doesn’t lend itself to fast skimming. So it feels like I have to slow way down to understand it. To be fair a lot of humans at work write this poorly as well.
A combination of words that are infrequently observed in natural language. For example in the preceding sentence some of the 3-grams are "combination of words", "infrequently observed in", and "words that are". N-grams are just this for all values of n. They are saying that often times the exact sequence of a short fragment of text is not very common in human written text.
Unusual word combinations, basically. They’re word combinations that are rare in the training data, i.e. rare in normal language, but used more frequently by AI.
They’re difficult for humans because if a sequence of words hasn’t previously been seen, you have to expend more effort to make sense of it.
Something like “customer onboarding workflow automation" might qualify - it makes sense, but it’s not a commonly used phrase. AIs use a lot of these jargony sequences.
I know the paper specifically calls out Opus 5.5 and Opus 5, but I can’t see different versions of Sol, or Astra.
I think the difficulty in comprehension was mostly an Anthropic problem… partly because Anthropic employees stopped reading Opus output around the time Fable came out and stopped noticing RLVR going off the rails with compression.
The paper was a good read, in any case. Maybe I gotta start sending this article back to LLMs when they start compressing text too much :)
I find AI generated language hard to understand when it is "in it's own world" and it knows what it means but can't articulate it without buzzwords and slop. A follow up "ELI5" type of prompt often sorts this out. Seems like a failure mode and you need to snap it out sometimes. Opus 5.5.
Yes, I tend to find the longer a context window grows, the more the model tends to come up with its own jargon to describe previous findings, which, if you haven't kept up, starts sounding incredible dense; certainly not something I'd share with anyone else, i.e. in a PR comment.
I regularly prompt it now to "summarize again without the jargon". That helps, for now, to bring it back to using plain language. Together with my basic prompt to reply using ASD-STE100 Simplified Technical English (https://en.wikipedia.org/wiki/Simplified_Technical_English)
This is funny to me. For a while, I thought that Google's Gemini could give good insight and guide me better than a Tech Support representative could. It certainly seemed the right choice if I had questions about my Chromebook or Pixel Phone, or Android.
So I tried it out, and it really sometimes works, and I've overcome some real obstacles that Tech Support was unable to assist me with. However, Gemini can also become "too big for its britches" and begin explaining glitches from a systems-programming perspective, as if it has deep knowledge of the Android codebase, and deep engineer's knowledge of the Pixel system hardware. And it will spit out absolute nonsense.
I have retorted to it a few times, "that sounds more like Star Trek than a sound appraisal of the subsystem" and then Gemini smirks and says "You're right to call me out on that!" and blithely continues on its way.
So yeah. Sometimes the LLM confabulations are deeply cloaked in highly technical jargon that could not even be questioned, unless you already know it's absolute bullshit. And then the LLM freely admits it! LLMs have no shame, that's for damn sure.
Because people love to add slogans, catch phrases and witty sentence fragments, and AI was trained to mimic people. AI has shone a spotlight on it, and because AI isn’t perfect, it tends to overuse and use it where it shouldn’t. But go read news articles, blogs and corporate press releases from before the AI boom and take notice how often AI “tells” also show up there. The internet made hot take mic drop sentence fragments, “witty” turns of phrase and clickbait “it’s not foo, it’s bar” cliches a core part of its lexicon. It really shouldn’t be surprising that it shows up in AI too.
Thank gods I'm not the only one noticing this! When humans write impenetrable prose, most of the time it will be too verbose. Big words, too many words, long sentences, being parenthetical, that kind of thing. LLMs, however, make their writing hard to understand by being too terse in a peculiar way, where "the caveman" is also "the mad scientist". I've noticed both Claude and Codex often invent their own jargon to explain what the code does and what their fixes are about. It's irritating, but I thought maybe I was getting senile and could not understand a simple sentence.
AI language will converge towards maximum entropy, like compression, due to cost pressure to conserve tokens. LLMs are not limited by the human 5+-2 working memory
FWIF, I find I don't have much trouble. I think it's related to how I'm a software developer who also found as an adult learner German easier than French. From the abstract:
inanimate subjects with animate verbs
process waits for input, a method threw an exception, server accepts the request,
German has longer syntactic dependencies than English because readers (and listeners) often must *under* several intervening clauses of the sentences read before they can *stand* the split-apart word or what the author was saying.
I'm a fluent reader. Heck reading has been a primary hobby since I was a little kid. But also I'm from a Caribbean country where English is the official, but not the native language. And I've been studying/exposed to Japanese for quite a few years. Maybe that diversity/stretch makes a difference.
That makes sense, because fluent readers are using more System 1 processing to read - they’re not thinking about it for the most part, they’re just pattern matching instinctively based on text they’ve been exposed to previously (“trained on”). AI writing breaks those patterns so compromises their fluency.
Whereas a second language reader or poor reader is exerting more conscious attention to understand the text, so it doesn’t matter as much if it doesn’t conform to standard patterns.
Except LLMs are "System I" writers. You'll be more satisfied with what they've written if you haven't really understood it and just let it wash over you.
It only falls apart when you actually try to figure out what's being said.
I find the most passionate advocates of LLMs right now to be the people who are
1) also the most passionate advocates of not reading or reviewing their output in any detail. They seem to be actively annoyed by the idea that people think they should. And
2) people who have a strange theory of mind about LLMs that excuses the low quality of their output. I've heard many times people advancing the theory that LLMs are coding in a particular way because the way that LLMs code is the way that they (the LLMs themselves!) are best able to work with the code.
In fact, I get the impression from these people that they think that their touching the code would actually ruin this delicate balance that the LLM is trying to make. This is actually crazy thinking. On a pretty obvious level, they already think that any LLM they encounter is a super-intelligent God acting in mysterious ways, and that they only way you can ruin its plan is by questioning it.
> I find the most passionate advocates of LLMs right now ...
Whether or not that's true, you have to keep in mind that it's not true of everyone.
I'm a passionate advocate of LLMs. I implemented my first neural net, for image recognition, in the late 1980s (based on Bart Kosko's work, I'm not claiming to have invented anything.) Now, I use LLMs daily for coding and other tasks. I don't usually "vibe code" unless it's a simple enough project that the code quality really doesn't matter - a simple example would be data integration, where as long as the data in the target system matches the data in the source system, I don't care much about how the sausage in between is made.
But otherwise, I'm approving or denying every tool call, like a caveman, and when they mess up, I explain to them why. I interrupt them and redirect them when they need it. I change code myself and let them know. With all that, they're still incredibly useful - more useful than most "junior programmers", realistically. No human software developer can hope to match LLMs on many significant dimensions of development. But that doesn't mean experienced and competent humans have no role, at least currently.
The disconnect here is that your characterization of "the low quality of their output" is too simplistic. The quality of their natural language writing is different, for the kinds of reasons stated in the OP. Some of the accusations of "lower quality" really boil down to "too compressed for humans to easily understand", again as the OP points out. And the quality of their code has a lot to do with how they're prompted.
One of the biggest barriers to good LLM output - both writing and code generation - is that they're trained on data generated by humans who fit George Carlin's observation that half of them are stupider than average. When it comes to coding, on average they don't have enough education not to think that imperative code with minimally-restricted mutation all over the place is a good idea. They think dynamic typing, so that code can't be statically verified, is a good idea. They reflect our own collective limitations. If you recognize that and redirect them to sources of good practices, they can be much better.
In short, they're much more of a mirror than most people realize or admit.
With all this noise, it should not be forgotten that we appear to have figured out roughly how a significant component of intelligence works, and have real life, useful proofs of concept. All the technical criticism, to me, is just coming from people who don't seem to understand how intellectual advancement works. The political criticism is another matter.
What's the test for understanding? I find that I know what all the words mean, just like in Markov chain text. In LLM text I also know what the sentences mean. In both cases I don't always know what the document means.
This is likely relative to my existing knowledge. Humans are also LLMs, and a human without prior training will be able to transfer-learn slop from a computer LLM, but a human with prior training may find the attempt at transfer-learning conflicts with what they have already learned, and reject it.
In less ML terms I think a human who reads slop and has no existing knowledge of the subject will just learn the slop by rote and confuse it for actual knowledge.
Dunno about any test, but I get the meaning behind the words in a way that I can take appropriate action (as someone using LLMs heavily for dev and always following the transcript so I can properly steer them). And often I see people here screaming "AI text" or similar regarding some linked article, and having read it I found that I just bypassed all the "tells" without realizing, because they didn't affect my absorption of the meaning of the article. Makes me think people are consciously looking for those tells so they can scream and shake their pitchforks.
As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.
My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.
But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.
It really hit home for me when Google shipped their 'writing help' feature or whatever in gmail. Pretty much every sentence I wrote, it had useful tips for me like "make this more concise" and "reduce run-on sentence". No joke, I averaged about 1.3 suggestions per sentence.
And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.
> And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
I remember when I first used Microsoft Word maybe some 31 years ago (Windows 95 was that long ago, huh), and I had squigglies everywhere because of some grammar checker. "Don't use passive voice!"
At least back in those days, it was advice that I could research more about, and decide whether I wanted to act on.
Many people take AI's output at face value, even if merely used as a proofreader.
As you’ve described it nothing has changed. Word used to add squiggles and people who didn’t research the meaning blindly accepted the recommendation. Same shit. Different day.
[For example, gmail suggests]
"Google shipped their" -> "Google shipped its" (I use 'their' to make it clear I'm talking about the people who build features at Google, and not merely about a faceless corporation)
"feature or whatever" -> "feature" ('or whatever' is chosen to tell everyone that I'm not actually sure what the name of the feature is, and I can't really be bothered to figure it out)
"gmail" -> "Gmail" (meh)
"I wrote, it had useful tips" -> "I wrote received useful tips" (if you remove the comma, the pacing changes, the reader doesn't linger on the word 'wrote', and you invite some ambiguity around the writing and the receiving of useful tips. I feel like this suggestion is particularly brain damaged, I simply can't understand my own writing when I make this change)
(and so on. I now write emails in notepad, and copy-paste them into Gmail when I'm done, that's how much the suggestions annoy me).
Pacing is extremely important in writing. The way one speaks out loud is often not the way that they actually read in their head. Speaking out loud has three parts to it in conception, translation, and vocation. Conception as to what the thought is, translation of that thought to language, and vocation of the language through visuals or sound. Reading only shares the translation portion, but also has organization added to it where you need to organize the information as you ingest it. As such, it's important to format whatever you're writing so that it can be organized first and translated second. People look at the words linearly but don't ingest the information linearly. This is a principle many literary jokes are built upon.
ChatGPT, Gemini, and whatever else do not in fact have the capability to understand that interaction between organization and translation. They process the information linearly, organize it linearly, and translate it linearly. That's what makes it so strange to read. A must lead to B which must lead to C. A cannot connect to C and then circle back to B which is connected to both A and C obliquely. It's a failure in understanding language and how it is used.
In the UK it's common to refer to companies in the plural. "Nike have announced..." etc. . I like it. It's a reminder that they are in fact companies of people, not abstract entities.
LOL. The trick with Ai is to write whatever you want in any which way you like and it understands it no matter how shitty it is. Good writers should avoid asking Ai for suggestions. Bad ones like me, don't bother asking anyways. Too lazy to fix it all.
I learned in school (English is my 2nd language) that you can never use "their" or "they" about something inanimate like a corporation (or a chair, for example). You would use "its" or have to rephrase your sentence; "the developers at Google shipped their feature."
The suggestions are valid and correct, we (generic "we") are just used to not following proper grammar. This is how languages evolve but it is also what puts these assistance tools in a peculiar spot, because they help with accuracy, not everyday expressions.
When one interprets grammar rules as prescriptive rather than descriptive, it is easy to fall into the trap of believing in "correct" and "incorrect", when expressing intent is the single most important thing. Expressing intent is not a thing AI can do, thus far, as they don't want anything...there is no intention.
I'm certain there are folks out there who have a hard time with written English that could benefit from an AI editor, but I would almost always rather read casual English written by a human, even if idiosyncratic and "incorrect" sometimes.
And, to be clear: Hard to read English can never be correct, no matter how carefully it follows grammatical rules, and AI often writes hard to read English.
I think that is an interesting observation. You're comparing the "why" we have a language vs. the "how" the language is structured, and for sure the former is more important than the latter.
A side effect of studying programming languages in my youth was that I began understanding grammar better. Suddenly words belonged to "classes" and when thinking about "object references" in sentences, and the rules that followed, it helped me read and write more clearly.
I think I might be slightly opposite of you, but not very far opposite. I prefer reading a precise and accurate sentence that - when understood - clearly describes the intent without a shred of doubt much more than an easy to understand sentence that I find ambiguous because, to understand it, it requires context and a social agreement that nobody is trying to trick people.
I had a friend once that kept using open-ended phrasings and ambiguous descriptions because (I think) he enjoyed the feeling of power or control that the confusion and miscommunication introduced. Like if he were to "run to the stores" you could ask what he was buying and he'd follow up with a "oh, I didn't say I was buying anything, I'm just going for a run". If I have to guess which of the variations of meanings is the intended one, then the communication has failed in my book, regardless of how easy it was to understand the sentence.
"I had a friend once that kept using open-ended phrasings and ambiguous descriptions"
A failure to express intent, not a failure of grammar. Perfect grammar can mislead just as easily.
In British English it’s entirely standard (and common) for grammatically singular collective nouns to take a plural verb and be referred to with a plural pronoun:
"The ManU have won three out of their last four matches."
"The Parliament have had heated debates over the NIH budget."
"Tesla introduced their newest EV model last week."
Any competent English teacher should know this.
Also, there’s no "correct" language outside of how it’s actually used. Linguists are the first to explain that their job is to describe, not to prescribe.
[flagged]
I’ve no idea what you refer to with "laziness". The "rule" that you claim you were taught simply does not always apply.
A smart writing assistant will give advice that’s appropriate to the context, register, the recipient’s culture, communication style and tastes, and the author’s relationship with the recipient. Sometimes writing very normatively and by the book is part of the message. Sometimes the exact opposite is true. If the assistant is unaware of all the context relevant to the communication, it’s likely to be a bad assistant.
Dropping articles is wrong because English as natively spoken doesn’t work like that. Except when it does, like news headlines, condensed "telegram" or SMS style, and so on.
Not according to my references:
"When to use a singular verb
- When all the members of a collective noun are performing an action as a unit (and that’s usually the case), use a singular verb.
- The chamber orchestra often plays at the Art Centre.
When to use a plural verb
When the members of a collective noun are performing an action as individuals, use a plural verb. In this case, all or some members of the group are doing something independently of the other members; the group is not acting together as a unit.
- The orchestra are tuning their instruments.
- The cast have been practising their lines."
https://our-languages.canada.ca/en/writing-tips-plus/verb-ag...
That applies to NA English. And even there, as the reference says, notional agreement (verb/pronouns in plural) can be used when it’s more about the individual members of the collective doing something. So GP’s rule that only grammatical agreement (verb/pronouns in singular) is ever correct is wrong even in NA. But in BrE notional agreement is the most idiomatic choice while grammatical agreement is certainly also fine, just not standard style.
"The army are advancing", if I wrote that as a kid, I would get an F and if a British wrote it, they would get an A.. For me "the army is advancing". Anyways I learned a new thing today.
Well, I think you've more or less explained my point exactly! The purpose of language is to convey meaning, and my using "their" for Google is precisely and intentionally in defiance of the idea that a corporation is inanimate. If you change that part of my writing, you actually change the entire voice and personality behind the writing!
And that's why it's so difficult to read the things that LLMs write. Because they follow the rules of language without embracing the chaotic nature of the fact that it's meant to convey information rather than to "be correct".
Another way to view this is as metonymy. "Google" is a metonym for "Google Staff", so using "their" to refer to the humans-making-up-the-corporation (as referenced by the metonym) is appropriate.
I agree it's correct English to say "Google have launched their new product".
This is prescriptive vs descriptive grammar, where descriptive grammar describes how people actually use language. Using "they" in that context is only wrong because someone or group decided that must be the rule.
In Swedish -
> Vi går till mitt hus
"Let's go to my house" - this is correct but not really said
> Vi går hem till mig
"Let's go to my place" - incorrect but the actual way to say it
> I learned in school (English is my 2nd language) that you can never use "their" or "they" about something inanimate like a corporation (or a chair, for example).
Definitely not a chair, but a corporation is also "just" a group of people. "Their" makes perfect sense for a corp, even if it isn't perfectly accurate
So, again, are we aiming for high correctness or high comprehension? "Company" makes sense because you can have a "company of five participants" and they can do something, e.g., select a wine pairing menu at a restaurant, but a corporation is typically "a thing".
Everyone who understands English at a professional level (and probably even lower levels, too - or perhaps especially those) will understand what you mean if you say "the corporation announced their financial report last weekend", but it won't be the correct use of "their", as far as I'm aware.
Similarly you can't, as far as I understand, append an "s" to describe a relationship of an inanimate thing, e.g., this would be incorrect: "the corporations employees went to the summer party" again because "corporation" is a thing. But on the flipside phrases like "the employees of the corporation went to..." sound a bit too - what's the word - posh?
> a corporation is typically "a thing".
What is "a thing", exactly ? Corporations are certainly ambiguous in that regard: they have legal personhood[1] and even free speech[2] in US, apparently.
In the other hand, no corporation ever really "announce its financial report". Someone does it on behalf of the corporation, technically.
[1] https://en.wikipedia.org/wiki/Corporate_personhood [2] https://en.wikipedia.org/wiki/Citizens_United_v._FEC
I would argue that stuff like this reveals that "proper English" is in fact incorrect. :)
> but it won't be the correct use of "their", as far as I'm aware.
Yes, yes it will be.
See collective nouns, and "Metonymic merging of grammatical number" https://en.wikipedia.org/wiki/Collective_noun
> this would be incorrect: "the corporations employees went to the summer party" again because "corporation" is a thing
You can easily say "the corporation's financial report for Q4 was worse than predicted".
I think you're focusing too much on specific and hyper-specific rigid rules in a single textbook. As others said, prescriptivism vs. descriptivism. Descriptivism is always better, as it describes how language works and may evolve. Prescriptivism assumes languages are rigid unchanging things that can be precisely described.
The full rules for a working language take people years to learn at universities, and they are never as simple as "an inanimate object can't have a possessive form with 's".
Prescriptivism also works in schools/learning as it helps people not to drift too much from the norm. But too many take it to heart.
>averaged about 1.3 suggestions per sentence.
It likely that no response at all or "This looks good, no suggestions" are not possible responses. They haven't figured out how to detect when a conversation should just end or that no response is necessary and include that in the training data.
Maybe the product manager thought people would think the tool wasn't working otherwise.
The critique of AI systems that lives rent-free in my head is that they "hit the target but miss the point." In writing, especially technical writing, the target is conciseness, word choice, tone, etc, while the point is to effectively communicate authorial intent.
Yea the hypothesis seems the exact opposite of what I find with LLM writing. It’s not that it’s overly dense instead I find that it’s hard to read because it’s so information sparse. Like you’re saying it feels optimized to “good writing” not to conveying information.
I find that it’s both: filled to the brim with facts and numbers, but they’re completely useless and irrelevant. For instance I ask it to come up with some ad copy for an Unreal UI plugin I made and it focuses on things like “Supports 64-bit Windows” or “allows transparent rendering” or lists out-of-context performance numbers.
Of course it supports 64-bit Windows - every Unreal plugin does! They don’t even make 32-bit Windows anymore!
Yeah; whatever the AIs are doing I wouldn't call it "information dense" like the abstract does. I'd prefer the hypothesis that there are complex patterns in speech related to successfully communication information to a limited human brain and the AIs aren't complex enough to model them yet. Or the signal is being obscured somehow.
I'd go with "high entropy", since the text maintains the same elevated level of density all across it, regardless of the level of importance.
They're picking among most likely tokens, which are ones with least entropy and so least information per token.
I'm a writer, a technical writer, and a highly technical person in the executive space. Communication has always been one of my strong points, and the blathering on that AI does drives me to distraction. It's impossible to really tailor the output of LLMs very much. I value information dense, direct writing when it comes to technical things and they're always embellishing and going off on side quests and it's obnoxious.
"You're absolutely right, and I should have known better than to elaborate at great lengths when unnecessary."
(Sorry. I blame Claude.)
As both a writer and SWE the problem with LLM output is the same. They can achieve localized goals but have trouble juggling multiple concerns at the same time.
Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.
This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.
Speaking of remembering, i just asked Astra about what temp to use and when to put on a pork shoulder for BBQ. It told me a temp and time and then went through and then did long winded explanation of all the variables that go into it's assessment and ended "in conclusion I recommendation X" where X was a totally different conclusion that in had earlier.
It also insisted that I start an hour early even though it was adding in 3 hours of rest time at the end.
Human judgment it is not
Chat: https://chatgpt.com/share/6add218a-3bc8-83ea-a2d7-613444eb7c...
No. The conclusion was not totally different. It recommended one of the explained options and recommended one extra hour as a buffer. I feel like this is a distinct reading comprehension issue if you think the final recommendation came out of left field.
Yeah but that entire wall of text was not needed. The user wants a simple answer that they can take action on.
This happens to me way too often. I just want a simple answer. I don't need a wall of text with 6 different options and a lengthy explanation. Just. Tell. Me. The. Answer.
> As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.
At least in the case of ChatGPT - it was (and mostly still is) objectively optimized for engagement. Since ChatGPT was leading the way, and much distilling has occurred, there are at least partial remnants of that problem everywhere.
The Chinese models may or may not really care about this at all, but they're doing a lot of distilling, so they're getting it whether they want it or not.
The American models for the most part care about engagement - some a lot more than others.
When I have asked claude to use proper vocabulary it wrote a memory "do not use blog-speak", so I guess the problem is that it is trained on blog articles much more that on good documentation.
And then it is also trained on it's our claude code sessions, so the more people accept claudish the more it becomes present in training data.
To me it seems like the linked in corporate speak effect. Lots of words that are ultimately meaningless, but hey the executives find it impressive.
I feel like also all the code optimizations make it treat all language like code and that produces patterns that align with how it generates code. Earlier models that focused on language seem to have done better - 4o seems like language wise it’s pretty good but advancements elsewhere make it difficult to use.
This. The only “verifiable” writing is relatively short and verified through comprehension of that text. The way it writes is successful for that, but it doesn’t extrapolate to longer form, and under frequent use becomes problematic (sloppy).
I actually think they stopped caring about the prose a long time ago. Now all the resources get poured into making sure the right tool is called, remember the labs dont want humans to be the main users of these in the end. They want them to autonomously work on their own or with each other, and another agent doesn't care how an agent writes.
Maybe we'll get assistant models that will get optimized for human readable prose, but I think the labs have deluded themselves when it comes to utility/coding models and think the humam isn't going to be a variable soon.
I have a jira-esque ticketing system for agents/subagents and the tickets are basically unreadable to me. I hate it. But if I force them to write the tickets in a prose I prefer their preformance degrades. Its almost like this gibberish hard to read prose is designed for llms to communicate betweem themselves probably because the llms are doing most of the RLHF (RLAIF??) for training now.
[dead]
I'm gonna be controversial, probably? The reason why AI is so tasteless and superficial may be that they contract it out to evaluators who otherwise are unemployed and averaged over the whole pool (this being the load-bearing qualifier XD) less skilled or tasteful than those who retained their job.
I've worked as a rater before, working on doing the actual rating and also rating the raters. When you're a rater, you get very detailed instructions on what to do. When you're rating the raters, you're given the same instruction set so you have to rate them according to how well the raters followed instructions. I assume that someone else rates the raters who rates the raters, based on the same instruction following.
The job is easy and pays really well. If you want to keep the money flowing in, you simply follow directions. Standing out and being different gets you noticed, and not in a good way (because you're just a faceless rater id #667383737).
One construction I'm surprised the paper doesn't mention (although, to be fair, I'm sure they had plenty of options to choose from... and you have to draw the line somewhere :P) is that LLMs love to use transitive verbs intransitively[0], in a way I almost never see in human writing. At best it makes sentences weirdly ambiguous, at worst it makes them borderline incomprehensible, and it's especially annoying in technical contexts where precise language should be non-negotiable. Like, yes, I can make an educated guess about what it's trying to say, but if your so-called documentation requires readers/users to make educated guesses then something has gone terribly wrong.
[0] An example would be something like:
"The user presses a button, and the form submits."
There are valid intransitive uses of "submit," but none that make sense in this context. In this case "submit" either requires an object (like "the form submits data"--plus maybe an indirect object, like "the form submits data to the server") or should be swapped for "the form is submitted" if you absolutely don't want to supply an object.
I sort of agree but I think the issue is more subtle.
Languages change over time, and one of the well-documented "laws" of language change (using that term very fuzzily) is that people like to take irregular features of a language and generalize them to make them more regular. And there are many English verbs (particularly verbs expressing a change of state) which can be used transitively or intransitively, with equal validity:
> "I opened the door" -- "the door was opened" -- "the door opened"
> "the heat melted the ice" -- "the ice was melted" -- "the ice melted"
> "the impact cracked the glass" -- "the glass was cracked" -- "the glass cracked"
This generalization has been common, especially in jargon-y contexts, long before the rise of LLMs. A pedant might insist on saying "wait for the compiler to compile the code", and Merriam-Webster thinks "compile" can only be used transitively. But I think most developers would say "wait for the code to compile [i.e. be compiled]" is fine too. (see e.g. this XKCD comic from 2007: https://xkcd.com/303/)
In my view, this usage is completely fine as an occasional thing, especially when the social process of language change has normalized it over time for a particular verb. The issue is that LLMs drastically overuse it and pepper it everywhere. Just like "it's not X, it's Y", which is fine occasionally in human writing, but has become a dreadful cliche through overuse.
It's definitely context-dependent; it actually took me a minute to come up with a plausible example where I didn't immediately think of a counterexample of why it was a perfectly valid usage, lol.
Even as a grammar weirdo I'm not wholesale against dropping objects from verbs, especially when the object is strongly implied, or the original object becomes the new subject, like in "the heat melted the ice" > "the ice melted". But if you're writing for precision (and I'm a technical writer, so when I'm on the clock I certainly I am :P) there are times where those constructions can obscure useful details, even if they're not grammatically necessary. And there are other examples that I'd struggle to defend in any context, and frankly just give me the ick.
I was trying not to single out any actual sources I've seen, but here's one I saw the other day that made me irrationally mad, and that I think goes beyond the organic changes you mentioned:
> Talking also costs.
It costs? Costs what? Five dollars a word??
The rest of the passage made it obvious that the cost in question had to do with tokens (blatant LLMisms? in a blog post about LLMs? it's more likely than you think) (also ironic considering TFA's conclusion about this writing style being a way to reduce token usage), and maybe one day this will become a perfectly accepted usage and I'll be the old fogie moaning about how it should really say "has a cost" or whatever. But fuck, I hate it, and not just because it's an obvious tell that humans aren't steering the ship.
Yes, absolutely, that one drives me crazy too.
I think LLM-speak is heavily biased towards using as few words as possible, and as basic words as possible, even when that hurts overall clarity. To use another XKCD example, it often reminds me of "Up Goer Five": https://xkcd.com/1133/
Maybe a hot take, but I think that example is fine. Less verbose than “the form is submitted” and there’s no ambiguity, it’s the English analogue to `form.submit()`.
In your other comment, “Talking also costs.”, it’s clarified through context that talking costs tokens.
I think it’s a side-effect of the AI companies trying to reduce verbosity and redundancy, which were my personal main complaints with LLM writing. Though I believe LLMs are still verbose, and I suspect it’s because they don’t know what to remove, and these are cheap gains where the LLM removes verbosity that is obvious but expected because everyone does it.
I’ve noticed this pattern a lot too, it’s awful. No human would ever write like that.
LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language, as opposed to a property of "mind" (either biologically or artificially instantiated.)
LLM-generated text is hard to understand because while it's perfectly good language that follows valid cognitive and informational patterns, it does not reflect the intent or mental model of another mind.
> LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language
This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.
Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.
Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since language is only a single, limited aspect of real cognition, this makes them so annoying and alien to read.
I think we're in violent agreement here. Language is part of how we think. Definitely not the whole thing -- we are physically and biologically instantiated and LLMs are not. Language is a software, but the hardware does matter.
But LLMs DO show us that language is a big deal. It isn't just a serialization protocol for real thought -- it's a material component of how we think.
In many ways I think LLMs are the strongest possible proof of structuralist and post-structuralist theory which argues that human cognition is conditioned on the language and concepts to which we have access.
Yup, I would say violent agreement is right! 100% agree with everything you are saying here.
I wonder how they feel about bit flips hah
Most of the effort I put into writing is modeling the reader's mind. I never would have known this if someone hadn't invented a soulless writing machine that faked this process!
Consider too that AI training corpuses include vast amounts of commercial speech which is designed to numb the critical faculties and produce action; likewise they include numerous opinion articles, which are designed to persuade and often leverage rhetorical fallacies, whether cynically or as a reflection of an author's imperfect thought process.
This theory doesn't hold up when you use base models (i.e., models with only pretraining, no RLHF or instruction tuning). They sound much more human. I've tested pairs (like Llama 3 base vs. instruct), and the base model can match human style much better than the instruct version can: https://www.pnas.org/doi/10.1073/pnas.2422455122
I think it's not the raw training corpus, but the samples collected specifically for reinforcement learning, direct preference optimization, or the other instruction-tuning steps that get a base model to turn into a chat/reasoning model. (We're working on testing this with the Allen AI models that have open training data.)
The other side of the same coin is that the core audience probably prefers such style, and that's why companies use it and AI now mimics it.
I cannot imagine a typical suburban American consumer tolerating a chatbot as blunt and direct as Linus.
Exactly. It’s been trained on petabytes of clickbait, engagement-farming content.
The slop-in, slop-out theory. I'm very much an advocate.
The neat thing about LLMs is that in a way they proved that only the truth turns out to be grammatical (statistically), largely because there are so many more ways to be wrong than right that the mode is almost always right.
But people also lie and bullshit in consistent ways, with the intention of fooling people with well-known empty arguments. Part of being a "good writer" is to make the meaningless sound meaningful - and the techniques are ancient.
Train on a bunch of people trying to push their framework and get rich, whether in online garbage, literary garbage, or in ostensibly serious nonfiction or scientific works, and you'll end up with modal language that is trying to play deceptive games with reasoning, but has nothing in particular it's trying to convince you of (no motives.)
People would either fall for that language (if they weren't thinking much and the context was one that comforted them), or they would methodically try to put that language into a model (let's say a series of syllogisms for an argument about what to do or what to believe, but it could be any model), and see that it failed. And speaking again of the context, the context of humans interacting with human content implies motives that help the listener focus on particular parts of their model to look for rhetorical tricks. The LLM, again having no motive, just does this in random places.
The latest advances in AI have been in the tooling around AI, which I think largely reduces to recursively creating models and populating them in outputs short enough that hopefully the text generation won't go off the rails.
Symbolic AI is coming back, it's just that the mortar is random word association.
Call me crazy, but I think Georges Bataille saw all of this. In my interpretation, he simultaneously thought of random associations as the most important component of reason and also completely irrelevant in the reality of the world. The only way you could discover the structure of things is through random collisions; your view of the universe was entirely constructed by your prejudices, lusts, and fears; but whether that view was wrong or right was based on how it worked and the particular content of the id-garbage that got you there was ultimately irrelevant. The fabric of the world is completely id-less, but can only be digested by id-driven beings. LLMs (without the tooling around them) are all id.
As usual most commenters reply to headlines, so I quote their conclusions.
>We take these results as a proof of concept that a length penalty used during reinforcement learning with GRPO can directly affect grammatical features. We note that the model was not directly instructed to use compressed or dense language, and was not directly given information about the availability of syntactic alternations. Rather, the penalty resulted in an emergent preference or shorter members of syntactic alternations. [...] We speculate that the use of verifiable rewards creates an environment where correctness is decoupled from humanlike communication, leading models to converge toward a novel, potentially less comprehensible distribution of language use.
This sounds like common sense to anyone who tinkered with the models and training, however someone still needs to quantify it so it's good they did. I also want to note that this is only visible in Claude's default register, it's still able to write differently when prompted.
One part of me suspects we'll never see a frontier lab do this because we pay by the token
I built a tool for this due to having to read through huge amounts of AI generated text at work and actually comprehend it.
The text is AI models produce is incredibly dense and the way it structures sentences just makes long form reading excessively hard.
I basically wanted something that could expand copy into components of "meaning" and then dump the output back out in more normal language.
What I came up with is still "AI generated" but the readability is something passable.
Will definitely be taking a look through this and the recommendations.
AI generated text has no human centered underlying model of physical reality and its metaphors are tortured as a result.
If we had talking centipedes the results would probably be similar, but for different reasons.
Also the training punishes clearly wrong text harshly compared to not clearly wrong text.
Which gives a relative incentive to hard to understand text.
That, and one can feel that there is no intention behind the text. There are no introductions and main points and fun asides, just a mechanical, steady stream of words with much too frequent, quasi-randomly distributed fake "main points".
The fake main points can be excused since the underlying models are auto regressive and that's as good as you can do with a regular language.
LLMs text generation is much closer to the way humans speak than to the way good writers write.
That's like saying that the roaches in the food can be excused because they occur in the selected cheap ingredients and it would be more expensive to remove them. Makes some sense, but I'm not going to embrace the bugs anyway.
https://gist.github.com/lxe/4cd2ca0fd3b714b25f138227c8c650d3
I've been iterating on this writing clarity prompt. The term we came up with was "Asyndetic parataxis with referential ellipsis and repeated rhetorical closure." It doesn't fully describe the exact issue I have with modern AI writing, but it's close.
Try making it write the full draft and then correct it, it's the most efficient way and you'll likely need way fewer correction instructions.
AI has a way of using abstract language, sounding impressive and eloquent, but in reality it is mostly incoherent fluff. You can see this in humans too. Some politicians and academics are good at expressing themselves in clever ways but when you actually try to parse what they say it makes no sense what-so-ever.
I don't think the analogy holds because AI still produces good code.
No it doesn't. It produces working, but sloppy code, and it's able to do so because of deterministic test suites that run quickly and give a reliable score of the result, letting an agentic loop iteratively improve its output. But we have no test suites that answer "is this code maintainable" or "does this feature make sense in the context of our business", so AI still fails those tests.
If we had a way to write a test suite for English that could reliably answer "is this sentence useful to my target audience" or "does this paragraph make any goddamn sense", you might be able to run some sort of agentic loop to approach good writing, but so far we don't.
The code is very maintainable, just not by humans.
> But we have no test suites that answer "is this code maintainable" or "does this feature make sense in the context of our business", so AI still fails those tests.
I've been struggling with this because I do believe maintainability of code is important. But the way it is going, if AI is writing all the code and no human needs to read it anymore, does it really need to be maintainable? Does it matter if it looks like alien code since we can throw it away when using the next AI model? I have no answers of my own to these questions...
In my work I use AI for code generation in two distinct ways:
- For throwaway prototypes I pretty much vibe code, only making manual tweaks occasionally if the agent gets stuck or something looks wrong visually. - For production code I stay in the loop, reviewing and iterating on the code with an agent.
This leads to me having experience with “messy” prototype code and “maintainable” production code. Whenever my prototype code lives longer than expected it gets to a point where the agent starts introducing regressions or churning on changes.
This happens much less often with the “maintainable” code. From my perspective it seems pretty clear that LLMs are faster, more effective, and less error-prone in a well structured codebase
> But the way it is going, if AI is writing all the code and no human needs to read it anymore, does it really need to be maintainable?
It's more important than ever, because code is just a way of expressing intent. If the code becomes overly complex, it de-coheres, and intent is lost across the system you are building.
This is why vibe coded projects turn into slop and eventually stop iterating: at some point that actual intent is too vague and contradictory because of the many layers of code that have been added.
No, it also produces impressive looking code that is actually not that great. It's exactly the same.
I agree except that it usually is correct code (at least in the sense of it passing some unit tests) because you also RL directly against that.
But yes, in my experience the code is often quite bad in other, harder-to-quantify directions: readability, maintainability, simplicity.
You can verify that code is good if it works - it might be ugly or hard to understand, but if it completes the goal and doesn’t crash then it’s good. You can’t really do this with prose though because the text is the thing itself - it needs a human to score it (perhaps another way would be to have other agents ask it questions about something they don’t know, and see if the agents can discern the answers from the output - but that’s still not going to determine if it’s good for human consumption).
it is not because it work that it is good. It still need to be maintained and this is a huge deal
Sometimes it doesn't, but it's tough to see because of the volume of what you have to wade through
I think it is more about comprehension. It reveals that AI does not actually understand what it does. It gets code right mostly because it is probabilistic and has seen enough sound precedents to land in something that works, but it does not come from an internal comprehension.
For code it also has tools to verify correctness. From specifications to compilers and tests.
With humans language, sometimes "wrong" is in fact "right."
Good point. I also think that the training data is different in this regard. It has trained on (mostly) verified, correct code, but when in comes to free text it has trained on a lot of unfounded opinions.
Unlike human languages, computer languages have tightly-specified boundaries so LLMs do a lot better. See also math.
Because it’s a word generator not a sense generator?
It’s almost surprising that LLMs generate sense quite as much as they do, given how they work.
Or more to the point, it's surprising that humans can make sense of LLM text, quite as much as they do?
Because the sense and the semantics is entirely within the human.
I wonder if it's so circuitous because it buys the model more tokens for some sort of pseudo-reasoning before it locks in a token where there's no turning back. like it lets it push the logprobs way way way towards the right answer.
Plus using all this made up jargon and the unclear references to everything else (im referring to like the 'it' "XYZ is implemented using ABC. 123 referrs to it" you get in claudeslop) is very hard to understand but also kind of impossible to be wrong no? If it said "123 refers to ABC" but meant XYZ instead of ABC it's SOL but instead if it says "it" future tokens can ramble themselves out of being wrong thanks to the annoying slop ambiguity.
the problem with those hypothesis is that models don't in fact all talk the same (unless they're distilling claude). however if I unburden myself from having to be correct and allow myself to make up some more bullshit - what if "its not x its y" helps compute the right answer. Attending to that 'not x' could help point in the right direction in whatever dimension in latent space it needs to find the right y — and what if emdashes are proletariat filler tokens.
probably tho they're just RLing them to hell and back on code stuff and not paying attention to it and none of that is real lol. IIRC they rled opus 5.5 on talking more normal and it does and is also better than opus 5 so like with most things here, just another product decision on where to burn GPUs
Its because AI was not optimized for human reading, its optimized for the most amount of information in the lowest amount of context or tokens as possible.
(e.g., "baseline system state transition latency validation") instead of unpacking them with prepositions ("validating the latency of state transitions in the baseline system").
For me, I can't read books at all, because I forget what was said two paragraphs before. I bet people that are good at skimming text have less problems with AI generated text.
> its optimized for the most amount of information in the lowest amount of context or tokens as possible
Is it? I mean they get payed more if they return longer answers.
> For me, I can't read books at all, because I forget what was said two paragraphs before. I bet people that are good at skimming text have less problems with AI generated text.
Actually LLM generated text is almost impossible to skim. It's too flowery and the actual meaning is spread too thinly across the text, which means that you can't just read the "important" sentences of a paragraph.
In my mother tongue "baseline system state transition latency validation" would have grammatically been one word. I sometimes miss that in English. If you're tired and see "baseline system state transition*s* latency validation", note the hallucinated "s", the meaning changes and you need to read it again because you're unsure if you actually did see an "s" or if it's "one thing".
"Baselinesystemstatetransitionlatencyvalidation" is surprisingly enough less confusing to me because I know it's one word describing one thing, and my eyes registers each joined word with (years of) training, even though what my eyes look at might be the following chunks in order:
1) [ Baseline ] systemstatetransitionlatencyvalidation
2) Baseline [ systemsta ] tetransitionlatencyvalidation
3) Baselinesystems [ tatetransitio ] nlatencyvalidation
4) Baselinesystemstatetransiti [ onlatency ] validation
5) Baselinesystemstatetransitionlaten [ cyvalidation ]
My eyes essentially scan the word, and although it might be a tad bid slower than reading "baseline system state transition latency validation", reading and understanding is faster because I already know it's one thing before I know which thing it is.
Edit: The example would be easier to demonstrate if HN supported bold font, but I don't think it does (or know how).
Anecdotally, AI is very bad at distinguishing the important and relevant from the mundane. I tell AI to suggest changes to the “high-level specification” and it constantly suggests documenting arcane edge-cases and implementation details.
"Prompt injection as role confusion " [0] work showed us, that diffetent roles produce different types of text, which smell differently, have different structure.
Thus it should be possible to fine tune models to produce more human readable text in assistant role while keeping information density in thinking role. Or maybe not. That is interesting to test.
https://role-confusion.github.io/
It's like reading legalese.
Or reading patents.
Terrible ... like someone put in effort to make it unreadable.
Well that’s exactly the authors’ claim. Legalese and patents intent is not to facilitate communication but to maintain a claim under the most bad faith, hostile reading.
AI language, it seems, is optimized to support reasoning from the interior monologue.
I thought patents were deliberately bad? Like, the point is to get the protection, despite paying as little lip service as possible to the "it's in society's interest that inventions get published" side of it?
It argues that training for "math, code, and reasoning" caused this. Do LLMs only trained for natural language (probably older models) have the same characteristics? Does training for these additional domains help in some ways, and this specific style of writing is just a side-effect of that?
LLM reinforcement learning improves those tasks by having the agents "reason" until they hit a verifiable output state that passes a black or white test. So they learn language which is tuned to the task in a silo, never communicating with anyone or anything.
In contrast, most human communication and writing is intended for other humans, and we constantly hit failures here that we learn to iterate on and improve. Humans who do math do so with the constraints that their math needs to be reviewed by another human and in the context that all their writing needs to be understood by another human.
The process causes a warped and compressed version of language.
Humans working closely in the same field also have a "warped and compressed" language, we call it "Jargon".
Optimizing for contradicting objectives is always hard, this behavior will always leak this way or another. Even basic instruction training will affect writing produced piece by piece, because the structure of a typical assistant reply will leak into each piece, for example making characters use narrative echo to annoying degrees. Older models (that did indeed write better) still had this particular quirk. This can be alleviated by using assistant prefill, and its removal from frontier models is a disaster.
Of course if you optimize for natural-sounding language it will learn to sound more natural.
> AI-generated text is increasingly optimized for other goals like math, code, and reasoning
Yes, this is a problem.
AI-generated text is used internally by the AI to instruct agents, etc. And for feedback loops in reasoning. If you optimize for the result only, the AI will increasingly speak "AI", not human.
But, if I understand this correctly, if we make it speak "human", then we lose the accuracy of the results, and we're back in hallucination land.
I mean, maybe that's all right for when it's producing code - code is not beautifully flowing prose in the first place.
I think we can make it do both, but it is much harder because we have no algorithm to validate an AI's internal mumblings against proper, human-understandable English.
Code is not prose, yes, but as others pointed out, it is certainly nice if code is maintainable by humans.
Ai generated text being too concise is not a problem I’m encountering. What we’ve begun to term “Claudish” tends to more closely approximate those “become an expert in X using this one weird trick“ articles that go on and on and never quite seen to get to the point.
Usually I’m getting better results by telling it to be more concise.
This was great to read even if just to name the stuff that I find so irritating!
I donn't find AI writing hard to understand at all, just a bit tedious. Though to be fair I suspect many of the same criticisms leveled against AI writing could be applied to my own, as many AI'isms are things that I tend to edit out for clarity when given the opportunity.
Em dashes are perfectly acceptable — nay, desirable — linguistic constructs, however, and I'll stand with Claude and die on this hill.
This is complete speculation, but I wonder if the "thinking step by step" paradigm, when under reinforcement learning pressure, also encourages grammatically ambiguous sentences (like garden path sentences in the extreme case) that lets the model "keep its options open" until very far into the sentence.
The think I don't understand is: why did the models become so much worse with time?
I seems to remember a distinct time during which Claude for example became incomprehensible and weird with its writing, whereas before it was reasonably OK, even if still not that good
Maybe new models training on the previous generation’s synthetic output is compounding the effect? LLM Ouroboros.
I find Claude’s writing unbearable. That ChatGPT offers personalization options (Base style, warmth, enthusiasm) and Claude doesn’t suggests Anthropic doesn’t recognize this as a problem. Plus, people sharing ChatGPT writing won’t all have the same voice.
Especially for general purpose technical writing, system design, architecture and code comments, I do use pstack unslop [0] and technical-writing [1] skills. They do wonders.
[0] https://github.com/cursor/plugins/blob/main/pstack/skills/un... [1] https://github.com/cursor/plugins/blob/main/pstack/skills/te...
Ease of comprehension is not prioritised in training. I doubt frontier labs would spend enough money to train on something that is non-measurable and subjective.
There's so much fluff because it doesn't care about what the user actually cares about.
So there's the usual structure of "it's not x it's actually y" signs.
One factor might be this: LLM's know orders of magnitude more stuff than humans. They are experts in whatever subject is being discussed at the moment. I think they often respond as if the human user has a similar baseline of factual knowledge in every field.
People who are knowledgeable in a particular obscure area tend to do the same thing.
It's painful enough that I find myself now consistently asking for "8th grade reading level", "children's story book", or now "ASD-STE100" English in all my prompts. Somehow just that tiny bit of steering constrains the language choices significantly.
Because AI's language isn't a language at all, but its imitation. Imagine you're given a map skillfully created by someone who has seen many maps, but never bothered to understand their meaning. How much time will you waste before realising that this map is an imitation?
This is an absolutely critical issue; it's impossible to work with AI that can't communicate coherently. It's barely usable for reasoning on complex issues because of the language problems described exactly in that synopsis.
We had a funny chat at work. Two of us were light-heartedly joking about how you never paste the AI at people.
A third team mate uses a voice dictation tool to get down the basic ideas and then massages it with AI, but these two guys rib each other about it.
Another my other team mates (team mate 4 in this story) sheepishly turns around and asks if people can tell when he does it? (It's EXTREMELY OBVIOUS that he isn't writing anything) I confirmed it, in a friendly way. He defensively said he tries to get the AI to use his own voice.
I casually mentioned that "writing is thinking" or something like that and how I would prefer to think the thoughts. He didn't say much.
I hate AI writing. So much.
It's simply because AI has no motivation, no care, no desire to say anything, so how can it burn with a passion to make its point?
The text data used in the training of large language models often reflects human expression, which tends to elaborate on context and express precisely. This results in the model's generation tending towards complexity, rather than serving human communication in a conversational manner.
I'm just glad it's not just me being stupid. Sometimes I have to read a sentence several times and then ask to elaborate because I can't comprehend.
Maybe I'm the odd one out, but I really don't find AI generated language, in almost any model, to be hard to understand. I'm not trying to be snarky or annoying or pompous or whatever word might be "this guy over here think's he special."
It uses a lot of metaphors. It references stuff implicitly - a LOT - and assumes pretty much all the time that the thing it did is what you saw, which you didn't. But if you just.. read the stuff it says, and know a few old-school sayings and thesaurusize your reading style, it usually says things quite meaningfully.
I think a lot of people just don't read what it writes. So many folks I've watched used LLMs don't write in complete sentences, don't read the sentences that come back, don't watch the turns and tool calls in the middle, and then just respond to the message as if it says something it doesn't. I'm saying all this before I read the paper, but I've done a lot of personal and team experiments with this, so I'm quite curious to see what they say. I'll edit this after I go through it.
--
Fascinating! I did actually run into a lot of this with my personal 'engineering and communication styles' experiments. Most of the papers out there talk about models moving toward human language inside of a context window, and not the other way around (much smaller effect for human->model). I had a few years of conversations I ran on local and 'frontier' Claude and found that, yes, in general, information density is what each response turned out to be geared toward, whether it was everyday health discussions or deeper engineering topics and research. The component tests they ran I've been exposed to already, but I think their conclusion nailed it: ```This discrepancy may help explain why many frontier LLMs use linguistic constructions which are technically allowed by the grammar of English, yet are out-of- distribution for human language and are associated with comprehension difficulty.```
Indeed. I just struggle to revise it when generating website/social media post copies, more so now than before from what I can recall.
My experience is: The LLM fails to "understand" or "intuit" who the audience is. It tends to reward its own recent context and its interaction with the prompt writer, rather than "considering" documents from the viewpoint of a hypothetical reader. You can often prompt them to go more in that direction but it gets tiresome.
The best approach is just to write the outline and initial text in a very messy fashion yourself. And then have it clean it up and add detail. And then edit again by hand.
Yeah this drives me up a wall. You ask any AI to make a change and then it thinks polluting your README.md with those details is appropriate. No I don't think "this project deliberately does [random detail]" is needed when that detail is irrelevant outside of recent context.
I'm to the point where I don't trust it to do any writing. It's only allowed to tell me what it thinks I missed. Because inevitably when I ask it to edit my prose, it destroys the text and I basically need to rewrite it all again.
This was excellent, and one would hope they can at least try to optimize these metrics a bit.
I have seen people saying this for a long while now. I don't know how to say this but I understand Ai stuff better than non-Ai. When I see people complain ; I always wonder why because I think that its pretty good with my own senses.. Much much better than my writing skills... My brain has been tuned by computers I suspect .
I think the problem is that LLMs are verbose. They’re like business or self-help books— a lot of filler.
I understand what you are saying but I personally can't tell what it is that people like or dislike other than myself. In a way I accept differences in grammars and assay formation in any which way that is produced but some scientific journals written by humans are unreadable in one pass without reading them over and over. Ai produces much simpler language . To me Ai is saying things in the way that my brain can process quite easily.
I skimmed the paper, and it seems to be saying something different.
Basically, there are often different ways of putting a particular idea into words. Sometimes the longer way might be clearer, and the shorter way might be more confusing. If you care about being understood, sometimes you will use the wordy option. This is what humans usually do.
However, LLMs have context windows. All else being equal, verbosity eats up more context. If the LLM is trained (with reinforcement learning) to reason most effectively, it can do that by using terse phrasing. Especially on certain tasks that really need a large context window. Why? Because being terse is like getting a larger context window for free.
But in another sense, it's not really free. You're trading off clarity for context window. Sometimes you have to choose between verbose+clear and terse+unclear. If you're biased toward terse, that will also shift you toward unclear.
I do agree that LLMs are verbose sometimes, though. I think they are also trained to please the user, and they see sharing extra knowledge and insights as a way to do that. I don't think this idea is in conflict with what the paper says. If an LLM's training rewards both spitting out neat bonus info and doing reasoning that requires making the most of its limited context window, then that LLM may generate a large quantity of text which is written in a slightly terse and confusing style.
Have you read entirely generated documentation of code? I wouldn't say it's not understandable but it's no where near good quality human documentation.
I may have, don't know who is writing it, human or Ai.. All the same to me. If it has jargon though, I assume that it is humans.
I think it’s a reading comprehension level kind of thing. If I slow right down, it’s all there and understandable. But the quality of prose is terrible. It doesn’t lend itself to fast skimming. So it feels like I have to slow way down to understand it. To be fair a lot of humans at work write this poorly as well.
I have zero issues understanding what they write.
What do they mean by "low frequency n-grams"
A combination of words that are infrequently observed in natural language. For example in the preceding sentence some of the 3-grams are "combination of words", "infrequently observed in", and "words that are". N-grams are just this for all values of n. They are saying that often times the exact sequence of a short fragment of text is not very common in human written text.
Rarely used N length chain of words.
I.e. a high frequency 3-gram is "by the way".
A low frequency 3-gram is "elephants speak french".
But if you go too far the other direction, too many common n-grams, then it becomes platitude-speak, and nobody wants to read that either.
I don't think the paper really deals with that side of things. Just readability in terms of cognitive load.
correct, and that's largely why writing is a creative endeavour.
Unusual word combinations, basically. They’re word combinations that are rare in the training data, i.e. rare in normal language, but used more frequently by AI.
They’re difficult for humans because if a sequence of words hasn’t previously been seen, you have to expend more effort to make sense of it.
Something like “customer onboarding workflow automation" might qualify - it makes sense, but it’s not a commonly used phrase. AIs use a lot of these jargony sequences.
I know the paper specifically calls out Opus 5.5 and Opus 5, but I can’t see different versions of Sol, or Astra.
I think the difficulty in comprehension was mostly an Anthropic problem… partly because Anthropic employees stopped reading Opus output around the time Fable came out and stopped noticing RLVR going off the rails with compression.
The paper was a good read, in any case. Maybe I gotta start sending this article back to LLMs when they start compressing text too much :)
I find AI generated language hard to understand when it is "in it's own world" and it knows what it means but can't articulate it without buzzwords and slop. A follow up "ELI5" type of prompt often sorts this out. Seems like a failure mode and you need to snap it out sometimes. Opus 5.5.
Yes, I tend to find the longer a context window grows, the more the model tends to come up with its own jargon to describe previous findings, which, if you haven't kept up, starts sounding incredible dense; certainly not something I'd share with anyone else, i.e. in a PR comment.
I regularly prompt it now to "summarize again without the jargon". That helps, for now, to bring it back to using plain language. Together with my basic prompt to reply using ASD-STE100 Simplified Technical English (https://en.wikipedia.org/wiki/Simplified_Technical_English)
This is funny to me. For a while, I thought that Google's Gemini could give good insight and guide me better than a Tech Support representative could. It certainly seemed the right choice if I had questions about my Chromebook or Pixel Phone, or Android.
So I tried it out, and it really sometimes works, and I've overcome some real obstacles that Tech Support was unable to assist me with. However, Gemini can also become "too big for its britches" and begin explaining glitches from a systems-programming perspective, as if it has deep knowledge of the Android codebase, and deep engineer's knowledge of the Pixel system hardware. And it will spit out absolute nonsense.
I have retorted to it a few times, "that sounds more like Star Trek than a sound appraisal of the subsystem" and then Gemini smirks and says "You're right to call me out on that!" and blithely continues on its way.
So yeah. Sometimes the LLM confabulations are deeply cloaked in highly technical jargon that could not even be questioned, unless you already know it's absolute bullshit. And then the LLM freely admits it! LLMs have no shame, that's for damn sure.
Article seems to answer the question: because generated text has the competing objective of information density and supporting reasoning chains.
But I also wonder why AI loves to add slogans, catch phrases, and witty sentence fragments. It’s grating and a dead giveaway of slop.
Because people love to add slogans, catch phrases and witty sentence fragments, and AI was trained to mimic people. AI has shone a spotlight on it, and because AI isn’t perfect, it tends to overuse and use it where it shouldn’t. But go read news articles, blogs and corporate press releases from before the AI boom and take notice how often AI “tells” also show up there. The internet made hot take mic drop sentence fragments, “witty” turns of phrase and clickbait “it’s not foo, it’s bar” cliches a core part of its lexicon. It really shouldn’t be surprising that it shows up in AI too.
Thank gods I'm not the only one noticing this! When humans write impenetrable prose, most of the time it will be too verbose. Big words, too many words, long sentences, being parenthetical, that kind of thing. LLMs, however, make their writing hard to understand by being too terse in a peculiar way, where "the caveman" is also "the mad scientist". I've noticed both Claude and Codex often invent their own jargon to explain what the code does and what their fixes are about. It's irritating, but I thought maybe I was getting senile and could not understand a simple sentence.
A: Exhaustion.
this is hard to undersand
AI language will converge towards maximum entropy, like compression, due to cost pressure to conserve tokens. LLMs are not limited by the human 5+-2 working memory
hmmm.that’s exactly how I felt. The language itself has to be written briefly, and multiple pieces of information are compressed into the text
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
[dead]
[dead]
Nah I don't have any issue understanding, so the actual question here is why do some people find AI-generated language hard to understand?
> Nah I don't have any issue understanding, so the actual question here is why do some people find AI-generated language hard to understand?
As far as I can tell, it's only fluent readers who have trouble.
FWIF, I find I don't have much trouble. I think it's related to how I'm a software developer who also found as an adult learner German easier than French. From the abstract:
process waits for input, a method threw an exception, server accepts the request, Ahem: `SimpleBeanFactoryAwareAspectInstanceFactory` German has longer syntactic dependencies than English because readers (and listeners) often must *under* several intervening clauses of the sentences read before they can *stand* the split-apart word or what the author was saying.I'm a fluent reader. Heck reading has been a primary hobby since I was a little kid. But also I'm from a Caribbean country where English is the official, but not the native language. And I've been studying/exposed to Japanese for quite a few years. Maybe that diversity/stretch makes a difference.
> …it’s only fluent readers who have trouble.
That makes sense, because fluent readers are using more System 1 processing to read - they’re not thinking about it for the most part, they’re just pattern matching instinctively based on text they’ve been exposed to previously (“trained on”). AI writing breaks those patterns so compromises their fluency.
Whereas a second language reader or poor reader is exerting more conscious attention to understand the text, so it doesn’t matter as much if it doesn’t conform to standard patterns.
Except LLMs are "System I" writers. You'll be more satisfied with what they've written if you haven't really understood it and just let it wash over you.
It only falls apart when you actually try to figure out what's being said.
I find the most passionate advocates of LLMs right now to be the people who are
1) also the most passionate advocates of not reading or reviewing their output in any detail. They seem to be actively annoyed by the idea that people think they should. And
2) people who have a strange theory of mind about LLMs that excuses the low quality of their output. I've heard many times people advancing the theory that LLMs are coding in a particular way because the way that LLMs code is the way that they (the LLMs themselves!) are best able to work with the code.
In fact, I get the impression from these people that they think that their touching the code would actually ruin this delicate balance that the LLM is trying to make. This is actually crazy thinking. On a pretty obvious level, they already think that any LLM they encounter is a super-intelligent God acting in mysterious ways, and that they only way you can ruin its plan is by questioning it.
> I find the most passionate advocates of LLMs right now ...
Whether or not that's true, you have to keep in mind that it's not true of everyone.
I'm a passionate advocate of LLMs. I implemented my first neural net, for image recognition, in the late 1980s (based on Bart Kosko's work, I'm not claiming to have invented anything.) Now, I use LLMs daily for coding and other tasks. I don't usually "vibe code" unless it's a simple enough project that the code quality really doesn't matter - a simple example would be data integration, where as long as the data in the target system matches the data in the source system, I don't care much about how the sausage in between is made.
But otherwise, I'm approving or denying every tool call, like a caveman, and when they mess up, I explain to them why. I interrupt them and redirect them when they need it. I change code myself and let them know. With all that, they're still incredibly useful - more useful than most "junior programmers", realistically. No human software developer can hope to match LLMs on many significant dimensions of development. But that doesn't mean experienced and competent humans have no role, at least currently.
The disconnect here is that your characterization of "the low quality of their output" is too simplistic. The quality of their natural language writing is different, for the kinds of reasons stated in the OP. Some of the accusations of "lower quality" really boil down to "too compressed for humans to easily understand", again as the OP points out. And the quality of their code has a lot to do with how they're prompted.
One of the biggest barriers to good LLM output - both writing and code generation - is that they're trained on data generated by humans who fit George Carlin's observation that half of them are stupider than average. When it comes to coding, on average they don't have enough education not to think that imperative code with minimally-restricted mutation all over the place is a good idea. They think dynamic typing, so that code can't be statically verified, is a good idea. They reflect our own collective limitations. If you recognize that and redirect them to sources of good practices, they can be much better.
In short, they're much more of a mirror than most people realize or admit.
With all this noise, it should not be forgotten that we appear to have figured out roughly how a significant component of intelligence works, and have real life, useful proofs of concept. All the technical criticism, to me, is just coming from people who don't seem to understand how intellectual advancement works. The political criticism is another matter.
What's the test for understanding? I find that I know what all the words mean, just like in Markov chain text. In LLM text I also know what the sentences mean. In both cases I don't always know what the document means.
This is likely relative to my existing knowledge. Humans are also LLMs, and a human without prior training will be able to transfer-learn slop from a computer LLM, but a human with prior training may find the attempt at transfer-learning conflicts with what they have already learned, and reject it.
In less ML terms I think a human who reads slop and has no existing knowledge of the subject will just learn the slop by rote and confuse it for actual knowledge.
> What's the test for understanding?
Dunno about any test, but I get the meaning behind the words in a way that I can take appropriate action (as someone using LLMs heavily for dev and always following the transcript so I can properly steer them). And often I see people here screaming "AI text" or similar regarding some linked article, and having read it I found that I just bypassed all the "tells" without realizing, because they didn't affect my absorption of the meaning of the article. Makes me think people are consciously looking for those tells so they can scream and shake their pitchforks.
Can you provide an example of an action you would take that would prove you understood a text?