I have in the past made fun of the Linux kernel devs, supposedly some of the best C developers in the world, for not knowing how to make stringbuffer and stringview types, but to be fair to them we didn't have the consensus we have today on the topic.
You know who did have the right idea though? Dennis Ritchie, who proposed a fat pointer type for C all the way back in 1990. Would have made for a perfect addition to C99. Imagine how different the world might have been had the committee added that in.
We had a second chance with the release of the "C's greatest mistake" blog article from Walter Bright in 2007, essentially pushing for the same idea as Ritchie (slices/stringviews) but explained with much clearer language.
Alas, didn't make it to C11.
We're now in C23, still nothing. But we did get _Generic and VLAs! Party hard.
VLA has been demoted to an optional feature in C11 (good).
IMHO the current main problem is that the C stdlib is stuck in the K&R era and the stdlib APIs haven't even been updated to the language features added in C99 (e.g. make use of struct args and return values). A range struct (ptr/size pair) in the stdlib and new or updated string functions to use such ranges would already go a long way.
* NUL terminated strings (and now, non UTF-8 encoded strings on input/output)
* Using LF or CR or CRLF as line terminators, and pipe/comma-delimited fields when there were other unambiguous ASCII characters that could have been used (eg, GS, FS, RS) that would have made the encoding/decoding of line termination an I/O thing keeping HT/VT/CR/LF/FF as literally print related codes.
UTF-8 on stdin/stdout works perfectly fine (unless you are on Windows of course, which is stuck in in the early 90s when it comes to international text encoding).
> Using LF or CR or CRLF as line terminators
This is also an operating system convention, and it would be better if programming languages wouldn't try to "guess" the correct line endings, since this causes more problems than it solves - but again, this is mostly a Windows specific problem, and it's Microsoft's job to finally bring Windows into the current century.
"The strncpy function within the Linux kernel has been a "persistent source of bugs" for years due to counter-intuitive semantics and behavior around NUL termination along with performance issues due to redundant zero-filling of the destination."
Huh. Whenever I've been asked to review C code, I always looked for strncpy and always found a bug with it.
I wonder what is the difficulty in rewriting strncpy uses that makes it take six years? Was it widespread? Or was it more of a long going effort, where it was only changed if there were some changes in the same file? Or is there some other thing that makes it difficult?
This sort of boring grind is where the real work of systems engineering is done. Big infrastructure projects like this work on making the Linux kernel more reliable while still keeping it workable throughout the process move on the scale of decades, not months.
On one hand I understand why it’s decade scale (the long tail of users/dependencies is really, really long) but on the other hand it doesn’t feel like a tenable pace at which we can make meaningful long term progress. Less of a gripe and I guess more a paradox of critical infrastructure.
wow, very humbling. I'm actually amazed how many people contributed to this. It's easy to get attribution for "cool new features", but arguable removing bad features is even more important for something as fundamental as the kernel. Cudos!
I'm sure these are the sorts of things that will go down as folklore from the "founding ages", when everyone will have forgotten how to understand source code in 50 years and the Claude/Codex cruft just silently keeps piling on and burning the majority of our planets energy.
Reminds me of Deepness in the Sky (Vernor Vinge) where a guy maintains a ship by doing software archeology. He is the only guy who knows what the Unix epoch is.
What's a way they could get a strong data type here? Wouldn't that also require a large refactor of the code around strncpy to use the type and its functions?
Today yes, but 40 years ago someone made the decision that a string was a char array and that every string manipulation going forward would require manipulating arrays. Talking about costly decisions.
It’s actually interesting to compare the pain and suffering of switching to a string datatype in the 80s (refactoring the limited code base then) vs the next 40 years of unnecessary boiler plate syntax and bugs for not having this type in key APIs.
Linux doesn't exist in the 1980s, Linus started this work in the 1990s.
But yes, the string slice type should have existed in C89 and it's very obvious from here that not having something of this sort - maybe what Rust would call &[u8] the reference to a slice of bytes - was a big problem for C.
The correct way to represent this is what's called a "fat pointer". A pair of values, one is a conventional "thin" pointer to the start of the slice, and the other is a count. Your register pressure increases in the compiler backend but problems are significantly reduced because you have fewer bounds misses.
That's what pascal did back in the day, but 255 byte strings were all that was needed back then so only a byte was needed to store the length. Does that still sound maintainable? Anyhow, some developers put data into strings when they shouldn't, and require doing that in the APIs they publish. Strings, whether NUL terminated or with stored length, aren't always the best choice architecturally so making them easy to use isn't necessarily a good idea.
I'd be curious to see how much CPU time is wasted on looking for a null every time strlen is called. The extra length integer is probably insignificant compared to that.
It is very expensive if you repeatedly measure and forget the length, this is presumably some of the price in Google's problem where some engineers wanted to use 0-terminated char* as the type of a string but others wanted C++ std::string and so the software ends up measuring how long the string is, allocating and copying, then immediately forgetting that length, only to once again measure how long it is, allocate and copy again.
That's a language design defect, C++ got its string slice reference (named std::string_view) only in 2017, years after Rust 1.0 shipped this as a core language feature, even though C++ is decades older.
On the other hand I can well believe on a 1970s computer where you'd be lucky to have 64kB of RAM the trade looks very different, I just think that by C89 it should have been fixed.
> Today yes, but 40 years ago someone made the decision that a string was a char array and that every string manipulation going forward would require manipulating arrays
That's not a bad thing, Common Lisp does the same and it Just Werks. The real problem is the more general "array to pointer decay", not arrays, really.
around the same time Wirth decided to have a length prefix in his Pascal strings (that's why string adressing began at 1, because 0 would be the length of the string)
There is a middle ground that Visual Basic (and then COM) took, with the BSTR type: It’s still a pointer to a zero-terminated char array, but there is a length field immediately preceding the first pointed-to byte. This is still compatible with a C string (assuming no embedded null characters), but BSTR-typed functions can take advantage of the length value.
Partly agree but there would have been squabbling on the data type of the size, unless it was variable length. The latter would have had other issues too.
For a while, 16bit would probably have seemed too extravagant. Now 32bit would probably seem too small.
For a “strongly typed” language, C is pretty damn loose where would have mattered.
I like the D approach where arrays are just `struct { size_t length; T* ptr; }` internally --- and strings are just arrays of `immutable(char)`.
It has a big advantage over the Pascal approach in that you can do zero-copy slicing, since the length is separate from the actual data.
And `size_t` makes perfect sense for the length here. If your strings are longer than the address space (which `size_t` technically isn't, but is practically very strongly correlated to it), then you're going to have a problem regardless of the number of bits for the length anyway.
This only makes a difference in terms of memory size, not in terms of speed, because for decades processors and compilers have been optimized for moving bytes around.
But one would note that in order to gain memory for this particular case of slicing, one introduces 2 extra words (size and pointer) for every other cases. Like perhaps the second most common string operation, concatenation. In those other cases, the benefit is slightly negative.
I've had extensive experience with "counted strings" because I implemented a bunch of Forth interpreters which also uses this scheme. Including the common trick of using counted and zero-terminated strings, which is the worst of both worlds in the end. Forth is the kind of language that quickly show you how bad your choices are.
I eventually dropped all that and adopted ASCIIZ strings because they are generally more efficient (if you pay attention to the strlen() performance pitfalls) and having a dead simple interface with the rest of the world (OS, libraries) is more valuable.
It turns out that the machine is much better at the sort of boring mechanical tasks where thoroughness counts and imagination doesn't and so languages which do more, and more, and more checking pay off very well. Rust's borrowck is the obvious first thought today but say WUFFS will check that you've proved certain key properties, WUFFS doesn't need to insert runtime bounds checks for example because you've proved, before the code would compile, that you don't have any bounds misses. You might have proved it by writing bounds checks yourself of course, or likely you have an inherent mathematical rationale for why your algorithm has no misses, but either way the compiler checked your work.
This is something that has irritated me for a long time.
Bounds checks and sized arrays and strings are mechanically very easy to perform by a machine. These are highly automated tasks.
There are some extreme cases where they ruin performance, but in the vast majority of cases they don't matter.
If you look at the type of tasks that cannot be automated, if going from no to full automation required an efficiency loss of 5%, most people would see taking the hit as an obvious choice.
And this is where the problem becomes recursive. You can build a language where the runtime check becomes a compile time check.
We ought to abandon the C paradigm of shifting all the work to the developer and shift more work to the machine.
Zero terminated strings were the basis for an awful lot of useful software. Calling them the biggest mistake in computing is a bit OTT.
I haven’t programmed anything Pascal related for 30+ years but I dimly remember thinking at the time that I wished the string system wasn’t so hard to use.
That argument isn’t valid. The argument would be “this string design enabled a whole lot of useful software” but that’s a different matter. (And it could very well be the case.)
Clang and GCC both let you use Pascal strings in C if you would like (with `\p`). But Pascal strings aren't that useful today because the maximum length is too short.
A pascal string is a single byte with the length, followed by the data.
Some implementations use more bytes for the length data, such as Delphi which changed over to a 4 byte prefix length, though those aren't technically Pascal strings anymore. I can't find anything about a Pascal string being two pointers?
It is conceivable, for both Pascal and C, to have more than one string implementation side by side, so the developer can choose to use the best-fitting one.
In C++23, variant<> permits to do what Rust's typed enums introduced (e.g. Result sum type that is either a "real" result - with result type - or an error - with error type -, each strongly typed).
If you do that, a definition like
class IString { /* basic string functions */ };
class MiniString : public IString {};
class CZeroTerminatedString : public IString {};
class PascalString : public IString {};
class CppString : public IString {};
use String = std::variant<MiniString, CZeroTerminatedString, PascalString, CppString>; // define one type for all impl.
permits to define string functions that operate over the sum type String, and which use the methods defined in the interface IString, and which then work for all string implementations.
The developer can then pick the most suitable implementation, i.e. CMiniString for very, very short strings (that fit into 64 bits, so approx. <= 8 UTF-8 characters), CZeroTerminatedString (for char *co = "test\n"; zero-terminated old style C strings), CPascalStrings for strings that carry a length in s[0] or as a struct member or class field, and CppString as a wrapper for the C++ std::string that implements IString.
Sum types are a type-safe and memory-preserving way to do what in the older days was sometimes implemented using a "union {}" (which was not type-safe).
Zero terminated string is a special case of sentinel value termination.
And sentinel value terminations make a lot of sense when you have punch cards and fixed length records that you need to carve into pieces.
Nobody expected any decisions they were making in the 1960s and 1970s to have any bearing on computing a half-century later. They all expected to have their mistakes long papered over by smarter people at some point.
But we ALL make the mistake of underestimating inertia.
It was definitely an interesting way to allocate pointers.
I did once have a very large project where devs didnt understand this and resolved hundreds or more off by one and memory overwrites in C due to this feature.
But at the same time, I think blaming the software was kind of a cop out. Devs were in a hurry and simply didnt respect the rules. Given todays software engineer at large. Nerfing programming languages so they cant destroy things might not be a bad idea. But AI will nerf everything.
why is AI gonna nerf everything? sure it could be used as the easy button, but I just spent two hours this morning learning about the neuroscience of how memory works in the brain that I didn't mean to and now I want to run studies on how memory works.
Why do you assume that AI is gonna nerf everything?
See, AI was trained on existing data - on all that existing C code out there (sure, and also on all the papers and articles saying what was wrong with that C code). Those bugs are in the training data, and often not marked as bugs. So when AI generates C code, is it going to avoid making the mistakes that human code made? No, it's going to generate the kind of code it was trained on. How could it be otherwise?
> See, AI was trained on existing data - on all that existing C code out there (sure, and also on all the papers and articles saying what was wrong with that C code). Those bugs are in the training data, and often not marked as bugs. So when AI generates C code, is it going to avoid making the mistakes that human code made? No, it's going to generate the kind of code it was trained on. How could it be otherwise?
The generalization of this is why I think all these AI companies writing blog posts where the marketing department is just jer—ranting endlessly about how AI will improve itself into the singularity is just crazy talk. They generate a random statistically likely output, and the most statistically likely output is mid. Exceptional outputs — the ones that wow us or move the needle are exactly that, unlikely. AGI is sci-fi, and LLMs will not change that.
You can see the same effect when AI emits bash, too, and especially so since most bash is terrible, and most users of bash do not put in the effort to learn bash and its foibles. So it outputs what most people write, which is not great.
In far from an AI fanatic, but I would argue training it on GitHub PRs and general software patches already provides that. Instead of just seeing the static snapshot it sees “this code was replaced by this (hopefully better) code”
Would you even know? Serious question. The volume of code the models can produce, the subtle ways these bugs can manifest (or even only manifest when under attack), it seems like they would be easy to overlook.
I have a habit of getting GPT 5.5 to review everything Opus writes for me, and vice versa. The model in the reviewer role frequently finds things I overlooked myself. Occasionally in parts of the code I wrote.
No modern LLM has found any buffer overflow bugs in parts of my code that originated from another LLM. Again, though, they have found one or two that were my fault.
I use Zig, which has slices, so so far none. But man, it can't get ref counting right to save its life. There have been remarkably few times it's gotten it right on the first try. My codebase considers OOM recoverable, so it keeps forgetting to clean up memory when OOM is raised. Even in the happy path though it still messes up ref counting. I use Kimi k2.6.
In addition to having to pick a size for the length counter and then, later, having to differentiate between lengths in bytes, codepoints, and glyphs, you can't subdivide a Pascal string using pointer arithmetic. To pass just the end of a string into a function, you have to either copy the tail of one Pascal-style string to another with a smaller size value, or your string has to be a struct with an integer and a pointer to the actual data instead of just an integer stuck on the beginning of the string. The first is a lot of copying in some cases, the second raises the specter of structs with invalid pointers. That's not to mention the potential problems that would cause with caches.
You can have a universal variable length field, for example 2 bytes for strings < 32768, then four bytes, 8 bytes etc. On the critical short string path, it costs just a single bit test. The glyph vs byte issues need to be dealt with in both formats.
The subdivision issue is a good perspective, but i would argue the performance impact of cloning substrings is dwarfed by the redundant full string reads to find length.
> You can have a universal variable length field, for example 2 bytes for strings < 32768, then four bytes, 8 bytes etc.
To hold the length of a string, I'd do something similar to unicode:
7-bits for size + 1-bit for continuation, then 15 bits for size + 1 bit for continuation, then 23-bits for size + 1 bit for continuation, etc.
Or maybe even do it exactly the same as unicode:
0XXX XXXX -> length of string is in those 7 bits
1XXX XXXX XXXX XXXX -> length of string is in those 7+8 bits
11XX XXXX XXXX XXXX XXXX XXXX-> length of string is in those 6+8+8 bits
...
> On the critical short string path, it costs just a single bit test.
A few more clock cycles compared to NULL-termination, although my alternatives above require even more clock cycles.
If the hardware had instructions for sentinel values, things would be easier (Like how DOS calls used '$' termination for strings) and safer.
Load a sentinel byte into a register and have dedicated copy and compare instructions that take each two addresses (src and dst) and copies (or compares) src/dst until the terminator is reached (with copy copying the sentinel as well).
Considering that sentinel values are needed so often, and are so useful, it's surprising that this is not in any ISA. What we have now is kludgy workarounds in the HLL for this. It's hard to blame the HLL, because some workaround has to be implemented.
Personally, I would avoid UTF-8 levels of complexity because you only pay the size cost once per string. A simple 2-bytes + optional 4 bytes continuation scheme handles strings up to 140TB and increases the size of the average string by just 2 bytes (compared to 1 byte for nul termination).
> A zero is a sentinel value and is catered to by all ISAs.
> Why would using a "$" be any easier/safer than a NUL?
I didn't say it had to be '$'; I specifically said that the sentinel would be loaded into a register. In that case it could be anything, including zero (for the snippet you posted), or INT_MAX if the code iterated across an array of integers, etc.
By having rep/mov variants that use sentinels, a lot of the HLL problems go away - Java, C#, Python, etc would all look very different today if the ISAs from the 80s included sentinal variant of memory instructions.
The third option is to have a variable width length: the top most bit signals whether the next byte corresponds to the length or to the start of the string.
Prompt injection only works because there isn't two streams of input to give to the LLM. Von Neumann being the architecture with a single shared memory for both data and instructions. If there were a clean way for the LLM model to distinguish between system messages vs user messages, we wouldn't have that problem.
The limitations were brutal. Initially you could only have 255 bytes in a string. The length of a string and the size of the allocation are now separate and you may need to think about that unused memory in your design. The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.
If you want to create an array of strings you either need to specify the length of all strings and accept the memory overhead or have an array of pointers to strings. If you use an array of pointers you may end up choosing to use the 'nil' value as a sentinel that means "end of list." So we're right back where we started.
--
Because someone decided to downvote this HN has limited the speed at which I can reply. This site is tragic and I'm fully done with it now. You can spread propaganda and poorly sourced zeitgeist and be among friends but if you try to have a genuine conversation about programming languages you are made to be unwelcome immediately. Screw this.
--
> No other data structure works like this.
The linked list.
> You can't mess this up in an array
C happily decomposes arrays into pointers. You can erase your length information from the type. This was an intentional decision.
> Strings are the only data structure that assume there will be a NULL at end.
Which is why almost every string API has a version that allows you to specify the maximum length. The fact that you can use a NUL doesn't mean you have to. Which is why the concept of "sentinel values" is broadly used in many types of applications you haven't considered here.
> You can spread propaganda and poorly sourced zeitgeist and be among friends but if you try to have a genuine conversation about programming languages you are made to be unwelcome immediately.
Indeed. And the ignorance of computing history in this discussion is particularly disturbing.
The context of this particular thread is "zero terminated string is ... computing's biggest mistake". This completely ignores the situation on the ground when C was developed. At the time, people were striving for a system programming language that sat above the level of assembly but was compact enough to run within the limited resources of the then emerging mini-computer systems. The PDP-11 on which C was developed was certainly not the first mini-computer, but it was among the earliest to have a regular enough instruction set and addressing model to make a general purpose, high-level system's language possible. These systems were extremely limited in memory; the PDP-11's instruction set is limited to directly addressing at most 64KiB (code and data) and many systems of the era were hardware limited to less than that. (Indeed, I regularly run an early version of Unix, including an early C compiler, on my PDP-11/05 which is maxed out at 56KiB [of actual core]). There was no way that even a brilliant engineer like Dennis Richie was going to be able to shoe-horn in "optional" types, or the mechanics of length-value strings into a compiler that has to run in such limited space, and produce code (e.g. the Unix kernel) that has to run in even less. The fact that strings and arrays are thin abstractions on top of pointers is both a brilliant compromise in design as well as a nod to then-prevalent assembly practice. It was the exactly kind of pragmatic decision that was needed to move computing along at the time. Of course the designs from this era are antiquated now. But they were not mistakes.
All those limitations were sorted out in 1978 with Modula-2 and open arrays, aka spans.
What about the UNIX and C folks propaganda of C being the first systems language, or always focusing on the original Pascal used for teaching and not everything else that followed up with Mesa, Modula-2, Ada, Object Pascal and friends, none of them with said limitations.
Successful systems languages trace back all the way to JOVIAL in 1958.
You missed quite a few between JOVIAL, and C being adopted outside Bell Labs.
Modula-2 was as widely available as C was outside UNIX and universities with access to UNIX source code.
It took a while for proper C to actually be "used for everything else", until the early 1990s actually, and by then anyone sensible would be much better with Typescript for C, aka C++.
You mean "C with Classes", later to be replaced by "C++" (Stroustrup pick this as favorite from a list of candidate names he crowdsourced) as implemented by Cfront.
>The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.
That isn't really a problem.
The problem with null-terminated strings is specifically what happens when you reach the end of the allocated array and there ISN'T a NULL character.
Every string function is designed to keep going until it finds the NULL character, so if a hacker gets rid of the NULL character, he can exploit pretty much any standard string manipulation function being used elsewhere in the program to manipulate whatever memory comes AFTER the string data structure.
No other data structure works like this. You can't mess this up in an array, because no function that manipulates arrays is just going to keep going until there is a null. That would be stupid because it would require users of the function to add a NULL to the end of their arrays before passing it to the function, so instead we just pass the size of the array to everything. Strings are the only data structure that assume there will be a NULL at end.
By the way, I read once that if you use UTF-32 every code point will be 4 bytes, constantly, but even then a single code point isn't necessarily a single character. Text is just complicated.
> Every string function is designed to keep going until it finds the NULL character, so if a hacker gets rid of the NULL character,
What sort of situation are you envisioning where a hacker can remove the sentinel (in the case of nul-termination) but not modify the length bytes (in the case of fat pointers)?
In C most data structures work like this, you keep going until you find NUL (character) or NULL (pointer). E.g. Strings, array of pointers, linked lists, etc. Of course you can add length to most of those, but it isn't the canonical/traditional way of doing things.
That can't be true. If you have an array of pointers it can be terminated in NULL. But an array of integers can't have a NULL value, since NULL would probably be just 0 which is a normal integer.
The null in a linked list is the null in the .next field, right? That's the way you would implemented linked lists independent of language. It's not the .value that is null.
A string is an array of characters (well, for characters representable in one byte at least) that has a specific value to represent the end of string.
It would be like if Int::MAX was reduced by 1 to make space for an Int:NUL constant that represented the end of an integer array. Or if you were creating your own ENUM, let's say for NORTH, SOUTH, EAST, WEST, and you added a fifth enumeration called Direction.NUL for use in arrays.
With an variable length array of structs, you can set all the fields all to 0 at the cost of an extra member at the end. In the cases where this is, the structures are such that (either intentionally or by consequence) something with all fields zero is outside of the function's domain
If I zero out the destination buffer of a strcpy, and the string is longer than the destination buffer I will run into a buffer overflow problem despite every byte being a zero byte. The absence or presence of the zero byte doesn't seem to be the deciding factor.
I think it was NULL itself. It was a long way until we realised we don't want invalid values and could use the type system to help us use special values safely.
The problem here is that null kinda is consequential of intentional design of the type system itself. In this way, I do think that null was discovered, rather than invented. Remember, C is a kinda "portable assembler" so the constructs in it are based relatively closely to how low level data structures are mapped out in memory.
This is, and continues to be, an incredibly useful feature that makes C and C structs immensely useful concepts. Part of that does need an invalid value[1]. NULL is convenient for this and although there are some very weird JavaScript-trinity-meme-style consequences for this[2], it's such a useful concept that basically all languages that have the ability to construct pointers have a null pointer[3].
The alternative world looks like everyone inventing their own invalid values. Invalid, non-null, pointers are typically MUCH worse than null pointers for debuggability and security. If you unintentionally read/write/execute memory at 0x0 (by far the most common value for NULL), most operating systems will trap this, whereas may not necessarily if 0x12345678 is your invalid value.
[1]: Stuff like IA64 had NaT bits which were effectively an extra bit for what I assume to be this sorta thing. The problem with this is that it costs an extra bit. I don't really know much about IA64, but presumably [NaT 1] + [don't care] would be your null pointers here. I think?
[2]: Really what the standard, in my opinion, should have done is probably not make use of the null pointer UB for many different functions. A lot of compilers took the UB surrounding that to make incredibly dubious "optimizations" that broke stuff with zero actual performance benefit whatsoever
[3]: Yes, even Rust. Although some (again in my opinion) unfortunate design decisions made it so that C-Rust FFI isn't zero cost because of how it treats spans/slices
>[3]: Yes, even Rust. Although some (again in my opinion) unfortunate design decisions made it so that C-Rust FFI isn't zero cost because of how it treats spans/slices
If Rust slices already make you sad, then the thing I'm cooking up will make you cry for days.
You're missing the point - give me a Rust compiler that can run and compile in 64KB memory, then you'll understand that the language C was constrained not just by what the output is running on, but by what the machines of the time could actually handle during compilation.
A naive sum type is just a tag plus a payload. There is no problem here. If you have enums you could have had sum types.
The historical argument and appeal to assembly is illogical here. The only real argument is that niche value optimization is too complex or too clever for the time so even if sum types were in C, nullable pointers would still exist either way.
Rust is a significantly higher level language than C, but it can be used it almost all environments where C is used; provided there's a supported compiler target for it. In (safe) Rust, null is basically a guaranteed compiler optimization. Optional / nullable values are represented via Option<T>, which is a sum type of Some(T) and None. When a reference or other pointer-like value (e.g. Box<T>, an owned heap allocation) is wrapped in Option, the compiler can use the invalid bit patterns of T (such as null) to represent the None variant. This is called niche optimization.
So yes, it's nulls underneath, but the developer never has to think about them.
Eventually you end up with registers that probably allow for 2^N values. But the point is not thinking about the machine executing the instructions, but the construction on top of it that has a safer design.
Seeking performance we've been very prone to avoid abstractions and over and over again have shown why we need the safe abstractions.
Compared to scripting languages with actual tagged types, C doesn't really have a type system, and that's readily apparent to anyone who has written C in the last 43 years and debugged a program written in it.
C pretends types exist with you, but once bytes hit the road, it's all real-life and segmentation faults.
C actually does have a type system and it's one of the bigger issues with the language. If it didn't, unaligned pointers and signed overflow would be totally fine.
Problems with unaligned pointers are basically a hardware defect. Signed overflow is an issue because academics are unhappy computers only can do finite math.
Issue with types and C is while the compiler knows about them the standards committees don't want you to be able to. If C had first class types more people would abandon C++ and that can't be allowed to happen.
Meh, I think NULL is fine in C. It's an extra, valid state to represent pointers at no cost. Unlike the more hand holdy languages, it's quite rare for a pointer in C to have the ability to be NULL since, more often than not, it's pointing at something known. It's actually quite rare to see NULL checks unless it's API code or something like that. I can see this being more of a problem in a managed language where anything can be NULL at any time.
NULL as a concept is fine. Inability to declare something as non-null is not.
There is a huge gap between developer expectation "it's pointing at something known" and hard reality confirmed by zillions of CVE. That's the reason optionality is prevalent in modern languages and type checkers (python, typescript), nowdays even Java has sane non-nullable types.
Struct foo has various members, including a bar*. But a foo may or may not be associated with a bar. If there's no associated bar, the bar* pointer is NULL. Seen and done this all the time
This precise mindset is why the world has suffered for decades (wrt security/integrity/availability) at the hands of what can only be described as an industry led by completely unjustified male confidence.
Why are there still people fighting the “it’s not that bad, guys! you’ve just got to be a good developer like ME!” fight?
Python is interpreted so None is always tested for and will throw an exception if used in the wrong context. This is quite different from a SEGVIO.
> But IMO having a “there is no value here” designation is not a bad thing.
Sure ... if it's done via the type system so that errors are caught at compile time. There's a reason that modern languages all either do this or are moving towards doing it. (And a reason that C programmers have no idea what we're talking about when we refer to type systems.)
> NULL in C just doesn’t belong at the end of a string.
The problem with let's get rid of NULL is that it's a real, required state. The vast majority of computing is actually not binary: any real input generally has at least 3 possible states: not set, true and false.
In practice really 4 because "indeterminate" is a reasonable error condition you'd like to know about.
And it keeps increasing anyway: e.g. not set has subcategories: not set due to lack of user input, not set because we're loading state from the backend etc.
NULL is the first expression of that basic problem: it's definitely not enough to eliminate NULL because the first thing which happens is your non pointer default value takes it's place.
What you are describing is option types, which are an entirely valid and very useful construct that helps make programs more rather than less reliable. But you need proper language type system support and compile-time enforcement to make it work, and C does neither of those.
I worked on a Win32 app that used space-padded strings, i.e. the destination string was padded with spaces, but there was still a null on the last byte. You had to use special versions of the string functions for length, copy etc.
I’m not sure why this was - the source base was so old it might have had its origins in Pascal struct behaviour.
I've seen a lot of confusion, where people are talking about checking for a NULL at the end of a list of pointers which is very different to a NUL at the end of a string.
Yes it was an abbreviation in ASCII, as are all the non-printable first 32 codes.
I would imagine 360 patches removed way more than 360 uses of strncpy. But yeah, it’s not a given that each of these patches addressed a bug. (Also not a given that there were only 360 bugs fixed.)
I was curious: Why have it, instead of just using memcpy_and_pad?
AI's answer (paraphrased) was
* Avoid possible bugs from manually write sizeof(dest)
* Enforces the __nonstring Attribute
* signals: "I am converting an actual C-string into a fixed-width legacy memory field." vs copy binary data & pad it.
Interesting to learn about the __nonstring attribute:
I wonder, why not use a string buffer paired with its length? For example, maybe use struct that has char pointer, and 2 ints (occupied length + total buffer length). Almost like c++'s std::string. This null terminator thing really sucks, it's potentially insecure and often unperformant.
It's definitely possible. And common, at least in some projects. The only real drawback is that sloppiness will lead to multiple slightly different nonstandard string types in the same project.
Reminds me of an article[1] that described how he cut GTA Online loading times by 70% because strlen was getting called for effectively every character in a string
I remember reading this blog post when it was first published, but the subsequent updates are better than I would've ever expected this to turn out. Worth checking it out again if you've seen it before :)
Exactly, you can't write clean concise code when working with c strings. Almost every c string manipulation requires cognitive load: "Is the buffer size enough (including null terminator), should I reallocate it?", "I need to have the offset from the last concat, to make next concats performant", "Umm, shold I put null terminator at i or i + 1?"... It really sucks, it's akin to death by thousands of cuts.
The size overhead of that is 2*sizeof(int) while the overhead of null termination is sizeof(char). If I remember the standard right, the former is worse by at least sizeof(char), and usually more in practice. This used to matter, sometimes still does.
I would assume the difference is mostly negligible in practice due to the allocator rounding up the allocated memory size at least by the word size anyway (for alignment and simpler bookkeeping). You can also use variable-length encoding in the header to use 1 byte for most cases, similar to how UTF-8 does it: if the most significant bit is not set, we assume a 7-bit encoding, which can represent string lengths up to 127 using 1 byte, which is probably 99% of strings.
Well, not saying to always use it, but if the string size is big enough, the overhead of 2 ints becomes relatively vanishing. For generic dynamically sized strings it probably has more advantages than disadvantages. But in any case, sure, if every single byte matters or some structure requires specific memory layout, then fine. I just don't think these things are the majority of use cases. Keep in mind that the cached lengths can increase performance, since you don't have to recalculate string lengths.
I am a terrible hobby c programmer that doesn't understand pointers but surely a symmetric approach doesn't have the overhead or the bug.
that is to say that if the language was designed to work in single bit pairs of a string character in conjunction of a string length character assuming a fail safe design of one dummy string character then if a bug happens in the code then there's no overflow because the length can never be shorter than the character.
Pascal did/does this, but eventually someone wants a string longer than the size portion can handle. Or wants the number of characters not the number of bytes.
I wasn't a programmer in these days, so I don't know if there's some other major concern that would kill this, but I sometimes wonder about whether we could have / should have used variable-length integers. That is, something like, 0-127 byte strings get their length prefixed, 128 - 16383 get two bytes of prefix, and the probably-rare 16384 - 2097151 strings would end up with three, though proportionally by that point it's hardly anything. Or you could use the UTF-8 mechanism for packing the bytes, though that costs more and probably doesn't get anything we'd care about in the 1980s or 1990s.
It's a bit of extra code, yes. Not necessarily all that much, but some. On average it is only slightly more expensive than null termination, and considered as a proportion of the size of the strings themselves it's hardly anything. It's probably better than the strings getting hard-limited to 0-255, though, which was quite frequently a user-visible quirk.
Dude, every sane language out there does this. Just generally with 4byte prefix. Null-terminated stuff has always been backwards compat stuff.
Pascal strings - historically and why people even remember this being an issue - were up to 255 chars in size, if not you had to use different string type.
You might still want raw pointers for all sorts of low level stuff, but you almost never want to have null-terminated strings for anything but back-compat, one of the worst things ever, even on memory constrained systems.
The purpose of strncpy, which was originally part of the UNIX kernel code, was to copy file names to and from directory entries that consisted of a 2 byte inode number and a 14 byte zero-padded but not zero-terminated name field.
I started warning my colleagues against using it the moment I saw it for the first time about 50 years ago.
In Unix Seventh Edition, ls and others read directory entries with fread() and parsed the struct direct themselves in application-mode code. The C library and application mode matter, here.
On the gripping hand, there is no strncpy in the Spinellis 7th Edition source code; 4.2BSD was using strncpy() inside readdir() in 1982, though.
The code for strncpy was in the UNIX kernel since at least V6. It was eventually added to the C library under the name strncpy. Sometimes those entries were processed in userland, e.g., by fsck. The utility of strncpy is noted in the C89 rationale (FWIW I was once a member of X3J11, the C89 standards committee):
"strncpy was initially introduced into the C library to deal with fixed-length name fields in structures such as directory entries. Such fields are not used in the same way as strings: the trailing null is unnecessary for a maximum-length field, and setting trailing bytes for shorter names to null assures efficient field-wise comparisons. strncpy is not by origin a "bounded strcpy," and the Committee has preferred to recognize existing practice rather than alter the function to better suit it to such use."
And I just found this comment from John Mashey (I never met John but he and I both worked under Ted Dolotta, John at Bell Labs and me at ISC in Santa Monica):
"I can answer definitively, since I wrote the originals ~1977, having moved from BTL Piscataway to Murray Hill. They were first named str*n, but were later renamed strn*, as there was some system in BTL that needed first 6 letters of external names to be unique.
I was working on kernel & user code that supported rudimentary per-process accounting, which started with someone else, but needed extensions due to big increase in UNIX systems in computer centers, who wanted more performance analysis. I.e. this was supported by commands like accton(1), acctcms(1),acctcom(1), acctmerge(1) (all in UNIX/TS 1.0, Nov 1978, which was ~Research V7 with first steps of PWB/UNIX influence. Think of that as 1.0, then PWB/UNIX 2.0, then UNIX System III...
The records described in acct(5) held the last 8 characters of the command pathname,truncated if necessary and thus possibly not null-terminated. I found multiple instances of inline code to manipulate these, which seemed a bad idea, so I wrote the str*n functions and replaced the inline code, and also used them in the various commands.
I also thought it was a good idea for better code safety.:-) Sigh."
A reminder that we've had strlcpy[1] for ~ 30 years but it was never accepted into the Linux world because of typical petty open source bullshit. This is why we can't have nice things.
The Linux kernel had strlcpy over 20 years ago. It was removed in favor of strscpy because the latter was judged a better interface. Here's a 2022 article: https://lwn.net/Articles/905777/
Returning an error is better but you're using ssize_t which is a tradeoff.
The race conditions appear to be a result of the Linux kernel implementation but UNIX style syscalls introduce these races by default. It is not an inherent flaw of the API or even the implementation Linux was using.
The only useable C string API has always been memcpy anyways.
Huh, are you confusing me with someone else? I don't doubt Claude Code did that, I do the same for refactors all the time.
But xscreensaver theme tweaks for personal use have a much lower standard for quality control, regression testing, side effects, etc than a kernel used by billions of devices with thousands of interconnected drivers and subsystems.
Not to mention the coordination problem to get every maintainer on board and patches approved for each specific area when working on a project of that scale, even for a relatively narrow change.
Claude Code doesn't really help with that so don't see why the expectation would be a significant speed up (and doing it all in a single patch would definitely be rejected).
Not sure why you'd refuse to believe that when a single, simple patch in Linux can take months to make it into a kernel release. Here we're looking at 300 patches scattered throughout a kernel with millions of LoC. That's going to translate to a lot of mailing list back and forth even if every change was accepted on the first try without a fuss.
The lag there is not due to the review time. How many maintainers were involved? 300? Because I'm still finding it hard to understand how the work of 300 people handling 300 commits cannot be parallelized into months (per your own stat.)
To be clear my original statement was that the bottleneck was most likely not mechanical code changes (where CC would have the most direct speedup) but everything else involved in the process (testing, discussion/approval, inclination towards caution, deliberately narrowly scoped changes, etc).
Not that the Linux kernel approval procedures couldn't be streamlined, work couldn't be parallelized, or anything else like that, which would be a different discussion entirely.
You stated that Claude Code could have significantly sped up the process, so the burden of evidence here should be on how specifically these patches would have benefited/time saved from using LLMs. Hand wavingly saying "LLMs = faster" is too vague/broad of a claim without providing any evidence (and also unfalsifiable).
And what I'm saying is I refuse to believe the Linux kernel approval procedures are that inefficient. Therefore, your belief "bottleneck was most likely not mechanical code changes" is most likely incorrect.
It would be interesting to get the actual answer to this question.
EDIT: Substantially changing your argument after posting isn't nice. But to answer your charge - no - I never made that claim.
Sorry, I didn't feel like this thread needed to be dragged out any longer since it's going in circles at this point and expanded my comment, but I didn't realize you had already replied.
> In another comment here I explained that I have run a test: asking Claude Code to add a substantial feature to 270 different C programs.
That's a different scenario, though.
Would Claude have performed adequately if it had to add a specific feature to 270 programs buried in a set of 270m program, each of which may or may not have a dependency on one or more of the others, with virtually unbounded results to test?
In terms of tokens alone, that would have been cost-prohibitive. But lets assume that you had the money to do this: it still might not even be possible.
You're confusing "I have these 270 independent programs and want to make this change to all of them" with "I have these 270m lines of code, of which only 270 needs to be changed".
> Of course I get downvoted for saying this. HN isn't interested in reality any more.
I suspect that rather many of us are simply just tired of Claude and friends getting shoehorned into any conversation about programming at this point. It is about as fun as the Rust Brigade entering any discussion about C. It adds nothing new to the discussion and it is frankly tiring since we pretty much at any time have a handful of conversations on the front page already covering "AI" topics anyway (counting four at the time of writing this).
Well - except in this conversation it's incredibly relevant. It took six years to do this work when the work is likely mostly mechanical and could have been done much more quickly and safely with an automated system.
I thought automation would be interesting to HN - given the context and the fact it was not used.
An LLM is not a mechanical automated system. A deterministic search and replace would be a mechanical automated system. Clearly it wasn't that simple of a problem though.
why would you start there instead of creating something from scratch ?if you can port drivers just as easily meaning you don't especially give a shit about hardware you're running on in the first place, why even deal with linux? The battle tested LRU cache system?
I've seen several workalike kernels in various stages of completion. at least one of them was able to run some pretty substantial applications (Postgres, nginx, that kind of thing), and that is still I guess around 250kloc. but it only really has drivers to support hypervisor devices.
unfortunately as time goes by, the linux api surface gets larger and more convoluted. so there's going to be some coverage you're just never going to get.
but in the abstract, definitely. linux is so bloated at this point that its not clear that it can ever be 'made safe'.
I have in the past made fun of the Linux kernel devs, supposedly some of the best C developers in the world, for not knowing how to make stringbuffer and stringview types, but to be fair to them we didn't have the consensus we have today on the topic.
You know who did have the right idea though? Dennis Ritchie, who proposed a fat pointer type for C all the way back in 1990. Would have made for a perfect addition to C99. Imagine how different the world might have been had the committee added that in.
We had a second chance with the release of the "C's greatest mistake" blog article from Walter Bright in 2007, essentially pushing for the same idea as Ritchie (slices/stringviews) but explained with much clearer language.
Alas, didn't make it to C11.
We're now in C23, still nothing. But we did get _Generic and VLAs! Party hard.
> But we did get _Generic and VLAs! Party hard.
VLA has been demoted to an optional feature in C11 (good).
IMHO the current main problem is that the C stdlib is stuck in the K&R era and the stdlib APIs haven't even been updated to the language features added in C99 (e.g. make use of struct args and return values). A range struct (ptr/size pair) in the stdlib and new or updated string functions to use such ranges would already go a long way.
> but to be fair to them we didn't have the consensus we have today on the topic.
This is my pet peeve of teamwork. We can choose solutions A, B or C. Each has upsides and downsides. We debate for two weeks, then we choose nothing.
That only goes to show where WG14 priorities are.
Things that have bugged me for 40 years...
* NUL terminated strings (and now, non UTF-8 encoded strings on input/output)
* Using LF or CR or CRLF as line terminators, and pipe/comma-delimited fields when there were other unambiguous ASCII characters that could have been used (eg, GS, FS, RS) that would have made the encoding/decoding of line termination an I/O thing keeping HT/VT/CR/LF/FF as literally print related codes.
> non UTF-8 encoded strings on input/output
UTF-8 on stdin/stdout works perfectly fine (unless you are on Windows of course, which is stuck in in the early 90s when it comes to international text encoding).
> Using LF or CR or CRLF as line terminators
This is also an operating system convention, and it would be better if programming languages wouldn't try to "guess" the correct line endings, since this causes more problems than it solves - but again, this is mostly a Windows specific problem, and it's Microsoft's job to finally bring Windows into the current century.
"The strncpy function within the Linux kernel has been a "persistent source of bugs" for years due to counter-intuitive semantics and behavior around NUL termination along with performance issues due to redundant zero-filling of the destination."
Huh. Whenever I've been asked to review C code, I always looked for strncpy and always found a bug with it.
I wonder what is the difficulty in rewriting strncpy uses that makes it take six years? Was it widespread? Or was it more of a long going effort, where it was only changed if there were some changes in the same file? Or is there some other thing that makes it difficult?
This sort of boring grind is where the real work of systems engineering is done. Big infrastructure projects like this work on making the Linux kernel more reliable while still keeping it workable throughout the process move on the scale of decades, not months.
On one hand I understand why it’s decade scale (the long tail of users/dependencies is really, really long) but on the other hand it doesn’t feel like a tenable pace at which we can make meaningful long term progress. Less of a gripe and I guess more a paradox of critical infrastructure.
wow, very humbling. I'm actually amazed how many people contributed to this. It's easy to get attribution for "cool new features", but arguable removing bad features is even more important for something as fundamental as the kernel. Cudos!
I'm sure these are the sorts of things that will go down as folklore from the "founding ages", when everyone will have forgotten how to understand source code in 50 years and the Claude/Codex cruft just silently keeps piling on and burning the majority of our planets energy.
Reminds me of Deepness in the Sky (Vernor Vinge) where a guy maintains a ship by doing software archeology. He is the only guy who knows what the Unix epoch is.
I'm of the opinion AI slop code will become untenable way before that.
A lot of pain and suffering to avoid having a string datatype.
> A lot of pain and suffering to avoid having a string datatype
No, a lot of pain and suffering to work around the lack of a string datatype in C.
What's a way they could get a strong data type here? Wouldn't that also require a large refactor of the code around strncpy to use the type and its functions?
Today yes, but 40 years ago someone made the decision that a string was a char array and that every string manipulation going forward would require manipulating arrays. Talking about costly decisions.
It’s actually interesting to compare the pain and suffering of switching to a string datatype in the 80s (refactoring the limited code base then) vs the next 40 years of unnecessary boiler plate syntax and bugs for not having this type in key APIs.
Linux doesn't exist in the 1980s, Linus started this work in the 1990s.
But yes, the string slice type should have existed in C89 and it's very obvious from here that not having something of this sort - maybe what Rust would call &[u8] the reference to a slice of bytes - was a big problem for C.
The correct way to represent this is what's called a "fat pointer". A pair of values, one is a conventional "thin" pointer to the start of the slice, and the other is a count. Your register pressure increases in the compiler backend but problems are significantly reduced because you have fewer bounds misses.
That's what pascal did back in the day, but 255 byte strings were all that was needed back then so only a byte was needed to store the length. Does that still sound maintainable? Anyhow, some developers put data into strings when they shouldn't, and require doing that in the APIs they publish. Strings, whether NUL terminated or with stored length, aren't always the best choice architecturally so making them easy to use isn't necessarily a good idea.
Are you muddling the Pascal strings with the string slice?
I'd be curious to see how much CPU time is wasted on looking for a null every time strlen is called. The extra length integer is probably insignificant compared to that.
It is very expensive if you repeatedly measure and forget the length, this is presumably some of the price in Google's problem where some engineers wanted to use 0-terminated char* as the type of a string but others wanted C++ std::string and so the software ends up measuring how long the string is, allocating and copying, then immediately forgetting that length, only to once again measure how long it is, allocate and copy again.
That's a language design defect, C++ got its string slice reference (named std::string_view) only in 2017, years after Rust 1.0 shipped this as a core language feature, even though C++ is decades older.
On the other hand I can well believe on a 1970s computer where you'd be lucky to have 64kB of RAM the trade looks very different, I just think that by C89 it should have been fixed.
> Today yes, but 40 years ago someone made the decision that a string was a char array and that every string manipulation going forward would require manipulating arrays
That's not a bad thing, Common Lisp does the same and it Just Werks. The real problem is the more general "array to pointer decay", not arrays, really.
The decision was made almost 55 years ago for C and Unix: https://en.wikipedia.org/wiki/Null-terminated_string#History
around the same time Wirth decided to have a length prefix in his Pascal strings (that's why string adressing began at 1, because 0 would be the length of the string)
I wonder if an early built-in string class would have been 8-bit or 16-bit UCS-2? Would have been hard to stomach 16-bit storage and performance.
strncpy is 99.999% of the time NOT the correct function to call, so this is a huge win.
It's just a shame that such a confusing name was chosen for such a niche use case (fixed width records that require null padding).
the zero terminated string is I think is computing's biggest mistake. Pascal style strings were much safer.
There is a middle ground that Visual Basic (and then COM) took, with the BSTR type: It’s still a pointer to a zero-terminated char array, but there is a length field immediately preceding the first pointed-to byte. This is still compatible with a C string (assuming no embedded null characters), but BSTR-typed functions can take advantage of the length value.
Partly agree but there would have been squabbling on the data type of the size, unless it was variable length. The latter would have had other issues too.
For a while, 16bit would probably have seemed too extravagant. Now 32bit would probably seem too small.
For a “strongly typed” language, C is pretty damn loose where would have mattered.
I like the D approach where arrays are just `struct { size_t length; T* ptr; }` internally --- and strings are just arrays of `immutable(char)`.
It has a big advantage over the Pascal approach in that you can do zero-copy slicing, since the length is separate from the actual data.
And `size_t` makes perfect sense for the length here. If your strings are longer than the address space (which `size_t` technically isn't, but is practically very strongly correlated to it), then you're going to have a problem regardless of the number of bits for the length anyway.
This only makes a difference in terms of memory size, not in terms of speed, because for decades processors and compilers have been optimized for moving bytes around.
But one would note that in order to gain memory for this particular case of slicing, one introduces 2 extra words (size and pointer) for every other cases. Like perhaps the second most common string operation, concatenation. In those other cases, the benefit is slightly negative.
I've had extensive experience with "counted strings" because I implemented a bunch of Forth interpreters which also uses this scheme. Including the common trick of using counted and zero-terminated strings, which is the worst of both worlds in the end. Forth is the kind of language that quickly show you how bad your choices are.
I eventually dropped all that and adopted ASCIIZ strings because they are generally more efficient (if you pay attention to the strlen() performance pitfalls) and having a dead simple interface with the rest of the world (OS, libraries) is more valuable.
C is not really strongly typed.
"Strongly typed but weakly checked"
It turns out that the machine is much better at the sort of boring mechanical tasks where thoroughness counts and imagination doesn't and so languages which do more, and more, and more checking pay off very well. Rust's borrowck is the obvious first thought today but say WUFFS will check that you've proved certain key properties, WUFFS doesn't need to insert runtime bounds checks for example because you've proved, before the code would compile, that you don't have any bounds misses. You might have proved it by writing bounds checks yourself of course, or likely you have an inherent mathematical rationale for why your algorithm has no misses, but either way the compiler checked your work.
This is something that has irritated me for a long time.
Bounds checks and sized arrays and strings are mechanically very easy to perform by a machine. These are highly automated tasks.
There are some extreme cases where they ruin performance, but in the vast majority of cases they don't matter.
If you look at the type of tasks that cannot be automated, if going from no to full automation required an efficiency loss of 5%, most people would see taking the hit as an obvious choice.
And this is where the problem becomes recursive. You can build a language where the runtime check becomes a compile time check.
We ought to abandon the C paradigm of shifting all the work to the developer and shift more work to the machine.
No, there would not have been and this is most likely not the reason. size_t exists for precisely this use case. It has existed since C89.
Zero terminated strings were the basis for an awful lot of useful software. Calling them the biggest mistake in computing is a bit OTT.
I haven’t programmed anything Pascal related for 30+ years but I dimly remember thinking at the time that I wished the string system wasn’t so hard to use.
That useful software would not have been less useful if the strings in it were represented as size + buf.
That argument isn’t valid. The argument would be “this string design enabled a whole lot of useful software” but that’s a different matter. (And it could very well be the case.)
Lead was the basis for an awful lot of useful gasoline. Doesn't mean it was the only solution or the best one.
A more accurate re-phrased version of the original is that they are the biggest mistake in the C language.
* https://news.ycombinator.com/item?id=48614913
* https://news.ycombinator.com/item?id=24454369
* https://news.ycombinator.com/item?id=1014533
255 characters ought to be enough for everybody, right?
Clang and GCC both let you use Pascal strings in C if you would like (with `\p`). But Pascal strings aren't that useful today because the maximum length is too short.
Why would a pascal string be any shorter than a C string?
A C string is one pointer reaching all of memory, a Pascal string is two pointers reaching all of memory
A pascal string is a single byte with the length, followed by the data.
Some implementations use more bytes for the length data, such as Delphi which changed over to a 4 byte prefix length, though those aren't technically Pascal strings anymore. I can't find anything about a Pascal string being two pointers?
It is conceivable, for both Pascal and C, to have more than one string implementation side by side, so the developer can choose to use the best-fitting one.
In C++23, variant<> permits to do what Rust's typed enums introduced (e.g. Result sum type that is either a "real" result - with result type - or an error - with error type -, each strongly typed).
If you do that, a definition like
permits to define string functions that operate over the sum type String, and which use the methods defined in the interface IString, and which then work for all string implementations.The developer can then pick the most suitable implementation, i.e. CMiniString for very, very short strings (that fit into 64 bits, so approx. <= 8 UTF-8 characters), CZeroTerminatedString (for char *co = "test\n"; zero-terminated old style C strings), CPascalStrings for strings that carry a length in s[0] or as a struct member or class field, and CppString as a wrapper for the C++ std::string that implements IString.
Sum types are a type-safe and memory-preserving way to do what in the older days was sometimes implemented using a "union {}" (which was not type-safe).
A Pascal string has a leading length byte. Because that is one byte, the text can't exceed 255 characters.
Almost as bad as newline-terminated lines. ;)
Zero terminated string is a special case of sentinel value termination.
And sentinel value terminations make a lot of sense when you have punch cards and fixed length records that you need to carve into pieces.
Nobody expected any decisions they were making in the 1960s and 1970s to have any bearing on computing a half-century later. They all expected to have their mistakes long papered over by smarter people at some point.
But we ALL make the mistake of underestimating inertia.
It was definitely an interesting way to allocate pointers. I did once have a very large project where devs didnt understand this and resolved hundreds or more off by one and memory overwrites in C due to this feature.
But at the same time, I think blaming the software was kind of a cop out. Devs were in a hurry and simply didnt respect the rules. Given todays software engineer at large. Nerfing programming languages so they cant destroy things might not be a bad idea. But AI will nerf everything.
why is AI gonna nerf everything? sure it could be used as the easy button, but I just spent two hours this morning learning about the neuroscience of how memory works in the brain that I didn't mean to and now I want to run studies on how memory works.
Why do you assume that AI is gonna nerf everything?
AGI might. AI? No way.
See, AI was trained on existing data - on all that existing C code out there (sure, and also on all the papers and articles saying what was wrong with that C code). Those bugs are in the training data, and often not marked as bugs. So when AI generates C code, is it going to avoid making the mistakes that human code made? No, it's going to generate the kind of code it was trained on. How could it be otherwise?
That's not going to nerf anything.
> See, AI was trained on existing data - on all that existing C code out there (sure, and also on all the papers and articles saying what was wrong with that C code). Those bugs are in the training data, and often not marked as bugs. So when AI generates C code, is it going to avoid making the mistakes that human code made? No, it's going to generate the kind of code it was trained on. How could it be otherwise?
The generalization of this is why I think all these AI companies writing blog posts where the marketing department is just jer—ranting endlessly about how AI will improve itself into the singularity is just crazy talk. They generate a random statistically likely output, and the most statistically likely output is mid. Exceptional outputs — the ones that wow us or move the needle are exactly that, unlikely. AGI is sci-fi, and LLMs will not change that.
You can see the same effect when AI emits bash, too, and especially so since most bash is terrible, and most users of bash do not put in the effort to learn bash and its foibles. So it outputs what most people write, which is not great.
It still could happen, if they had a way to judge the exceptional outputs from the mid and terrible ones. But I'm not sure they have that...
In far from an AI fanatic, but I would argue training it on GitHub PRs and general software patches already provides that. Instead of just seeing the static snapshot it sees “this code was replaced by this (hopefully better) code”
When's the last time you saw a decent coding model create a buffer-overflow bug while trying to use C strings?
Serious question. Anyone else seen this happen in the last 12-18 months? If so, which model and version were you using?
I had Claude write a bit of stupid C# the other day that had an off by one string truncate. Surprised the hell out of me.
Would you even know? Serious question. The volume of code the models can produce, the subtle ways these bugs can manifest (or even only manifest when under attack), it seems like they would be easy to overlook.
I have a habit of getting GPT 5.5 to review everything Opus writes for me, and vice versa. The model in the reviewer role frequently finds things I overlooked myself. Occasionally in parts of the code I wrote.
No modern LLM has found any buffer overflow bugs in parts of my code that originated from another LLM. Again, though, they have found one or two that were my fault.
How many people are writing C code with LLMs? I get the impression it's mostly JavaScript web apps
All the time. C, C++, occasionally some VHDL or Verilog.
I use Zig, which has slices, so so far none. But man, it can't get ref counting right to save its life. There have been remarkably few times it's gotten it right on the first try. My codebase considers OOM recoverable, so it keeps forgetting to clean up memory when OOM is raised. Even in the happy path though it still messes up ref counting. I use Kimi k2.6.
> the zero terminated string is I think is computing's biggest mistake.
No. They had trade-offs to make, and sentinel-based sequences are a needed thing, even outside of strings.
The mistake was that ISAs never looked at what HLL needed, then add the necessary instructions (I posted more about this below).
Even NULL is not a big mistake, when looked at in context of the time in which it was developed.
In addition to having to pick a size for the length counter and then, later, having to differentiate between lengths in bytes, codepoints, and glyphs, you can't subdivide a Pascal string using pointer arithmetic. To pass just the end of a string into a function, you have to either copy the tail of one Pascal-style string to another with a smaller size value, or your string has to be a struct with an integer and a pointer to the actual data instead of just an integer stuck on the beginning of the string. The first is a lot of copying in some cases, the second raises the specter of structs with invalid pointers. That's not to mention the potential problems that would cause with caches.
You can have a universal variable length field, for example 2 bytes for strings < 32768, then four bytes, 8 bytes etc. On the critical short string path, it costs just a single bit test. The glyph vs byte issues need to be dealt with in both formats.
The subdivision issue is a good perspective, but i would argue the performance impact of cloning substrings is dwarfed by the redundant full string reads to find length.
> You can have a universal variable length field, for example 2 bytes for strings < 32768, then four bytes, 8 bytes etc.
To hold the length of a string, I'd do something similar to unicode:
7-bits for size + 1-bit for continuation, then 15 bits for size + 1 bit for continuation, then 23-bits for size + 1 bit for continuation, etc.
Or maybe even do it exactly the same as unicode:
> On the critical short string path, it costs just a single bit test.A few more clock cycles compared to NULL-termination, although my alternatives above require even more clock cycles.
If the hardware had instructions for sentinel values, things would be easier (Like how DOS calls used '$' termination for strings) and safer.
Load a sentinel byte into a register and have dedicated copy and compare instructions that take each two addresses (src and dst) and copies (or compares) src/dst until the terminator is reached (with copy copying the sentinel as well).
Considering that sentinel values are needed so often, and are so useful, it's surprising that this is not in any ISA. What we have now is kludgy workarounds in the HLL for this. It's hard to blame the HLL, because some workaround has to be implemented.
Personally, I would avoid UTF-8 levels of complexity because you only pay the size cost once per string. A simple 2-bytes + optional 4 bytes continuation scheme handles strings up to 140TB and increases the size of the average string by just 2 bytes (compared to 1 byte for nul termination).
So what exactly is the NUL/0 in the code below other than a sentinel value?
I am not sure what this is in response to. Can you explain which point of mine you are responding to?
> If the hardware had instructions for sentinel values, things would be easier (Like how DOS calls used '$' termination for strings) and safer.
A zero is a sentinel value and is catered to by all ISAs.
Why would using a "$" be any easier/safer than a NUL?
> A zero is a sentinel value and is catered to by all ISAs.
> Why would using a "$" be any easier/safer than a NUL?
I didn't say it had to be '$'; I specifically said that the sentinel would be loaded into a register. In that case it could be anything, including zero (for the snippet you posted), or INT_MAX if the code iterated across an array of integers, etc.
By having rep/mov variants that use sentinels, a lot of the HLL problems go away - Java, C#, Python, etc would all look very different today if the ISAs from the 80s included sentinal variant of memory instructions.
The third option is to have a variable width length: the top most bit signals whether the next byte corresponds to the length or to the start of the string.
.. which is why you need a second type, the one dotnet calls "Span". A substring.
compared to Von Newman versus Harvard architecture for LLMs? I think that's a far bigger mistake.
Neumann, and .. what? In what way?
Prompt injection only works because there isn't two streams of input to give to the LLM. Von Neumann being the architecture with a single shared memory for both data and instructions. If there were a clean way for the LLM model to distinguish between system messages vs user messages, we wouldn't have that problem.
> Pascal style strings were much safer.
The limitations were brutal. Initially you could only have 255 bytes in a string. The length of a string and the size of the allocation are now separate and you may need to think about that unused memory in your design. The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.
If you want to create an array of strings you either need to specify the length of all strings and accept the memory overhead or have an array of pointers to strings. If you use an array of pointers you may end up choosing to use the 'nil' value as a sentinel that means "end of list." So we're right back where we started.
--
Because someone decided to downvote this HN has limited the speed at which I can reply. This site is tragic and I'm fully done with it now. You can spread propaganda and poorly sourced zeitgeist and be among friends but if you try to have a genuine conversation about programming languages you are made to be unwelcome immediately. Screw this.
--
> No other data structure works like this.
The linked list.
> You can't mess this up in an array
C happily decomposes arrays into pointers. You can erase your length information from the type. This was an intentional decision.
> Strings are the only data structure that assume there will be a NULL at end.
Which is why almost every string API has a version that allows you to specify the maximum length. The fact that you can use a NUL doesn't mean you have to. Which is why the concept of "sentinel values" is broadly used in many types of applications you haven't considered here.
> You can spread propaganda and poorly sourced zeitgeist and be among friends but if you try to have a genuine conversation about programming languages you are made to be unwelcome immediately.
Indeed. And the ignorance of computing history in this discussion is particularly disturbing.
The context of this particular thread is "zero terminated string is ... computing's biggest mistake". This completely ignores the situation on the ground when C was developed. At the time, people were striving for a system programming language that sat above the level of assembly but was compact enough to run within the limited resources of the then emerging mini-computer systems. The PDP-11 on which C was developed was certainly not the first mini-computer, but it was among the earliest to have a regular enough instruction set and addressing model to make a general purpose, high-level system's language possible. These systems were extremely limited in memory; the PDP-11's instruction set is limited to directly addressing at most 64KiB (code and data) and many systems of the era were hardware limited to less than that. (Indeed, I regularly run an early version of Unix, including an early C compiler, on my PDP-11/05 which is maxed out at 56KiB [of actual core]). There was no way that even a brilliant engineer like Dennis Richie was going to be able to shoe-horn in "optional" types, or the mechanics of length-value strings into a compiler that has to run in such limited space, and produce code (e.g. the Unix kernel) that has to run in even less. The fact that strings and arrays are thin abstractions on top of pointers is both a brilliant compromise in design as well as a nod to then-prevalent assembly practice. It was the exactly kind of pragmatic decision that was needed to move computing along at the time. Of course the designs from this era are antiquated now. But they were not mistakes.
The C code for strcpy is:
On a PDP-11 that is:All those limitations were sorted out in 1978 with Modula-2 and open arrays, aka spans.
What about the UNIX and C folks propaganda of C being the first systems language, or always focusing on the original Pascal used for teaching and not everything else that followed up with Mesa, Modula-2, Ada, Object Pascal and friends, none of them with said limitations.
C was specifically developed to allow Unix to be ported.
It was a systems programming language and the first well known/successful one.
There was BCPL and then B before that, which is why the language is called "C".
Pascal was considered a teaching language, along with "Algorithms + Data Structures = Programs" by Wirth etc.
The UCSD P-system was one of the first "IDEs" and used Pascal and a bytecode interpreter of the compiled code.
Modula-2 was barely available in the early 1980s.
Ada was mired in MIL-SPEC and expensive compilers etc.
People used FORTRAN for scientific programming, C for most everything else in the non-IBM mainframe world.
Successful systems languages trace back all the way to JOVIAL in 1958.
You missed quite a few between JOVIAL, and C being adopted outside Bell Labs.
Modula-2 was as widely available as C was outside UNIX and universities with access to UNIX source code.
It took a while for proper C to actually be "used for everything else", until the early 1990s actually, and by then anyone sensible would be much better with Typescript for C, aka C++.
> Typescript for C
You mean "C with Classes", later to be replaced by "C++" (Stroustrup pick this as favorite from a list of candidate names he crowdsourced) as implemented by Cfront.
> You can erase your length information from the type. This was an intentional decision.
Well yes, but given the number of security issues the argument is that it was in retrospect the wrong decision.
>The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.
That isn't really a problem.
The problem with null-terminated strings is specifically what happens when you reach the end of the allocated array and there ISN'T a NULL character.
Every string function is designed to keep going until it finds the NULL character, so if a hacker gets rid of the NULL character, he can exploit pretty much any standard string manipulation function being used elsewhere in the program to manipulate whatever memory comes AFTER the string data structure.
No other data structure works like this. You can't mess this up in an array, because no function that manipulates arrays is just going to keep going until there is a null. That would be stupid because it would require users of the function to add a NULL to the end of their arrays before passing it to the function, so instead we just pass the size of the array to everything. Strings are the only data structure that assume there will be a NULL at end.
By the way, I read once that if you use UTF-32 every code point will be 4 bytes, constantly, but even then a single code point isn't necessarily a single character. Text is just complicated.
> Every string function is designed to keep going until it finds the NULL character, so if a hacker gets rid of the NULL character,
What sort of situation are you envisioning where a hacker can remove the sentinel (in the case of nul-termination) but not modify the length bytes (in the case of fat pointers)?
> No other data structure works like this.
In C most data structures work like this, you keep going until you find NUL (character) or NULL (pointer). E.g. Strings, array of pointers, linked lists, etc. Of course you can add length to most of those, but it isn't the canonical/traditional way of doing things.
That can't be true. If you have an array of pointers it can be terminated in NULL. But an array of integers can't have a NULL value, since NULL would probably be just 0 which is a normal integer.
The null in a linked list is the null in the .next field, right? That's the way you would implemented linked lists independent of language. It's not the .value that is null.
A string is an array of characters (well, for characters representable in one byte at least) that has a specific value to represent the end of string.
It would be like if Int::MAX was reduced by 1 to make space for an Int:NUL constant that represented the end of an integer array. Or if you were creating your own ENUM, let's say for NORTH, SOUTH, EAST, WEST, and you added a fifth enumeration called Direction.NUL for use in arrays.
With an variable length array of structs, you can set all the fields all to 0 at the cost of an extra member at the end. In the cases where this is, the structures are such that (either intentionally or by consequence) something with all fields zero is outside of the function's domain
A little bit related: https://devblogs.microsoft.com/oldnewthing/20091008-00/?p=16...
If I zero out the destination buffer of a strcpy, and the string is longer than the destination buffer I will run into a buffer overflow problem despite every byte being a zero byte. The absence or presence of the zero byte doesn't seem to be the deciding factor.
> Your string size is in bytes and you need to track characters separately
No worse than C strings then.
I think it was NULL itself. It was a long way until we realised we don't want invalid values and could use the type system to help us use special values safely.
The problem here is that null kinda is consequential of intentional design of the type system itself. In this way, I do think that null was discovered, rather than invented. Remember, C is a kinda "portable assembler" so the constructs in it are based relatively closely to how low level data structures are mapped out in memory.
This is, and continues to be, an incredibly useful feature that makes C and C structs immensely useful concepts. Part of that does need an invalid value[1]. NULL is convenient for this and although there are some very weird JavaScript-trinity-meme-style consequences for this[2], it's such a useful concept that basically all languages that have the ability to construct pointers have a null pointer[3].
The alternative world looks like everyone inventing their own invalid values. Invalid, non-null, pointers are typically MUCH worse than null pointers for debuggability and security. If you unintentionally read/write/execute memory at 0x0 (by far the most common value for NULL), most operating systems will trap this, whereas may not necessarily if 0x12345678 is your invalid value.
[1]: Stuff like IA64 had NaT bits which were effectively an extra bit for what I assume to be this sorta thing. The problem with this is that it costs an extra bit. I don't really know much about IA64, but presumably [NaT 1] + [don't care] would be your null pointers here. I think?
[2]: Really what the standard, in my opinion, should have done is probably not make use of the null pointer UB for many different functions. A lot of compilers took the UB surrounding that to make incredibly dubious "optimizations" that broke stuff with zero actual performance benefit whatsoever
[3]: Yes, even Rust. Although some (again in my opinion) unfortunate design decisions made it so that C-Rust FFI isn't zero cost because of how it treats spans/slices
>[3]: Yes, even Rust. Although some (again in my opinion) unfortunate design decisions made it so that C-Rust FFI isn't zero cost because of how it treats spans/slices
If Rust slices already make you sad, then the thing I'm cooking up will make you cry for days.
Genuinely curious, how would you handle cases where a value is unset without NULL? This is a legitimate case that happens a lot in eg data modeling
Sum types, of course.
How do you expect to use sum types in assembly? Remember where C came from and why it was designed the way it was.
I remember why C stayed what it is at least: elitism and gatekeeping. And YAGNI, repeated millions of times, of which only the first few were correct.
You're telling me OCaml / Rust / Haskell compile to fairy pixie dust? Obviously their compilers figured it out and it works.
> I remember why C stayed what it is at least: elitism and gatekeeping.
If that was the goal, it failed horribly - the gatekeeping didn't work because the popularity exploded.
> You're telling me OCaml / Rust / Haskell compile to fairy pixie dust? Obviously their compilers figured it out and it works.
I said nothing of the sort.
You asked how sum types work in assembly. I'm telling you that at least 3 compilers figured that part out.
> You asked how sum types work in assembly.
No, I didn't - I asked how sum types were supposed to work in an era of 64KB memory systems.
They don't need extra memory in Rust for the case of nullable pointers.
The boring cases require an enum tag in C too.
By bringing up the one thing that doesn't matter, your argument becomes purely ideological.
You're missing the point - give me a Rust compiler that can run and compile in 64KB memory, then you'll understand that the language C was constrained not just by what the output is running on, but by what the machines of the time could actually handle during compilation.
A naive sum type is just a tag plus a payload. There is no problem here. If you have enums you could have had sum types.
The historical argument and appeal to assembly is illogical here. The only real argument is that niche value optimization is too complex or too clever for the time so even if sum types were in C, nullable pointers would still exist either way.
How are you going to build sum types in a way where you can interact with assembly or machine code? The CPU doesn't know about that stuff
OCaml / Rust / Haskell.
Apparently they found a way to have the CPU know... about "this stuff".
The way we do it in modern languages with things like std::optional and even that is not the best example.
And higher level languages that works. But what do you do when you get down to low level C or assembly?
You basically end up with null/0 don’t you?
Rust is a significantly higher level language than C, but it can be used it almost all environments where C is used; provided there's a supported compiler target for it. In (safe) Rust, null is basically a guaranteed compiler optimization. Optional / nullable values are represented via Option<T>, which is a sum type of Some(T) and None. When a reference or other pointer-like value (e.g. Box<T>, an owned heap allocation) is wrapped in Option, the compiler can use the invalid bit patterns of T (such as null) to represent the None variant. This is called niche optimization.
So yes, it's nulls underneath, but the developer never has to think about them.
Eventually you end up with registers that probably allow for 2^N values. But the point is not thinking about the machine executing the instructions, but the construction on top of it that has a safer design.
Seeking performance we've been very prone to avoid abstractions and over and over again have shown why we need the safe abstractions.
They already said:
> use the type system to help us use special values safely
... but this is not the place to explain what a type system is or what sum types/maybe/optional/etc. are.
Compared to scripting languages with actual tagged types, C doesn't really have a type system, and that's readily apparent to anyone who has written C in the last 43 years and debugged a program written in it.
C pretends types exist with you, but once bytes hit the road, it's all real-life and segmentation faults.
C actually does have a type system and it's one of the bigger issues with the language. If it didn't, unaligned pointers and signed overflow would be totally fine.
Problems with unaligned pointers are basically a hardware defect. Signed overflow is an issue because academics are unhappy computers only can do finite math.
Issue with types and C is while the compiler knows about them the standards committees don't want you to be able to. If C had first class types more people would abandon C++ and that can't be allowed to happen.
By that logic, no natively-compiled language has a type system.
Though I should note that in a way, even some ISAs have one, what with e.g. separate float vs integer registers.
Meh, I think NULL is fine in C. It's an extra, valid state to represent pointers at no cost. Unlike the more hand holdy languages, it's quite rare for a pointer in C to have the ability to be NULL since, more often than not, it's pointing at something known. It's actually quite rare to see NULL checks unless it's API code or something like that. I can see this being more of a problem in a managed language where anything can be NULL at any time.
NULL as a concept is fine. Inability to declare something as non-null is not.
There is a huge gap between developer expectation "it's pointing at something known" and hard reality confirmed by zillions of CVE. That's the reason optionality is prevalent in modern languages and type checkers (python, typescript), nowdays even Java has sane non-nullable types.
> to represent pointers at no cost
I wouldn't call "cause of bugs and security issues" "no cost".
> it's quite rare for a pointer in C to have the ability to be NULL
As a C programmer for more than 25 years, that is the exact opposite of my experience.
Struct foo has various members, including a bar*. But a foo may or may not be associated with a bar. If there's no associated bar, the bar* pointer is NULL. Seen and done this all the time
This precise mindset is why the world has suffered for decades (wrt security/integrity/availability) at the hands of what can only be described as an industry led by completely unjustified male confidence. Why are there still people fighting the “it’s not that bad, guys! you’ve just got to be a good developer like ME!” fight?
Is None OK in Python?
NULL in C just doesn’t belong at the end of a string. But IMO having a “there is no value here” designation is not a bad thing.
I think you're mixing up the NULL pointer and the NULL (sometimes NUL) character.
Python is interpreted so None is always tested for and will throw an exception if used in the wrong context. This is quite different from a SEGVIO.
> But IMO having a “there is no value here” designation is not a bad thing.
Sure ... if it's done via the type system so that errors are caught at compile time. There's a reason that modern languages all either do this or are moving towards doing it. (And a reason that C programmers have no idea what we're talking about when we refer to type systems.)
> NULL in C just doesn’t belong at the end of a string.
Different discussion. (And NUL, not NULL.)
The problem with let's get rid of NULL is that it's a real, required state. The vast majority of computing is actually not binary: any real input generally has at least 3 possible states: not set, true and false.
In practice really 4 because "indeterminate" is a reasonable error condition you'd like to know about.
And it keeps increasing anyway: e.g. not set has subcategories: not set due to lack of user input, not set because we're loading state from the backend etc.
NULL is the first expression of that basic problem: it's definitely not enough to eliminate NULL because the first thing which happens is your non pointer default value takes it's place.
What you are describing is option types, which are an entirely valid and very useful construct that helps make programs more rather than less reliable. But you need proper language type system support and compile-time enforcement to make it work, and C does neither of those.
C++ and rust make these optionals ugly. Zig does it right. Zig also forbids null pointers and requires use of optionals.
I worked on a Win32 app that used space-padded strings, i.e. the destination string was padded with spaces, but there was still a null on the last byte. You had to use special versions of the string functions for length, copy etc.
I’m not sure why this was - the source base was so old it might have had its origins in Pascal struct behaviour.
It can perhaps be due to the string originating from a sql database ”char” field, I.e. not ”varchar”. Char fields in databases are space padded.
I think this behavior has its roots in COBOL, not pascal.
Which has its roots in punch cards, where pre-computer hardware operated on fixed-sized fields and an unpunched column is equivalent to a space.
Perhaps prevent realocation when string size changes? Or aligning cpu cache lines?
In all the comments in this thread it's interesting how people confuse:
* NUL: An ASCII non-printing character with the byte value of 0
* NULL: A pointer that does not point to usable memory with the value that compiles in C to be equal to ((void *) 0).
NUL was always just an abbreviation for null: https://www.rfc-editor.org/rfc/rfc20.html#section-4
I don’t think anyone in this thread is confusing the null character with the null pointer.
I've seen a lot of confusion, where people are talking about checking for a NULL at the end of a list of pointers which is very different to a NUL at the end of a string.
Yes it was an abbreviation in ASCII, as are all the non-printable first 32 codes.
Note that "360 Patches" is 360 uses of strncpy that have been removed, not necessarily bugs.
I would imagine 360 patches removed way more than 360 uses of strncpy. But yeah, it’s not a given that each of these patches addressed a bug. (Also not a given that there were only 360 bugs fixed.)
strtomem_pad seems redundant with memcpy_and_pad, and also it requires the preprocessor: https://github.com/torvalds/linux/blob/1a3746ccbb0a97bed3c06...
I was curious: Why have it, instead of just using memcpy_and_pad?
AI's answer (paraphrased) was * Avoid possible bugs from manually write sizeof(dest) * Enforces the __nonstring Attribute * signals: "I am converting an actual C-string into a fixed-width legacy memory field." vs copy binary data & pad it.
Interesting to learn about the __nonstring attribute:
https://github.com/torvalds/linux/blob/1a3746ccbb0a97bed3c06... https://github.com/search?q=repo%3Atorvalds%2Flinux+__nonstr...
I wonder, why not use a string buffer paired with its length? For example, maybe use struct that has char pointer, and 2 ints (occupied length + total buffer length). Almost like c++'s std::string. This null terminator thing really sucks, it's potentially insecure and often unperformant.
Wonder no longer!
https://dlang.org/spec/arrays.html#dynamic-arrays
and
https://dlang.org/spec/arrays.html#strings
and for C:
https://digitalmars.com/articles/C-biggest-mistake.html
It's definitely possible. And common, at least in some projects. The only real drawback is that sloppiness will lead to multiple slightly different nonstandard string types in the same project.
That's called a fat pointer. Null terminated c strings is the majority of memory errors out there.
Yes I have seen it happen a few times with `strlen` being called in a loop silently causing O(N) to turn to O(N^2)
Reminds me of an article[1] that described how he cut GTA Online loading times by 70% because strlen was getting called for effectively every character in a string
[1]: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
I remember reading this blog post when it was first published, but the subsequent updates are better than I would've ever expected this to turn out. Worth checking it out again if you've seen it before :)
Exactly, you can't write clean concise code when working with c strings. Almost every c string manipulation requires cognitive load: "Is the buffer size enough (including null terminator), should I reallocate it?", "I need to have the offset from the last concat, to make next concats performant", "Umm, shold I put null terminator at i or i + 1?"... It really sucks, it's akin to death by thousands of cuts.
Joel Spolsky coined the term “Shlemiel the Painter’s Algorithm” for this type of thing back in 2001: https://www.joelonsoftware.com/2001/12/11/back-to-basics/
The size overhead of that is 2*sizeof(int) while the overhead of null termination is sizeof(char). If I remember the standard right, the former is worse by at least sizeof(char), and usually more in practice. This used to matter, sometimes still does.
I would assume the difference is mostly negligible in practice due to the allocator rounding up the allocated memory size at least by the word size anyway (for alignment and simpler bookkeeping). You can also use variable-length encoding in the header to use 1 byte for most cases, similar to how UTF-8 does it: if the most significant bit is not set, we assume a 7-bit encoding, which can represent string lengths up to 127 using 1 byte, which is probably 99% of strings.
Well, not saying to always use it, but if the string size is big enough, the overhead of 2 ints becomes relatively vanishing. For generic dynamically sized strings it probably has more advantages than disadvantages. But in any case, sure, if every single byte matters or some structure requires specific memory layout, then fine. I just don't think these things are the majority of use cases. Keep in mind that the cached lengths can increase performance, since you don't have to recalculate string lengths.
> Well, not saying to always use it, but if the string size is big enough, the overhead of 2 ints becomes relatively vanishing.
In that case, the fix is not to change C strings (breaking a lot of existing code), but to introduce a stringbuilder type.
You can still use null terminator for compatibility (std::string does use this), but just not rely on that in your own code.
I am a terrible hobby c programmer that doesn't understand pointers but surely a symmetric approach doesn't have the overhead or the bug. that is to say that if the language was designed to work in single bit pairs of a string character in conjunction of a string length character assuming a fail safe design of one dummy string character then if a bug happens in the code then there's no overflow because the length can never be shorter than the character.
Pascal did/does this, but eventually someone wants a string longer than the size portion can handle. Or wants the number of characters not the number of bytes.
I wasn't a programmer in these days, so I don't know if there's some other major concern that would kill this, but I sometimes wonder about whether we could have / should have used variable-length integers. That is, something like, 0-127 byte strings get their length prefixed, 128 - 16383 get two bytes of prefix, and the probably-rare 16384 - 2097151 strings would end up with three, though proportionally by that point it's hardly anything. Or you could use the UTF-8 mechanism for packing the bytes, though that costs more and probably doesn't get anything we'd care about in the 1980s or 1990s.
It's a bit of extra code, yes. Not necessarily all that much, but some. On average it is only slightly more expensive than null termination, and considered as a proportion of the size of the strings themselves it's hardly anything. It's probably better than the strings getting hard-limited to 0-255, though, which was quite frequently a user-visible quirk.
And then anyone that isn't stuck in 1976 will use open arrays.
Dude, every sane language out there does this. Just generally with 4byte prefix. Null-terminated stuff has always been backwards compat stuff.
Pascal strings - historically and why people even remember this being an issue - were up to 255 chars in size, if not you had to use different string type.
You might still want raw pointers for all sorts of low level stuff, but you almost never want to have null-terminated strings for anything but back-compat, one of the worst things ever, even on memory constrained systems.
A lot of them are strings coming from or going to user space right? So wouldn’t you have to do constant conversions?
Did anybody else misunderstand the title as removing strncpy func for linux users ?
For a moment, I misunderstood it as (g)libc removing strncpy and was worried about the trouble its going to cause.
The purpose of strncpy, which was originally part of the UNIX kernel code, was to copy file names to and from directory entries that consisted of a 2 byte inode number and a 14 byte zero-padded but not zero-terminated name field.
I started warning my colleagues against using it the moment I saw it for the first time about 50 years ago.
strncpy appears somewhere around the Unix v7 time frame, however only as function in the standard C library. It is not used in the v7 kernel itself.
In Unix Seventh Edition, ls and others read directory entries with fread() and parsed the struct direct themselves in application-mode code. The C library and application mode matter, here.
On the gripping hand, there is no strncpy in the Spinellis 7th Edition source code; 4.2BSD was using strncpy() inside readdir() in 1982, though.
Here it is in 2.9BSD: https://www.tuhs.org/cgi-bin/utree.pl?file=2.9BSD/usr/src/uc...
login used it, but not for directory entries:
https://www.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/lo...
The code, but not the function, occurred in multiple places in the V6 kernel and userland.
The code for strncpy was in the UNIX kernel since at least V6. It was eventually added to the C library under the name strncpy. Sometimes those entries were processed in userland, e.g., by fsck. The utility of strncpy is noted in the C89 rationale (FWIW I was once a member of X3J11, the C89 standards committee):
"strncpy was initially introduced into the C library to deal with fixed-length name fields in structures such as directory entries. Such fields are not used in the same way as strings: the trailing null is unnecessary for a maximum-length field, and setting trailing bytes for shorter names to null assures efficient field-wise comparisons. strncpy is not by origin a "bounded strcpy," and the Committee has preferred to recognize existing practice rather than alter the function to better suit it to such use."
And I just found this comment from John Mashey (I never met John but he and I both worked under Ted Dolotta, John at Bell Labs and me at ISC in Santa Monica):
https://softwareengineering.stackexchange.com/questions/4380...
"I can answer definitively, since I wrote the originals ~1977, having moved from BTL Piscataway to Murray Hill. They were first named str*n, but were later renamed strn*, as there was some system in BTL that needed first 6 letters of external names to be unique.
I was working on kernel & user code that supported rudimentary per-process accounting, which started with someone else, but needed extensions due to big increase in UNIX systems in computer centers, who wanted more performance analysis. I.e. this was supported by commands like accton(1), acctcms(1),acctcom(1), acctmerge(1) (all in UNIX/TS 1.0, Nov 1978, which was ~Research V7 with first steps of PWB/UNIX influence. Think of that as 1.0, then PWB/UNIX 2.0, then UNIX System III...
The records described in acct(5) held the last 8 characters of the command pathname,truncated if necessary and thus possibly not null-terminated. I found multiple instances of inline code to manipulate these, which seemed a bad idea, so I wrote the str*n functions and replaced the inline code, and also used them in the various commands.
I also thought it was a good idea for better code safety.:-) Sigh."
Now lets put that work into money, to assert what was the cost impact of replacing strncpy().
A reminder that we've had strlcpy[1] for ~ 30 years but it was never accepted into the Linux world because of typical petty open source bullshit. This is why we can't have nice things.
[1] https://man.openbsd.org/strlcpy
The Linux kernel had strlcpy over 20 years ago. It was removed in favor of strscpy because the latter was judged a better interface. Here's a 2022 article: https://lwn.net/Articles/905777/
Returning an error is better but you're using ssize_t which is a tradeoff.
The race conditions appear to be a result of the Linux kernel implementation but UNIX style syscalls introduce these races by default. It is not an inherent flaw of the API or even the implementation Linux was using.
The only useable C string API has always been memcpy anyways.
Actually, glibc 2.38 has it.
Wow it only took them 26 years to import a 30 line C function, a third of which is comments?
I should have sent them a nice fruit basket to commemorate the occasion.
Am I going to be the first person to ask this after five hours? Really?
Wouldn't this work be extremely easy to implement with an LLM coder?
I don't think the bottleneck was that it took six years to Ctrl-F strncpy and type in new code for each file.
It's a shame you're misrepresenting what is actually going on.
In another comment here I explained that I have run a test: asking Claude Code to add a substantial feature to 270 different C programs.
Despite your beliefs - it went extremely well.
Huh, are you confusing me with someone else? I don't doubt Claude Code did that, I do the same for refactors all the time.
But xscreensaver theme tweaks for personal use have a much lower standard for quality control, regression testing, side effects, etc than a kernel used by billions of devices with thousands of interconnected drivers and subsystems.
Not to mention the coordination problem to get every maintainer on board and patches approved for each specific area when working on a project of that scale, even for a relatively narrow change.
Claude Code doesn't really help with that so don't see why the expectation would be a significant speed up (and doing it all in a single patch would definitely be rejected).
Yes, I understand the difference in rigor.
I refuse to believe the six year delay here was getting people to test a patch.
Which, actually, Claude Code will also do quite well.
Not sure why you'd refuse to believe that when a single, simple patch in Linux can take months to make it into a kernel release. Here we're looking at 300 patches scattered throughout a kernel with millions of LoC. That's going to translate to a lot of mailing list back and forth even if every change was accepted on the first try without a fuss.
The lag there is not due to the review time. How many maintainers were involved? 300? Because I'm still finding it hard to understand how the work of 300 people handling 300 commits cannot be parallelized into months (per your own stat.)
To be clear my original statement was that the bottleneck was most likely not mechanical code changes (where CC would have the most direct speedup) but everything else involved in the process (testing, discussion/approval, inclination towards caution, deliberately narrowly scoped changes, etc).
Not that the Linux kernel approval procedures couldn't be streamlined, work couldn't be parallelized, or anything else like that, which would be a different discussion entirely.
You stated that Claude Code could have significantly sped up the process, so the burden of evidence here should be on how specifically these patches would have benefited/time saved from using LLMs. Hand wavingly saying "LLMs = faster" is too vague/broad of a claim without providing any evidence (and also unfalsifiable).
Right.
And what I'm saying is I refuse to believe the Linux kernel approval procedures are that inefficient. Therefore, your belief "bottleneck was most likely not mechanical code changes" is most likely incorrect.
It would be interesting to get the actual answer to this question.
EDIT: Substantially changing your argument after posting isn't nice. But to answer your charge - no - I never made that claim.
Sorry, I didn't feel like this thread needed to be dragged out any longer since it's going in circles at this point and expanded my comment, but I didn't realize you had already replied.
Well - not really a circle. I keep saying the same thing over and over and you keep throwing arguments at it, unsuccessfully.
I don’t think it’s the arguments that are unsuccessful.
If you have an actual criticism of my argument you'll need to be more clear to be understood.
> In another comment here I explained that I have run a test: asking Claude Code to add a substantial feature to 270 different C programs.
That's a different scenario, though.
Would Claude have performed adequately if it had to add a specific feature to 270 programs buried in a set of 270m program, each of which may or may not have a dependency on one or more of the others, with virtually unbounded results to test?
In terms of tokens alone, that would have been cost-prohibitive. But lets assume that you had the money to do this: it still might not even be possible.
You're confusing "I have these 270 independent programs and want to make this change to all of them" with "I have these 270m lines of code, of which only 270 needs to be changed".
This is a job for Claude!
What happens if you turn a job like that over to Claude Code? A mess? Good results? Code bloat? Worth trying on existing C programs.
I ran a test where I added a "light" mode to xscreensaver: unique changes to over 270 different C programs.
It mostly did an amazing job in a short period of time.
EDIT: Of course I get downvoted for saying this. HN isn't interested in reality any more.
These stories with no details and no proof are not interesting or helpful.
My apologies. I did not want to be downvoted for promoting my own material.
https://github.com/qarl/qscreensaver
A diff.[1]
[1] https://github.com/qarl/qscreensaver/commit/2843caba683495d5...
> Of course I get downvoted for saying this. HN isn't interested in reality any more.
I suspect that rather many of us are simply just tired of Claude and friends getting shoehorned into any conversation about programming at this point. It is about as fun as the Rust Brigade entering any discussion about C. It adds nothing new to the discussion and it is frankly tiring since we pretty much at any time have a handful of conversations on the front page already covering "AI" topics anyway (counting four at the time of writing this).
Well - except in this conversation it's incredibly relevant. It took six years to do this work when the work is likely mostly mechanical and could have been done much more quickly and safely with an automated system.
I thought automation would be interesting to HN - given the context and the fact it was not used.
An LLM is not a mechanical automated system. A deterministic search and replace would be a mechanical automated system. Clearly it wasn't that simple of a problem though.
> An LLM is not a mechanical automated system.
Pretty sure that's exactly what LLMs in coding harnesses are.
Right. Six years of work on a grunt job. That's what automation is for.
Wonder when is someone going to brave and fork the linux kernel and try to ffwd it with automatic programming.
why would you start there instead of creating something from scratch ?if you can port drivers just as easily meaning you don't especially give a shit about hardware you're running on in the first place, why even deal with linux? The battle tested LRU cache system?
It's much easier to use something with all the edge cases already handled as a starting point.
I've seen several workalike kernels in various stages of completion. at least one of them was able to run some pretty substantial applications (Postgres, nginx, that kind of thing), and that is still I guess around 250kloc. but it only really has drivers to support hypervisor devices.
unfortunately as time goes by, the linux api surface gets larger and more convoluted. so there's going to be some coverage you're just never going to get.
but in the abstract, definitely. linux is so bloated at this point that its not clear that it can ever be 'made safe'.