I was reading the `fontconfig` source recently (to help me understand what, exactly, is the difference between the various `fc-*` tools and their options) and some of the code scared me.
I thought, "for sure this is a use-after-free" ... but it happened to be safe because some function performed an undocumented incref, and the return value of that function was kept alive.
So this is definitely a high priority thing to port!
... I suppose the interesting question is: if some of my dependencies are ported, and some are not, is there any way a normal compiler can call into a Fil-C-compiled library for just a few functions?
> ... I suppose the interesting question is: if some of my dependencies are ported, and some are not, is there any way a normal compiler can call into a Fil-C-compiled library for just a few functions?
To my understanding, there's no way to do this: Fil-C is basically a managed runtime for native code, so the FFI implications are similar to those for Go or any other intrusive managed runtime. Which is to say that Fil-C could offer an FFI, but not without losing the blanket exit-instead-of-memory-unsafety guarantees it aims to offer.
What is the actual catch with Fil-C? Memory safety for C based projects without a complete rewrite sounds like a very good idea, but there must be some pitfalls in this approach which this would not work in some cases.
In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
- Not ABI compatible with Yolo-C. So, you have to recompile the whole stack. This is both a major catch, and it is both a bug and a feature. It's a bug for obvious reasons. It's a feature because it means I'll probably be the first to have a totally usable, totally memory safe desktop environment.
- In my testing, it's between 1.2x and 4x slower than Yolo-C. It uses between 2x and 3x more memory. Others have observed higher overheads in certain tests (I've heard of some things being 8x slower). How much this matters depends on your perspective. Imagine running your desktop environment on a 4x slower computer with 3x less memory. You've probably done exactly this and you probably survived the experience. So the catch is: Fil-C is for folks who want the security benefits badly enough.
It's not obvious that either of these catches are fundamental. The Fil-C development philosophy is very much to get something working first, which means downscoping. That's why I didn't include Yolo-C ABI compat as a goal at all and it's also why I haven't done as many optimizations as I should have. If you look at my GitHub you'll see >20 open issues with performance optimization ideas.
> In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
Of course this has happened.
Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C). Fil-C doesn't have an `unsafe` statement and the dependencies are all recompiled with Fil-C.
> Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Have you considered making... Fil-Zig? Would it be easier to do because Zig is already fairly far ahead of C in terms of safety?
> Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C).
To be fair `sudo-rs`'s usage of unsafe is all there just to interface with C code (e.g. PAM) so it isn't really sudo-rs that is "less safe", it's just that PAM isn't any safer (because it's unmodified C). Also you can use Rust without libc - https://github.com/sunfishcode/mustang - unfortunately it doesn't seem to have gained much traction which I think is a shame because glibc is a curse.
Most Rust programs actually have very few C dependencies. The big exceptions are libc and OpenSSL, but I think they'll be excised eventually.
> sudo-rs: uses pam compiled with Yolo-C, so it's not actually safe
Well it is, it's just that it doesn't magically make PAM safe either.
We're not disagreeing about anything technical, I just think it's slightly unfair to say Rust isn't as safe as Fil-C based on that. It is as safe; it just can't automatically make all C code safer like Fil-C can (and CHERI, etc.).
Almost all uses of unsafe in Rust are either for FFI, or to avoid the overhead of things like ref-counting. If you use, eg, Rc RefCell everywhere you will achieve safety without issue.
> Almost all uses of unsafe in Rust are either for FFI
Which makes Rust less safe than Fil-C.
Consider that you have a dependency like libc, PAM, openssl, or the like.
In Rust: you will `unsafe` call into the unsafe versions of those libraries. Hence, even if your Rust code is safe in isolation, your program is not safe overall because you're pulling in unsafe code.
In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
> On the other hand fil-c won't help with dependencies written in C++, Fortran, and lots of others.
IIRC Fil-C is already documented to work with C++. It's not immediately obvious to me that there are any fundamental reasons Fil-C wouldn't work for Fortran and "lots of others" as well; a fairly large chunk of its work is done via a pass on LLVM IR [0] so anything that compiles to LLVM could reap Fil-C's benefits with (probably?) relatively minor alterations to frontends.
> In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
Great work on Fil-C by the way. Very technically impressive stuff!
It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this). More and more C libraries are getting pure rust ports these days. It’s increasingly rare to actually need to pull in C libraries at all. Sticking to safe rust is a bit inconvenient - but I suspect so is using fil-c. I suppose with fil-C you can use any libraries you want, you’ve just gotta recompile them. That’s cool!
Rust has a bunch of unsafe code in std, so you’re still at risk of a bug there causing safety problems. But the same is true of Fil-C.
I could definitely see myself reaching for Fil-C when running legacy code. It’d be nice if there was a way to use fil-C to protect my rust program from memory bugs in C dependencies. But I can think of dozens of ways why that would be tricky to pull off.
Also I’m curious - can Fil-C protect against threading bugs like rust can? Rust has Send & Sync traits to mark which objects can be safely shared between threads. I can’t think of a way to do that in an automated way without help from the programmer.
> It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
Say you wanted to write a tool like sudo in C# or Go. Here's what would happen: you'd end up having to rely on dependent libraries, like pam, which then relies on other things like libselinux and libaudit. And libselinux depends on pcre2. Those things would be compiled with Yolo-C, so while your sudo tool's new code written by you would be safe, the process would mostly contain unsafe code.
But if you write sudo in Fil-C, then you can rely on all of those dependencies being compiled with Fil-C. The whole process is safe.
So, the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
> If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this).
That's impractical for the reasons above.
> Also I’m curious - can Fil-C protect against threading bugs like rust can?
Not like Rust can, but yes, it does.
Fil-C preserves memory safety under races. This means that debugging multithreaded C/C++ programs in Fil-C is a much nicer experience than in Yolo-C. If you do something wrong, you're going to get a panic - similar to how multithreading bugs in Java or C# lead to exceptions.
But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
> the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
Right; that makes a lot of sense. So currently the only implementation of pam is written in Yolo-C. If you want to build a safe, secure tool which depends on pam - like sudo - then you either need to trust pam (like sudo-rs does). Or you need some way to compile pam to safe code. Fil-C is a way to do that.
Just to name them - I can think of two other solutions to this problem:
1. Sandbox pam within a wasm container. This is probably less convenient than Fil-C (although it would allow that wasm container to be linked from yolo-c, rust, go, js, etc). For pam in particular, this would also be much less safe than Fil-C, since memory bugs in pam would still be possible, and might still result in privilege escalation. Wasm only sandboxes memory at the boundary. It wouldn't protect pam itself from its own bugs like Fil-C does.
2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
I think this also reaffirms the role I see for Fil-C: as a tool to make legacy C code (like pam) safe, at the cost of performance. For new code, I'd rather pick a fully GC language if I don't care about performance, or use rust if performance matters. Again, it would be nice if I could have the best of both worlds - use fil-c to wrap and sandbox pam, but write the rest of my program in rust or C# or Go or something else that is already safe without needing to pay fil-c's cost.
> But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
It depends what you're doing, but this can be a pretty big deal. I suppose this puts Fil-C in about the same safety and performance camp as Go, Java and C#. Ie, it has some protections that rust is missing - like there aren't any unsafe blocks. And its missing some protections rust has - like better thread safety. The unique selling point is it works with existing C code, like old C libraries that we don't want to rewrite. Thanks for your work on it!
> 2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
OK, let's think about that for a moment.
Go and Rust do not have dynamic linking. Well, Rust sort of does, if you're OK with using C ABI (i.e. `unsafe`).
C# only has dynamic loading if you aren't using the AOT. That means you're JITing. I don't think JITing scales to every process on your system doing it (you get no sharing of code between processes, and if you consider that I've got >500 processes on the machine I'm using right now, you can imagine how much of a memory burden JIT-in-everything would be).
And pam is engineered around dynamic linking. That's sort of the whole point of how it works. Your pam stack refers to modules that get dynamically loaded, which then allows pam modules to pull in things loads of other libraries.
So: there is currently no scalable alternative to Fil-C for making pam safe.
> Well, Rust sort of does, if you're OK with using C ABI (i.e. `unsafe`).
Rust has full blown safe dynamic linking - so long as you use rust, and the the same compiler version, for everything. What it doesn't have is runtime libdl support for that. Which would be a problem for the current pam architecture.
I'm not particularly convinced you couldn't just re-architect pam to use static linking though. Or a scripting language.
> Rust has full blown safe dynamic linking - so long as you use rust, and the the same compiler version, for everything.
This is UB in rust as far as I understand:
1) you compile bar using version 1.5 of libfoo.so at link/compile time
2) you update libfoo.so to version 1.6. It has a lot of changes, but is still source compatible and some kind of reasonable precautions were taken, like that struct layouts are the same
3) you run bar with the new libfoo.so version 1.6
This is UB even if you use the same compiler.
If this is UB then that’s not “full blown safe dynamic linking”.
Note that Fil-C does safely support this. This is safe in Fil-C even if you don’t take appropriate precautions (like if you change function signatures, change struct layouts, etc - it’ll still be safe and have well defined outcomes in Fil-C).
What you're looking for is a stable ABI, not just dynamic linking. Honestly not sure if rust guarantees a stable ABI under some set of "reasonable" precautions... likely not but it's never been something I've needed when I've even wanted in the context of matching rust's other constraint of matching compiler versions.
Does Fil-C support dynamic linking to "yolo C" code or only modules that have also been compiled using fil-C? Would all the pam modules (eg pam_systemd.so, pam_selinux.so, etc) also need to be compiled with fil-c? I assume so, to make it safe?
In general dynamic linking support is something that rust, go and C# could definitely improve upon. As I understand it, Rust does support dynamic linking for pure rust code. It just doesn't have a stable ABI yet. So dynamic linking is only supported within the same version of the rust compiler.
Taking a look at pam, it looks like there's 2 parts:
- The configuration parser & dynamic loader, which would be trivial to rewrite in rust or C# or something.
- The actual dynamically loaded modules, and the ABI bridge to call into them.
I think the obvious way to port pam to rust or C# would be to rewrite the core. Then just statically link in all the standard, built-in pam modules (like pam_unix). And - at least for now - handle "external" modules in exactly the same "yolo C" dyld way they are handled today. Yes, that means a dozen lines of boilerplate unsafe rust code across the C ABI bridge. It wouldn't be quite as safe as Fil-C or C# JITting for everything. But its within my personal safety paranoia tolerance. It would certainly be much safer than pam is today, given currently every single line is in yolo mode. It would also mean those modules wouldn't need to all be rewritten in rust all at once, or at all.
If you want a fully safe replacement for the pam bridge maybe wasm+wasi could work here? I'm not sure. I certainly hear you that this is a problem in rust, go and c#, and I'm not entirely sure what a good solution would look like. Well, other than rust stabilizing its ABI. At the pace rust moves at, that could be any decade now.
> I don't think JITing scales to every process on your system doing it (you get no sharing of code between processes, and if you consider that I've got >500 processes on the machine I'm using right now, you can imagine how much of a memory burden JIT-in-everything would be).
Forget every program for a moment. C#'s JIT would certainly be fast enough for pam. And most linux programs don't make such heavy use of dynamic linking to work. So they could be AOT compiled.
On the topic of performance, I'm not sure I want to pay the fil-C performance overhead for all 500+ processes on my system either. I assume the performance ceiling for fil-c is going to be somewhere around the performance of other well optimized GC runtimes. I don't know of any GC language that does better than about a 2x memory usage overhead.
Call me crazy, but for general computing, I like the speed of native code.
> As I understand it, Rust does support dynamic linking for pure rust code. It just doesn't have a stable ABI yet.
Not quite. To safely run a Rust program against a Rust dynamic library, you had to have compiled and linked that program against exactly that version of the Rust library. This is necessary for fundamental Rust type system reasons. It’s not simply a matter of stabilizing ABI. It’s not like updating the library without relinking would work if you used exactly the same rustc version - it would still be unsafe.
So no, Rust doesn’t support dynamic linking in any meaningful way.
> And most linux programs don't make such heavy use of dynamic linking to work.
This is not true at all.
Linux relies on dynamic linking extensively in two ways:
- A dynamic library may be updated independently of its dependents. You can’t do this in Rust, see above.
- Lots of Linux programs rely on the dynamically loaded module architecture like what pam does. All of the scripting language runtimes do this for example.
> Linux relies on dynamic linking extensively in two ways:
>
> - A dynamic library may be updated independently of its dependents. You can’t do this in Rust, see above.
This isn't a linux thing. Its a popular linux distribution thing. Linux distributions want to be in charge of which version of which dependencies every package uses. But that means the same piece of software on different linux machines will use subtly different versions of all their dependencies.
Distribution maintainers love this, because they can patch libjpeg to fix a bug and it'll affect all programs on the system. But its hell for the actual developers of those packages. If I use a dependency at version 1.2, ubuntu might silently replace that with version 1.1 or 1.2.1 or something else - none of which I've actually tested. Debian might only have version 0.9 available, so they quietly try and patch my software to use that instead. If there's a bug as a result, people will blame me for the bug. Sometimes the debian maintainers maintain their own patch sets to fix bugs in old versions - so they ship version 0.9.0-1 or something. When a user is having problems its now almost completely impossible for me to reproduce their environment on my computer.
Linux doesn't have to do any of this. It all works great with static linked binaries. Eg alpine linux uses musl instead of glibc and just statically links everything. This provides better performance in many cases because library code can be inlined.
Apt is in theory language agnostic. But in practice, this whole dynamic linking thing only really works for pure C libraries. Even C++ is a bit too fragile in most cases for this to work. (I think C++ is in theory ABI compatible, but in practice it only seems to work reliably if you compile all binaries with the same compiler, against the same version of the C++ STL.)
The reason Go doesn't support dynamic linking is that Rob Pike thinks its stupid. That dynamic linking only ever made sense when we didn't have much RAM, and those days are way behind us. I think I generally agree with him. I have no idea where that leaves rust.
> This isn't a linux thing. Its a popular linux distribution thing
You mean: it’s an OS thing.
Windows relies on it. MacOS and iOS rely on it.
When shared frameworks get to the multiple GB size then you can’t boot the system without dynamic linking.
> Even C++ is a bit too fragile in most cases for this to work. (I think C++ is in theory ABI compatible, but in practice it only seems to work reliably if you compile all binaries with the same compiler, against the same version of the C++ STL.)
Maybe about a quarter of the shared libraries, and maybe half of the large ones (as measured by source size) on Linux are C++.
It’s true that the preferred idiom is to use C ABI around the edges, but not everyone does this.
C++ has stable ABI even if you use different compilers these days. That’s been true for at least a decade now.
> The reason Go doesn't support dynamic linking is that Rob Pike thinks its stupid
I don’t care about what Rob Pike thinks is stupid.
> That dynamic linking only ever made sense when we didn't have much RAM, and those days are way behind us
Are you ok with me making thay same argument in favor of Fil-C’s current weak sauce optimization story?
“Optimizing your language implementation only even made sense when we didn’t have much RAM and when computers were slow, and those days are way behind us.”
What shared frameworks are multiple gigabytes in size?? If that’s the case I don’t think the problem is dynamic linking. A hoarder shouldn’t solve their problem by renting more houses.
Honestly I’ve been thinking of making my own toy OS and I’m still a bit on the fence about dynamic linking. As I understand it, macOS puts a lot of their UI components into dynamic libraries. That lets them update the look and feel of the OS between versions by swapping out the implementation of the controls for all apps. I think that’s a good design. I like that they have some way to do that. And for what it’s worth, I agree with your earlier point about scripting languages and whatnot using dynamic linking in Linux for extensibility.
In general I want more programs to be able to do what Pam does and be programmatically extensible. Dynamic linking is a very efficient way to do that. I think it’s really cool that Fil-C can transparently provide safety across those API boundaries. That’s something rust can’t really do until it has a stable ABI. In practice it doesn’t usually take much code to wrap / expose a C api from rust. But it’s inconvenient and definitely unsafe.
> Are you ok with me making thay same argument in favor of Fil-C’s current weak sauce optimization story?
No of course not. Static linking is a mixed bag optimisation wise because - while code is duplicated in ram, doing so also allows inlining and dead code elimination. So it’s usually not that bad in practice. And binaries are usually tiny. It’s an application’s data where I want efficiency.
I think Fil-C’s optimisation story is fine for many use cases. It should be able to be as efficient as C#, Go and other similar languages - which are pretty fast languages. More than fast enough for most applications. But there’s a lot of software I’d like to keep in fast native code, like databases and browsers. So I wouldn’t use Fil-C there. And in the places I don’t mind slower code, I’d rather use a C# or go or typescript. So for me that leaves Fil-C as an interesting tool primarily for sandboxing old C code.
If that was how Fil-C was talked about and positioned, that would be fine. But that’s not the narrative. I’m frustrated with some of the hypocrisy I’ve seen online from “influencers” like Casey M. Last week C and systems code was good because of how fast bare metal C runs. But now C is still great because it’s memory safe through Fil-C, so why even bother with rust? What liars! They just like C and don’t want to admit it. If they really cared about performance and systems code, they wouldn’t find Fil-C exciting. And if they actually cared about safety they wouldn’t have been so dismissive of memory safe languages before Fil-C came out. Liking C is fine. Hating rust is fine. There’s a lot to dislike. I just don’t like being lied to.
“I like Fil-C because it lets the Linux ecosystem have safety without needing to change to another language. I like C and want us to use it forever” is - I think - what a lot of people actually believe. If that’s the case, I wish people would come out and say it.
The ability to replace libraries without recompiling downstream dependents is not a necessary component of dynamic linking. However if rust wanted to gain this ability, it basically is just a matter of a stable ABI. The only other thing you would need is to get a hash of all the parts of the system that are equivalent to C header files and have the program check the hash in the dylib matched the version it was compiled against at load time.
I believe in fact the sdylib crate type in nightly rust does (or at least is intended to do, not sure it's fully implemented) exactly this.
It's not a bug, it's just a lack of an additional feature on top of dynamic linking.
In fact it prevents bugs as it provides a clear bar on the misuse of dynamic linking as is so common in C/C++/(as designed) Fil-C to replace libraries with likely-ABI-incompatible replacements. Fil-C limits the blast radius of bugs caused by this, but doesn't prevent the bugs. Just saying "you can't do that" like current rust actually prevents them. (Alternatively growing that feature properly, including checking ABI compatibility, as intended with sdylib's would also prevent them. To the best of my knowledge no C-like language does this today)
Dynmaic linking used responsibly to reduce linking times in development cycles, and to reduce memory usage when you have memory things using the same library, remains perfectly safe without this feature. The former is reasonably common in the rust ecosystem today, and I've literally never hit or heard of anyone else hitting a safety issue as a result. Minimizing memory usage is a niche use case in the modern world... but it's there if you want it. You just have to not modify the application post compilation - just like you have to not modify any application post compilation to retain memory safety guarantees.
I mean, that's a real advantage. But personally I'm not so keen for all my system software to have the performance of garbage collected languages. I don't want my software to suddenly get 1.5x-4x slower and start using twice as much RAM. For pam and sudo it wouldn't matter, sure. But across the whole linux system? I'd rather a slow and careful rewrite in rust over that sort of system wide slowdown.
There will never be a slow and careful rewrite in Rust. Nobody wants to do it, and doing it without introducing new bugs and security issues (not linked to memory) is even more difficult.
> There will never be a slow and careful rewrite in Rust. Nobody wants to do it
Lots of people want to do it. Whether you want to run any of their code is an open question - but rewriting old things from scratch is a very popular passtime.
In fact, people are already doing it. For example, the recent sudo-rs rewrite. Or the uutils coreutils package:
> doing it without introducing new bugs and security issues (not linked to memory) is even more difficult.
Difficult, but not impossible. Plenty of alternate C compilers exist. And alternate SQL databases. And some web browsers. And email servers. And TLS implementations. And on and on. Arguably there already are several reimplementations of the "linux userland", with various tweaks. They're just called Freebsd. And Illumos. And MacOS.
Why would the linux userland be uniquely impossible to reimplement in another language? I feel like people celebrate when someone makes a new text editor or gameboy emulator. Why so much hate at the idea of more implementations of unix tools?
> More and more C libraries are getting pure rust ports these days.
I don't think these rewrites will ever be mainstream. There is so much C out there, and typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
> I don't think these rewrites will ever be mainstream.
Forever is a long time. And there's always a huge appetite amongst smart young developers to create their own ecosystem. After all, you don't become Linus Torvalds by contributing to linux. You become linus torvalds by creating your own operating system from scratch.
Whether or not new implementations become valuable and get maintained over time is an open question. But the energy is definitely there. (Even though its not necessarily pointed in the direction of economically useful, long lived software.)
> typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
I hear a claim like: "The current C version of most tools is the only place, and only project which will ever bother solving its chosen problem properly." I'm not sure if tahts quite what you mean, but its a wildly bold claim. Even in C and C++, this seems already false. There's several good, solid, standards compliant SQL databases. And OS kernels. And compilers. And web browsers. If there's room for several C based OS kernels, do you really think there'll never be a viable OS kernel in rust, Zig or Odin? Or a web browser or compiler?
> And there's always a huge appetite amongst smart young developers to create their own ecosystem. After all, you don't become Linus Torvalds by contributing to linux. You become linus torvalds by creating your own operating system from scratch.
The tech landscape was completely different back then. Linus himself stated that had BSD been free at the time, he wouldn't have started working on a free MINIX clone. He worked to solve a concrete problem (the lack of a free UNIX-like kernel), he didn't partially rewrite something that exists and works well just to "create his own ecosystem."
Ok, that might be true of Linus himself. But I'm not sure what point you're making by bringing that up. Give up then? Linux forever, lets never try and iterate and improve?
Its weird talking to people in this thread. I get the impression lots of folks would really like it if the actual C code making up linux stayed in place forever. Like it would be a good thing if humans exploring the cosmos in the year 3025 were still logging in using pam, written in C. And not just C, but the current lines of code we have right now. Why? Because! Because C is the best at things. Things like dynamic linking! Because don't change what works. Because if you rewrite something you might get bugs. Because C is fast and efficient. Oh, never mind fast and efficient - lets compile it with Fil-C even though it makes it really slow, just so long as we don't have to change the code!
I think gnu/linux could be way better than it is now, in lots of ways. I get the impression that we - as a species - have barely scratched the surface of the design space for operating systems. The UNIX model is a really early design attempt. Is it the best design that will ever exist? Of course not. Do you know how the unix socket API came about? It was an intern / grad student at berkley who happened to be at the right place and right time. They invented an API and its basically never been changed since then. Because its the right API? No. Just because they were there and it was too hard to change it later.
I want proper capabilities. I want a microkernel. I want faster syscalls. I want storage that's relational, not just file based. I want atomic disk operations. I want programs which can be migrated between computers while they're running. I want supervisor trees. I want data that can be moved or shared between computers. I want my programming languages which can talk to each other using something richer than the C ABI. I want debugging tools that work with lots of languages. I want to be able to download and run programs from the internet safely, without risking all my data. This is all technically possible today.
But the idea of just making a bug for bug compatible, drop in replacement of a single tool like sudo is met with so much hostility and distrust from the community. Sometimes I think I live in a different world than most people. I think things can be better. I want things to be better. But we can't get there if we can't accept any change.
It sounds like Fil-C combines the ergonomics of C with the performance of Python.
It might have a beautiful niche for safely sandboxing legacy code. But I don’t see any compelling reason to use it for new code. Modern GC languages are much more ergonomic than C and better optimised. C#, Java, JavaScript and Go are all easier to write, they have better tooling and they will probably all perform as well or better than Fil-C in its current state.
The ergonomics of C are excellent and the performance seems to be more like Java or C# than Python. The point is that you don't have to rewrite anything, so even the ergonomics of C don't matter.
Personally when I write C, I miss memory safety. And thread safety. I miss non-nullable types. I miss having a good way to return results. And generics. And tuples. And real enums (sum types). I miss having a sane build system. I miss having IDEs that just work with my projects. I miss having a well designed standard library. I miss being able to hit build and have it more or less work on every OS. I miss having a good package manager. And a sane macro language. I wish I didn't need to write header files. And I wish my header files didn't need to be parsed MxN times. I miss fixed integer types just being built into the language - like u8/u16/u32/... And real strings. And on and on.
C compilers are fast and mature, so that's a plus. And C binaries are tiny. I mean, C has aged surprisingly well given the language is 53 years old. But its a really old language. And it shows. I would never use it for a new project, because it seems worse in every way compared to languages like Zig, Odin and Rust.
> and the performance seems to be more like Java or C# than Python.
Alright, I was exaggerating. But I still don't want my linux workstation to feel like a big java program.
> The point is that you don't have to rewrite anything, so even the ergonomics of C don't matter.
As I said in another comment, I really agree with this. Fil-C makes a lot of sense to me if the goal is to simply take existing legacy software and recompile it in a more memory safe way. But I don't think that's how the developers see it. I think they want fil-C so rust developers shut up, and they can go back to writing C code forever and ever and ever.
On the topic of Java.... did you know this? The original inventor of Java is a convicted pedophile. It's so embarassing that they scrubbed his name from pretty much everything, and when you Google "creator of Java" it gives you a different person (someone who got involved later). Some useful trivia for the future. Might come in handy.
I was reading the `fontconfig` source recently (to help me understand what, exactly, is the difference between the various `fc-*` tools and their options) and some of the code scared me.
I thought, "for sure this is a use-after-free" ... but it happened to be safe because some function performed an undocumented incref, and the return value of that function was kept alive.
So this is definitely a high priority thing to port!
... I suppose the interesting question is: if some of my dependencies are ported, and some are not, is there any way a normal compiler can call into a Fil-C-compiled library for just a few functions?
> ... I suppose the interesting question is: if some of my dependencies are ported, and some are not, is there any way a normal compiler can call into a Fil-C-compiled library for just a few functions?
To my understanding, there's no way to do this: Fil-C is basically a managed runtime for native code, so the FFI implications are similar to those for Go or any other intrusive managed runtime. Which is to say that Fil-C could offer an FFI, but not without losing the blanket exit-instead-of-memory-unsafety guarantees it aims to offer.
Exactly right
You could use a sandbox process and use some existing out of process calling system for it. This also disentangles the malloc systems of the modules.
If Fil-C requires complete recompile of ".c" code how does it deal with calls to the OS - Does it rewrap them (like Go?). I'm bit uncertain here...
It uses a sandwich; see https://fil-c.org/runtime
Thank you!!!
What is the actual catch with Fil-C? Memory safety for C based projects without a complete rewrite sounds like a very good idea, but there must be some pitfalls in this approach which this would not work in some cases.
In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
But great work on this nonetheless.
Author here! :-)
> What is the actual catch with Fil-C?
Two things:
- Not ABI compatible with Yolo-C. So, you have to recompile the whole stack. This is both a major catch, and it is both a bug and a feature. It's a bug for obvious reasons. It's a feature because it means I'll probably be the first to have a totally usable, totally memory safe desktop environment.
- In my testing, it's between 1.2x and 4x slower than Yolo-C. It uses between 2x and 3x more memory. Others have observed higher overheads in certain tests (I've heard of some things being 8x slower). How much this matters depends on your perspective. Imagine running your desktop environment on a 4x slower computer with 3x less memory. You've probably done exactly this and you probably survived the experience. So the catch is: Fil-C is for folks who want the security benefits badly enough.
It's not obvious that either of these catches are fundamental. The Fil-C development philosophy is very much to get something working first, which means downscoping. That's why I didn't include Yolo-C ABI compat as a goal at all and it's also why I haven't done as many optimizations as I should have. If you look at my GitHub you'll see >20 open issues with performance optimization ideas.
> In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
Of course this has happened.
Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C). Fil-C doesn't have an `unsafe` statement and the dependencies are all recompiled with Fil-C.
> Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Have you considered making... Fil-Zig? Would it be easier to do because Zig is already fairly far ahead of C in terms of safety?
> Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C).
To be fair `sudo-rs`'s usage of unsafe is all there just to interface with C code (e.g. PAM) so it isn't really sudo-rs that is "less safe", it's just that PAM isn't any safer (because it's unmodified C). Also you can use Rust without libc - https://github.com/sunfishcode/mustang - unfortunately it doesn't seem to have gained much traction which I think is a shame because glibc is a curse.
Most Rust programs actually have very few C dependencies. The big exceptions are libc and OpenSSL, but I think they'll be excised eventually.
> To be fair `sudo-rs`'s usage of unsafe is all there just to interface with C code (e.g. PAM) so it isn't really sudo-rs that is "less safe"
That's exactly my point.
sudo compiled with Fil-C: uses pam compiled with Fil-C, and all of pam's dependencies are compiled with Fil-C, so the whole thing is memory safe.
sudo-rs: uses pam compiled with Yolo-C, so it's not actually safe. pam is quite big and pulls in other unsafe dependencies
> sudo-rs: uses pam compiled with Yolo-C, so it's not actually safe
Well it is, it's just that it doesn't magically make PAM safe either.
We're not disagreeing about anything technical, I just think it's slightly unfair to say Rust isn't as safe as Fil-C based on that. It is as safe; it just can't automatically make all C code safer like Fil-C can (and CHERI, etc.).
[flagged]
Almost all uses of unsafe in Rust are either for FFI, or to avoid the overhead of things like ref-counting. If you use, eg, Rc RefCell everywhere you will achieve safety without issue.
> Almost all uses of unsafe in Rust are either for FFI
Which makes Rust less safe than Fil-C.
Consider that you have a dependency like libc, PAM, openssl, or the like.
In Rust: you will `unsafe` call into the unsafe versions of those libraries. Hence, even if your Rust code is safe in isolation, your program is not safe overall because you're pulling in unsafe code.
In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
> you will `unsafe` call into the unsafe versions of those libraries
Maybe. There's rust version of libc and SSL already. On the other hand fil-c won't help with dependencies written in C++, Fortran, and lots of others.
The "less safe" gets very context dependent.
> On the other hand fil-c won't help with dependencies written in C++, Fortran, and lots of others.
IIRC Fil-C is already documented to work with C++. It's not immediately obvious to me that there are any fundamental reasons Fil-C wouldn't work for Fortran and "lots of others" as well; a fairly large chunk of its work is done via a pass on LLVM IR [0] so anything that compiles to LLVM could reap Fil-C's benefits with (probably?) relatively minor alterations to frontends.
[0]: https://fil-c.org/compiler
Yeah, putting Fortran through the Fil-C pipeline is just a matter of a handful of hacks in flang.
> In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
Great work on Fil-C by the way. Very technically impressive stuff!
It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this). More and more C libraries are getting pure rust ports these days. It’s increasingly rare to actually need to pull in C libraries at all. Sticking to safe rust is a bit inconvenient - but I suspect so is using fil-c. I suppose with fil-C you can use any libraries you want, you’ve just gotta recompile them. That’s cool!
Rust has a bunch of unsafe code in std, so you’re still at risk of a bug there causing safety problems. But the same is true of Fil-C.
I could definitely see myself reaching for Fil-C when running legacy code. It’d be nice if there was a way to use fil-C to protect my rust program from memory bugs in C dependencies. But I can think of dozens of ways why that would be tricky to pull off.
Also I’m curious - can Fil-C protect against threading bugs like rust can? Rust has Send & Sync traits to mark which objects can be safely shared between threads. I can’t think of a way to do that in an automated way without help from the programmer.
> It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
Say you wanted to write a tool like sudo in C# or Go. Here's what would happen: you'd end up having to rely on dependent libraries, like pam, which then relies on other things like libselinux and libaudit. And libselinux depends on pcre2. Those things would be compiled with Yolo-C, so while your sudo tool's new code written by you would be safe, the process would mostly contain unsafe code.
But if you write sudo in Fil-C, then you can rely on all of those dependencies being compiled with Fil-C. The whole process is safe.
So, the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
> If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this).
That's impractical for the reasons above.
> Also I’m curious - can Fil-C protect against threading bugs like rust can?
Not like Rust can, but yes, it does.
Fil-C preserves memory safety under races. This means that debugging multithreaded C/C++ programs in Fil-C is a much nicer experience than in Yolo-C. If you do something wrong, you're going to get a panic - similar to how multithreading bugs in Java or C# lead to exceptions.
But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
> the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
Right; that makes a lot of sense. So currently the only implementation of pam is written in Yolo-C. If you want to build a safe, secure tool which depends on pam - like sudo - then you either need to trust pam (like sudo-rs does). Or you need some way to compile pam to safe code. Fil-C is a way to do that.
Just to name them - I can think of two other solutions to this problem:
1. Sandbox pam within a wasm container. This is probably less convenient than Fil-C (although it would allow that wasm container to be linked from yolo-c, rust, go, js, etc). For pam in particular, this would also be much less safe than Fil-C, since memory bugs in pam would still be possible, and might still result in privilege escalation. Wasm only sandboxes memory at the boundary. It wouldn't protect pam itself from its own bugs like Fil-C does.
2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
I think this also reaffirms the role I see for Fil-C: as a tool to make legacy C code (like pam) safe, at the cost of performance. For new code, I'd rather pick a fully GC language if I don't care about performance, or use rust if performance matters. Again, it would be nice if I could have the best of both worlds - use fil-c to wrap and sandbox pam, but write the rest of my program in rust or C# or Go or something else that is already safe without needing to pay fil-c's cost.
> But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
It depends what you're doing, but this can be a pretty big deal. I suppose this puts Fil-C in about the same safety and performance camp as Go, Java and C#. Ie, it has some protections that rust is missing - like there aren't any unsafe blocks. And its missing some protections rust has - like better thread safety. The unique selling point is it works with existing C code, like old C libraries that we don't want to rewrite. Thanks for your work on it!
> 2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
OK, let's think about that for a moment.
Go and Rust do not have dynamic linking. Well, Rust sort of does, if you're OK with using C ABI (i.e. `unsafe`).
C# only has dynamic loading if you aren't using the AOT. That means you're JITing. I don't think JITing scales to every process on your system doing it (you get no sharing of code between processes, and if you consider that I've got >500 processes on the machine I'm using right now, you can imagine how much of a memory burden JIT-in-everything would be).
And pam is engineered around dynamic linking. That's sort of the whole point of how it works. Your pam stack refers to modules that get dynamically loaded, which then allows pam modules to pull in things loads of other libraries.
So: there is currently no scalable alternative to Fil-C for making pam safe.
C# supports dynamic loading in AOT compilation.
Native AOT, and the oldie NGEN only do dynamic linking.
https://learn.microsoft.com/en-us/dotnet/core/deploying/nati...
.NET Native can do both,
Applications are statically compiled, libraries are compiled into WinRT components, COM's evolution, introduced in Windows 8.
https://learn.microsoft.com/en-us/windows/uwp/dotnet-native/...
Just watched the 2024 talk by the way, quite interesting.
[dead]
> Well, Rust sort of does, if you're OK with using C ABI (i.e. `unsafe`).
Rust has full blown safe dynamic linking - so long as you use rust, and the the same compiler version, for everything. What it doesn't have is runtime libdl support for that. Which would be a problem for the current pam architecture.
I'm not particularly convinced you couldn't just re-architect pam to use static linking though. Or a scripting language.
> Rust has full blown safe dynamic linking - so long as you use rust, and the the same compiler version, for everything.
This is UB in rust as far as I understand:
1) you compile bar using version 1.5 of libfoo.so at link/compile time
2) you update libfoo.so to version 1.6. It has a lot of changes, but is still source compatible and some kind of reasonable precautions were taken, like that struct layouts are the same
3) you run bar with the new libfoo.so version 1.6
This is UB even if you use the same compiler.
If this is UB then that’s not “full blown safe dynamic linking”.
Note that Fil-C does safely support this. This is safe in Fil-C even if you don’t take appropriate precautions (like if you change function signatures, change struct layouts, etc - it’ll still be safe and have well defined outcomes in Fil-C).
What you're looking for is a stable ABI, not just dynamic linking. Honestly not sure if rust guarantees a stable ABI under some set of "reasonable" precautions... likely not but it's never been something I've needed when I've even wanted in the context of matching rust's other constraint of matching compiler versions.
Does Fil-C support dynamic linking to "yolo C" code or only modules that have also been compiled using fil-C? Would all the pam modules (eg pam_systemd.so, pam_selinux.so, etc) also need to be compiled with fil-c? I assume so, to make it safe?
In general dynamic linking support is something that rust, go and C# could definitely improve upon. As I understand it, Rust does support dynamic linking for pure rust code. It just doesn't have a stable ABI yet. So dynamic linking is only supported within the same version of the rust compiler.
Taking a look at pam, it looks like there's 2 parts:
- The configuration parser & dynamic loader, which would be trivial to rewrite in rust or C# or something.
- The actual dynamically loaded modules, and the ABI bridge to call into them.
I think the obvious way to port pam to rust or C# would be to rewrite the core. Then just statically link in all the standard, built-in pam modules (like pam_unix). And - at least for now - handle "external" modules in exactly the same "yolo C" dyld way they are handled today. Yes, that means a dozen lines of boilerplate unsafe rust code across the C ABI bridge. It wouldn't be quite as safe as Fil-C or C# JITting for everything. But its within my personal safety paranoia tolerance. It would certainly be much safer than pam is today, given currently every single line is in yolo mode. It would also mean those modules wouldn't need to all be rewritten in rust all at once, or at all.
If you want a fully safe replacement for the pam bridge maybe wasm+wasi could work here? I'm not sure. I certainly hear you that this is a problem in rust, go and c#, and I'm not entirely sure what a good solution would look like. Well, other than rust stabilizing its ABI. At the pace rust moves at, that could be any decade now.
> I don't think JITing scales to every process on your system doing it (you get no sharing of code between processes, and if you consider that I've got >500 processes on the machine I'm using right now, you can imagine how much of a memory burden JIT-in-everything would be).
Forget every program for a moment. C#'s JIT would certainly be fast enough for pam. And most linux programs don't make such heavy use of dynamic linking to work. So they could be AOT compiled.
On the topic of performance, I'm not sure I want to pay the fil-C performance overhead for all 500+ processes on my system either. I assume the performance ceiling for fil-c is going to be somewhere around the performance of other well optimized GC runtimes. I don't know of any GC language that does better than about a 2x memory usage overhead.
Call me crazy, but for general computing, I like the speed of native code.
> As I understand it, Rust does support dynamic linking for pure rust code. It just doesn't have a stable ABI yet.
Not quite. To safely run a Rust program against a Rust dynamic library, you had to have compiled and linked that program against exactly that version of the Rust library. This is necessary for fundamental Rust type system reasons. It’s not simply a matter of stabilizing ABI. It’s not like updating the library without relinking would work if you used exactly the same rustc version - it would still be unsafe.
So no, Rust doesn’t support dynamic linking in any meaningful way.
> And most linux programs don't make such heavy use of dynamic linking to work.
This is not true at all.
Linux relies on dynamic linking extensively in two ways:
- A dynamic library may be updated independently of its dependents. You can’t do this in Rust, see above.
- Lots of Linux programs rely on the dynamically loaded module architecture like what pam does. All of the scripting language runtimes do this for example.
> Linux relies on dynamic linking extensively in two ways: > > - A dynamic library may be updated independently of its dependents. You can’t do this in Rust, see above.
This isn't a linux thing. Its a popular linux distribution thing. Linux distributions want to be in charge of which version of which dependencies every package uses. But that means the same piece of software on different linux machines will use subtly different versions of all their dependencies.
Distribution maintainers love this, because they can patch libjpeg to fix a bug and it'll affect all programs on the system. But its hell for the actual developers of those packages. If I use a dependency at version 1.2, ubuntu might silently replace that with version 1.1 or 1.2.1 or something else - none of which I've actually tested. Debian might only have version 0.9 available, so they quietly try and patch my software to use that instead. If there's a bug as a result, people will blame me for the bug. Sometimes the debian maintainers maintain their own patch sets to fix bugs in old versions - so they ship version 0.9.0-1 or something. When a user is having problems its now almost completely impossible for me to reproduce their environment on my computer.
Linux doesn't have to do any of this. It all works great with static linked binaries. Eg alpine linux uses musl instead of glibc and just statically links everything. This provides better performance in many cases because library code can be inlined.
Apt is in theory language agnostic. But in practice, this whole dynamic linking thing only really works for pure C libraries. Even C++ is a bit too fragile in most cases for this to work. (I think C++ is in theory ABI compatible, but in practice it only seems to work reliably if you compile all binaries with the same compiler, against the same version of the C++ STL.)
The reason Go doesn't support dynamic linking is that Rob Pike thinks its stupid. That dynamic linking only ever made sense when we didn't have much RAM, and those days are way behind us. I think I generally agree with him. I have no idea where that leaves rust.
> This isn't a linux thing. Its a popular linux distribution thing
You mean: it’s an OS thing.
Windows relies on it. MacOS and iOS rely on it.
When shared frameworks get to the multiple GB size then you can’t boot the system without dynamic linking.
> Even C++ is a bit too fragile in most cases for this to work. (I think C++ is in theory ABI compatible, but in practice it only seems to work reliably if you compile all binaries with the same compiler, against the same version of the C++ STL.)
Maybe about a quarter of the shared libraries, and maybe half of the large ones (as measured by source size) on Linux are C++.
It’s true that the preferred idiom is to use C ABI around the edges, but not everyone does this.
C++ has stable ABI even if you use different compilers these days. That’s been true for at least a decade now.
> The reason Go doesn't support dynamic linking is that Rob Pike thinks its stupid
I don’t care about what Rob Pike thinks is stupid.
> That dynamic linking only ever made sense when we didn't have much RAM, and those days are way behind us
Are you ok with me making thay same argument in favor of Fil-C’s current weak sauce optimization story?
“Optimizing your language implementation only even made sense when we didn’t have much RAM and when computers were slow, and those days are way behind us.”
What shared frameworks are multiple gigabytes in size?? If that’s the case I don’t think the problem is dynamic linking. A hoarder shouldn’t solve their problem by renting more houses.
Honestly I’ve been thinking of making my own toy OS and I’m still a bit on the fence about dynamic linking. As I understand it, macOS puts a lot of their UI components into dynamic libraries. That lets them update the look and feel of the OS between versions by swapping out the implementation of the controls for all apps. I think that’s a good design. I like that they have some way to do that. And for what it’s worth, I agree with your earlier point about scripting languages and whatnot using dynamic linking in Linux for extensibility.
In general I want more programs to be able to do what Pam does and be programmatically extensible. Dynamic linking is a very efficient way to do that. I think it’s really cool that Fil-C can transparently provide safety across those API boundaries. That’s something rust can’t really do until it has a stable ABI. In practice it doesn’t usually take much code to wrap / expose a C api from rust. But it’s inconvenient and definitely unsafe.
> Are you ok with me making thay same argument in favor of Fil-C’s current weak sauce optimization story?
No of course not. Static linking is a mixed bag optimisation wise because - while code is duplicated in ram, doing so also allows inlining and dead code elimination. So it’s usually not that bad in practice. And binaries are usually tiny. It’s an application’s data where I want efficiency.
I think Fil-C’s optimisation story is fine for many use cases. It should be able to be as efficient as C#, Go and other similar languages - which are pretty fast languages. More than fast enough for most applications. But there’s a lot of software I’d like to keep in fast native code, like databases and browsers. So I wouldn’t use Fil-C there. And in the places I don’t mind slower code, I’d rather use a C# or go or typescript. So for me that leaves Fil-C as an interesting tool primarily for sandboxing old C code.
If that was how Fil-C was talked about and positioned, that would be fine. But that’s not the narrative. I’m frustrated with some of the hypocrisy I’ve seen online from “influencers” like Casey M. Last week C and systems code was good because of how fast bare metal C runs. But now C is still great because it’s memory safe through Fil-C, so why even bother with rust? What liars! They just like C and don’t want to admit it. If they really cared about performance and systems code, they wouldn’t find Fil-C exciting. And if they actually cared about safety they wouldn’t have been so dismissive of memory safe languages before Fil-C came out. Liking C is fine. Hating rust is fine. There’s a lot to dislike. I just don’t like being lied to.
“I like Fil-C because it lets the Linux ecosystem have safety without needing to change to another language. I like C and want us to use it forever” is - I think - what a lot of people actually believe. If that’s the case, I wish people would come out and say it.
The ability to replace libraries without recompiling downstream dependents is not a necessary component of dynamic linking. However if rust wanted to gain this ability, it basically is just a matter of a stable ABI. The only other thing you would need is to get a hash of all the parts of the system that are equivalent to C header files and have the program check the hash in the dylib matched the version it was compiled against at load time.
I believe in fact the sdylib crate type in nightly rust does (or at least is intended to do, not sure it's fully implemented) exactly this.
> The ability to replace libraries without recompiling downstream dependents is not a necessary component of dynamic linking.
Yes it is, in the sense that not having this is a massive bug.
Dynamic linking becomes horrifically unsafe without this property, rendering it mostly useless.
It's not a bug, it's just a lack of an additional feature on top of dynamic linking.
In fact it prevents bugs as it provides a clear bar on the misuse of dynamic linking as is so common in C/C++/(as designed) Fil-C to replace libraries with likely-ABI-incompatible replacements. Fil-C limits the blast radius of bugs caused by this, but doesn't prevent the bugs. Just saying "you can't do that" like current rust actually prevents them. (Alternatively growing that feature properly, including checking ABI compatibility, as intended with sdylib's would also prevent them. To the best of my knowledge no C-like language does this today)
Dynmaic linking used responsibly to reduce linking times in development cycles, and to reduce memory usage when you have memory things using the same library, remains perfectly safe without this feature. The former is reasonably common in the rust ecosystem today, and I've literally never hit or heard of anyone else hitting a safety issue as a result. Minimizing memory usage is a niche use case in the modern world... but it's there if you want it. You just have to not modify the application post compilation - just like you have to not modify any application post compilation to retain memory safety guarantees.
I think the whole point is to _not_ rewrite anything.
I mean, that's a real advantage. But personally I'm not so keen for all my system software to have the performance of garbage collected languages. I don't want my software to suddenly get 1.5x-4x slower and start using twice as much RAM. For pam and sudo it wouldn't matter, sure. But across the whole linux system? I'd rather a slow and careful rewrite in rust over that sort of system wide slowdown.
It works perfectly fine in the legions of schools using Chromebooks, TVs with WebOS, and about 70% of mobile devices.
The only three Linux kernel powered systems that have actually achieved The Year of Linux Desktop success among non technical users.
There will never be a slow and careful rewrite in Rust. Nobody wants to do it, and doing it without introducing new bugs and security issues (not linked to memory) is even more difficult.
> There will never be a slow and careful rewrite in Rust. Nobody wants to do it
Lots of people want to do it. Whether you want to run any of their code is an open question - but rewriting old things from scratch is a very popular passtime.
In fact, people are already doing it. For example, the recent sudo-rs rewrite. Or the uutils coreutils package:
https://github.com/uutils/coreutils
- Which passes almost all conformance tests now by the way. (Though they could certainly use a few more tests):
https://uutils.github.io/coreutils/docs/test_coverage.html
> doing it without introducing new bugs and security issues (not linked to memory) is even more difficult.
Difficult, but not impossible. Plenty of alternate C compilers exist. And alternate SQL databases. And some web browsers. And email servers. And TLS implementations. And on and on. Arguably there already are several reimplementations of the "linux userland", with various tweaks. They're just called Freebsd. And Illumos. And MacOS.
Why would the linux userland be uniquely impossible to reimplement in another language? I feel like people celebrate when someone makes a new text editor or gameboy emulator. Why so much hate at the idea of more implementations of unix tools?
> More and more C libraries are getting pure rust ports these days.
I don't think these rewrites will ever be mainstream. There is so much C out there, and typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
> I don't think these rewrites will ever be mainstream.
Forever is a long time. And there's always a huge appetite amongst smart young developers to create their own ecosystem. After all, you don't become Linus Torvalds by contributing to linux. You become linus torvalds by creating your own operating system from scratch.
Whether or not new implementations become valuable and get maintained over time is an open question. But the energy is definitely there. (Even though its not necessarily pointed in the direction of economically useful, long lived software.)
> typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
I hear a claim like: "The current C version of most tools is the only place, and only project which will ever bother solving its chosen problem properly." I'm not sure if tahts quite what you mean, but its a wildly bold claim. Even in C and C++, this seems already false. There's several good, solid, standards compliant SQL databases. And OS kernels. And compilers. And web browsers. If there's room for several C based OS kernels, do you really think there'll never be a viable OS kernel in rust, Zig or Odin? Or a web browser or compiler?
> And there's always a huge appetite amongst smart young developers to create their own ecosystem. After all, you don't become Linus Torvalds by contributing to linux. You become linus torvalds by creating your own operating system from scratch.
The tech landscape was completely different back then. Linus himself stated that had BSD been free at the time, he wouldn't have started working on a free MINIX clone. He worked to solve a concrete problem (the lack of a free UNIX-like kernel), he didn't partially rewrite something that exists and works well just to "create his own ecosystem."
Ok, that might be true of Linus himself. But I'm not sure what point you're making by bringing that up. Give up then? Linux forever, lets never try and iterate and improve?
Its weird talking to people in this thread. I get the impression lots of folks would really like it if the actual C code making up linux stayed in place forever. Like it would be a good thing if humans exploring the cosmos in the year 3025 were still logging in using pam, written in C. And not just C, but the current lines of code we have right now. Why? Because! Because C is the best at things. Things like dynamic linking! Because don't change what works. Because if you rewrite something you might get bugs. Because C is fast and efficient. Oh, never mind fast and efficient - lets compile it with Fil-C even though it makes it really slow, just so long as we don't have to change the code!
I think gnu/linux could be way better than it is now, in lots of ways. I get the impression that we - as a species - have barely scratched the surface of the design space for operating systems. The UNIX model is a really early design attempt. Is it the best design that will ever exist? Of course not. Do you know how the unix socket API came about? It was an intern / grad student at berkley who happened to be at the right place and right time. They invented an API and its basically never been changed since then. Because its the right API? No. Just because they were there and it was too hard to change it later.
I want proper capabilities. I want a microkernel. I want faster syscalls. I want storage that's relational, not just file based. I want atomic disk operations. I want programs which can be migrated between computers while they're running. I want supervisor trees. I want data that can be moved or shared between computers. I want my programming languages which can talk to each other using something richer than the C ABI. I want debugging tools that work with lots of languages. I want to be able to download and run programs from the internet safely, without risking all my data. This is all technically possible today.
But the idea of just making a bug for bug compatible, drop in replacement of a single tool like sudo is met with so much hostility and distrust from the community. Sometimes I think I live in a different world than most people. I think things can be better. I want things to be better. But we can't get there if we can't accept any change.
> What is the actual catch with Fil-C?
It sounds like Fil-C combines the ergonomics of C with the performance of Python.
It might have a beautiful niche for safely sandboxing legacy code. But I don’t see any compelling reason to use it for new code. Modern GC languages are much more ergonomic than C and better optimised. C#, Java, JavaScript and Go are all easier to write, they have better tooling and they will probably all perform as well or better than Fil-C in its current state.
The ergonomics of C are excellent and the performance seems to be more like Java or C# than Python. The point is that you don't have to rewrite anything, so even the ergonomics of C don't matter.
> The ergonomics of C are excellent
That is a matter of opinion.
Personally when I write C, I miss memory safety. And thread safety. I miss non-nullable types. I miss having a good way to return results. And generics. And tuples. And real enums (sum types). I miss having a sane build system. I miss having IDEs that just work with my projects. I miss having a well designed standard library. I miss being able to hit build and have it more or less work on every OS. I miss having a good package manager. And a sane macro language. I wish I didn't need to write header files. And I wish my header files didn't need to be parsed MxN times. I miss fixed integer types just being built into the language - like u8/u16/u32/... And real strings. And on and on.
C compilers are fast and mature, so that's a plus. And C binaries are tiny. I mean, C has aged surprisingly well given the language is 53 years old. But its a really old language. And it shows. I would never use it for a new project, because it seems worse in every way compared to languages like Zig, Odin and Rust.
> and the performance seems to be more like Java or C# than Python.
Alright, I was exaggerating. But I still don't want my linux workstation to feel like a big java program.
> The point is that you don't have to rewrite anything, so even the ergonomics of C don't matter.
As I said in another comment, I really agree with this. Fil-C makes a lot of sense to me if the goal is to simply take existing legacy software and recompile it in a more memory safe way. But I don't think that's how the developers see it. I think they want fil-C so rust developers shut up, and they can go back to writing C code forever and ever and ever.
Indeed, black hats across the globe appreciate C's ergonomics making their jobs easier.
On the topic of Java.... did you know this? The original inventor of Java is a convicted pedophile. It's so embarassing that they scrubbed his name from pretty much everything, and when you Google "creator of Java" it gives you a different person (someone who got involved later). Some useful trivia for the future. Might come in handy.
This is super gross, but I don't see it is relevant... Does java's ecosystem suddenly not solve the same problems it did before?
I think it's interesting! Don't you? A fun fact of history most people don't know.