That's under very light CPU load. So an async approach, with no preemption, can work. If there's any significant compute going on, it won't work as well.
A useful number to measure on a scope is worst case interrupt latency. This is what matters if there's a hard real-time constraint. They measured standard deviation, but not worst case. The usual test setup is that an input signal (typically a square wave) goes to an input pin, interrupt happens if interrupts not prevented, task starts, task turns on an output pin. You watch input to output delay on a scope and look for outliers.
If you're running entirely run to completion, the outliers are determined by the longest compute task. This is a problem if there's a compute task.
This is historically where QNX shines. Interrupt is processed and schedules a thread. About all that happens at interrupt level is thread activation. The thread turns on the output pin. You can look on a scope for scheduling outliers. The best case latency is higher than doing the work at interrupt level, but the worst case latency is constant, even if lower priority threads are compute bound.
This is the difference between real time and "near real time" scheduling.
Embassy can (these days, not sure if this is more recent than the article) do preemption, but it works by setting up multiple task pools and executors for each priority level: https://docs.embassy.dev/embassy-executor/git/cortex-m/struc... Non-realtime compute heavy tasks can be processed in the background executor and interrupted by latency sensitive ones.
> The best case latency is higher than doing the work at interrupt level
One approach is to do everything in ISRs, a la RTIC. That requires efficient, vectored, nested, tail-chained, base priority-ed interrupt silicon, and a lot of it, but it is feasible and elegant where this exists, such as Cortex NVIC. Emerging RISC-V devices with xCLIC (ch32v, gd32v, newer ESP32 and others) are potentially even better.
I really appreciate that the author took the time to add the Embassy vs RTIC addendum.
FWIW, just having a mask in the interrupt controller is normally enough to give you the same thing at the cost of a dozen or so cycles in the critical path. Basically you just keep a mask per priority that can be built up cheaply at init time (or even compile time if you're cute about it), you apply the appropriate mask in the interrupt prologues and epilogues, and pretty much as soon as you apply the new mask in the prologue you go ahead and acknowledge the interrupt.
You can do this on x86 as well at a cost a merely tens to hundreds (possibly lots of hundreds) of thousands of cycles. This is part of why x86 is so popular in the embedded space.
(I’m being sarcastic, obviously. x86 interrupts and interrupt returns are hilariously slow. FRED may improve this by quite a bit.)
Mostly tons of speculative state that needs to be unwound, combined with spectre mitigations, plus tons of committed state that the interrupt prologue needs to save, plus a huge song and dance to do that correctly (that FRED should help with).
All combined with the fact that there's a good chance the memory the interrupt handler is going to touch isn't in the cached working set anymore, both in the actual L* caches and in subtler places like the branch predictors and TLBs.
You’re missing the big ones: both the interrupt delivery and the IRET (interrupt return) mechanisms use incredibly complicated data structures to determine what the new state should be. They need to dig around in the IDT, the GDT, the TSS and possibly the LDT to find all the register values they need to set, and they need to handle all kinds of backwards compatibility. And they “serialize”, which is an extra heavyweight fence, although that only likely accounts for a few hundred cycles in each direction.
Check out the pseudocode in the SDM — there are pages of it, and the pseudocode isn’t even complete.
FRED simplifies the state transitions such that the new state is mostly a foregone conclusion based on MSR contents.
I am aware. That "dozen or so" is a problem: when everything is an interrupt, there are no interrupts: it's just scheduling, and things that must be scheduled frequently can't suffer "a dozen or so" overhead. For the SRP model to really hum, you need the silicon that solves this.
I've found that it doesn't matter except for something that you want at the absolute highest priority anyway, which then by definition doesn't need to jump through the same hoops because nothing can preempt it anyway.
> One approach is to do everything in ISRs, a la RTIC.
That only works for really simple systems. On more complex systems there is a pretty good chance you will end up with locked up hardware if your ISR is long enough. Interrupts need servicing to keep the data flowing, prioritization is a job for the OS, not the hardware.
The model works pretty well up to much larger systems than you'd expect.
If a particular interrupt has a hard real time constraint, it sounds like a great candidate for a higher priority interrupt which will let it meet that requirement.
The biggest constraint is that this is really a single core model. You need something different if you go to SMP. Though there, AMP where the main core runs this 'interrrupt controller is your scheduler' scheme, and the other cores run against a work stealing scheduler for compute bound work items still is a very nice system to program against.
In my experience, it is pretty rare to run much actual work on a microcontroller.
Testing at effectively idle represents most of my usecases.
For the times where there is a background load: RTIC has task priorities and pre-emption, so you can run your compute-intensive task with a lower priority and react to interrupts in a timely manner.
I wanted to switch to Rust for new embedded project. I was looking for native RTOS, and embassy came up. But i always feel like it's hacky to me. What i want is RTOS that is similar to FreeRTOS or Zephyr.
What is this lol. A test where your consumer is orders of magnitude slower than your produce, but you focus on button press latency, as though the task isn’t completed dominated by the slow-ass USART print. 20 bytes is like 1.7 ms to print. They’re also running freeRTOS preemptively even though it doesn’t help here. Just use the cooperative mode. Or better yet, just write one event loop, since all the workloads are extremely bounded. Or even better yet, use a 555 or something, because this workload literally doesn’t even need a processor. I don’t even dislike embassy or freeRTOS, but this comparison reaches depths of stupidity I thought were impossible to reach without switching to some sort of hypoxic trimix.
The title in the article is literally "Async Rust vs RTOS showdown!" and the article shows "Embassy/Rust against FreeRTOS/C". There should also be a (2022) appended.
You just need to read the Reddit comments to see why this is not a useful comparison [1] [2].
Technically RTOS’s can also be cooperatively multithreaded, to some degree at least (see FreeRTOS). The comparison is valid IMO as threading with fixed yield points is an abstraction that isn’t exclusive to async Rust.
This article is almost 5 years old now, which makes it fairly ancient in Embedded Rust terms. I think the general landscape hasn't changed all that much, but I'd be cautious of relying on any details from it.
Meanwhile nothing really seems to matter anymore but Zephyr which has become almost Linux like in its escape velocity. Every vendor on earth now supports it, and these guys follow the money.
Rust seems to have modest support in some places.
With AI I don’t know language really matters anymore.
Language still matters because writing robust code in C requires discipline on many fronts, and LLMs will often do the minimum to get code running (as will most humans) unless prompted further. Rust protects you from some of these issues and gives you greater confidence that changes will not introduce bugs.
ha! I was happily surprised to see this article posted because it felt like it was picking up where the technical discourse was before LLM's took over.
2022 explains that perfectly, albeit leaves me less happy.
> In the web world async/await has already won from threads
Slightly off topic but threads were never supported by the web (even now you only have message passing between workers) so it's a bit hollow to say async won.
Your comment seems to imply that async/await should be or is in some setting not just sugar for promises. But, I am under the impression the async/await is (and always has been) sugar over explicit promises. Not just in JavaScript but in C#, where the sugar originates, as well.
Is there somewhere that async/await is implemented as a different concurrency mechanism?
That's under very light CPU load. So an async approach, with no preemption, can work. If there's any significant compute going on, it won't work as well.
A useful number to measure on a scope is worst case interrupt latency. This is what matters if there's a hard real-time constraint. They measured standard deviation, but not worst case. The usual test setup is that an input signal (typically a square wave) goes to an input pin, interrupt happens if interrupts not prevented, task starts, task turns on an output pin. You watch input to output delay on a scope and look for outliers.
If you're running entirely run to completion, the outliers are determined by the longest compute task. This is a problem if there's a compute task.
This is historically where QNX shines. Interrupt is processed and schedules a thread. About all that happens at interrupt level is thread activation. The thread turns on the output pin. You can look on a scope for scheduling outliers. The best case latency is higher than doing the work at interrupt level, but the worst case latency is constant, even if lower priority threads are compute bound.
This is the difference between real time and "near real time" scheduling.
Embassy can (these days, not sure if this is more recent than the article) do preemption, but it works by setting up multiple task pools and executors for each priority level: https://docs.embassy.dev/embassy-executor/git/cortex-m/struc... Non-realtime compute heavy tasks can be processed in the background executor and interrupted by latency sensitive ones.
> The best case latency is higher than doing the work at interrupt level
One approach is to do everything in ISRs, a la RTIC. That requires efficient, vectored, nested, tail-chained, base priority-ed interrupt silicon, and a lot of it, but it is feasible and elegant where this exists, such as Cortex NVIC. Emerging RISC-V devices with xCLIC (ch32v, gd32v, newer ESP32 and others) are potentially even better.
I really appreciate that the author took the time to add the Embassy vs RTIC addendum.
FWIW, just having a mask in the interrupt controller is normally enough to give you the same thing at the cost of a dozen or so cycles in the critical path. Basically you just keep a mask per priority that can be built up cheaply at init time (or even compile time if you're cute about it), you apply the appropriate mask in the interrupt prologues and epilogues, and pretty much as soon as you apply the new mask in the prologue you go ahead and acknowledge the interrupt.
You can do this on x86 as well at a cost a merely tens to hundreds (possibly lots of hundreds) of thousands of cycles. This is part of why x86 is so popular in the embedded space.
(I’m being sarcastic, obviously. x86 interrupts and interrupt returns are hilariously slow. FRED may improve this by quite a bit.)
But you’re also comparing dozens of cycles at 10mhz to 100ks at 5ghz. That’s probably comparable in terms of wall clock, no?
What's the technical reason for them being slow? Book keeping with caches or something?
Mostly tons of speculative state that needs to be unwound, combined with spectre mitigations, plus tons of committed state that the interrupt prologue needs to save, plus a huge song and dance to do that correctly (that FRED should help with).
All combined with the fact that there's a good chance the memory the interrupt handler is going to touch isn't in the cached working set anymore, both in the actual L* caches and in subtler places like the branch predictors and TLBs.
You’re missing the big ones: both the interrupt delivery and the IRET (interrupt return) mechanisms use incredibly complicated data structures to determine what the new state should be. They need to dig around in the IDT, the GDT, the TSS and possibly the LDT to find all the register values they need to set, and they need to handle all kinds of backwards compatibility. And they “serialize”, which is an extra heavyweight fence, although that only likely accounts for a few hundred cycles in each direction.
Check out the pseudocode in the SDM — there are pages of it, and the pseudocode isn’t even complete.
FRED simplifies the state transitions such that the new state is mostly a foregone conclusion based on MSR contents.
Thanks. Surely this was a performance hit even before Spectre?
Also, any good technical resources that concisely describe FRED?
I found this, which isn't bad but it's a bit more dumbed down than I'd like: https://www.tomshardware.com/pc-components/cpus/amd-adopts-f...
I am aware. That "dozen or so" is a problem: when everything is an interrupt, there are no interrupts: it's just scheduling, and things that must be scheduled frequently can't suffer "a dozen or so" overhead. For the SRP model to really hum, you need the silicon that solves this.
I've found that it doesn't matter except for something that you want at the absolute highest priority anyway, which then by definition doesn't need to jump through the same hoops because nothing can preempt it anyway.
> One approach is to do everything in ISRs, a la RTIC.
That only works for really simple systems. On more complex systems there is a pretty good chance you will end up with locked up hardware if your ISR is long enough. Interrupts need servicing to keep the data flowing, prioritization is a job for the OS, not the hardware.
The model works pretty well up to much larger systems than you'd expect.
If a particular interrupt has a hard real time constraint, it sounds like a great candidate for a higher priority interrupt which will let it meet that requirement.
The biggest constraint is that this is really a single core model. You need something different if you go to SMP. Though there, AMP where the main core runs this 'interrrupt controller is your scheduler' scheme, and the other cores run against a work stealing scheduler for compute bound work items still is a very nice system to program against.
How would it handle kernel/user space if everything runs inside ISR context ?
In my experience, it is pretty rare to run much actual work on a microcontroller. Testing at effectively idle represents most of my usecases.
For the times where there is a background load: RTIC has task priorities and pre-emption, so you can run your compute-intensive task with a lower priority and react to interrupts in a timely manner.
Agreed. I really like Embassy, and the write up is a fun read. But, this isn't what "real" embedded software looks like.
I wanted to switch to Rust for new embedded project. I was looking for native RTOS, and embassy came up. But i always feel like it's hacky to me. What i want is RTOS that is similar to FreeRTOS or Zephyr.
> You watch input to output delay on a scope and look for outliers.
better to acquire these or use timestamped gpio and compute real statistics- but for the sake of illustrative metaphor, sure.
When your goal is to show your pet language is 'better', you pick the benchmarks that 'prove' it.
No, this is really the whole point of an RTOS. It can preempt low priority tasks to respond to critical events.
I'm quite clear on what the point of an RTOS is, thank you. But I wasn't addressing that, which is the point you seem to have missed.
True RTOS scheduling is equally possible on Rust. RTIC does it already.
What is this lol. A test where your consumer is orders of magnitude slower than your produce, but you focus on button press latency, as though the task isn’t completed dominated by the slow-ass USART print. 20 bytes is like 1.7 ms to print. They’re also running freeRTOS preemptively even though it doesn’t help here. Just use the cooperative mode. Or better yet, just write one event loop, since all the workloads are extremely bounded. Or even better yet, use a 555 or something, because this workload literally doesn’t even need a processor. I don’t even dislike embassy or freeRTOS, but this comparison reaches depths of stupidity I thought were impossible to reach without switching to some sort of hypoxic trimix.
Title should be changed. Async Rust != RTOS. RTOS's are preemptively multithreaded while async is done cooperatively with yield points.
Perhaps "Embedded async Rust vs. C RTOS"
The title in the article is literally "Async Rust vs RTOS showdown!" and the article shows "Embassy/Rust against FreeRTOS/C". There should also be a (2022) appended.
You just need to read the Reddit comments to see why this is not a useful comparison [1] [2].
[1] https://www.reddit.com/r/rust/comments/sik3g0/async_rust_vs_...
[2] https://www.reddit.com/r/embedded/comments/she3u9/async_rust...
Cooperatively multi-threaded RTOSs exist.
The article itself seems to have the right title: Async Rust vs RTOS showdown!
The title on HN should probably be updated to that
Technically RTOS’s can also be cooperatively multithreaded, to some degree at least (see FreeRTOS). The comparison is valid IMO as threading with fixed yield points is an abstraction that isn’t exclusive to async Rust.
This article is almost 5 years old now, which makes it fairly ancient in Embedded Rust terms. I think the general landscape hasn't changed all that much, but I'd be cautious of relying on any details from it.
dude talks about measuring C while the code he shows is C++.
Meanwhile nothing really seems to matter anymore but Zephyr which has become almost Linux like in its escape velocity. Every vendor on earth now supports it, and these guys follow the money.
Rust seems to have modest support in some places.
With AI I don’t know language really matters anymore.
Language still matters because writing robust code in C requires discipline on many fronts, and LLMs will often do the minimum to get code running (as will most humans) unless prompted further. Rust protects you from some of these issues and gives you greater confidence that changes will not introduce bugs.
(2022)
ha! I was happily surprised to see this article posted because it felt like it was picking up where the technical discourse was before LLM's took over.
2022 explains that perfectly, albeit leaves me less happy.
> In the web world async/await has already won from threads
Slightly off topic but threads were never supported by the web (even now you only have message passing between workers) so it's a bit hollow to say async won.
Yeah, async/await in JavaScript is syntactic sugar for promises.
Your comment seems to imply that async/await should be or is in some setting not just sugar for promises. But, I am under the impression the async/await is (and always has been) sugar over explicit promises. Not just in JavaScript but in C#, where the sugar originates, as well.
Is there somewhere that async/await is implemented as a different concurrency mechanism?
Those damn compilers are going to take our jobs!!!