Cool stuff. At my previous (startup / research) job I had to set up similar system (but with franka arm and multi view camera) alone because I was the only one with robotics background.
>I do not intend to calibrate the camera’s extrinsics or intrinsics for now.
Sensible choice, although I suggest it's good in the long run to do at early stage in your setup, especially if you intend to collect data for policy learning.
Debugging trained policies for visual manipulation task can be a headache and having as much context on variables that can change is a good practice.
My previous setup was in Japan, a earthquake prone place and I wasted some time after realizing the camera got misaligned due to earthquake. A simple solution is just to place an Aruco marker on the table that tracks the relative extrinsic position of camera, and add it as metadata to collected teleoperation dataset.
Great points and I very much appreciate the input!
Right now the static camera is probably really bad: It's mounted on my desk, so its very easy to bump into it and move it. So yeah, it's position for sure will change over time. I think I need a better solution, maybe a rail system that's more rigidly attached to the robot arm so that at least the camera stays fixed relative to that point of reference.
Good move getting a nice robot! I'm doing something similar to you, but I went with a cheap robot, the "HIWONDER 6DOF Robotic Arm Kit". It was only $600, but wow is it bad. The precision and repeatability are both "are you drunk?" level. I can hear the gears grind when it moves. I suppose I should upgrade. But if my system can work with a terrible robot, I assume it would work even better with a nice one!
Hey this is cool! I am doing something similar myself with the SO101 arm robot from Robot Studio using a patchwork of my own code and LeRobot. Would love to collaborate with you if you are open to it. You can find me on Discord as `.avilay`. https://www.linkedin.com/posts/avilay_lerobot-huggingface-ro...
Would like to know your reasoning on not going with LeRobot.
Looks very cool! I’m not a huge discord user but how about you shoot me an email and we can figure out how to share notes? (I don’t want to post it directly here but it’s easy to find on my personal website, just google my name)
Re why not SO-101: the article has a footnote about this; I actually bought the SO-101 as well! I want to integrate it into the same setup so I can switch depending on task.
Somewhat surprisingly the xarm was actually much faster to arrive; I got it within 2 days of ordering. I don’t have a 3D printer and getting the SO-101 from the vendor I ordered it at took almost 4 weeks. So partially it just came down to what I had access to more quickly.
Second point is reliability: I think the SO-101 is cool but I’d be surprised if it doesn’t break more quickly than the xarm. I wanted something that’s going to last a long time without headaches. And these industrial arms are really mature hardware wise now.
Thanks for your response! I totally get your point about the delay in getting the robot, I ordered mine from PartaBot and they did take a couple of weeks to get here. But when they did, they worked great out-of-the-box :-)
I'm very interested in the SO101. I've never done any robotics and that seems a palatable entry level thing to try things out.
How have you found it?
(The author does explain his reasons for not using LeRobot in the post - although "I also use LeRobot for training and running baseline policies, and the vendor SDKs for the hardware.")
> The author does explain his reasons for not using LeRobot in the post - although "I also use LeRobot for training and running baseline policies, and the vendor SDKs for the hardware.")
Ah! That is exactly what I use it for as well :-)
I am liking the SO101 - teleop and robot both work great. For sure, it is very easy to get started with. I was able to collect around 50 demos with them and train my first ACT policy within days of setting up the robots. Happy to share more detailed learnings from this if/when you get started with it. https://github.com/avilay/learn-robotics
And yeah I feel you re humanoid. I worked on the Rubik's cube project at OpenAI, which used a humanoid hand, and it was insanely painful and hard. Also fun anecdote: it was completely impossible to teleop the shadow hand. We had a data glove to capture hand movements but as soon as contact / haptics come in, you're lost. We could never even get a single rotation on the Rubik's cube via teleop.
I do think simpler hardware like the one described in my post works really though and it's so much easier to do something with it.
Thanks! I did have fun tinkering at the intersection of both media.
I would have liked to explore a bit for but doing innovation readiness on technologies related to XR I basically had to move on to the next project. This though happened mostly because, as pointed out by the essay by Brooks, dexterity is hard, on both ends. Namely yes one can get a basic robot "arm" for cheap... but doing a robotic arm with a hand is something else, one with fine motor control is ... well I'm not an expert in the field but basically it doesn't exist yet IMHO. Sure we have grippers but that has basically nothing to do with a human hand. It's amazing how much flexibility we have at our disposal in such a compact and efficient form, sensors for touch obviously but also heat, proprioception of course, part that are smooth and flexibly while other are hard. The range of craftmanship we can do is... mindblowing. If you don't believe me just look at a basic magician, not even a good one (like me, I confess), doing sleight of hand, it's just amazing.
So that was on the robotic part, discovering what it could do, amazing, but more importantly for my work what it could NOT do.
A seemingly simple project was how to remove a 3D print from our 3D printer in order to free it up and move to the next job. This sounds trivial ... until you try to actually do it. I won't get into details but we didn't manage. It's of course feasible in the ideal scenario, e.g. successful print that is mostly rigid with attachment and support to the plate that requires just the right amount of force. It can be done. Now doing that in a realistic set of scenarii that a 3D printing house would do... well maybe it's feasible but I professionally didn't know (and still don't know) how to do with a realistic set of constraints (time wise, economically speaking too).
Moving on then to the other end, or hand (sorry for the pun) tracking in VR is good, honestly. It's quite fun for games... then trying to do so in a professional scenario then 1mm difference or occlusion for .1s is not acceptable anymore.
TL;DR (sorry I have to run and it might be longer than what you even asked for!) : the concept itself is obviously good, especially from a programmer standpoint. We are expert at automating, in fact I'd argue that's the 1 thing we excel at. The implementation though, in real life, is much harder that we naively consider, even with a LOT more computing power.
TL;DR (short): quick wins, yes, harder wins... not intractactable but at least beyond my own ability.
Very interesting article. I'm looking to get my first robot arm soon-ish, probably something in the SO-101 category.
Can someone get reasonably far using recorded sessions compared to training in a simulated environment¹?
Do you have any experience with third-party or DIY attachments for robot arms? I assume it's going to be more difficult for something like the Ufactory arm vs. the open models.
I'm only starting down this road but my sense is that ACT and Diffusion Policy both make it pretty feasible to start on real data only. LeRobot also makes it easier to train these. But that's the next step that I'm working on, so I don't know yet.
On attachments: during this project I really wanted a 3D printer several times. So that's probably next on the shopping list.
The Ufactory arm is actually quite extensible: it exposes digital input/output and you have a standard wrist mount where you can mount different end effectors or attachments.
Love this, I’m playing around with the cheapo esp32+servos version of this, super fun.
Something I’m working on is a hardware CLI for agents to run experiments, with a “CICD” pipeline that validates everything and means I can delegate more of the experiments to the agents. I wonder if you have any thoughts on this?
The idea is to allow the coding agent to run the full loop of experiments and validations, with vision, audio, button pressing, speaking etc to interact in place of the human
This seems indeed useful. I haven't seen this for robotics but I'm sure people need this for larger deployments (either for a distributed fleet or for an "arm farm"-like setup where there's many robots in one location for data collection / eval). Interesting idea!
Nice, I will be following your posts! I just bought a robot arm myself, the seeed studio B601DM (€1500 6+1 axis), it works great and is open source hardware as well and a bit more solid then the so101. I also opted to not use ros, I don't want to give up control by putting another framework in between. Is your plan to see whats possible right now or do you also have ideas on how to improve sota?
Oh very cool! Looks a bit like the TRLC-DK1 (I was looking at this one for a bit).
I think pushing the sota is quite hard to do solo but we'll see. Mostly I want to get back up to speed after having not done much robotics during the last 6 years. Best way for me to learn is to just do it, so here we are. We'll see how far I get (I suspect at some point compute will be the main bottleneck)
Great article. I'll be following along. Would like to learn more about the robotics space.
- I've heard the advantage of ROS besides the architecture is the ecosystem (driver integrations, etc). Is that not an issue because the arm supports a Python SDK OOTB?
- Any issues you've been running into with this setup?
- How do you determine if a session recording is good enough for training? Is 50/100 samples really all you need?
- The driver situation turned out totally fine; I intentionally picked HW with good python sdk support so that was very painless.
- The static camera (the C920) is not super great; it drops frames and sometimes cuts out. We’ll see how that goes but it’s probably the clostest thing I want to swap right now. Another issue is reach of the arm when forcing the worst to be axis parallel with the table; you cannot get very far away. The chess setup demo in the video gives an example: I can just reach the row of pawns and beyond that it’s out of reach.
- I don’t know yet! The 50-100 figure comes from the ACT and diffusion policy papers but it depends on the type of task. For fine tuning my sense is that you only need a few hours worth of demos to get good results with pi0.5 etc. a big reason I’m doing this project is that I want to try all of this myself, so the next posts definitely will talk about that
As impressive as this setup may be, I'm still amazed at how slow this type of robot is, whether amateur or professional grade.
I have no expertise in this field, but as an observer, the apparent progress in this area seems very limited.
I guess my expectations are too high and my understanding of the problems to solve is too low.
It’s partially my fault I currently clip the max speed _and_ I only input soft control changes when teleoperating to avoid crashing into things. The robot itself could definitely move more quickly than what you see in the video.
It would be interesting to explore how RL can be applied on top of my (flawed) human demos to optimize beyond what I’m able to do.
I had a very similar setup. Really happy with the xarm 6 lite. I played around with the diffusion policy paper experiments and was thinking to buy a webcam as a top camera as well but I ended up buying two intel realsense ones because of the timestamp drift issues. How did you solve that? Or is camera feed syncing not necessary for your intended projects?
I timestamp everything twice: once with the hardware clock (if available, like for the realsense camera) and once within my robot stack once it gets read from the device (using `time.monotonic_ns()`). Both are stored and alignment can happen with either timestamp. I think the 2nd timestamp is actually more meaningful since ultimately I want to reconstruct the state that the policy would've seen; so if one modality is delayed I should actually include that effect during training.
That being said, I might switch to a realsense for the static tabletop camera as well; the realsense wrist is clearly much more reliable than the cheap Logitech C920 that I currently use.
Both timestamps are useful in different ways. The early-as-possible hardware stamp is best for reasoning about causality, while the later-and-full-o-jitter middleware stamps are good for compensating for that inevitable jitter.
Time is one of the hard problems in robots, because they are inevitably but non-obviously distributed systems.
For understanding: I think the level is much deeper if I wrote the code vs reading someone else’s. Same applies to coding agents of course which is why I wrote most of it myself and only delegate some tasks (for example codex was great help at setting up telemetry dashboards or writing the custom glfw renderer).
On control: LeRobot will change all the time and I’ll be unaware of what changed. If something suddenly doesn’t work anymore, it’s a pain to find out. I can of course fork or pin but that defeats the purpose a bit.
At the end it’s also partially just preference: I wanted to write this layer myself, and I have opinions about how it should be architected, so I did.
Cool stuff. At my previous (startup / research) job I had to set up similar system (but with franka arm and multi view camera) alone because I was the only one with robotics background.
>I do not intend to calibrate the camera’s extrinsics or intrinsics for now.
Sensible choice, although I suggest it's good in the long run to do at early stage in your setup, especially if you intend to collect data for policy learning.
Debugging trained policies for visual manipulation task can be a headache and having as much context on variables that can change is a good practice.
My previous setup was in Japan, a earthquake prone place and I wasted some time after realizing the camera got misaligned due to earthquake. A simple solution is just to place an Aruco marker on the table that tracks the relative extrinsic position of camera, and add it as metadata to collected teleoperation dataset.
Great points and I very much appreciate the input!
Right now the static camera is probably really bad: It's mounted on my desk, so its very easy to bump into it and move it. So yeah, it's position for sure will change over time. I think I need a better solution, maybe a rail system that's more rigidly attached to the robot arm so that at least the camera stays fixed relative to that point of reference.
Good move getting a nice robot! I'm doing something similar to you, but I went with a cheap robot, the "HIWONDER 6DOF Robotic Arm Kit". It was only $600, but wow is it bad. The precision and repeatability are both "are you drunk?" level. I can hear the gears grind when it moves. I suppose I should upgrade. But if my system can work with a terrible robot, I assume it would work even better with a nice one!
My project is https://github.com/colinator/Ariel - basically, no VLAs - instead, "just write code". Or have the agents do it.
I don't have a writeup yet about applying Ariel to _this_ robot, but this is for a previous one: https://colinator.github.io/Ariel/post1.html.
Excited to follow your progress!
Wow very cool project and thanks for the Ariel pointer! Added to my reading list.
And yes, having a nice robot makes life so much easier. Very happy with my choice so far!
Hey this is cool! I am doing something similar myself with the SO101 arm robot from Robot Studio using a patchwork of my own code and LeRobot. Would love to collaborate with you if you are open to it. You can find me on Discord as `.avilay`. https://www.linkedin.com/posts/avilay_lerobot-huggingface-ro...
Would like to know your reasoning on not going with LeRobot.
Looks very cool! I’m not a huge discord user but how about you shoot me an email and we can figure out how to share notes? (I don’t want to post it directly here but it’s easy to find on my personal website, just google my name)
Re why not SO-101: the article has a footnote about this; I actually bought the SO-101 as well! I want to integrate it into the same setup so I can switch depending on task.
Somewhat surprisingly the xarm was actually much faster to arrive; I got it within 2 days of ordering. I don’t have a 3D printer and getting the SO-101 from the vendor I ordered it at took almost 4 weeks. So partially it just came down to what I had access to more quickly.
Second point is reliability: I think the SO-101 is cool but I’d be surprised if it doesn’t break more quickly than the xarm. I wanted something that’s going to last a long time without headaches. And these industrial arms are really mature hardware wise now.
Hope this helps!
Thanks for your response! I totally get your point about the delay in getting the robot, I ordered mine from PartaBot and they did take a couple of weeks to get here. But when they did, they worked great out-of-the-box :-)
Will email you to compare notes.
I'm very interested in the SO101. I've never done any robotics and that seems a palatable entry level thing to try things out.
How have you found it?
(The author does explain his reasons for not using LeRobot in the post - although "I also use LeRobot for training and running baseline policies, and the vendor SDKs for the hardware.")
> The author does explain his reasons for not using LeRobot in the post - although "I also use LeRobot for training and running baseline policies, and the vendor SDKs for the hardware.")
Ah! That is exactly what I use it for as well :-)
I am liking the SO101 - teleop and robot both work great. For sure, it is very easy to get started with. I was able to collect around 50 demos with them and train my first ACT policy within days of setting up the robots. Happy to share more detailed learnings from this if/when you get started with it. https://github.com/avilay/learn-robotics
Ah nice I've tinkered a bit with robotics few years ago, e.g. https://twitter-archive.benetou.fr/utopiah/status/1760260544... and it's pretty empowering... but also showcases how amazing the human body is. Our dexterity, ability to sense, etc.
Reminds me of https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... which is basically a stark warning against the hype.
Cool video! What made you stop if I can ask?
And yeah I feel you re humanoid. I worked on the Rubik's cube project at OpenAI, which used a humanoid hand, and it was insanely painful and hard. Also fun anecdote: it was completely impossible to teleop the shadow hand. We had a data glove to capture hand movements but as soon as contact / haptics come in, you're lost. We could never even get a single rotation on the Rubik's cube via teleop.
I do think simpler hardware like the one described in my post works really though and it's so much easier to do something with it.
Thanks! I did have fun tinkering at the intersection of both media.
I would have liked to explore a bit for but doing innovation readiness on technologies related to XR I basically had to move on to the next project. This though happened mostly because, as pointed out by the essay by Brooks, dexterity is hard, on both ends. Namely yes one can get a basic robot "arm" for cheap... but doing a robotic arm with a hand is something else, one with fine motor control is ... well I'm not an expert in the field but basically it doesn't exist yet IMHO. Sure we have grippers but that has basically nothing to do with a human hand. It's amazing how much flexibility we have at our disposal in such a compact and efficient form, sensors for touch obviously but also heat, proprioception of course, part that are smooth and flexibly while other are hard. The range of craftmanship we can do is... mindblowing. If you don't believe me just look at a basic magician, not even a good one (like me, I confess), doing sleight of hand, it's just amazing.
So that was on the robotic part, discovering what it could do, amazing, but more importantly for my work what it could NOT do.
A seemingly simple project was how to remove a 3D print from our 3D printer in order to free it up and move to the next job. This sounds trivial ... until you try to actually do it. I won't get into details but we didn't manage. It's of course feasible in the ideal scenario, e.g. successful print that is mostly rigid with attachment and support to the plate that requires just the right amount of force. It can be done. Now doing that in a realistic set of scenarii that a 3D printing house would do... well maybe it's feasible but I professionally didn't know (and still don't know) how to do with a realistic set of constraints (time wise, economically speaking too).
Moving on then to the other end, or hand (sorry for the pun) tracking in VR is good, honestly. It's quite fun for games... then trying to do so in a professional scenario then 1mm difference or occlusion for .1s is not acceptable anymore.
TL;DR (sorry I have to run and it might be longer than what you even asked for!) : the concept itself is obviously good, especially from a programmer standpoint. We are expert at automating, in fact I'd argue that's the 1 thing we excel at. The implementation though, in real life, is much harder that we naively consider, even with a LOT more computing power.
TL;DR (short): quick wins, yes, harder wins... not intractactable but at least beyond my own ability.
Very interesting article. I'm looking to get my first robot arm soon-ish, probably something in the SO-101 category. Can someone get reasonably far using recorded sessions compared to training in a simulated environment¹? Do you have any experience with third-party or DIY attachments for robot arms? I assume it's going to be more difficult for something like the Ufactory arm vs. the open models.
¹https://blog.comma.ai/mlsim/
I'm only starting down this road but my sense is that ACT and Diffusion Policy both make it pretty feasible to start on real data only. LeRobot also makes it easier to train these. But that's the next step that I'm working on, so I don't know yet.
On attachments: during this project I really wanted a 3D printer several times. So that's probably next on the shopping list.
The Ufactory arm is actually quite extensible: it exposes digital input/output and you have a standard wrist mount where you can mount different end effectors or attachments.
Love this, I’m playing around with the cheapo esp32+servos version of this, super fun.
Something I’m working on is a hardware CLI for agents to run experiments, with a “CICD” pipeline that validates everything and means I can delegate more of the experiments to the agents. I wonder if you have any thoughts on this?
The idea is to allow the coding agent to run the full loop of experiments and validations, with vision, audio, button pressing, speaking etc to interact in place of the human
Very cool!
Have you seen the recent nvidia thing? They do this at scale for robotics manipulation: https://research.nvidia.com/labs/gear/enpire/
Ah no I hadn't seen that, very interesting.
I'm finding a gap just before running those experiments.
The process of updating firmware, doing basic smoke tests on each device and validating it is live, and can function correctly.
Basically the pre-deployment green light that you get on github, but for hardware.
Have you seen or thought about that at all?
This seems indeed useful. I haven't seen this for robotics but I'm sure people need this for larger deployments (either for a distributed fleet or for an "arm farm"-like setup where there's many robots in one location for data collection / eval). Interesting idea!
Nice, I will be following your posts! I just bought a robot arm myself, the seeed studio B601DM (€1500 6+1 axis), it works great and is open source hardware as well and a bit more solid then the so101. I also opted to not use ros, I don't want to give up control by putting another framework in between. Is your plan to see whats possible right now or do you also have ideas on how to improve sota?
Oh very cool! Looks a bit like the TRLC-DK1 (I was looking at this one for a bit).
I think pushing the sota is quite hard to do solo but we'll see. Mostly I want to get back up to speed after having not done much robotics during the last 6 years. Best way for me to learn is to just do it, so here we are. We'll see how far I get (I suspect at some point compute will be the main bottleneck)
It looks like one stole the design from the other, I don't know which one, haha.
Great article. I'll be following along. Would like to learn more about the robotics space.
- I've heard the advantage of ROS besides the architecture is the ecosystem (driver integrations, etc). Is that not an issue because the arm supports a Python SDK OOTB?
- Any issues you've been running into with this setup?
- How do you determine if a session recording is good enough for training? Is 50/100 samples really all you need?
Glad you like it!
Re your questions:
- The driver situation turned out totally fine; I intentionally picked HW with good python sdk support so that was very painless.
- The static camera (the C920) is not super great; it drops frames and sometimes cuts out. We’ll see how that goes but it’s probably the clostest thing I want to swap right now. Another issue is reach of the arm when forcing the worst to be axis parallel with the table; you cannot get very far away. The chess setup demo in the video gives an example: I can just reach the row of pawns and beyond that it’s out of reach.
- I don’t know yet! The 50-100 figure comes from the ACT and diffusion policy papers but it depends on the type of task. For fine tuning my sense is that you only need a few hours worth of demos to get good results with pi0.5 etc. a big reason I’m doing this project is that I want to try all of this myself, so the next posts definitely will talk about that
As impressive as this setup may be, I'm still amazed at how slow this type of robot is, whether amateur or professional grade. I have no expertise in this field, but as an observer, the apparent progress in this area seems very limited. I guess my expectations are too high and my understanding of the problems to solve is too low.
It’s partially my fault I currently clip the max speed _and_ I only input soft control changes when teleoperating to avoid crashing into things. The robot itself could definitely move more quickly than what you see in the video.
It would be interesting to explore how RL can be applied on top of my (flawed) human demos to optimize beyond what I’m able to do.
Fascinating article. Keep up the work, Matthias!
I had a very similar setup. Really happy with the xarm 6 lite. I played around with the diffusion policy paper experiments and was thinking to buy a webcam as a top camera as well but I ended up buying two intel realsense ones because of the timestamp drift issues. How did you solve that? Or is camera feed syncing not necessary for your intended projects?
I timestamp everything twice: once with the hardware clock (if available, like for the realsense camera) and once within my robot stack once it gets read from the device (using `time.monotonic_ns()`). Both are stored and alignment can happen with either timestamp. I think the 2nd timestamp is actually more meaningful since ultimately I want to reconstruct the state that the policy would've seen; so if one modality is delayed I should actually include that effect during training.
That being said, I might switch to a realsense for the static tabletop camera as well; the realsense wrist is clearly much more reliable than the cheap Logitech C920 that I currently use.
Both timestamps are useful in different ways. The early-as-possible hardware stamp is best for reasoning about causality, while the later-and-full-o-jitter middleware stamps are good for compensating for that inevitable jitter.
Time is one of the hard problems in robots, because they are inevitably but non-obviously distributed systems.
Robots are annoyingly, wonderfully difficult.
How does Lerobot prevent « full control » and « understanding »? I thought this was an open source library.
I am not an official supporter of the library but am asking out of curiosity.
For understanding: I think the level is much deeper if I wrote the code vs reading someone else’s. Same applies to coding agents of course which is why I wrote most of it myself and only delegate some tasks (for example codex was great help at setting up telemetry dashboards or writing the custom glfw renderer).
On control: LeRobot will change all the time and I’ll be unaware of what changed. If something suddenly doesn’t work anymore, it’s a pain to find out. I can of course fork or pin but that defeats the purpose a bit.
At the end it’s also partially just preference: I wanted to write this layer myself, and I have opinions about how it should be architected, so I did.
This is really exciting. Incredible that you can do this for this budget at home. Unthinkable a couple years ago.
Thanks Tim and fun seeing you here :)
ROS sucks, good move. Too complicated
That was my take 8 years ago glad to hear it’s still that
whats the good alternative