This was built as part of the Nebius x NVIDIA hackathon. This is a piece of middleware to allow interaction between lightweight models on the robot and large vision language models on the cloud.
In our example, we had a raspberry pi based sorting system. When a Kiwi was placed on the platform, our detection model (YOLOX-Nano) didn't recognise it (COCO classes). It therefore wanted to discard it. But because this action is irreversible, we had to query the cloud. However, as we had disconnected the robot's Wi-Fi, it had to wait for 37 seconds for the network to come back. The cloud responded by telling us to place it in the basket. You can see this decision here: https://demo.cerebellumlabs.dev/journal?decision=01a174de-46... (the robot is no longer running).
There are 3 different types of actions:
Reflex: Robot makes decision. No wait time. If someone is in sight, stop.
Reversible: Robot makes instant decision. If confidence is below the threshold set for this action (here 0.9), the cloud checks it after robot has made a decision and can correct it.
Irreversible: Only cloud model makes decision. Waits in queue whilst offline.
The cloud model can only make one of the robot's choices. It will never directly control the motors.
The agent was written in Python with a SQLite Journal/Queue. The server was written in Go + Postgres. Any vision model with a compatible OpenAI API can be used. We used Nemotron on the Nebius Token Factory. Our first mistake was that our token limit was too low. By default the model will think before answering, so with 300 tokens we never received any responses. We fixed this by turning off the thinking for any queries that the robot was waiting for.
You can run everything on 1 machine using a simulated camera (see README). You'll need a Token Factory key to use the cloud portion. The sorting paddle was just a popsicle stick stuck to a servo.
Overall this was a hackathon project, there was only 1 robot and 19 decisions, nothing was stress tested. How would you decide what actions should be irreversible?
This was built as part of the Nebius x NVIDIA hackathon. This is a piece of middleware to allow interaction between lightweight models on the robot and large vision language models on the cloud.
In our example, we had a raspberry pi based sorting system. When a Kiwi was placed on the platform, our detection model (YOLOX-Nano) didn't recognise it (COCO classes). It therefore wanted to discard it. But because this action is irreversible, we had to query the cloud. However, as we had disconnected the robot's Wi-Fi, it had to wait for 37 seconds for the network to come back. The cloud responded by telling us to place it in the basket. You can see this decision here: https://demo.cerebellumlabs.dev/journal?decision=01a174de-46... (the robot is no longer running).
There are 3 different types of actions:
Reflex: Robot makes decision. No wait time. If someone is in sight, stop.
Reversible: Robot makes instant decision. If confidence is below the threshold set for this action (here 0.9), the cloud checks it after robot has made a decision and can correct it.
Irreversible: Only cloud model makes decision. Waits in queue whilst offline.
The cloud model can only make one of the robot's choices. It will never directly control the motors.
The agent was written in Python with a SQLite Journal/Queue. The server was written in Go + Postgres. Any vision model with a compatible OpenAI API can be used. We used Nemotron on the Nebius Token Factory. Our first mistake was that our token limit was too low. By default the model will think before answering, so with 300 tokens we never received any responses. We fixed this by turning off the thinking for any queries that the robot was waiting for.
You can run everything on 1 machine using a simulated camera (see README). You'll need a Token Factory key to use the cloud portion. The sorting paddle was just a popsicle stick stuck to a servo.
Overall this was a hackathon project, there was only 1 robot and 19 decisions, nothing was stress tested. How would you decide what actions should be irreversible?