How it works
The robot observes an action such as someone picking up a cup and translates that observation into natural language, thinking something like, "The person took the cup, this internal narration transforms an image into a sentence and that changes everything." By verbalizing what it is seeing, the robot manages to create connections between the visual world and its own actions, even without having ever seen that object before.
This process not only accelerates learning, but also drastically reduces the memory and processing load required, it is what scientists call zero-shot learning, learning something new without prior training, all of this happens locally, without depending on the cloud.
The new version of the model called Gemini Robotic AI runs directly on the robot, this guarantees a real-time response, ideal for environments where latency can be critical, such as in hospitals, for example. Furthermore, by not sending data to external servers, privacy remains intact.
The promise goes beyond domestic robots, with this ability to talk to themselves, robots will be able to adapt to complex situations such as folding clothes, opening bags or performing delicate tasks in real time, all with almost human fluidity. They have already been tested on different platforms such as Google's Aloha robot, Trony's Apollo, even on the Frenca FR3 robotic arm. In each of them, the learning was successfully transferred, demonstrating the potential for use in hospitals, factories, warehouses and even our homes.
