Language-guided manipulation on a low-cost robot arm

I tested GPT-6 Astra with xhigh reasoning to control my LeRobot SO101 for a real-world pickup using a third-person RGB camera.

I deliberately chose a pen placement that required a backward-reaching configuration in this run.

My prompts

1. “We have connected a camera (for a third-person view) and a LeRobot SO101 robotic arm for control. Please control the robotic arm to pick up the red pen and hold it. You need to figure out for yourself how to adjust the robotic arm's pose.”
2. “Please record a video while operating.”
3. “Did you record a video of our entire process? Give me a final MP4.” (after I saw the robot complete the task)

What it did

It automatically reused some of my existing code for motor control, joint coordinate conversion, configuration loading, camera capture, and robot kinematics. Similar code and robot models are available online. It wrote new motion and recording scripts, adjusted its approach from camera feedback, and successfully picked up and held the pen.

Completion time and feedback

The pickup took 22 minutes end to end. This completion time does not specify movement speed or control frequency.

Each attempt returns camera feedback, allowing the agent to inspect the outcome and revise its next command. The successful pickup does not, by itself, reveal how accurately the model estimated the scene geometry or physical properties.

The reused control and kinematics code supplies part of the system's capability. The result reflects the agent's use of those tools together with visual feedback; it does not isolate manipulation ability attributable to the model alone.

Event-triggered visual replanning

One direction is to separate the timing of agent decisions from the execution of motion segments. A low-level controller could execute short segments, while unexpected visual feedback triggers another agent decision. The research problem is to decide which changes in the observed state warrant replanning. The aim is to reduce planning overhead while retaining the ability to detect and recover from grasp failure.

My original experiment on X.

@misc{dou2026so101pickup,
  author = {Dou, Zhiyang},
  title = {Language-guided manipulation on a low-cost robot arm},
  year = {2026},
  month = sep,
  howpublished = {Research Notes},
  url = {https://frank-zy-dou.github.io/blog/so101-pen-pickup/}
}