Full-stack robot learning & Haptic-ACT

Problem

Customers could teleoperate robots in VR, but turning those demonstrations into reliable autonomous policies was slow, manual and fragile. And imitation policies trained on vision alone squeeze too hard on anything delicate.

What I did

I built the backend and cloud infrastructure behind Extend’s robot learning, end to end (the system diagram below).

Data in. Demonstrations captured through the AMAS XR interface land on the edge, then upload to S3. A Lambda trigger queues each upload in SQS for processing on EC2 into LeRobot, HDF5 or rosbag datasets, alongside synthetic data from Isaac Lab, NVIDIA Cosmos and MimicGen.

Training and serving. Step Functions orchestrate SageMaker training runs for ACT, GR00T N1.6 and π0, with artifacts versioned in S3. Trained models are served through TorchServe, evaluated in simulation, then downloaded and converted to TensorRT for the robot. A web console ties it together, so a customer can inspect data, launch training and deploy a model without touching a terminal.

Real time on the robot. On the Jetson Orin I split execution into an inference engine and a control engine that run asynchronously: observations stream in at 30 Hz and the controller runs at 50 Hz, so the robot never waits on a slow transformer forward pass. Each cycle the policy predicts an action chunk but only the first action is executed before re-planning on the newest camera, joint and force readings, a receding-horizon loop that reacts to contact as it happens. Actions are interpolated before they reach the arm, so commands stay smooth.

Then we added touch. Haptic-ACT, which I co-developed with Kelin Li and Imperial College’s Robot Intelligence Lab, feeds fingertip contact forces into ACT alongside two camera streams and the arm and hand joint positions. The same forces go back to the demonstrator through a SenseGlove, so both the human and the policy learn how hard to grip. A later version adds the wrist force-torque sensor as a further input.

Result

In the IROS paper, across 50 simulated pick-and-place runs, the Haptic-ACT thumb pressed about 30% less than vanilla ACT (p < 0.001). On the real robot, average fingertip force across all five fingers dropped about 15%, and success rate held up (86% vs 83% for ACT).

Force feedback also made the XR teleoperation itself better on contact-rich work. With feedback, battery insertion success rose from 78% to 95% and got 22% faster, button pressing from 65% to 90%, and plugging an MSD connector with under a millimetre of clearance from 50% to 77%, while the low-contact tomato pick barely changed. Beginners gained the most. For customers: better demonstrations, shorter training cycles and policies that are gentler on what they handle.