sw/
← projects
Q-Bot·2021–22

Self-supervised spray targeting & auto-drive

A U-Net trained on years of operator behaviour, with no hand labels, that tells Q-Bot's Mk7 spray robot where to insulate next and lets it drive there on its own. First-author paper at IEEE MMAR 2022.

Problem

Q-Bot had been insulating homes with a fleet of robots for years, almost entirely by teleoperation. The operator decided every spray location, repositioned the robot and adjusted the spray pattern by hand, inside a dark, cramped void they could only see through a camera. It worked, but it was a constant cognitive load, and the robot could not suggest or do anything on its own.

What I did

I led the work to automate the core decision, where to spray next, and published it as first author with Ashley Napier, Maddy Clifford, Tom Lipinski and Peter Childs (Q-Bot and Imperial College London).

Labels for free. Every on-site install was already recorded: joypad presses, sensor readings, images and point clouds, from thousands of properties across the UK and Europe. I turned that history into training data with no human labelling: each spray polygon an operator chose became a segmentation mask of valid spray region, no-spray region (robot body and void walls) and unknown. That gave about 100,000 images.

The model. A simplified U-Net, trained at 800 × 600 with PyTorch Lightning. Operator selections are noisy and the classes are badly imbalanced, so I used a weighted symmetric cross-entropy loss: weighted cross-entropy for the imbalance plus reverse cross-entropy for robustness to noisy labels.

On the robot. I integrated the Spray Region Estimator into the Mk7’s ROS stack. It predicts the next spray polygon from the camera image and shows it to the operator, who can accept or adjust it. One click and the planning stack drives the robot to the best spraying position, using a global planner for the route and a local planner for fine manoeuvres, ready to apply insulation.

Result

Weighted symmetric cross-entropy raised spray-region mIoU from 0.645 with plain cross-entropy to 0.706, and average mIoU to 0.722. On site, the robot proposes where to spray next and drives itself there, and the model adapts to different operators’ preferences, generalises to voids it has not seen and gets better as more installs are recorded. The model was running in shadow mode at the time of publication.

system

scroll sideways on small screens

Mk7 software architectureSensors feed the Spray Region Estimator, wheel odometry and visual-odometry localisation. The estimator's region goes to the Spray Region Manager, which drives the Spray Lines Generator and the Optimal Spray Position Estimator. Planning drives the robot through a priority drive multiplexer. Teleoperation from the operator's gamepad can feed every stage, and a dead man switch overrides spraying and driving. Everything reaches the Mk7 platform through a serial bridge.HIDgamepadTeleoperationSensorscameras · T265Spray Region EstimatorU-Net · this workSpray Region ManagerSpray Lines GeneratorSerial BridgePlatformMk7 robotWheel odometryLocalisationvisual odometryOptimal sprayposition estimatorPlanningglobal + local plannerPriority drivemultiplexerGUIoperatorhardwaresoftwaresafety override
diagramWhere the Spray Region Estimator sits in the Mk7 software: next to teleoperation, odometry and planning, with a dead man switch able to override spraying and driving.Source: Wagh et al., MMAR 2022

footage

muted, looping

Manual spray region, auto-navigation: the operator draws the region and the robot drives itself to the spraying position.
AI assisting the installer: the model's predicted spray region (yellow, left) is proposed in the operator view (right), the installer accepts or adjusts it, and the robot drives there.
Limitations: when predictions scatter, the proposed region can span several joists and needs the installer to step in.

figures

click to enlarge

fig. 01(a) Onboard camera view of the void. (b) U-Net prediction of the next spray area in yellow. (c) Post-processed spray polygon for the operator. (d) Robot auto-driven to the spray location.Wagh et al., MMAR 2022
fig. 02Q-Bot's Mk7 spray robot: front camera, RealSense T265, actuated camera-laser system and pivoting spray gun. Fits through gaps as low as 175 mm.Wagh et al., MMAR 2022
fig. 03Self-supervised labels: the region an operator actually sprayed becomes the mask. Valid spray region, no-spray region (robot body, walls) and unknown.Wagh et al., MMAR 2022
fig. 04Four on-site images, the self-supervised label taken from where the operator sprayed, and the U-Net's predicted spray region (yellow) with three loss functions. Weighted symmetric cross-entropy matches the operator best and stays quiet when there is nothing to spray (bottom row).Wagh et al., MMAR 2022
fig. 05Inference: camera image to U-Net, predicted mask, spray polygon in 3D, then the path planner.Wagh et al., MMAR 2022
fig. 06Limitations: scattered predictions can produce a spray polygon that spans several joists.Wagh et al., MMAR 2022
esc