sw/
← projects
Heriot-Watt University·2018

Underwater motion & 3D reconstruction

MSc thesis at the Ocean Systems Laboratory: two unsupervised CNN frameworks that learn depth maps (camera) or elevation maps (sonar) and ego-motion from raw underwater video, optical and acoustic, with no ground truth. Supervised by Dr Sen Wang and Prof. Yvan Petillot.

Problem

Autonomous underwater vehicles inspect ship hulls, pipelines and reefs, but they still struggle to know where they are. Cameras see only a short distance through murky water, forward-looking sonar images are noisy with acoustic clutter, and the extra navigation sensors that fill the gap add cost, payload and power. Deep networks had started learning depth and ego-motion from ordinary video on land, but nobody had tried it underwater, and there was no ground-truth depth, elevation or pose to train on.

What I did

For my MSc thesis at Heriot-Watt’s Ocean Systems Laboratory, supervised by Dr Sen Wang and Prof. Yvan Petillot, I built two unsupervised learning frameworks that learn 3D structure and motion directly from raw image sequences, so no labels are needed.

Optical. Following the SfMLearner design, a single-view depth network (DispNet-style, multi-scale) and a pose network are trained together: the predicted depth and relative poses are used to warp neighbouring frames onto the target frame, and the mismatch is the training signal. For the underwater setting I added a structural similarity (SSIM) loss to the image reconstruction loss and made the smoothness loss edge-aware using the image gradients. I trained it in TensorFlow on a TITAN Xp, on 9,772 three-frame sequences from the Universitat de Girona underwater cave dataset, for 200,000 iterations.

Sonar. I then extended the idea to forward-looking sonar, which had not been done before. I worked out the sonar projection geometry, proved it by synthesising a new sonar view from an elevation map and a chosen pose, and built SonarNet: an elevation map network and a pose network trained on polar sonar frames with a reconstruction loss and a second-order smoothness loss. It was trained on 20,550 sequences from a sonar dataset recorded in a water tank at the Ocean Systems Laboratory, for 300,000 iterations; the proof-of-concept frames came from an ARIS 3000 sonar.

Result

Both models produce plausible depth or elevation maps and point clouds from a single frame, and SonarNet’s predicted trajectory matches the motion visible in the sonar frames, mostly rotation about the vertical axis with some sideways movement. With no ground truth available, the evaluation is qualitative, and the thesis is candid about the limits: small datasets, artificial lighting in the caves and sparse, noisy sonar images all held the models back, and the optical trajectory drifts. As a first attempt at learned depth, elevation and motion underwater, it laid out the data and model changes needed next.

system

scroll sideways on small screens

Unsupervised depth and ego-motion from view synthesisThe depth network predicts a per-pixel depth map from the target frame (an elevation map for sonar in SonarNet). The pose network predicts the relative poses to the neighbouring frames. Projecting the target through the depth and poses and inverse-warping the source views reconstructs the target view; the reconstruction, SSIM and smoothness losses train both networks without any ground truth.Target view I_tcentre frameSource viewsI_t-1 · I_t+1Depth / elevation netDispNet-style, multi-scalePose network6-DoF relative poseDepth map D_televation map E_t for sonarRelative posesT t→t-1 · T t→t+1Project + inverse warpQ_t = D_t K⁻¹ p_tReconstructedtarget viewLossphotometric · SSIM · smoothcompared with I_ttrains both networks, no ground truthsource views to warp
diagramThe training loop behind both models: predict a depth map (an elevation map for sonar) and pose, rebuild the target frame from its neighbours, and learn from how well it matches. No ground-truth depth, elevation or pose is needed.Source: the MSc thesis, Fig. 4.1 and 4.7

footage

muted, looping

SonarNet on water-tank data: a polar sonar frame, its predicted elevation map and the resulting point cloud.
Sonar frames in Cartesian form beside the trajectory SonarNet predicts for them, built up from overlapping three-frame pose groups.

figures

click to enlarge

fig. 01Single-view depth predicted from monocular frames of the Girona underwater cave dataset. The bottom row shows how artificial lighting misleads the model.MSc thesis, 2018, Fig. 5.1
fig. 02A dense 3D point cloud rebuilt from one input frame and its predicted depth, seen from three viewpoints. Depth scaled to a 10 m range.MSc thesis, 2018, Fig. 5.2
fig. 03SonarNet elevation map predictions for polar sonar frames recorded in a water tank.MSc thesis, 2018, Fig. 5.8
fig. 04Proof of concept: (a) polar sonar image of a tyre in a water tank, (b) its Cartesian form, (c) a synthetically generated elevation map.MSc thesis, 2018, Fig. 4.3
esc