Vision Reading Group, 2nd of May 2018 Martin Rünz CodeSLAM — Learning a Compact, Optimisable Representation for Dense Visual SLAM Vision Reading Group, 2nd of May 2018 Martin Rünz

An alternative representation to depth maps is presented In a nutshell An alternative representation to depth maps is presented

Background – Common Map Representations Surfels Mesh Pose-Graph + Depth-Maps TSDF

Ideas Depth-maps are not random

Ideas Depth-maps are not random (especially in man-made environments)

Ideas Depth-maps are not random A compact representation – code – can be learned by encoder Decoders are differentiable → code can be optimised (w.r.t. photometric loss...) This is useful in SfM or SLAM scenarios Reconstruct high-frequency details from color

Depth Reconstruction Depth-from-mono Depth-from-mono + code Given this differentiable function, warping constraints can be used to optimise c and pose

Architecture Laplace distribution Color feature extractor + uncertainty predictor Laplace distribution Only for training, ground-truth depth Decoder: CODE Variational autoencoder, to increase smoothness of mapping between code and depth Small changes in code → small changes in depth

Video

Inverse depth parametrization average Error behaves more Gaussian original depth

Warping Both functions differentiable to inputs Transform → B 3D Photometric error: Both functions differentiable to inputs Expensive (convolutions) Pre-computed if decoder is linear!

Application SfM Cost function: Initialise poses and code with zero-vector Use residuals + Jacobians in Gauss-Newton style optimisation Cost function: Functionality of : Mask invalid correspondences Relative weighting Huber weighting Down-weight slanted surfaces Down-weight occluded pixels

Experiments SfM

Experiments SfM

Experiments SLAM

Experiments Setups

Experiments Setups

Experiments Influence code entries

Video

Thanks for listening!