Yuping Lin and Gérard Medioni

 Introduction  Method  Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to Map registration ▪ Interleaving image to image and image to map ▪ Partial local mosaic  Synchronization of multiple video streams  Conclusion

 Input:  Multiple UAV video streams  Position of moving objects in each video stream  Goal: Synchronize using a common moving object

 Register UAV streams to a global reference image (a map), then  Synchronize the streams using the unique path of a common moving object on the map

 Introduction  Method  Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to Map registration ▪ Interleaving image to image and image to map ▪ Partial local mosaic  Synchronization of multiple video streams  Conclusion

 Input:  Global reference image (Map)  UAV stream  The homography of the first frame of the UAV stream to the map

ISSUES  UAV images and the map are different in terms of viewpoints, sensors and time of capture  Direct matching is difficult APPROACH  Given the homography of the first UAV frame to the map,  Two step registration  Consecutive UAV image registration, then  UAV to Map registration

 Introduction  Method  Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to Map registration ▪ Interleaving image to image and image to map ▪ Partial local mosaic  Synchronization of multiple video streams  Conclusion

 Method:  extract features in each frame  Establish feature correspondences between consecutive images  estimate the transformation

ISSUES  Features should be descriptive for matching and sufficient to give good transform estimation  Feature matching  Transform estimation APPROACH  SIFT feature extraction  128 dimension feature descriptor  Avg features in each image  Nearest neighbor matching  Avg matches in each pair of images  RANSAC  Avg. 600 inliers in each pair of images

 Illustration

 Problem: error is accumulated

 Introduction  Method  Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to Map registration ▪ Interleaving image to image and image to map ▪ Partial local mosaic  Synchronization of multiple video streams  Conclusion

 Method:  Perform local search for correspondences between the UAV image and the map

ISSUES  UAV images are very different from the map, SIFT features cannot always match APPROACHES  Sample points in the map  For each point, locally search for the most similar image patch in the UAV image  Use Mutual Information as similarity measurement

 Illustration

 Introduction  Method  Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to Map registration ▪ Interleaving image to image and image to map ▪ Partial local mosaic  Synchronization of multiple video streams  Conclusion

 Method:  Perform consecutive UAV image registration and UAV to Map registration iteratively ▪ Consecutive UAV image registration produce good initials for UAV to Map registration ▪ Register the partial local mosaic to the map

ISSUES  Correspondences in a single frame are not enough Registration is unstable APPROACH  Multiple frames in a time window forms a partial local mosaic which spans a larger region and provides more correspondences  More robust  Smooth transition

ISSUES  Correspondences in a single frame are not enough Registration is unstable APPROACH  Multiple frames in a time window forms a partial local mosaic which spans a larger region and provides more correspondences  More robust  Smooth transition

 Result Register single frameRegister partial local mosaic

 Illustration

 Result

 Introduction  Method  Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to Map registration ▪ Interleaving image to image and image to map ▪ Partial local mosaic  Synchronization of multiple video streams  Conclusion

 Input: UAV image sequences of different views, different frame rates, but capture the same area and overlap in time  An moving object on the ground plane which serves as a “clock” to synchronize the sequences

 The moving object should generate a single path on the map  Use sequence alignment algorithm to synchronize the UAV streams

 Two steps to register an UAV image to the map  Register each frame to its previous frame to derive an initial estimate  Register UAV image to the map to derive  Limitations  Initial estimate should be given  Unable to recover from a bad estimate