1. MACHINE VISION GROUP Graphics hardware accelerated panorama builder for mobile phones Miguel Bordallo López*, Jari Hannuksela*, Olli Silvén* and Markku Vehviläinen**, * University of Oulu, Finland ** Nokia Research Center, Tampere, Finland

2. MACHINE VISION GROUP Contents Introduction Document Panorama Builder Interactive user interface Quality evaluation and frame selection Re-registration and blending Computation times Scene Panorama Builder Graphics processor as a computing engine GPU accelerated panorama builder

3. MACHINE VISION GROUP Introduction Document capture with single snapshot ViewFinder ImageReal size section

4. MACHINE VISION GROUP Introduction Panorama Builder Panorama builder Used to combine several images into one wide-angle mosaic image Traditionally image mosaic is formed from a few separate images Video sequences can also be used as a source

5. MACHINE VISION GROUP Introduction Video Mosaicking Videos contain lots of information – most of it redundant Video mosaicking vs. image mosaicking – image registration is easier due to smaller motions between frames – some frames can be discarded without losing information

6. MACHINE VISION GROUP Document Panorama Builder Interactive user interface VGA video analysis Motion estimation system calculates shift, rotation and scale in real time When frame is suitable for stitching (high quality), the user receives feedback and instructions

7. MACHINE VISION GROUP Document Panorama Builder Quality determination and frame selection

8. MACHINE VISION GROUP Document Panorama Builder Re-registration and blending Selected framesFinal panorama

9. MACHINE VISION GROUP Document Panorama Builder Computation times 1. Registration with SIFT feature based RANSAC to find the warping and stitching parameters –approx. 6 frames/minute in software 2. Image Correction: warping with interpolation –approx. 2 frames/s in software 3. Quality determination and frame selection during the capture stage –approx. 10 frames/s in software 4. Motion estimation during capture stage –approx. 30 frames/s in software

10. MACHINE VISION GROUP Scene Panorama Builder >360 degrees panoramas from landscape Real-time registration Real-time frame evaluation and selection Real-time frame correction On-the-fly stitching Less memory requirements

11. MACHINE VISION GROUP GPU as a computing engine Newer phones include a GPU chipset OpenGL ES as a highly optimized and attractive accelerator interface Compatible data formats for graphics and camera sub- systems desirable Emerging platforms (CUDA, OpenCL) facilitate using the GPU as a computing resource

12. MACHINE VISION GROUP Fixed pipeline (OpenGL ES 1.1) vs. programmable pipeline (OpenGL ES 2.0)

13. MACHINE VISION GROUP Stream processing (OpenGL) vs. shared memory processing (CUDA)

14. MACHINE VISION GROUP OpenCL (Embedded Profile) Emerging platforms will offer needed flexibility Code executed concurrently on CPU & GPU (even DSP) High parallelization on image processing computations. High efficiency

15. MACHINE VISION GROUP GPU-accelerated panorama builder Open GL ES 1.1: –Warping with interpolations: >4 times faster –Pixel blending: Open GL ES 2.0: –Image features (SIFT, Haar,...) extraction: 10 times faster on PC, CPU load decreased 50% –Frame quality evaluation Emerging platforms: –RANSAC estimator –Better efficiency –Enabling the use of other chipsets

16. MACHINE VISION GROUP Image warping with interpolation on OpenGL ES 1.1 Create quad Transform Render texture

17. MACHINE VISION GROUP Summary Document mosaicing allows feedback to the user Quality can be measured while capturing Mobile phones are an excellent platform for this method GPUs can be used as a general purpose procesors New platforms will offer more efficiency and flexibility Not optimized interfaces include excesive overheads Lack of RAM in small devices sitll a problem

18. MACHINE VISION GROUP