1. Research Area B Leif Kobbelt

2. Communication System Interface Research Area B 2

3. 3 Interface System Communication A A C C D D

4. definition of „application“ within UMIC find new and relevant applications (killer app) combine existing technology identify application profiles types of data amount of data latency requirements input / output devices golden demo good : convey the message bad : no basis research, commercial competitors 4 Research Area B

5. Future Mobile Applications 5 fundamental algorithms system design evaluation commercialization

6. Future Mobile Applications 6 fundamental algorithms system design evaluation commercialization

7. Future Mobile Applications 7 fundamental algorithms system design evaluation commercialization communication it security computer graphics computer vision

8. Future Mobile Applications 8 fundamental algorithms system design evaluation commercialization software engineering security computer graphics & vision interface design

9. Future Mobile Applications 9 fundamental algorithms system design evaluation commercialization graphical UIs prototypes user studies...

10. Visual Computing for Future Mobile Applications Bastian Leibe

11. Target Scenario: Pedestrian Navigation 11 Aachen Cathedral  Mobile visual search  Simply point the camera to any object/building of interest.  Images are transmitted to a central server for recognition.

12. Target Scenario: Pedestrian Navigation 12 Aachen Cathedral  Mobile visual search  Simply point the camera to any object/building of interest.  Images are transmitted to a central server for recognition.  Object-specific content is sent back to for visualization on the mobile phone (mobile AR).

13. LocalizeMe Demo 13 P. Steingrube, T. Weyand, T. Sattler, A. Schmitz, B. Leibe, L. Kobbelt

14. Mobile Service Structure 14 Localization Service Image Database 3D Model Render Server Information Service Cultural Database Internet Compound Application Mobile Client Server User Interface

15. Localization: Large-Scale Image Matching  How can we perform this matching step efficiently? 15 Database with thousands (millions) of images ? ? Mobile photo

16. Localization: Large-Scale Image Matching 16 Database with thousands (millions) of images Mobile photo Local features (~1000 per image) … “Visual vocabulary” (~1M feature clusters) Shortlist of candidate matches (~100 images) Shortlist of candidate matches (~100 images) Matching (nearest neighbors in 128D space)

17. Localization: Geometric Verification 17 Mobile photo(for each image in shortlist) xAjxAj xBjxBj XjXj Assumption: corresponding 3D structure T. Sattler, B. Leibe, L. Kobbelt, SCRAMSAC: Improving RANSAC’s Efficiency with a Spatial Consistency Filter. International Conference on Computer Vision, 2009.  Find a rigid geometric transformation to verify that the matched features correspond to the same 3D structure.  Problem: efficient processing with many outliers.

18. Summary: Visualization Service 18 Image Local Feature Extractor Local Feature Database Image Database Feature Matching Image Candidate Matches Geometric Verification Determine Location On mobile device On server side

19. Mobile Service Structure 19 Localization Service Image Database 3D Model Render Server Information Service Cultural Database Internet Compound Application Mobile Client Server User Interface

20. World-Scale Mining for Content Creation 20 e.g. Wikipedia match Mining geotagged images Extracted Image clusters Automatic annotation & verification Frequent tags

21. Example: Automatic Landmark Detection 21 Matched images for Aachen city hall (subset) T. Weyand, B. Leibe

22. How Does This Scale?  Feasibility study  Pairwise matching on 500,000 geotagged images of Paris  How many matching images can we find at a certain location?  Touristic sites and central roads are well-covered. 22 T. Weyand, B. Leibe

23. Mobile Service Structure 23 Localization Service Image Database 3D Model Render Server Information Service Cultural Database Internet Compound Application Mobile Client Server User Interface

24. Virtual City Model 24 Floor Plan Map (2D) Street Graph (2D) Height Field (3D) Synthetic Textures Photographic Textures Building Model Landscape Model Optimized Octree Data Structure Estate Plan (2D)

25. Results: Virtual Aachen Model 25 G. Fabritius, J. Kraßnigg, L. Krecklau, C. Manthei, A. Hornung, M. Habbecke, L. Kobbelt

26. Alternative 1: Mobile Rendering 26  Rendering quality on a regular PC C. Schreder, A. Schmitz, L. Kobbelt  Mobile rendering  Very limited memory!  Need to precompute octree structure  Dynamic transmission of geometry & textures

27. Alternative 2: Stream Rendering 27 Rendering on server Internet  Current prototype  Transmit images in UDP packages  Tradeoff: compression vs. framerate

28. Mobile Service Structure 28 Localization Service Image Database 3D Model Render Server Information Service Cultural Database Internet Compound Application Mobile Client Server User Interface

29. Application Study: Looking Through Time  How did Aachen’s cathedral look in past centuries?  Camera phone as a “magic lens” to reveal past building states  How should the interface be designed for such an application?  Mock-up prototype (using Motion Capture system for tracking)  Evaluation in user studies 29 T. Palm, J. Borchers

30. Research in Novel Interaction Techniques 30 T. Karrer, M. Weiss, M. Wittenhagen, G. Herkenrath, J. Borchers  TWEND  Twisting and bending as new interaction gestures in future mobile devices  E.g. for interaction with an electronic map or an e-book.  PocketDRAGON  Directly drag objects along their movement trajectory to precisely navigate to a specific point in a video sequence  Mobile implementation made possible through communication with server