1. Virtual Dart: An Augmented Reality Game on Mobile Device Supervisor: Professor Michael R. Lyu Prepared by: Lai Chung Sum Siu Ho Tung

2. Outline Background Information Background Information Motivation Motivation Objective Objective Methods Methods Results Results Future Work Future Work Q & A Q & A

3. What is Augmented Reality (AR)? A combination of real world and computer generated data A combination of real world and computer generated data Add computer graphic into video Add computer graphic into video

4. Background Information Most mobile phones equipped with cameras Most mobile phones equipped with cameras Games written in J2ME & proprietary development platform Games written in J2ME & proprietary development platform

5. Background Information Typical mobile games Typical mobile games

6. Background Information Mobile games employed Augmented Reality Mobile games employed Augmented Reality

7. Motivation How can the game “remember” external environment? How can the game “remember” external environment?  Save external environment information

8. Objectives Demonstrate how a game “remember” its external environment for Augmented Reality (AR) Demonstrate how a game “remember” its external environment for Augmented Reality (AR) Virtual Dart is just a game for demonstration of the proposed methodology Virtual Dart is just a game for demonstration of the proposed methodology

9. Problems to be solved… 1. What information should we store? 2. How does the game recognize the information? 3. How does the game perform motion tracking?

10. Introduction to Mobile Video Object Tracking Engine (mVOTE) Convert the camera movement into translational movement and degree of rotation Convert the camera movement into translational movement and degree of rotation Feature Selection (Find a feature to trace) Motion Tracking of Translational Movement Motion Tracking of Rotational Movement

11. What is a feature? Section of an image that is easily highlighted for the purpose of detection and tracking Section of an image that is easily highlighted for the purpose of detection and tracking Have a high contrast in relation to its immediate surroundings Have a high contrast in relation to its immediate surroundings X

12.  What does our program need? Functions needed for our program Can mVOTE do this? Feature Selection (What information should we store?) Feature Recognition (How does the game recognize the information?)  Motion Tracking of Translational Movement (How does the game perform motion tracking?)

13. Program Flow – Initial Algorithm

15. Experiment of Feature Selection Feature Selection in mVOTE VS FAST Corner Detection Algorithm Feature Selection in mVOTE VS FAST Corner Detection Algorithm Testing Environment Testing Environment 1. Normal lighting 2. Insufficient lighting

16. Normal Lighting Condition Feature Selection in mVOTE FAST Corner Detector

17. Normal Lighting Condition Feature Selection in mVOTE FAST Corner Detector

18. Normal Lighting Condition Feature Selection in mVOTE FAST Corner Detector

19. Insufficient Lighting Condition Feature Selection in mVOTE FAST Corner Detector

20. Insufficient Lighting Condition Feature Selection in mVOTE FAST Corner Detector

21. Analysis Normal Lighting Normal Lighting  Both algorithms worked reasonably well Insufficient Lighting Insufficient Lighting  Only mVOTE’s Feature Selection could produce output Occasionally, Feature Selection in mVOTE selected some flat regions as features Occasionally, Feature Selection in mVOTE selected some flat regions as features FAST Corner worked better in terms of accuracy FAST Corner worked better in terms of accuracy

22. Experiment of Initial Approach Selected Features Recognized Features

23. Experiment of Initial Approach Selected Features Recognized Features

24. Experiment of Initial Approach Selected Features Recognized Features

25. Initial Feature Recognition Conclusion Accuracy?? LOW! Accuracy?? LOW! Selected Features Recognized Features

26. Analysis Matching accuracy is very bad Matching accuracy is very bad Store 3 25x25 pixels features blocks Store 3 25x25 pixels features blocks Feature Recognition on the 3 Blocks Feature Recognition on the 3 Blocks More than 1 point have same SSD More than 1 point have same SSD

27. Program Flow – Enhanced Feature Recognition Algorithm

28. Enhanced Feature Recognition Set 1 Set 2

29. Enhanced Feature Recognition Set 3 Set 4

30. Enhanced Feature Recognition Set 5 Set 6

31. Analysis Totally 10 set of sample photos Totally 10 set of sample photos 3 trials at each set 3 trials at each set Each run would produce a slightly different result Each run would produce a slightly different result May come from a small vibration during image capturing or maybe due to a small change in light intensity May come from a small vibration during image capturing or maybe due to a small change in light intensity

32. Algorithms Comparison Initial Feature Recognition VS Enhanced Feature Recognition Initial Feature Recognition VS Enhanced Feature Recognition Initial Approach: 3 Features Initial Approach: 3 Features New Approach: Whole selection area New Approach: Whole selection area Reason for LOW accuracy: (Initial Approach) Reason for LOW accuracy: (Initial Approach)  Features may not be descriptive enough

33. Improvement of Feature Selection Two conditions of a “Good” Feature: Two conditions of a “Good” Feature: Descriptive Descriptive Large internal intensity difference Large internal intensity difference  Corner Detector can help us to find out good features  Corner Detector can help us to find out good features

34. FAST Corner Detector Examine a small patch of image Examine a small patch of image Considering the Bresenham Circle of radius r around the candidate pixel which is called p Considering the Bresenham Circle of radius r around the candidate pixel which is called p Intensities of n continuous pixels on the circle are larger than p or smaller than p by barrier Intensities of n continuous pixels on the circle are larger than p or smaller than p by barrier  Potential corner

35. 215255200 39167 20132 1665152 91153 101165 114182167 e.g. r = 3, n = 12, barrier = 25 215 – 65 = 150 > 25 = barrier  Marked by red 65 – 39 = 26 > 25 = barrier  Marked by Blue

36. 2156321 8457 10812 659052 3115 3543 575862 e.g. r = 3, n = 12, barrier = 25

37. 2156321 8457 10812 659090 3115 3543 575862

38. FAST Corner Detector The typical values of r and n are 3 and 12 respectively The typical values of r and n are 3 and 12 respectively For the value of barrier, we did an experiment to choose the value For the value of barrier, we did an experiment to choose the value We chose “25” after the experiment (for what?) We chose “25” after the experiment (for what?)

39. FAST Corner Detector Advantage: Advantage: Fast Fast Disadvantages: Disadvantages: Cannot work well in noisy environment Cannot work well in noisy environment Accuracy depends on parameter – barrier Accuracy depends on parameter – barrier

40. FAST Corner Detector barrier = 10 barrier = 40

41. How does Feature Recognition works? Full screen as search window Full screen as search window Use Sum Square Difference (SSD) to calculate the similarity of blocks Use Sum Square Difference (SSD) to calculate the similarity of blocks Still slow in current stage (~20 – 60sec) Still slow in current stage (~20 – 60sec) Tried to use a smaller image and scale up to full screen Tried to use a smaller image and scale up to full screen Scaling step is too time consuming Scaling step is too time consuming

42. Motion Tracking during the game Keep track of three features Keep track of three features Use two features to locate dart board Use two features to locate dart board The last feature point is used for backup The last feature point is used for backup Use if either one of the feature points fail Use if either one of the feature points fail Condition for a feature point failure Condition for a feature point failure Feature point is at the edge of the screen Feature point is at the edge of the screen Two feature points are too close Two feature points are too close

43. Future Works Allow users to load saved features Allow users to load saved features Increase the speed of feature recognition Increase the speed of feature recognition Add physical calculation engine Add physical calculation engine

44. Q & A