Virtual Dart: An Augmented Reality Game on Mobile Device Supervisor: Professor Michael R. Lyu Prepared by: Lai Chung Sum Siu Ho Tung

Outline Background Information Background Information Motivation Motivation Objective Objective Methods Methods Results Results Future Work Future Work Q & A Q & A

What is Augmented Reality (AR)? A combination of real world and computer generated data A combination of real world and computer generated data Add computer graphic into video Add computer graphic into video

Background Information Most mobile phones equipped with cameras Most mobile phones equipped with cameras Games written in J2ME & proprietary development platform Games written in J2ME & proprietary development platform

Background Information Typical mobile games Typical mobile games

Background Information Mobile games employed Augmented Reality Mobile games employed Augmented Reality

Motivation How can the game “remember” external environment? How can the game “remember” external environment?  Save external environment information

Objectives Demonstrate how a game “remember” its external environment for Augmented Reality (AR) Demonstrate how a game “remember” its external environment for Augmented Reality (AR) Virtual Dart is just a game for demonstration of the proposed methodology Virtual Dart is just a game for demonstration of the proposed methodology

Problems to be solved… 1. What information should we store? 2. How does the game recognize the information? 3. How does the game perform motion tracking?

Introduction to Mobile Video Object Tracking Engine (mVOTE) Convert the camera movement into translational movement and degree of rotation Convert the camera movement into translational movement and degree of rotation Feature Selection (Find a feature to trace) Motion Tracking of Translational Movement Motion Tracking of Rotational Movement

What is a feature? Section of an image that is easily highlighted for the purpose of detection and tracking Section of an image that is easily highlighted for the purpose of detection and tracking Have a high contrast in relation to its immediate surroundings Have a high contrast in relation to its immediate surroundings X

 What does our program need? Functions needed for our program Can mVOTE do this? Feature Selection (What information should we store?) Feature Recognition (How does the game recognize the information?)  Motion Tracking of Translational Movement (How does the game perform motion tracking?)

Program Flow – Initial Algorithm

Experiment of Feature Selection Feature Selection in mVOTE VS FAST Corner Detection Algorithm Feature Selection in mVOTE VS FAST Corner Detection Algorithm Testing Environment Testing Environment 1. Normal lighting 2. Insufficient lighting

Normal Lighting Condition Feature Selection in mVOTE FAST Corner Detector

Normal Lighting Condition Feature Selection in mVOTE FAST Corner Detector

Normal Lighting Condition Feature Selection in mVOTE FAST Corner Detector

Insufficient Lighting Condition Feature Selection in mVOTE FAST Corner Detector

Insufficient Lighting Condition Feature Selection in mVOTE FAST Corner Detector

Analysis Normal Lighting Normal Lighting  Both algorithms worked reasonably well Insufficient Lighting Insufficient Lighting  Only mVOTE’s Feature Selection could produce output Occasionally, Feature Selection in mVOTE selected some flat regions as features Occasionally, Feature Selection in mVOTE selected some flat regions as features FAST Corner worked better in terms of accuracy FAST Corner worked better in terms of accuracy

Experiment of Initial Approach Selected Features Recognized Features

Experiment of Initial Approach Selected Features Recognized Features

Experiment of Initial Approach Selected Features Recognized Features

Initial Feature Recognition Conclusion Accuracy?? LOW! Accuracy?? LOW! Selected Features Recognized Features

Analysis Matching accuracy is very bad Matching accuracy is very bad Store 3 25x25 pixels features blocks Store 3 25x25 pixels features blocks Feature Recognition on the 3 Blocks Feature Recognition on the 3 Blocks More than 1 point have same SSD More than 1 point have same SSD

Program Flow – Enhanced Feature Recognition Algorithm

Enhanced Feature Recognition Set 1 Set 2

Enhanced Feature Recognition Set 3 Set 4

Enhanced Feature Recognition Set 5 Set 6

Analysis Totally 10 set of sample photos Totally 10 set of sample photos 3 trials at each set 3 trials at each set Each run would produce a slightly different result Each run would produce a slightly different result May come from a small vibration during image capturing or maybe due to a small change in light intensity May come from a small vibration during image capturing or maybe due to a small change in light intensity

Algorithms Comparison Initial Feature Recognition VS Enhanced Feature Recognition Initial Feature Recognition VS Enhanced Feature Recognition Initial Approach: 3 Features Initial Approach: 3 Features New Approach: Whole selection area New Approach: Whole selection area Reason for LOW accuracy: (Initial Approach) Reason for LOW accuracy: (Initial Approach)  Features may not be descriptive enough

Improvement of Feature Selection Two conditions of a “Good” Feature: Two conditions of a “Good” Feature: Descriptive Descriptive Large internal intensity difference Large internal intensity difference  Corner Detector can help us to find out good features  Corner Detector can help us to find out good features

FAST Corner Detector Examine a small patch of image Examine a small patch of image Considering the Bresenham Circle of radius r around the candidate pixel which is called p Considering the Bresenham Circle of radius r around the candidate pixel which is called p Intensities of n continuous pixels on the circle are larger than p or smaller than p by barrier Intensities of n continuous pixels on the circle are larger than p or smaller than p by barrier  Potential corner

e.g. r = 3, n = 12, barrier = – 65 = 150 > 25 = barrier  Marked by red 65 – 39 = 26 > 25 = barrier  Marked by Blue

e.g. r = 3, n = 12, barrier = 25

FAST Corner Detector The typical values of r and n are 3 and 12 respectively The typical values of r and n are 3 and 12 respectively For the value of barrier, we did an experiment to choose the value For the value of barrier, we did an experiment to choose the value We chose “25” after the experiment (for what?) We chose “25” after the experiment (for what?)

FAST Corner Detector Advantage: Advantage: Fast Fast Disadvantages: Disadvantages: Cannot work well in noisy environment Cannot work well in noisy environment Accuracy depends on parameter – barrier Accuracy depends on parameter – barrier

FAST Corner Detector barrier = 10 barrier = 40

How does Feature Recognition works? Full screen as search window Full screen as search window Use Sum Square Difference (SSD) to calculate the similarity of blocks Use Sum Square Difference (SSD) to calculate the similarity of blocks Still slow in current stage (~20 – 60sec) Still slow in current stage (~20 – 60sec) Tried to use a smaller image and scale up to full screen Tried to use a smaller image and scale up to full screen Scaling step is too time consuming Scaling step is too time consuming

Motion Tracking during the game Keep track of three features Keep track of three features Use two features to locate dart board Use two features to locate dart board The last feature point is used for backup The last feature point is used for backup Use if either one of the feature points fail Use if either one of the feature points fail Condition for a feature point failure Condition for a feature point failure Feature point is at the edge of the screen Feature point is at the edge of the screen Two feature points are too close Two feature points are too close

Future Works Allow users to load saved features Allow users to load saved features Increase the speed of feature recognition Increase the speed of feature recognition Add physical calculation engine Add physical calculation engine

Q & A