1. Final Year Project 2004/2005 LYU0402 Augmented Reality Table for Interactive Card Games Supervisor: Prof Michael Lyu Presented by: Kevin Chow, Albert Lam

2. Overview Introduction Introduction Objective Objective Trading Card Game Trading Card Game Architecture of ART Architecture of ART Implementation in First Term Implementation in First Term Experiment Experiment Conclusion Conclusion Future Work Future Work

3. Introduction What is Augmented Reality? What is Augmented Reality? Mixed Reality combines the content from the real world with virtual imaginaryMixed Reality combines the content from the real world with virtual imaginary Augmented Reality is a subset of this where virtual content is overlaid into real objects of the worldAugmented Reality is a subset of this where virtual content is overlaid into real objects of the world

4. Introduction (Con’t) Paul Milgram characterized Mixed Reality interfaces on his “Reality- Virtuality Continuum” in 1994Paul Milgram characterized Mixed Reality interfaces on his “Reality- Virtuality Continuum” in 1994 Mixed Reality (MR) Virtual Environment Augmented Virtuality (AV) Augmented Reality (AR) Real Environment Reality-Virtuality (RV) Continuum

5. Introduction (Con’t) An AR system has the following three main characteristics: An AR system has the following three main characteristics: Combines real and virtual objects in a real environmentCombines real and virtual objects in a real environment Runs interactively, and in real timeRuns interactively, and in real time Registers virtual objects onto the real worldRegisters virtual objects onto the real world

6. Introduction (Con’t) ART stands for Augmented Reality Table ART stands for Augmented Reality Table User can play Card Games on the ART.User can play Card Games on the ART. ART can enhance the visual effect during the playing of the card game.ART can enhance the visual effect during the playing of the card game.

7. Objective Develop a generic Interactive ART Trading Card Game application. Develop a generic Interactive ART Trading Card Game application. Enhance the traditional card games: Enhance the traditional card games: Visual effectVisual effect Sound effectSound effect Complex calculationComplex calculation

8. Trading Card Game Trading Card Games Trading Card Games Two players play a match.Two players play a match. Players play cards to fight with each other. It includes summoning monster, casting magic, or setting traps, etc.Players play cards to fight with each other. It includes summoning monster, casting magic, or setting traps, etc. We choose “YU-GI-OH” as our implementationWe choose “YU-GI-OH” as our implementation

9. Trading Card Game (Con’t) Trading card game includes five kinds of actions Trading card game includes five kinds of actions RestoreRestore Draw cardsDraw cards Play cardsPlay cards Attack/ChallengeAttack/Challenge Discard cardsDiscard cards

10. An Example - YU-GI-OH In YU-GI-OH, Different information are stored on the card. In YU-GI-OH, Different information are stored on the card.

11. An Example - YU-GI-OH (Con’t) Game Mat of YU-GI-OH Game Mat of YU-GI-OH Different kinds of card must be put inside particular card zones Different kinds of card must be put inside particular card zones

12. An Example - YU-GI-OH (Con’t) LP : 2000 ATK : 500 ATK : 1200 1200 – 500 = 700 -700 LP : 1300

13. An Example - YU-GI-OH (Con’t) LP : 1300LP : 2000 DEF : 1300 ATK : 1200 1300 – 1200 = 100 -100 LP : 1900

14. Architecture Hardware Setup Hardware Setup Plasma monitorPlasma monitor Overhead cameraOverhead camera System architecture System architecture Augmented Reality PerceptionAugmented Reality Perception Game CoreGame Core DatabaseDatabase Game EnhancementGame Enhancement

15. Hardware Setup

16. Hardware Setup (Con’t) Computer Computer processes perceiving imageprocesses perceiving image generates visual and audio enhancementgenerates visual and audio enhancement controls game flowcontrols game flow Plasma Monitor. Plasma Monitor. acts as the Game Tableacts as the Game Table displays computer-generated scenedisplays computer-generated scene Overhead mounted camera Overhead mounted camera captures the cards and the screen of the plasma.captures the cards and the screen of the plasma. only input of the systemonly input of the system

17. System Architecture Game Core Module Game Core Module Perception Module Perception Module Database Module Database Module Game Enhancement Module Game Enhancement Module

18. System Architecture (Con’t)

19. Microsoft DirectX SDK Microsoft DirectX SDK A set of low-level APIs for high- performance multimedia applicationsA set of low-level APIs for high- performance multimedia applications directly access the hardwaredirectly access the hardware provides device independent through HALprovides device independent through HAL We will use DirectShow, Direct3D and DirectSound in our implementationWe will use DirectShow, Direct3D and DirectSound in our implementation

20. Implementation Perception Module Perception Module CalibrationCalibration Search window locatorSearch window locator Card locatorCard locator Card RecognizerCard Recognizer Database Module Database Module Identify card typeIdentify card type Image retrievalImage retrieval Image matchingImage matching

21. Perception Module Read and process the raw video perceived from the camera Read and process the raw video perceived from the camera Detect input card and command Detect input card and command Some assumptions Some assumptions fixed camera, fixed tablefixed camera, fixed table camera is approximately right above the tablecamera is approximately right above the table cards can only be placed in predefined region called card zonescards can only be placed in predefined region called card zones

22. Calibration Why? Why? Captured image varies with environmental lighting conditionCaptured image varies with environmental lighting condition Take advantage of the assumptions madeTake advantage of the assumptions made

23. Calibration (Con’t) How? How? Set the card zone positionSet the card zone position Set the card area threshold for each card zoneSet the card area threshold for each card zone Calibrate the colorsCalibrate the colors Calibrate card type colorsCalibrate card type colors

24. Search Windows Locator Find region to search for cards Find region to search for cards The screen is partitioned into some fixed small search windows The screen is partitioned into some fixed small search windows Search windows which just stop changing will be processed Search windows which just stop changing will be processed

25. Search Windows Locator (Con ’ t) Located search windows that have changes Located search windows that have changes Image differentImage different Pixel changed ifPixel changed if A Search window is said to be changed ifA Search window is said to be changed if Activate search window from “change” to “unchange” state Activate search window from “change” to “unchange” state

26. Card Locator Locate four corner card positions given a search window Locate four corner card positions given a search window Apply Canny AlgorithmApply Canny Algorithm Extract ContoursExtract Contours Search for contours whichSearch for contours which contain 4 points contain 4 points contain nearly right angle corners contain nearly right angle corners have area within a certain threshold have area within a certain threshold

27. Card Locator (Con ’ t) Canny Algorithm Canny Algorithm Remove noise by Gaussian FilterRemove noise by Gaussian Filter Compute edge strength and edge direction by Sobel OperatorCompute edge strength and edge direction by Sobel Operator Apply Nonmaximum Suppression to trace edgesApply Nonmaximum Suppression to trace edges Use Hystersis to eliminate streakingUse Hystersis to eliminate streaking

28. Card Recognizer Card orientation Card orientation Compare the lengths of the four edgesCompare the lengths of the four edges Compare color difference between top and bottom parts of the cardCompare color difference between top and bottom parts of the card

29. Card Recognizer (Con’t) Get Card Image Get Card Image Extract the undistorted card image from a distorted imageExtract the undistorted card image from a distorted image

30. Card Recognizer (Con’t) Pixel coordinate transformationPixel coordinate transformation Map a point (x,y) on the distorted image to a point (x’,y’) on the undistorted image Map a point (x,y) on the distorted image to a point (x’,y’) on the undistorted image (x 0,y 0 ) (x 1,y 1 ) (x 2,y 2 ) (x 3,y 3 ). (x,y) h w. (x’,y’)

31. Card Recognizer (Con’t) Brightness interpolationBrightness interpolation Assign brightness value to the point (x,y) Assign brightness value to the point (x,y) Apply Bilinear Interpolation Apply Bilinear Interpolation

32. Card Recognizer (Con’t) Identify the card uniquely Identify the card uniquely Query the Image Database with the card image for unique card IdQuery the Image Database with the card image for unique card Id Query the Card Database with the card Id for card information and detailsQuery the Card Database with the card Id for card information and details

33. Database Module Contains all game information Contains all game information Identify card image by 3 steps: Identify card image by 3 steps: Identify card typeIdentify card type Classify card by card type Classify card by card type Image retrievalImage retrieval Retrieve several similar candidate cards Retrieve several similar candidate cards Image matchingImage matching Find the best match card Find the best match card

34. Identify Card Type Compare the background color of the card to the calibrated colorCompare the background color of the card to the calibrated color

35. Image Retrieval Retrieve similar images of same card type. Retrieve similar images of same card type. Use color-based retrieval method due to low resolution Use color-based retrieval method due to low resolution Color Histogram method select cards with largest color histogram intection Color Histogram method select cards with largest color histogram intection

36. Image Retrieval (Con’t) Color histogram intersection Color histogram intersection

37. Select the best matched card from the candidates Select the best matched card from the candidates Split the captured image into 4 channelsSplit the captured image into 4 channels Compare only inner imageCompare only inner image Compare the different channels to the database separatelyCompare the different channels to the database separately Image Matching

38. Find the pixel difference byFind the pixel difference by Reject the image if the pixel difference is larger than the thresholdReject the image if the pixel difference is larger than the threshold Accept the image with minimum pixel differenceAccept the image with minimum pixel difference

39. Experiment Match 15 cards to a set of different cards with threshold set to 1500 Match 15 cards to a set of different cards with threshold set to 1500 Here, we have chosen results of three cards for discussion Here, we have chosen results of three cards for discussion

40. Experimental Result

41. Experimental Result (Con’t)

43. Conclusion We have implemented a simplified version of the ART system We have implemented a simplified version of the ART system Concerning low resolution of the image, we have develop an algorithm to recognize card with high accuracy but less efficiency Concerning low resolution of the image, we have develop an algorithm to recognize card with high accuracy but less efficiency

44. Future Work Color Calibration Color Calibration Develop an efficient card recognition algorithm while retaining high accuracy Develop an efficient card recognition algorithm while retaining high accuracy Game Enhancement Game Enhancement Interactive input detector Interactive input detector

45. End of Presentation Thank you very much

46. Q & A Session