1. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Adam Hedji Mantas Pulinas Philip San III. Viktorious SSIP 2009, Debrecen, Hungary

2. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers

3. Project Natal X-Box 360 Microsoft Corporation Full Body Motion Detection New project that promises to use two cameras for full-body motion detection

4. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers CHECKERS!

5. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Overview of Code Board Plane Geometry Detection

6. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Overview of Code Relaying of Data with Checkers A.I. Feedback Identification of Square Status by Colour Classification Segmentation of Individual Game Squares Board Plane Geometry Detection This is blue!

7. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Geometry Detection Find the corners 2 approaches were used: Edge detection and region growing Hough transforms

8. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Geometry Detection – Edge Detection and Region Growing

9. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Geometry Detection – Edge Detection and Region Growing Edge detection with Sobel operator Dilate the image to fill the gaps in the border

10. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Geometry Detection – Edge Detection and Region Growing Edge detection with Sobel operator Dilate the image to fill the gaps in the border Thin the image to get the true border

11. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Geometry Detection – Edge Detection and Region Growing Edge detection with Sobel operator Dilate the image to fill the gaps in the border Thin the image to get the true border Dilate the image several times to remove useless edges

12. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Geometry Detection – Edge Detection and Region Growing Edge detection with Sobel operator Dilate the image to fill the gaps in the border Thin the image to get the true border Dilate the image several times to remove useless edges Region growing Determine the corner tiles Track corners in real-time

13. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Geometry Detection – Projective Correction Edge detection with Sobel operator Dilate the image to fill the gaps in the border Thin the image to get the true border Dilate the image several times to remove useless edges Region growing Determine the corner tiles Track corners in real-time

14. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Geometry Detection – Projective Correction Calculate homography using corner coordinates

15. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Geometry Detection – Hough Transform Hough transform –We had time so we developed a better solution –Based on Hough transformations –Better real-time line detection

16. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers

17. Colour Classification How can the computer classify the colours? (171,154,158) (32,61,105)

18. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Colour Classification Use image processing algorithms to make the RGB values only 0 or 255 (255,255,255) (0,0,255)

19. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Colour Classification (255,255,255) (0,0,0) (255,255,0) (0,0,255) Select individual tile Analyse the predominant colour inside to classify the square state (white, black, blue, yellow) Sample of pixels used as opposed to whole square

20. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Colour Classification How to determine if each R, G and B values are 0 or 255? -Need to choose threshold value e.g. A given pixel of value (15, 19, 250) A threshold of 126 Output is (0,0,255)

21. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Colour Classification - But how to choose the threshold? Webcams can be of relatively poor quality and provide poor contrast. –For example, blue pieces used were relatively hard to distinguish from the black tiles.

22. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers

23. But how to choose the threshold? Can instead normalise the image and use a threshold of 127, given by (256/2)-1. –Select a white tile and take an average of the colour values. –Do the same for a black tile. –Use these averages to normalise the image p 1 =(241,209,210) p 2 =(232,204,214) p 3 =(240,211,205)

24. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Colour Classification - Normalisation

25. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Colour Classification - Normalisation

26. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Colour Classification - Normalisation

27. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Colour Classification - Normalisation Then reduce the image to absolute values of 0 and 255 Use a threshold of 126 (half of full intensity value)

28. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Colour Classification - Normalisation

29. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Colour Classification - Normalisation

30. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers

32. Summary Main results –Successfully used 2 approaches to chess board detection Edge detection and region growing Hough transform –Removing perspective distortion –Identification of individual tiles and pieces, including classification –Connection to engine interface with feedback system

33. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Improvements Aim towards real-time 60fps processing Use a more efficient programming language such as C++ Use of GPU using CUDA or Open CL programming language More complex algorithms –Motion detection of hand Use of overlay of 3D structures onto camera image. Virtual humans...

34. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers Unused Virtual Human

35. Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers The Team Adam Hedji University of Zagreb, Croatia adam.hedi@fer.hr Mantas Paulinas Vilnius Gediminas Technical University, Lithuania mantas.paulinas@el.vgut.lt Philip San University College London, England p.san@ucl.ac.uk Viktor Blaskovics University of Szeged, Hungary blaskovics.viktor@stud.u-szeged.hu Viktor Blaskovics University of Szeged, Hungary blaskovics.viktor@stud.u-szeged.hu Philip San University College London, England p.san@ucl.ac.uk