1. Mobile Motion Tracking using Onboard Camera Supervisor: Prof. LYU, Rung Tsong Michael Prepared by: Lam Man Kit Wong Yuk Man

2. Agenda Motivation & Objective Previous Work Improvement & New components Sample Applications Conclusion Q&A

3. Motivation Rapid increase in the use of camera- phone Camera-phone can perform image processing tasks on the device itself Enhance human-computer interaction by mobile phone

4. Objective Implement real-time motion tracking on Symbian phone, without requiring additional hardware Translational motion tracking Rotational motion tracking To develop a tracking engine for other mobile devices developers to use

5. Previous Work A translational motion tracking engine Blocking matching SEA (Successive Elimination Algorithm) PPNM PDE (Partial Distortion Elimination) Adaptive Window Search Spiral Scan Feature selection A translational motion tracking engine

6. Block Matching Algorithm

7. Evaluate the "goodness" of a match by Sum of Absolute Difference Select the candidate block with the lowest error

8. Feature Selection A good feature block: High variance -> complex block Not in a repeated pattern If a good feature block is found, the performance of the motion tracking is improved We have made improvements to our existing algorithm

9. squares Old Feature Selection Divide the current frame into squares Apply Feature Selection Algorithm to each squares However, the best feature block may appear between two squares

10. New Feature Selection More blocks are sampled Sample 26x26 variances of blocks with size 15x15 in a 54x54 window Search in spiral way and stop searching when selection criteria are matched Prefer to find a feature block in the center Prevent out of bound problem (block appear out of screen)

11. New Feature Selection If a feature block is found on the edge, the following will happen: The tracking algorithm will fail to find the exact match Solution: Apply additional constraint

12. New Feature selection In order to find a feature block that is not on the edge, we apply a checking algorithm Important difference in all directions => interest point

13. New Feature Selection Now our newest feature selection algorithm becomes: Find a block with a large variance -> which indicate the complexity of the block Check if the block is on the edge or not by calculating the SAD between the candidate block and its 4 neighbors If either one of the SAD is small -> the block is on edge -> reject Else the block is not on edge

14. SSD is used instead of SAD SSD = Sum of Squared Difference Last term, SAD is used as matching criteria SAD is commonly used because of its lower complexity (faster) But we chose to use SSD finally

15. SSD is used instead of SAD Experiment shows that SSD has better performance in accuracy and Algorithm with SSD run as fast as that with SAD Reason (of why as fast as SAD): Elimination effects from SEA, PPNM & PDE become large when SSD is used Compensate the higher complexity of SSD

16. SSD is used instead of SAD SEA and PPNM lower bound is adjusted using the inequality (12) proposed in the following paper J.J. Francis and G. de Jager. A Sum Square Error based Successive Elimination Algorithm for Block Motion Estimation (2002)

17. Experimental Result on Symbian phone Frame rate Maximum frame rate supported by Nokia 6600 is about 14 frames/sec Running our algorithm (1/0.007 frames/sec) once only for each frame do not slow down the displaying frame rate, in other word, the frame rate can still be 14 frames/sec, our algorithm doesn ’ t lag the display

18. Rotation Tracking Engine As we rotate the phone, the object can still be tracked correctly (center of object roughly equals center of green box) Observation and Motivation

19. Rotation Tracking Engine Our approach If two blocks are tracked at the same time, angle of line connecting the two blocks reflect the tiling angle of the phone Two-block approach

20. Rotation Tracking Engine Another approach Track one block by simple motion tracking engine, then find which rotation gives the best match Fail if tracking object is the same for different angle One-block approach

21. Rotation Tracking Engine Modification of motion tracking algorithm to suit rotation tracking Linear Adaptive method used in translation motion tracking is not used here because phone ’ s motion can be both linear motion and circular motion

22. Rotation Tracking Engine What to predict? Given the coordinates of the tracking blocks in the last 2 previous frames Predict the coordinates of the blocks in the next frame θ θ A simplified mathematic problem (the following assumptions are approximately correct) All three lines pass through the circle ’ s center End points of the three lines lie on the circle Line L 2 Line L 1 Line L 3

23. Rotation Tracking Engine Solution Angle between L 2 and L 3 = θ θ = Tan -1 (slope(L 2 )) – Tan -1 (slope(L 3 )) Coordinates of the next tracking block (x L1, y L1 ) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix

24. Rotation Tracking Engine Solution The prediction of the position of the next tracking block should also take the translational movement into account Horizontal displacement = Tx Tx = ( x L2 1 + x L2 2 – x L3 1 - x L3 2 )/2 Vertical displacement = Ty Ty = ( y L2 1 + y L2 2 – y L3 1 - y L3 2 )/2 Coordinates of the next tracking block (x L1, y L1 ) is calculated by

25. Rotation Tracking Engine A simple drawing to show the detected rotation angle of the phone

26. Rotation Tracking Engine Reducing error by using level Apply threshold on output, increase/decrease one level only when change is large To give less sensitive but more desirable output for game e.g. skiing game: skier face only to 7 directions Reduce difficulty, increase reliability A small rotation or a small detection error will not increase/decrease one level

27. Conditions to be a good background for both engines Objective Max. performance measurement Increase usability Condition: Feature selection can always find a good feature point Within certain distance No repeat pattern Very distinct pattern Pattern is nearly the same when rotated. E.g. Circle which is good for rotation detection

28. Virtual Mouse An application that make full use of the translational motion detection engine Remote control the mouse of PC by Symbian phone using motion tracking Advantages: It allows input in all directions It provides high levels of control Joystick can still be used as rough moving while camera input can be used as fine translation

29. Virtual Mouse Server In server side (Windows), the Bluetooth is configured to provide RFComm Service and server regards the Bluetooth transmission port as Comm. Port Server receives message from that Comm. Port Call function in MouseAction.h to make the mouse move and trigger mouse click event

30. Virtual Mouse Client Search for Bluetooth device nearby Connect to the selected Bluetooth device Every time motion tracking algorithm finishes, results are sent to server (12 times/sec) Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)

31. Car Racing Game A “ Car Racing Game ” is developed using the motion tracking engine. Game lags. It is because: The engine takes time to find the motion vector CPU speed of the Symbian phones are low The game engine thread uses most of the CPU power while the thread for updating the screen cannot proceed Solutions: Double Buffering Direct Screen Access Single Thread

32. Double Buffered Area Double-buffering – two complete color buffers One is displayed while the other is calculating next scene to be drawn, When the drawing of the next frame is completed, the content of the back buffer is copied to the front screen Computing Frame 2 Show Frame 2 Buffer 1Buffer 2 copy Frame 1 Computing …… Frame 2 Front Screen Frame 2 ready

33. Direct Screen Access In Symbian, traditional drawing requires the help of Window Server Overhead in context switching Lower speed Access the screen directly in 3 ways: Creating and using CfbsScreenDevice Accessing screen memory directly. Using CdirectScreenAccess. Using DSA can bypass the Window Server Get rid of context switching Faster

34. Traditional Graphic Programming Update Display Device Screen and Keypad Window Server RWsSession Application 1 Key Presses

35. Direct Screen Access 2. If successful, a region for drawing to is returned to the application. Window Server 1. Request to Window Server to work in Direct Screen Access mode Application

36. Single Thread Prevent context switching No need to do synchronization

37. Car Racing Game With the use of efficient graphic programming algorithms, the tracking engine will not affect the performance of the game It shows that we can use the engine to develop other applications without performance degradation

38. Skiing Game Sample program from the book “ Developing Series 60 Applications: A Guide for Symbian Os C++ Developers ” Based on this game, we plug in our rotation tracking engine into it The game becomes more interactive Rotate the phone to control the angle of the skier

39. Skiing Game

40. The process of using the engine is very simple Create an instance of the tracking engine Call the function of the engine to find the motion vector Use the motion vector for the game logic

41. Skiing Game

42. Demo

43. Experimental Result on Symbian phone Testing Environment Platform Symbian Phone Nokia 6600 Algorithm Our final (hybrid-type) Algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method) Algorithm parameter Block size = 17 x 17 (pixels) Search window size = 16 x 16 (pixels) Number of block to track in each run of algorithm = 1 block ONLY

44. Experimental Result on Symbian phone Testing Result (avg time to run) Final algorithm 7ms Final algorithm without adaptive window method and SEA, PPNM method 22ms Full Exhaustive Search algorithm (the most simplest one) 55ms

45. Q & A