1. Portable Vision-Based HCI A Real-Time Hand Mouse System on Portable Devices 連矩鋒 (Burt C.F. Lien) Department of Computer Science and Information Engineering National Taiwan University

2. Problems A Portable Vision-Based HCI –Hand mouse operating on a projected interface –Real-time detection of user hand motion from a user PDA/SmartPhone’s video camera (target platform) Need an efficient method to run the idea on portable devices

3. Why important Vision-based HCI is a more instinct way to manipulate data

4. Related Works I A Portable System for Anywhere Interactions –Sukaviriya et al., IBM Research Real-time hand tracking using a set of cooperative classifiers based on Haar-like features –Barczak1 et al., Institute of Information & Mathematical Sciences Massey University

5. Everywhere Display (IBM) Figure 1: Interactive store application

6. Related Works II Rapid Object Detection Using a Boosted Cascade of Simple Features. –Viola, P., & Jones, M. (2001). Robust real-time object detection. –Viola, P., & Jones, M. Robust real-time face detection –P. Viola and M. Jones. Adaboost-based real-time pedestrian detection –P. Viola, M. Jones, and D. Snow. James W. Davis. "Hierarchical Motion History Images for Recognizing Human Motion," event, p. 39, IEEE Workshop on Detection and Recognition of Events in Video (EVENT'01), 2001 Tim Weingaertner, Stefan Hassfeld, Ruediger Dillmann. "Human Motion Analysis: A Review," nam, p. 0090, 1997 IEEE Workshop on Motion of Non-Rigid and Articulated Objects (NAM '97), 1997

7. Reference codes Intel OpenCV Libraries –Motion Template –Motion History Image

8. Contribution An efficient method to run a real-time vision- based HCI system on portable device –Experiment result: Typically 5~7% CPU Usage ( Intel Pentium M processor 730 (1.6G) ) with 640x320 resolution (3FPS) The motion method used in this system does not need a training process. This significantly reduced lots of training efforts and can be more robust (lighting proof) on object detection even with a blurred image.

9. Target Devices

10. System Configuration Hand motion capture and interpretation Wireless projector data transmission Interactive Interface

11. Platform and Tools Platform (prototype) –“Laptop” + “Low Cost Camera (USB) – NT300” Software tools –“MS VC++” + “Intel OpenCV library”

12. Assumption A rectangle screen shape Background is static most of the time 1 user only

13. Adaboost (old version) To recognize a “hand” –Adaboost training ( 1397 hand images + 3000 background images ) –Takes 2 days for training a 11-stage classifier ( Viola & Jones  order of weeks ) –Result: Classifier too weak to recognize and falsealarm rate is high

14. Haartraining Result Original test image Darkening the background Stress the outline of a hand manually

15. Motion Template Give up adaboost learning classifier Motion Template –Motion History Image : image ring buffer ( N=3) –To reduce the computation (take off complex mathematical computation and replace with some simple heuristics ) –To acquire and record the front edge of a motion –To define orientation (for different instruction) –To detect a “touch” behavior (density drop rate)

16. Motion History Image where each pixel (x,y) in the MHI is marked with a current timestamp if the function signals object presence (or motion) in the current video image I(x,y) ; the remaining timestamps in the MHI are removed if they are older than the decay value. This update function is called for every new video frame analyzed in the sequence.

17. Silhouette

18. Motion trajectory Note: Record the last 50 front edges

19. System Flow Chart Image Diff Capture from CAM Find the screen (edge detection) MHI Update Find frond point Motion interpretation Noise filter Mouse/keyboard events start

20. Find the Screen During initialization, to find the projected screen –Algorithm Canny edge detection –Find the screen Find all the squares in the image and choose the biggest one –Adaptive Adjust the screen every 10 second in case the camera is moved

21. Position (pixel) Mapping Screen mapping (camera and computer) –Define the scale for coordinate translation –eg. 800x600 (camera resolution)  1280x800 (computer resolution). –scale-x = 1280/800 –scale-y = 800/600 detected screen Computer 1280 800 600 800 Origin New Origin Camera Resolution

22. Event definition To define mouse or keyboard events –mouse click if image density dropped dramatically ( > 70%~80%), the position of last frond edge is defined coordinate of a mouse click –Page Up (PgUp) if above action happens from the left side of the screen, we define this as a “PgUp” event. –Close current windows application Consecutive 3 error detection within 8 seconds

23. Noise filtering False positives –motion trajectories are recorded to filter out false positive signals (partly implemented) Signal bouncing –A 10 second interval of bouncing is introduced after a valid mouse/keyboard event is detected

24. Performance CPU: Pentium M Processor 730 (1.6GHz) HaarDetectObjects (Typical) –5 fps (640x480) : 80% CPU Usage –3 fps (640x480) : 30% CPU Usage –3 fps (640x480, hand+face classifier) : 50% Motion Template (Typical) –3 fps (176x144) : 2~5% CPU Usage –3 fps (640*480) : 5~8% CPU Usage –3 fps (800x600) : 10% CPU Usage

25. System Limitation High error rate when moving fast –Can be solved by increasing the FPS Unexpected stop in the middle of the screen will cause falsealarm Shadow would impact the correctness If the screen is not well detected, or if the mapping is distorted, accuracy of position will be very low.

26. Future Work To improve the accuracy To port the system to a handheld device To advance to a real steerable interface (something like “Minority Report”) that a user can drag the icons directly on the projected screen.

27. Q&A