[cvPONG] A 3-D Pong Game Controlled Using Computer Vision Techniques Quan Yu and Chris Wagner

[Project Idea] <> 3-D adaptation of the classic Pong game <> Score by bouncing the ball past the opponent’s paddles <> Avoid letting the ball go past your paddles <> Calculate the user’s paddle positions based on the positions of the user’s hands and head <> Total of 3 Paddles <> Implementation: <> Unity3D + EmguCV (a OpenCV wrapper)

[Methods Used] <> Background Subtraction - Chris <> Differentiate user from background <> Skin Detection – Chris, Quan <> Find skin pixels in image <> Erosion & Dilation - Chris <> Clean image <> Finding Components - Quan <> Find head and hands

[Background Subtraction] <> Use background subtraction to obtain mask region for skin detection 1. Background Subtraction 1 <> Compare object pixels’ intensities in capture frame with those of previously captured background image <> How do we choose a good T?

[Background Subtraction] <> Use background subtraction to obtain mask region for skin detection 2. Mixture of Gaussians <> Each pixel modeled by a mixture of K Gaussian distributions <> Different Gaussians represent different colors <> Mixture weights determined by time proportions that colors stay in scene <> Learns probable background colors are the ones which stay longer and are more static <> Implemented in OpenCV: BackgroundSubtractorMOG()

[Background Subtraction] <> Use background subtraction to obtain mask region for skin detection 2. Mixture of Gaussians An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection P. KaewTraKulPong and R. Bowden

[Skin Detection] <> Apply skin detection in masked area to locate user’s hands and head <> Elliptical Boundary Model <> Convert RGB color space to normalized 2-D chrominance space by eliminating intensity (handles varying illumination conditions) <> Most non-skin chrominances concentrate on a single point (gray point) <> Want skin chrominance classification distribution to avoid overlapping this point (false positives) <> Skin chrominance distribution fits nicely into a skewed normal distribution towards the gray point

[Skin Detection] <> Apply skin detection in masked area to locate user’s hands and head <> Elliptical Boundary Model Chrominance DistributionSingle Gaussian ModelElliptical Boundary An Elliptical Boundary Model for Skin Color Detection Jae Y. Lee and Suk I. Yoo

[Finding Components] <> Skin detection gives binary image in which we can find three largest components (2 hands, head) 1. Erode then dilate image <> Removes small components and noise 2. Calculate contour boundaries of remaining components <> Use OpenCV method, findContours 3. Calculate bounding box around three largest contours 4. Take center of bounding boxes as position of user’s paddles

[Demo]

[Problems] <> Trade-off between accuracy and performance <> Shadows/Illumination <> Glasses! <> Similarly colored objects in foreground <> Speed – 8 fps in debug mode <> Erosion/Dilation removes too many skin pixels <> Low-fidelity between frames <> EmguCV is not completely compatible with Unity3D <> …

[Outcome and Lessons Learned] <> A controlled environment is better than a good algorithm in skin detection <> Vision-based input is a more fun experience than the keyboard/mouse <> Speed/interactivity is important in video games <> Simple background subtraction does not work well <> Simple skin detection works pretty good <> Noise is hard to remove

[Future Work] <> Incorporate MoG background subtraction to prune skin detection areas <> Utilize shape analysis/face detection to separate hands/head <> Smoother paddle movement between frames <> GPU programming to improve performance <> Add more game elements

[Thank you!] Questions, comments?