EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 1 Wenbing Zhao wenbing@ieee.org 11/18/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development Requirements Prerequisite: Moderate programming background Basic knowledge of data structure and algorithms Basic knowledge of operating systems 11/18/2018 EEC492/693/793 - iPhone Application Development

Textbooks and References Main textbook Kinect for Windows SDK Programming Guide, by Abhijit Jana, PACKT publishing, 2012. References: Kinect Hacks, by Jared St. Jean, O’Reilly, 2013 Hacking the Kinect, by Jeff Kramer et al., Apress, 2012.Reference materials C# in a Nutshell, By Peter Drayton, Ben Albahari, Ted Neward, O'Reilly, 1st Edition March 2002 11/18/2018 EEC492/693/793 - iPhone Application Development

Required Hardware & Software Windows PC or laptop Kinect sensor Software Visual Studio 2015 or later Kinect SDK v1.8 Unity 4 (preferred) OpenCV Other related libraries 11/18/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development Kinect’s Capability Capturing real-time video using the color sensor Tracking a human body and then responding to its movements and gestures as a natural user interface Generating depth map of the objects tracked Recognizing a human voice and developing hands-free applications that can be controlled by voice 11/18/2018 EEC492/693/793 - iPhone Application Development

Real-World Applications of Kinect Healthcare: exercise measurement, monitoring patients, their body movements, and so on Robotics: as a navigation system for robots either by tracking human gestures, voice commands, or by human body movements Education: help them to learn subjects either by their gesture and voice commands Security system: track human body movement or face and send the notifications Virtual Reality Trainer: measuring the movements of human body joints, providing live feedback to users if the joints are moving in an appropriate manner by comparing the movements with previously stored data Military: build intelligent drones to spy on enemy lines 11/18/2018 EEC492/693/793 - iPhone Application Development

Healthcare with Kinect Reflexion Health Advancing physical therapy http://blogs.msdn.com/b/kinectforwindows/archive/2013/05/09/reflexion-health-advancing-physical-therapy-with-kinect-for-windows.aspx 11/18/2018 EEC492/693/793 - iPhone Application Development

Healthcare with Kinect ReMotion360: uses Kinect to measure and provide on screen feedback for range of motion exercises either in a clinic or at home http://www.infostrat.com/solutions/Remotion360 https://www.youtube.com/watch?feature=player_embedded&v=-jhpYslVygs 11/18/2018 EEC492/693/793 - iPhone Application Development

Improving Safety at Workplace Siemens uses Kinect for Windows to improve nuclear plant employee safety virtual 3-D human avatars  integrated to model motion-controlled actions input with a Kinect http://blogs.msdn.com/b/kinectforwindows/archive/2013/04/19/siemens-uses-kinect-for-windows-to-improve-nuclear-plant-employee-safety.aspx 11/18/2018 EEC492/693/793 - iPhone Application Development

Enabling Retailers to Drive Business Benefits: Capture a customer's attention Educate customers about products Move a customer through the decision-making cycle to close a sale Ski Shop scenario: https://www.youtube.com/watch?feature=player_embedded&v=ORWYr1gx5eU Toy store scenario: https://www.youtube.com/watch?feature=player_embedded&v=wZfcDOsBs2E Retail clothing scenario https://www.youtube.com/watch?feature=player_embedded&v=Mr71jrkzWq8 11/18/2018 EEC492/693/793 - iPhone Application Development

Styku: Apparel Shopping Help find clothes that fit Virtually try on clothes before they buy To be offered by major retailers, such as Brooks Brothers, IM-Label, and others 11/18/2018 EEC492/693/793 - iPhone Application Development

Nissan Pathfinder Virtual Showroom Shoppers who approach the Kinect-based showroom are guided through an array of natural movements to activate hotspots on the Pathfinder model, allowing them to inspect the car inside and out https://www.youtube.com/watch?feature=player_embedded&v=rQVycUmeqB4 11/18/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development Education Tool nsquared sky spelling makes it fun for children to learn to spell in an interactive and collaborative environment capture letters that complete the spelling of various words  11/18/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development Hospitality Tool Nsquared Hotel Kiosk augments the concierge function in a hotel by providing guidance to hotel guests through an intuitive, interactive experience  use a “grip” gesture to drag content across the screen and a “push” gesture to select content 11/18/2018 EEC492/693/793 - iPhone Application Development

Ubi Interactive: Turn any Surface into a Touch Screen Instruct classes in an interactive lecture hall Access product information on a store’s window front Recipes can be projected onto kitchen countertops without having to worry about getting anything dirty Use wall to play interactive games like Angry Birds Video: https://www.youtube.com/watch?feature=player_embedded&v=Iu2XH5p_hMM 11/18/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development 3D Scanning Kinect fusion: perform 3D scanning with a single Kinect sensor! Nice match with 3D printers 11/18/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development New Development on Depth Cameras: PC Makers Bet on Gaze, Gesture, Voice, and Touch A combined 2-D and 3-D camera from Intel will be built into laptops and tablets from a range of manufacturers, Intel announced at the 2014 International Consumer Electronics Show (CES) 11/18/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development New Development on Depth Cameras: PC Makers Bet on Gaze, Gesture, Voice, and Touch Toshiba also demonstrated a dual-camera module for smartphones, tablets, and laptops at 2014 CES Gesture recognition, refocus images, etc. 11/18/2018 EEC492/693/793 - iPhone Application Development

Integrating Kinect with Wearable Computing 11/18/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development Main Topics Understand the Kinect sensor and how they work Learn how to capture data streams from the Kinect sensor Learn the uses of color, depth and IR data and how to leverage them in your applications Use skeleton-tracking for interactive applications Learn basic gesture recognition methods and develop gesture-enabled applications Learn 3D game development with Unity3D for Kinect (virtual reality!) Applied computer vision with OpenCV and Kinect 11/18/2018 EEC492/693/793 - iPhone Application Development

Assignments and Grading In-class exercises Term project Reading several research papers on the application of Kinect Develop a working prototype program using Kinect Project presentations and demonstration Grading policy Class participation: 60% (any missed session will lead to credit reduction) Project: 40% No exams 11/18/2018 EEC492/693/793 - iPhone Application Development