EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 1 Wenbing Zhao wenbing@ieee.org 4/16/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development Requirements Prerequisite: Moderate programming background Basic knowledge of data structure and algorithms Basic knowledge of operating systems 4/16/2018 EEC492/693/793 - iPhone Application Development

Textbooks and References Main textbook Kinect for Windows SDK Programming Guide, by Abhijit Jana, PACKT publishing, 2012. References: Kinect Hacks, by Jared St. Jean, O’Reilly, 2013 Hacking the Kinect, by Jeff Kramer et al., Apress, 2012.Reference materials C# in a Nutshell, By Peter Drayton, Ben Albahari, Ted Neward, O'Reilly, 1st Edition March 2002 4/16/2018 EEC492/693/793 - iPhone Application Development

Required Hardware & Software Windows PC or laptop Kinect sensor Software Visual Studio 2015 Kinect SDK v1.8 Unity 4 OpenCV Other related libraries 4/16/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development Kinect’s Capability Capturing real-time video using the color sensor Tracking a human body and then responding to its movements and gestures as a natural user interface Generating depth map of the objects tracked Recognizing a human voice and developing hands-free applications that can be controlled by voice 4/16/2018 EEC492/693/793 - iPhone Application Development

Real-World Applications of Kinect Healthcare: exercise measurement, monitoring patients, their body movements, and so on Robotics: as a navigation system for robots either by tracking human gestures, voice commands, or by human body movements Education: help them to learn subjects either by their gesture and voice commands Security system: track human body movement or face and send the notifications Virtual Reality Trainer: measuring the movements of human body joints, providing live feedback to users if the joints are moving in an appropriate manner by comparing the movements with previously stored data Military: build intelligent drones to spy on enemy lines 4/16/2018 EEC492/693/793 - iPhone Application Development

Healthcare with Kinect Reflexion Health Advancing physical therapy http://blogs.msdn.com/b/kinectforwindows/archive/2013/05/09/reflexion-health-advancing-physical-therapy-with-kinect-for-windows.aspx 4/16/2018 EEC492/693/793 - iPhone Application Development

Healthcare with Kinect ReMotion360: uses Kinect to measure and provide on screen feedback for range of motion exercises either in a clinic or at home http://www.infostrat.com/solutions/Remotion360 https://www.youtube.com/watch?feature=player_embedded&v=-jhpYslVygs 4/16/2018 EEC492/693/793 - iPhone Application Development

Improving Safety at Workplace Siemens uses Kinect for Windows to improve nuclear plant employee safety virtual 3-D human avatars  integrated to model motion-controlled actions input with a Kinect http://blogs.msdn.com/b/kinectforwindows/archive/2013/04/19/siemens-uses-kinect-for-windows-to-improve-nuclear-plant-employee-safety.aspx 4/16/2018 EEC492/693/793 - iPhone Application Development

Enabling Retailers to Drive Business Benefits: Capture a customer's attention Educate customers about products Move a customer through the decision-making cycle to close a sale Ski Shop scenario: https://www.youtube.com/watch?feature=player_embedded&v=ORWYr1gx5eU Toy store scenario: https://www.youtube.com/watch?feature=player_embedded&v=wZfcDOsBs2E Retail clothing scenario https://www.youtube.com/watch?feature=player_embedded&v=Mr71jrkzWq8 4/16/2018 EEC492/693/793 - iPhone Application Development

Styku: Apparel Shopping Help find clothes that fit Virtually try on clothes before they buy To be offered by major retailers, such as Brooks Brothers, IM-Label, and others 4/16/2018 EEC492/693/793 - iPhone Application Development

Nissan Pathfinder Virtual Showroom Shoppers who approach the Kinect-based showroom are guided through an array of natural movements to activate hotspots on the Pathfinder model, allowing them to inspect the car inside and out https://www.youtube.com/watch?feature=player_embedded&v=rQVycUmeqB4 4/16/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development Education Tool nsquared sky spelling makes it fun for children to learn to spell in an interactive and collaborative environment capture letters that complete the spelling of various words  4/16/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development Hospitality Tool Nsquared Hotel Kiosk augments the concierge function in a hotel by providing guidance to hotel guests through an intuitive, interactive experience  use a “grip” gesture to drag content across the screen and a “push” gesture to select content 4/16/2018 EEC492/693/793 - iPhone Application Development

Ubi Interactive: Turn any Surface into a Touch Screen Instruct classes in an interactive lecture hall Access product information on a store’s window front Recipes can be projected onto kitchen countertops without having to worry about getting anything dirty Use wall to play interactive games like Angry Birds Video: https://www.youtube.com/watch?feature=player_embedded&v=Iu2XH5p_hMM 4/16/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development 3D Scanning Kinect fusion: perform 3D scanning with a single Kinect sensor! Nice match with 3D printers 4/16/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development New Development on Depth Cameras: PC Makers Bet on Gaze, Gesture, Voice, and Touch A combined 2-D and 3-D camera from Intel will be built into laptops and tablets from a range of manufacturers, Intel announced at the 2014 International Consumer Electronics Show (CES) 4/16/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development New Development on Depth Cameras: PC Makers Bet on Gaze, Gesture, Voice, and Touch Toshiba also demonstrated a dual-camera module for smartphones, tablets, and laptops at 2014 CES Gesture recognition, refocus images, etc. 4/16/2018 EEC492/693/793 - iPhone Application Development

New Development on Depth Cameras SoftKinetic demonstrate a depth camera for 3D scanning Need to move the camera around the subject Apple will also come to the depth camera field Acquired Primesense, the company invented the technology behind Kinect Filed a patent in 3D camera for iOS devices 4/16/2018 EEC492/693/793 - iPhone Application Development

EEC492/693/793 - iPhone Application Development Main Topics Understand the Kinect sensor and how they work Learn how to capture data streams from the Kinect sensor Learn the uses of color, depth and IR data and how to leverage them in your applications Use skeleton-tracking for interactive applications Learn basic gesture recognition methods and develop gesture-enabled applications Learn 3D game development with Unity3D for Kinect (virtual reality!) Applied computer vision with OpenCV and Kinect 4/16/2018 EEC492/693/793 - iPhone Application Development

Assignments and Grading In-class exercises Term project Reading several research papers on the application of Kinect Develop a working prototype program using Kinect Project presentations and demonstration Grading policy Class participation: 60% (any missed session will lead to credit reduction) Project: 40% No exams 4/16/2018 EEC492/693/793 - iPhone Application Development