Sai Goud Durgappagari Dr. Yingcai Xiao https://www.leapmotion.com/

Slides:



Advertisements
Similar presentations
UNDERSTANDING JAVA APIS FOR MOBILE DEVICES v0.01.
Advertisements

Game Development with Kinect
Input and Interaction Dr. Yingcai Xiao. A good user interface allows users to perform interaction tasks with ease and joy. WYSIWYG (What you see is what.
Cambodia-India Entrepreneurship Development Centre - : :.... :-:-
 Introduction  Devices  Technology – Hardware & Software  Architecture  Applications.
Tools for Game Developmet Erik Harpstead Carnegie Mellon University 1.
Yingcai Xiao EDP Scripting Yingcai Xiao. Why do we need EDP? What is EDP? How to implement EDP? How can we take advantages of EDP in game design and implement?
Designing 3D Interfaces Examples of 3D interfaces Pros and cons of 3D interfaces Overview of 3D software and hardware Four key design issues: system performance,
Welcome to CIS 083 ! Events CIS 068.
INTRODUCTION Generally, after stroke, patient usually has cerebral cortex functional barrier, for example, the impairment in the following capabilities,
Yingcai Xiao Game Development Interactive Animation.
Virtual Reality Lecture2. Some VR Systems & Applications 고려대학교 그래픽스 연구실.
Augmented reality Prepared by: Khyati Kataria 08DCE001
Introduction to Processing. 2 What is processing? A simple programming environment that was created to make it easier to develop visually oriented applications.
Augmented Reality Authorized By: Miss.Trupti Pardeshi. NDMVP, Comp Dept. Augmented Reality 1/ 23.
Abstract A Structured Approach for Modular Design: A Plug and Play Middleware for Sensory Modules, Actuation Platforms, Task Descriptions and Implementations.
Realtime Robotic Radiation Oncology Brian Murphy 4 th Electronic & Computer Engineering.
HW#9 Clues CSCI 571 Fall, HW#9 Prototype
Interactive Computer Graphics
Haris Ali (15) Abdul Ghafoor (01) Kashif Zafar (27)
Made By: Pallavi Chhikara
Yingcai Xiao Event-driven Programming in Game Development Yingcai Xiao.
HRTF 8 AudioGraph Spatial audio source Can be input source or submix Support #emitters depends on multiple factors Orientation, Position,
TOUCHLESS TOUCHSCREEN USER INTERFACE
Matthew Locke November 2007 A Linux Power Management Architecture.
WHAT IS COMPUTER ? . A computer is a complex system consisting of both hardware and software components.
Game Development with Unity3D
Interactive Animation
Vijay Kumar Kolagani Dr. Yingcai Xiao
Events and Event Handling
EEC-693/793 Applied Computer Vision with Depth Cameras
Introducing virtual REALITY
5/5/2018 1:12 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS.
Musical Instrument Virtual
Event Loops and GUI Intro2CS – weeks
Introduction to Virtual Environments & Virtual Reality
CompSci 230 S Software Construction
EEC-693/793 Applied Computer Vision with Depth Cameras
Introduction to Event-Driven Programming
Human Computer Interaction
Mobile Navigation Control for Planetary Web Portals Team Members: John Calilung, Miguel Martinez, Frank Navarrete, Kevin Parton, Max Ru, Catherine Suh.
Emerging Trends in Information Technology
Augmented Reality And Virtual Reality.
Event Driven Programming Dick Steflik
CSC461 Lecture 8: Input Devices
Virtual Reality By: brady adger.
Vijay Kumar Kolagani Dr. Yingcai Xiao
TOUCHLESS TOUCHSCREEN USER INTERFACE
Managed DirectX Joe Gavin.
Programming HCI Yingcai Xiao Yingcai Xiao.
EEC-693/793 Applied Computer Vision with Depth Cameras
Graphical User Interfaces -- Introduction
Sai Goud Durgappagari, VIJAY KOLAGANI, Dr. Yingcai Xiao
EE 422C Java FX.
Vijay Kumar Kolagani Dr. Yingcai Xiao
Introduction to Computing Using Java
Event-driven Programming
A seminar on Touchless Technology
Oculus Rift DK2 + Leap Motion Unity Tutorial
A Prime Example of HCI Application
Event Driven Programming
Virtual Reality.
An Introduction to Software Architecture
Introduction to Computer Software
Chapter I Introduction
Professional Environment
Myo + Oculus Rift Tutorial
LEAP MOTION: GESTURAL BASED 3D INTERACTIONS
EEC-693/793 Applied Computer Vision with Depth Cameras
CMPE 135: Object-Oriented Analysis and Design March 14 Class Meeting
Presentation transcript:

Sai Goud Durgappagari Dr. Yingcai Xiao https://www.leapmotion.com/ Leap Motion Sai Goud Durgappagari Dr. Yingcai Xiao https://www.leapmotion.com/

Introduction NUI Input device Only tracks hands and fingers http://store-us.leapmotion.com/ Input device Only tracks hands and fingers https://www.leapmotion.com Can be integrated with VR systems https://www.leapmotion.com/product/controller-vr#104 https://www.leapmotion.com/product/vr#113 Over 200K programmers https://developer.leapmotion.com/

Introduction The Leap Motion Controller tracks hands and fingers and reports position, velocity, and orientation with low latency and good accuracy. The controller can be mounted on a VR headset or be used on a tabletop. The Leap Motion controller system consists of a hardware device and a software component which runs as a service or daemon on the host computer. The software component analyses images produced by the hardware and sends tracking information to applications. The Leap Motion Unity plugin connects to this service to get data.

Hand Tracking Leap Motion controller uses optical sensors and infrared light. Sensors field of view about 150 degrees. Approximately extends 0.03 to 0.6 meters above device. Detection and tracking work best when the controller has a clear, high-contrast, silhouette view of the hands and fingers. The Leap Motion software combines its sensor data with an internal model of the human hand to help cope with challenging tracking conditions.

Leap Motion Programming NUI OO-EDP

Keys to OO-EDP Events: NUI events are usually composite, and described by vender- defined classes (UDTs). Registration: registration APIs are usually similar to that of callback functions. Sometimes are hardcoded in the handlers’ names. Event Handlers: usually single argument functions with an event object as the input argument. Sometimes, with another argument to identify the sender (i.e., event generator).

Steps to OO-EDP Initialization: to setup the UI and to prepare the event loop. Sometimes the platform takes care of all of these, you just need to create the objects for the event/handlers. Registration: to register the event handers to interested events. No need if the names of the event handlers are hardcoded to the events. Event loop: to start the event loop, usually through a platform-provided API. Some platforms automatically starts the event loop after your code is done in the main function. Event handers: To process the event To re-register the event-handler To invoke post-processing functions, directly or indirectly. The latter is done by creating another event which is handled by the post-processing functions. Cleanup: release recourses.

Tracking Model

Tracking Model Frame Hand Arm Pointable, Finger Bone Image The Frame object is the root and provides access to all other tracked entities. Hand The Hand object describes the position and orientation of a hand, tracks its motion between frames and contains lists of the fingers associated with that hand Arm Arm objects describe the position, direction, and orientation of the arm to which a hand is attached. Arms can only be accessed through the Hand object. Pointable, Finger Pointable objects define the basic characteristics common to fingers. The Finger class extends Pointable with additional information specific to those entities. Bone Bone object represent the position and orientation of a bone. Image Image objects provide the raw sensor data and calibration grid for the Leap Motion cameras. Utility Classes Vector and Matrix

Tracking API for Application without Unity3D System Architecture for Native Application Development

Tracking API without Unity 3D System Architecture for WebSocket Interface

Steps to OO-EDP for Initialization: to setup the UI and to prepare the event loop. Sometimes the platform takes care of all of these, you just need to create the objects for the event/handlers. Registration: to register the event handers to interested events. No need if the names of the event handlers are hardcoded to the events. Event loop: to start the event loop, usually through a platform-provided API. Some platforms automatically starts the event loop after your code is done in the main function. Event handers: To process the event To re-register the event-handler To invoke post-processing functions, directly or indirectly. The latter is done by creating another event which is handled by the post-processing functions. Cleanup: release recourses.

Tracking API without Unity 3D To connect to the Leap Motion device, create a Controller object Controller controller = new Controller(); Get Frame objects containing tracking data by calling the Controller.frame() function. Foreground and Background Applications Leap Motion service/daemon only sends tracking data to the application that has the operating system input focus. Listening for Controller Events The Controller object dispatches a number of events using the Listener mechanism. To handle these events, you can extend the Listener class to implement callback functions.

Tracking API without Unity 3D import com.leapmotion.leap.*; public class LeapEventListener extends Listener { public void onFrame (Controller controller){ System.out.println("New Frame"); } public void onInit (Controller controller){ System.out.println("Initialized"); public void onConnect (Controller controller){ System.out.println("Connected");

Tracking API without Unity 3D public void onDisconnect (Controller controller){ System.out.println("Disconnected"); } public void onFocusGained (Controller controller){ System.out.println("Focus gained"); public void onFocusLost (Controller controller){ System.out.println("Focus lost");

Tracking API without Unity 3D Controller controller = new Controller(); LeapEventListener listener = new LeapEventListener(); controller.addListener(listener);

Tracking API without Unity 3D

Leap Motion Programming NUI OO-EDP with Unity3D

Setup in Unity3D Download the latest asset package from: https://developer.leapmotion.com/downloads/unity Open or create a project. Select the Unity Assets > Import Package > Custom Package menu command. Locate the downloaded asset package and click Open. The assets are imported into your project. Example scenes can be found in the LeapMotion/Scenes folder. Reference : https://www.leapmotion.com/

Tracking API for Unity3D The classes in the Leap.Unity namespace interact directly with GameObjects and other UnityEngine components. These scripts take tracking data and put it to use in Unity. LeapServiceProvider provides access to the tracking data and images as well as access to the Leap.|Controller|_ object. The LeapServiceProvider transforms the tracking data so that the coordinates are relative to the transform of the Unity GameObject to which it is attached. LeapHandController provides access to the active hand graphics and physics objects.

Coordinate Systems Unity3D uses left-hand convention for coordinate system Leap Motion API uses right-hand convention Unity3D default unit: meters. Leap Motion default unit: millimeters.

Unity Coordinate System (left-hand) Reference : https://www.leapmotion.com/

Demo http://www.cs.uakron.edu/~xiao/hci/EDP-Unity5-LeapMotion.zip EDP-Unity5-LeapMotion/Assets/Player.cs EDP-Unity5- LeapMotion/Assets/LeapMotion/DemoResources/HandCycler.cs

Leap Motion Recent Advancement with VR Hand Tracking in Virtual Reality Virtual Reality is a sensory input and stereo output system that transports the user into a virtual world. Current technology uses head-mounted displays as the primary means of sensory input. These displays use position and orientation tracking of various levels of sophistication to immerse the user in the virtual world and provide a sense of physical presence. Hand tracking can amplify that sense of presence.

Unity Coordinate System for VR Reference : https://www.leapmotion.com/

References https://www.leapmotion.com/ https://developer.leapmotion.com/ https://developer.leapmotion.com/documentation/java/index.html https://developer.leapmotion.com/documentation/unity/unity/Unity _Overview.html