1. Sai Goud Durgappagari, VIJAY KOLAGANI, Dr. Yingcai Xiao
Leap Motion
Sai Goud Durgappagari, VIJAY KOLAGANI, Dr. Yingcai Xiao

2. What is Leap?
The Leap Motion controller, also known as The Leap, was Launched in 2012 by Leap Motion, Inc.
iPod sized USB peripheral
It Can create custom gestures

3. Components Two Global Sutter Image Sensor Infrared LED’s
USB 3.0 Controller

4. How does it work Can track movements down to a 1/100th of a millimeter
The leap motion device acts as a camera, and continuously takes stereographic images of the area in front of the sensor. The images are scanned for hands, fingers and pointables.

5. Leap Motion Programming NUI OO-EDP

6. Programming Leap motion
Input device
NUI
Only tracks hands and fingers
Can be integrated with VR systems
Over 200K programmers

7. Programming Leapmotion
The Leap Motion Controller tracks hands and fingers and reports position, velocity, and orientation with low latency and good accuracy.
The controller can be mounted on a VR headset or be used on a tabletop.
The Leap Motion controller system consists of a hardware device and a software component which runs as a service or daemon on the host computer.
The software component analyses images produced by the hardware and sends tracking information to applications.
The Leap Motion Unity plugin connects to this service to get data.

8. Hand Tracking
Leap Motion controller uses optical sensors and infrared light.
Sensors field of view about 150 degrees.
Approximately extends 0.03 to 0.6 meters above device.
Detection and tracking work best when the controller has a clear, high- contrast, silhouette view of the hands and fingers.
The Leap Motion software combines its sensor data with an internal model of the human hand to help cope with challenging tracking conditions.

9. SDK V1.0
Download SDK here
how to set up C# Windows Forms and Presentation Foundation projects using Visual Studio Sample program up.html

10. Leap Motion Programming DATA STRUCTURE

11. Tracking Model

12. Tracking Model Frame Hand Arm Pointable, Finger Bone Image
The Frame object is the root and provides access to all other tracked entities.
Hand
The Hand object describes the position and orientation of a hand, tracks its motion between frames and contains lists of the fingers associated with that hand
Arm
Arm objects describe the position, direction, and orientation of the arm to which a hand is attached. Arms can only be accessed through the Hand object.
Pointable, Finger
Pointable objects define the basic characteristics common to fingers.
The Finger class extends Pointable with additional information specific to those entities.
Bone
Bone object represent the position and orientation of a bone.
Image
Image objects provide the raw sensor data and calibration grid for the Leap Motion cameras.
Utility Classes
Vector and Matrix

13. Leap Motion Programming Algorithms

14. Steps to OO-EDP
Initialization: to setup the UI and to prepare the event loop. Sometimes the platform takes care of all of these, you just need to create the objects for the event/handlers.
Registration: to register the event handers to interested events. No need if the names of the event handlers are hardcoded to the events.
Event loop: to start the event loop, usually through a platform-provided API. Some platforms automatically starts the event loop after your code is done in the main function.
Event handers:
To process the event
To re-register the event-handler
To invoke post-processing functions, directly or indirectly. The latter is done by creating another event which is handled by the post-processing functions.
Cleanup: release recourses.

15. Keys to OO-EDP
Events: NUI events are usually composite, and described by vender-defined classes (UDTs).
Registration: registration APIs are usually similar to that of callback functions. Sometimes are hardcoded in the handlers’ names.
Event Handlers: usually single argument functions with an event object as the input argument. Sometimes, with another argument to identify the sender (i.e., event generator).

16. Tracking API for Application without Unity3D
System Architecture for Native Application Development

17. Tracking API without Unity 3D
System Architecture for WebSocket Interface

18. Tracking API without Unity 3D
To connect to the Leap Motion device, create a Controller object
Controller controller = new Controller();
Get Frame objects containing tracking data by calling the Controller.frame() function.
Foreground and Background Applications
Leap Motion service/daemon only sends tracking data to the application that has the operating system input focus.
Listening for Controller Events
The Controller object dispatches a number of events using the Listener mechanism. To handle these events, you can extend the Listener class to implement callback functions.

19. Tracking API without Unity 3D
import com.leapmotion.leap.*; public class LeapEventListener extends Listener { public void onFrame (Controller controller){ System.out.println("New Frame"); } public void onInit (Controller controller){ System.out.println("Initialized"); public void onConnect (Controller controller){ System.out.println("Connected");

20. Tracking API without Unity 3D
public void onDisconnect (Controller controller){ System.out.println("Disconnected"); } public void onFocusGained (Controller controller){ System.out.println("Focus gained"); public void onFocusLost (Controller controller){ System.out.println("Focus lost");

21. Tracking API without Unity 3D
Controller controller = new Controller(); LeapEventListener listener = new LeapEventListener(); controller.addListener(listener);

22. SampleListener class to define event handler function
class SampleListener {
public void OnServiceConnect(object sender, ConnectionEventArgs args) {
Console.WriteLine("Service Connected"); }
public void OnConnect(object sender, DeviceEventArgs args)
Console.WriteLine("Connected");
public void OnFrame(object sender, FrameEventArgs args)
Console.WriteLine("Frame Available.");

23. properties of the Frame object:
public void OnFrame(object sender, FrameEventArgs args) {
// Get the most recent frame
Frame frame = args.frame;
Console.WriteLine( "Frame id: {0}, timestamp: {1}, hands: {2}", frame.Id, frame.Timestamp, frame.Hands.Count );
foreach (Hand hand in frame.Hands)
Console.WriteLine(" Hand id: {0}, palm position: {1}, fingers: {2}", hand.Id, hand.PalmPosition, hand.Fingers.Count);
// Get the hand's normal vector and direction
Vector normal = hand.PalmNormal;
Vector direction = hand.Direction;
// Calculate the hand's pitch, roll, and yaw angles
Console.WriteLine( " Hand pitch: {0} degrees, roll: {1} degrees, yaw: {2} degrees",
direction.Pitch * 180.0f / (float)Math.PI, normal.Roll * 180.0f / (float)Math.PI, direction.Yaw * 180.0f / (float)Math.PI ); }

24. Leap Motion Programming NUI OO-EDP with Unity3D

25. Setup in Unity3D
Download the latest asset package from:
Open or create a project.
Select the Unity Assets > Import Package > Custom Package menu command.
Locate the downloaded asset package and click Open.
The assets are imported into your project. Example scenes can be found in the LeapMotion/Scenes folder.
Reference :

26. Tracking API for Unity3D
The classes in the Leap.Unity namespace interact directly with GameObjects and other UnityEngine components. These scripts take tracking data and put it to use in Unity.
LeapServiceProvider
provides access to the tracking data and images as well as access to the Leap.|Controller|_ object. The LeapServiceProvider transforms the tracking data so that the coordinates are relative to the transform of the Unity GameObject to which it is attached.
LeapHandController
provides access to the active hand graphics and physics objects.

27. Coordinate Systems
Unity3D uses left-hand convention for coordinate system
Leap Motion API uses right-hand convention
Unity3D default unit: meters.
Leap Motion default unit: millimeters.

28. Unity Coordinate System (left-hand)
Reference :

29. Demo http://www.cs.uakron.edu/~xiao/hci/EDP-Unity5-LeapMotion.zip
EDP-Unity5-LeapMotion/Assets/Player.cs
EDP-Unity5-LeapMotion/Assets/LeapMotion/DemoResources/HandCycler.cs

30. Leap Motion Recent Advancement with VR
Hand Tracking in Virtual Reality
Virtual Reality is a sensory input and stereo output system that transports the user into a virtual world. Current technology uses head-mounted displays as the primary means of sensory input. These displays use position and orientation tracking of various levels of sophistication to immerse the user in the virtual world and provide a sense of physical presence. Hand tracking can amplify that sense of presence.

31. Unity Coordinate System for VR
Reference :

32. References https://www.leapmotion.com/ ml