1. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | From a certain point of view Eye tracking with Java Gerrit Grunwald Java Technology Evangelist Oracle Simon Ritter Head of Java Technology Evangelism Oracle

2. Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 2

3. Human Machine Interaction Eye tracking TheEyeTribe Java and Eye tracking Demos 1 2 3 4 3 5

4. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Human Machine Interaction Eye Tracking TheEyeTribe Java and Eye tracking Demos 1 2 3 4 4 5

5. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. How It All Started

6. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Progress Was Made…

7. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Multi-Touch Has Become Popular

8. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Gaming Has Driven Several Interfaces

9. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Gestures Rather Than Touch…

10. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Interactions Without Hands

11. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Human Machine Interaction Eye Tracking TheEyeTribe Java and Eye tracking Demos 2 1 3 4 11 5

12. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. “Eye tracking is the process of measuring either the point of gaze or the motion of an eye relative to the head.” – Wikipedia 12

13. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Of the stimuli that reach the brain are visual 80 % 13 The eyes are the windows to the soul

14. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Eyes Give Clues When Answering Questions Visually Constructe d Visually Remembered Auditory Constructed Auditory Remembered Feeling/Kinesthetic Internal Dialog This is for a right-handed person. Positions reversed if left-handed

15. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Where to use it ? 15

16. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 16 Eye gaze correction for video conferencing

17. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 17 Minimize dangers in air traffic displays

18. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 18 Developing Video Games and Graphics

19. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 19 Marketing and E-Commerce website development

20. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 20 Improved computer-human interaction

21. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. What can it do ? 21

22. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Determine whether a user is looking at the screen – Paying attention to course, video, etc If a user is reading or scanning – Speed of movement of the eyes over text Intensity of user’s attention – How long they look ata aprticular thing Whether the user is searching for specific information Capabilities

23. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Eye attached tracking Electric potential measurement Optical tracking 23 Types of Eye Trackers

24. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Special contact lense with embedded mirror or sensor (magnetic search coils) Very precise Allows measurement in horizontal, vertical and torsion directions Expensive 24 Eye attached tracking

25. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Measures electric potential field of eyes Can measure eye movement even if eyes are closed Wearable method – also used in sleep research Relatively poor accuracy in gaze direction detection Expensive 25 Electric potential measurement

26. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Measures reflection of IR light from the front of the cornea Contactless method Good for fast and slow eye movements Inexpensive 26 Optical tracking

27. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Optical Eye Tracking Principle 27

28. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 28 Optical Eye Tracking Principle IR-Light emission Eyeball Cornea Reflected IR-Light Eye Tracker

29. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Human Machine Interaction Eye Tracking TheEyeTribe Java and Eye tracking Demos 2 1 3 4 29 5

30. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. TheEyeTribe 30

31. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Sampling rate Accuracy Latency Operating range Tracking area Screen sizes Support for Connection 31 TheEyeTribe 30 Hz and 60 Hz 0.5° - 1° < 20ms at 60 Hz 45 cm - 75 cm 40 cm x 30 cm at 65 cm up to 24" C++, C# and Java USB 3.0

32. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 32 TheEyeTribe APPLICATION SDK EyeTribe SERVER NETWORK HW TRACKER Open API External applications (e.g. Java program) Language spec. wrappers (C++, Java etc.) The eye tracking process. Initializes device(s), frameworks, servers etc. No GUI - output goes through API

33. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Human Machine Interaction Eye Tracking TheEyeTribe Java and Eye tracking Demos 1 4 33 5 23

34. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 34 import com.theeyetribe.client.*; public enum EyeTracker implements IGazeListener { INSTANCE; private GazeData gazeData; … private EyeTracker { init(); initBuffers(); initEyeTracker(); timer.start(); } private void init() {…}; private void initBuffers() {…}; } Java and Eye Tracking Java package

35. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 35 import com.theeyetribe.client.*; public enum EyeTracker implements IGazeListener { INSTANCE; private GazeData gazeData; … private EyeTracker { init(); initBuffers(); initEyeTracker(); timer.start(); } private void init() {…}; private void initBuffers() {…}; Java and Eye Tracking implement Interface

36. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 36 import com.theeyetribe.client.*; public enum EyeTracker implements IGazeListener { INSTANCE; private GazeData gazeData; … private EyeTracker { init(); initBuffers(); initEyeTracker(); timer.start(); } private void init() {…}; private void initBuffers() {…}; } Java and Eye Tracking initialize EyeTracker

37. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 37 private void initEyeTracker() { GazeManager.getInstance().activate(GazeManager.ApiVersion.VERSION_1_0, GazeManager.ClientMode.PUSH); } Java and Eye Tracking initialize EyeTracker

38. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 38 public void start() { if (running.get()) return; running.set(true); GazeManager.getInstance().addGazeListener(this); timer.start(); } Java and Eye Tracking start EyeTracker

39. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 39 @Override public void onGazeUpdate(final GazeData GAZE_DATA) { gazeData = GAZE_DATA; xBuffer.remove(); yBuffer.remove(); xBuffer.addLast(gazeData.smoothedCoordinates.x); yBuffer.addLast(gazeData.smoothedCoordinates.y); xAvg = xBuffer.stream().mapToDouble((x) -> x).average(); yAvg = yBuffer.stream().mapToDouble((y) -> y).average(); } Java and Eye Tracking get Gaze data

40. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 40 public void stop() { GazeManager.getInstance().removeGazeListener(EyeTracker.this); GazeManager.getInstance().deactivate(); } Java and Eye Tracking stop EyeTracker

41. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. What data do you get ? 41

42. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Gaze Data delivered in frames… 42

43. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Tracking state State of the eye and gaze tracking process. (e.g. Person in field of view) Fixation If the tracked gaze is currently fixated, the frame will contain that information Gaze Coordinates The point on screen (pixels) that the user is currently looking at. (Needs calibration) Pupil coordinates The position of a tracked persons pupil relative to the Tracker sensor 43 Gaze data

44. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Problem… 44

45. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Problem…SPEED 45

46. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 46 timer = new AnimationTimer() { @Override public void handle(final long now) { if (now > lastTimerCall + UPDATE_INTERVAL) { if (null == gazeData) return; focusPoint.set(new Point2D(xAvg.getAsDouble(), yAvg.getAsDouble())); eventCounter++; leftEyeOpen.set(gazeData.leftEye.pupilSize > 0); rightEyeOpen.set(gazeData.rightEye.pupilSize > 0); state.set(gazeData.state); lastTimerCall = NOW; } }; } Problem…SPEED use AnimationTimer to update data in intervals

47. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Eye tracking provides a useful and different way to interact with a UI – Control is hard, tracking a user’s interaction is easier Many potential applications – Helping the physically challenged – Reducing potential problems in critical systems www.eyetribe.com dev.eyetribe.com/java Conclusions And Resources

48. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Human Machine Interaction Eye Tracking TheEyeTribe Java and Eye tracking Demos 1 4 48 5 23