1. BlueEyes Human Operator Monitoring System BlueEyes Human-Operator Monitoring System Poznań University of Technology, Poland http://www.put.poznan.pl Team mentor Team membersPaweł Kowalik Piotr Kubiaczyk Krystian Nowak Stanisław Osiński Tomasz Pużak Jan Kniat, Ph. D. Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration

2. BlueEyes Human Operator Monitoring System Agenda Motivation What is BlueEyes (and what is not) ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration

3. BlueEyes Human Operator Monitoring System Motivation Human error - a frequent reason of catastrophes and ecological disasters Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration Lack of complex solution exploiting wireless link technology, database systems and AI algorithms –temporal indisposition –weariness –oversight Monitoring of conscious brain involvement Capable sensors available

4. BlueEyes Human Operator Monitoring System What is BlueEyes ? The complex solution for human-operator monitoring: –conscious brain involvement monitoring –triggering user-defined actions –data recording Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration What is BlueEyes not? Doesn’t predict nor interfere with operator’s thoughts Cannot force directly the operator to work

5. BlueEyes Human Operator Monitoring System Who is BlueEyes for ? Generic control rooms –Power station –Captain bridge –Flight control centers –Operating theatres – anesthesiologists Common application –A simpler system version for drivers Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration

6. BlueEyes Human Operator Monitoring System BlueEyes - benefits Prevention from dangerous incidents Minimization of –ecological consequences –financial loss –a threat to a human life The reconstruction of the course of operator’s work Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration

7. BlueEyes Human Operator Monitoring System Physiological foundations Preconscious brain Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration manages the knowledge logical reasoning makes decisions alogical „automatic” Conscious brain Human brain

8. BlueEyes Human Operator Monitoring System Conscious brain - operation modes Exploratory modePlanning mode Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration

9. BlueEyes Human Operator Monitoring System Eye movements Saccade - a rapid eye jump to a new location assigned by the conscious attention process. Saccades presence  Exploratory mode Central vision and the conscious brain Retina - central vision (macula) Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration

10. BlueEyes Human Operator Monitoring System The eye movement sensor Eye position measuring - direct infrared oculography Oxy- and deoxyhemoglobin measurment Two axial accelerometer Ambient light sensor Jazz Multisensor Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration

11. BlueEyes Human Operator Monitoring System Wires, wires, wires... A serious limitation of: –mobility –comfort –safety Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration

12. BlueEyes Human Operator Monitoring System Personal Area Network ! A wireless Personal Area Network Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration Solution:

13. BlueEyes Human Operator Monitoring System A reliable real-time buffering, processing and recording A clear visualization interface Mechanisms for creating custom data analyzers To ensure a possibility of distributing the processing among a number of computers Design objectives A mobile data acquisition device Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration Objectives System overview DAU CSU

14. BlueEyes Human Operator Monitoring System System overview Central System UnitData Acquisition Unit Atmel 89C52 microcontroller Jazz Multisensor Connection Manager Data Analysis Visualization Module Bluetooth Bluetooth technology provides means for creating a Personal Area Network linking the operators and the central system. Data Logger Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration Objectives System overview DAU CSU

15. BlueEyes Human Operator Monitoring System DAU - features Lightweight Runs on batteries - low power consumption Easy to use - does not disturb the operator working ID cards for operator authorization Voice transmission using hardware PCM codec Data Acquisition Unit Atmel 89C52 microcontroller Jazz Multisensor Bluetooth Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration Objectives System overview DAU CSU

16. BlueEyes Human Operator Monitoring System CSU - features Central System Unit Connection Manager Data Analysis Visualization Module Bluetooth Connection management Data processing Visualization Data recording Access verification System maintenance Connection Manager Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration Objectives System overview DAU CSU Data Logger

17. BlueEyes Human Operator Monitoring System CSU - saccade detector Saccade detector and conscious brain involvement level calculation saccade Raw eye movement data Differentiation Conscious brain involvement level Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration Objectives System overview DAU CSU

18. BlueEyes Human Operator Monitoring System CSU - Visualization Module Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration Objectives System overview DAU CSU Raw and processed data visualization using: audio and video data playback –VU-meters –pie-charts –time series

19. BlueEyes Human Operator Monitoring System Data security Only registered mobile devices can connect to the system Bluetooth connection authentication Bluetooth connection encryption Access rights restrictions Personal and physiological data encryption Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration Objectives System overview DAU CSU

20. BlueEyes Human Operator Monitoring System Implementation - DAU Bluetooth module inside Jazz sensor interface Microphone and earphone sockets 5-key keyboard Two lines, 8 characters LCD Powered from 6 AA batteries Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration DAU CSU Made from scratch by the team

21. BlueEyes Human Operator Monitoring System Implementation - DAU Bluetooth module PCM codec board Batteries and voltage level monitor Microcontroller board Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration DAU CSU

22. BlueEyes Human Operator Monitoring System Implementation - CSU Custom-built Bluetooth communication routines Intel USB camera ID card programmer Developed using Visual C++ 6.0 Deployment environment: Windows 2000, MS SQL Server Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration DAU CSU Standard MDI user interface

23. BlueEyes Human Operator Monitoring System Future improvements DAU –small CMOS camera to monitor the operator’s point of gaze –single PCB (SMD technlogy) –low voltage ICs - LiIO batteries power CSU –data mining algorithms –advanced database encryption using e.g. AES algorithm Motivation What is BlueEyes ? Physiological foundations System design System implementation System demonstration Future improvements Project work summary

24. BlueEyes Human Operator Monitoring System Project work summary Bluetooth and PAN technology acquired New skills on designing software and hardware systems gained A mobile device implemented 31 KLOC of C++ code and 4 KLOC of 8051 assembler code written All parts of the Project Kit efficiently used New approach to system safety proposed and implemented Motivation What is BlueEyes ? Physiological foundations System design System implementation System demonstration Future improvements Project work summary

25. BlueEyes Human Operator Monitoring System BlueEyes Human-Operator Monitoring System System demonstration Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration

26. BlueEyes Human Operator Monitoring System Thank you for your attention BlueEyes Human-Operator Monitoring System Poznań University of Technology, Poland http://www.put.poznan.pl Motivation What is BlueEyes ? Physiological foundations System design System implementation System demonstration Future improvements Project work summary

27. BlueEyes Human Operator Monitoring System BlueEyes Data Acquisition Unit ID card DAU - components MC 145483 PCM codec ID card interface Simple keyboard Jazz Multisensor Beeper LCD display LED indicators Atmel 89C52 microcontroller UART MAX232 UART Bluetooth module microphone earphone

28. BlueEyes Human Operator Monitoring System Initializes Bluetooth modules Inquires for new devices in range Establishes connections Supports operator roaming Feeds Operator Manager with physiological data Bluetooth... CSU - Connection Manager Bluetooth Connection Manager Operator Data Manager Alerts Separated data streams... Operator Manager Alert buffer Raw Data Buffer Alerts Operator Manager Alert buffer Raw Data Buffer Data demultiplexer Data producer queues

29. BlueEyes Human Operator Monitoring System CSU - Data Analysis Custom data analyzers - decision tree based classifiers Example: Operator gesture recognition (simplified) Vertical Acceleration variance value high Horizontal Acceleration variance high YN YNYN No answer “NO”“YES”

30. BlueEyes Human Operator Monitoring System CSU - Data Analysis Custom data analyzers - decision tree based classifiers Automatic decision tree coefficients generation - C4.5 based algorithm Data samples Assigned category “Yes” No answer “Yes” “No” No answer...... Decision tree coefficients

31. BlueEyes Human Operator Monitoring System CSU - intermodule communication Uses buffered thread-safe queues and producer-consumer scheme Data producer Data queue Data consumer Data consumer creates Data consumer Processing Data consumer & producer Data consumer & producer Data queue Data queue register at Data consumer

32. BlueEyes Human Operator Monitoring System CSU - intermodule communication Data Analysis Visualization Module Data Logger Operator manager (OM)OM Connection Manager... Separated physiological data streams Processed data Alerts Recorded (off-line) data