1. Mining Minds Presenter12-July-2014KHU Information Curation Layer Low Level Context-awareness

2. / Information Curation Layer 2

3. / Low Level Context-awareness Introduction Motivation Related Works Architecture Tools and Technologies Development Timeline Current Status 3

4. / Introduction 4 Physical Activities Emotional States Social Interaction Heterogeneous information source around people Daily physical activities Social interactions Psychological states Automatically collect and provide basic information to the system

5. / Introduction 5 Physical Activities Emotional States Social Interaction Knowledge Information Data Extract information directly from raw data Play an important role to get useful knowledge from people Daily habit Behavior

6. / Motivation 6 Collect, process different kind of data Sensory data Social data Analyze and extract useful information at low-level Physical activities Social activities Emotional states Provide the interface to interact with higher level and database manager

7. / Related Works 7 Activity Recognizer [WiiRemote] The Wii Remote™ Plus controller is the heart of the motion gaming experience on your Wii console. [Han2012] tried to overcome the limitation of accelerometer based activity recognition. Accelerometer is used to recognize walking, running, and stay, and audio, GPS and wifi are used to recognize bus and subway. There lots of segmentation works such as graph-cut based segmentation by [Pourjam2013], and mean-shift algorithm by [Atefian2013] have been proposed for human body segmentation.

8. / Low Level Context-awareness 8 Emotion Recognizer [MITMindReader] MIT’s Mind Reader software can scan faces in a crowd to determine audience mood, a tool that may replace opinion polls and help public speakers tailor their words for maximum impact. Various types of classifiers have been used for the task of speech emotion recognition such as HMM, GMM, SVM, etc. [Ayadi2011]. Several emotion research works tried to separate the original complex multiple emotion classification problem by applying hierarchical approach with combination of different classifiers [Lee2011].

9. / Architecture 9 High Level Context-awareness HDFS Data Access Interface Low Level Context-awareness Raw Data Personal Information SNS Interaction Analyzer Activity Recognizer Emotion Recognizer

10. / Architecture 10 High Level Context-awareness HDFS Data Access Interface Low Level Context-awareness Social Data (Twitter) Attribute Thesaurus Thesaurus Manager System Data Morpheme Manager Analysis Syntax Analyzer Morpheme Analyzer Attribute Extractor Emotion Extractor Attribute-Emotion Mapping Module Attribute-Emotion Mapping Module Positive, Negative Analyzer Positive, Negative Analyzer Compilation DMBS Connector Archive Listener Remote Control Request Module Remote Control Request Module Twitter Analyzer Sentiment Activity Recognizer Wearable Sensor based AR Data Acquisition Data Acquisition Feature Extraction Feature Extraction Training Models Training Models Classifying Smartphone based AR Preprocessing Feature Extraction Feature Extraction GPS Validation GPS Validation Decision Maker Video based AR Data Acquisition Data Acquisition Segmentation Feature Extraction Feature Extraction Classifying Emotion Recognizer Audio based ER Preprocessing Classification Tree Construction Classification Tree Construction Feature Extraction Feature Extraction Classifying Video based ER Face Detection HMM Training Feature Extraction Feature Extraction HMM Testing Physiological sensor based ER Statistical Feature Extraction Statistical Feature Extraction Non-Param Cumulative Sum Non-Param Cumulative Sum Auto Associate Neural Network Auto Associate Neural Network Decision Fusion Synchronization Probability Computation Sensory Data (Acc, GPS, Video) Sensory Data (Heart rate, Video, Audio) Personal Information (behavior, interest) Activity Label (standing, sitting, running, …) Emotion Label (happy, angry, boredom, …)

11. / Architecture 11 High Level Context-awareness HDFS Data Access Interface Low Level Context-awareness Social Data (Twitter) Attribute Thesaurus Thesaurus Manager System Data Morpheme Manager Analysis Syntax Analyzer Morpheme Analyzer Attribute Extractor Emotion Extractor Attribute-Emotion Mapping Module Attribute-Emotion Mapping Module Positive, Negative Analyzer Positive, Negative Analyzer Compilation DMBS Connector Archive Listener Remote Control Request Module Remote Control Request Module Twitter Analyzer Sentiment Activity Recognizer Wearable Sensor based AR Data Acquisition Data Acquisition Feature Extraction Feature Extraction Training Models Training Models Classifying Smartphone based AR Preprocessing Feature Extraction Feature Extraction GPS Validation GPS Validation Decision Maker Video based AR Data Acquisition Data Acquisition Segmentation Feature Extraction Feature Extraction Classifying Emotion Recognizer Audio based ER Preprocessing Classification Tree Construction Classification Tree Construction Feature Extraction Feature Extraction Classifying Video based ER Face Detection HMM Training Feature Extraction Feature Extraction HMM Testing Physiological sensor based ER Statistical Feature Extraction Statistical Feature Extraction Non-Param Cumulative Sum Non-Param Cumulative Sum Auto Associate Neural Network Auto Associate Neural Network Decision Fusion Synchronization Probability Computation Sensory Data (Acc, GPS, Video) Sensory Data (Heart rate, Video, Audio) Personal Information (behavior, interest) Activity Label (standing, sitting, running, …) Emotion Label (happy, angry, boredom, …) SNS Analyzer - Twitter Take input from Twitter API in schema format Analyze Twitter data in different contexts Activity Emotion Behavior Provide the output based on keyword

12. / Architecture 12 High Level Context-awareness HDFS Data Access Interface Low Level Context-awareness Social Data (Twitter) Attribute Thesaurus Thesaurus Manager System Data Morpheme Manager Analysis Syntax Analyzer Morpheme Analyzer Attribute Extractor Emotion Extractor Attribute-Emotion Mapping Module Attribute-Emotion Mapping Module Positive, Negative Analyzer Positive, Negative Analyzer Compilation DMBS Connector Archive Listener Remote Control Request Module Remote Control Request Module Twitter Analyzer Sentiment Activity Recognizer Wearable Sensor based AR Data Acquisition Data Acquisition Feature Extraction Feature Extraction Training Models Training Models Classifying Smartphone based AR Preprocessing Feature Extraction Feature Extraction GPS Validation GPS Validation Decision Maker Video based AR Data Acquisition Data Acquisition Segmentation Feature Extraction Feature Extraction Classifying Emotion Recognizer Audio based ER Preprocessing Classification Tree Construction Classification Tree Construction Feature Extraction Feature Extraction Classifying Video based ER Face Detection HMM Training Feature Extraction Feature Extraction HMM Testing Physiological sensor based ER Statistical Feature Extraction Statistical Feature Extraction Non-Param Cumulative Sum Non-Param Cumulative Sum Auto Associate Neural Network Auto Associate Neural Network Decision Fusion Synchronization Probability Computation Sensory Data (Acc, GPS, Video) Sensory Data (Heart rate, Video, Audio) Personal Information (behavior, interest) Activity Label (standing, sitting, running, …) Emotion Label (happy, angry, boredom, …) Activity Recognizer Take input from different sensors Wearable sensors Smartphone’s sensors Video sensors Recognize activities based on specific machine learning algorithms for each kind of data Provide output as activity label in text format

13. / Architecture 13 High Level Context-awareness HDFS Data Access Interface Low Level Context-awareness Social Data (Twitter) Attribute Thesaurus Thesaurus Manager System Data Morpheme Manager Analysis Syntax Analyzer Morpheme Analyzer Attribute Extractor Emotion Extractor Attribute-Emotion Mapping Module Attribute-Emotion Mapping Module Positive, Negative Analyzer Positive, Negative Analyzer Compilation DMBS Connector Archive Listener Remote Control Request Module Remote Control Request Module Twitter Analyzer Sentiment Activity Recognizer Wearable Sensor based AR Data Acquisition Data Acquisition Feature Extraction Feature Extraction Training Models Training Models Classifying Smartphone based AR Preprocessing Feature Extraction Feature Extraction GPS Validation GPS Validation Decision Maker Video based AR Data Acquisition Data Acquisition Segmentation Feature Extraction Feature Extraction Classifying Emotion Recognizer Audio based ER Preprocessing Classification Tree Construction Classification Tree Construction Feature Extraction Feature Extraction Classifying Video based ER Face Detection HMM Training Feature Extraction Feature Extraction HMM Testing Physiological sensor based ER Statistical Feature Extraction Statistical Feature Extraction Non-Param Cumulative Sum Non-Param Cumulative Sum Auto Associate Neural Network Auto Associate Neural Network Decision Fusion Synchronization Probability Computation Sensory Data (Acc, GPS, Video) Sensory Data (Heart rate, Video, Audio) Personal Information (behavior, interest) Activity Label (standing, sitting, running, …) Emotion Label (happy, angry, boredom, …) Emotion Recognizer Take input from different sensors Audio sensor Video sensors Physiological sensors Recognize emotions based on specific machine learning algorithms for each kind of data Apply Fusion technique to increase confident of predict output from different decisions. Provide output as emotion label in text format

14. / Tools and Technologies 14 Tools for development MATLAB Eclipse Android SDK Technologies Machine Learning Platforms Microsoft Windows Android OS

15. / Development Timeline 15 1 st year 2 nd year Interface Definition Adapter Development Component Modification Component Validation Evaluate Components based on collected data Component Adjustment Output Modified Components Validated Components Evaluation Report 1 st Integration Phase 2 nd Evaluation Phase Final module Interface Report

16. / Current Status 16 Social Interaction Analyzer Need to define the input and output to interact with Tapacross’s engine Activity and Emotion Recognizer Each individual module is available and ready for integration

17. / References 17 [WiiRemote] https://www.nintendo.com/wii/what-is-wii/#/controls [Han2012] Manhyung Han, La The Vinh, Young-Koo Lee and Sungyoung Lee, "Comprehensive Context Recognizer Based on Multimodal Sensors in a Smartphone", Journal of Sensors, vol. 12, no. 9, pp. 12588-12605, 2012 [Pourjam2013] Pourjam, E., Ide, I., Deguchi, D., & Murase, H. Segmentation of Human Instances Using Grab-cut and Active Shape Model Feedback. In proceddings of MVA2013 IAPR International Conference on Machine Vision Applications, pp. 77–80, May 20–23, 2013. [Atefian2013] Atefian, M., & Mahdavi-Nasab, H. (2013). A Robust Mean-Shift Tracking Using GMM Background Subtraction, J. Basic. Appl. Sci. Res., vol. 3, no. 4, 596–607, 2013. [MITMindReader] http://trac.media.mit.edu/mindreader/ [Ayadi2011] Ayadi, M.E., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition 44 (3), 572 - 587 (2011). [Lee2011] C.-C. Lee, E. Mower, C. Busso, S. Lee, and S. Narayanan. Emotion recognition using a hierarchical binary decision tree approach. Speech Commun., 53(9-10):1162-1171, Nov. 2011.

18. Questions Thank You! lebavui@oslab.khu.ac.kr lebavui@oslab.khu.ac.kr 18