Sensor-based Situated, Individualized, and Personalized Interaction in Smart Environments Simone Hämmerle, Matthias Wimmer, Bernd Radig, Michael Beetz Technische Universität München – Informatik IX Sensor-based Situated, Individualized, and Personalized Interaction in Smart Environments

SIP via sensors Situation detection: information about persons: name, location, focus of attention, posture, motion,… information about persons: name, location, focus of attention, posture, motion,… Individualized settings: desktop, avatar, input settings (gestures, voice commands,…) desktop, avatar, input settings (gestures, voice commands,…) Personalized settings: user’s role, right management,… user’s role, right management,… SIP detection using sensors more comprehensive SIP information more comprehensive SIP information more intuitive HCI more intuitive HCI

Our Test Bed Sensors: cameras, microphones, laser-range-sensors Actuators: monitor, speaker, video-wall Scenarios: Scenarios: person localization person localization automatic login automatic login meeting reminder meeting reminder individualized gesture interaction individualized gesture interaction

Video

Techniques (Computer Vision) person detection OpenCV (Haar-Face-Detector) person recognition OpenCV (Hidden Markov Models) person tracking developed at TUM laser-scanner based multiple hypothesis tracking,… gesture recognition developed at TUM motion templates, multiple classifiers,… mimic recognition developed at TUM point distribution model, optical flow,…

Techniques (others) natural language input Java Sphinx 4 (origin CMU, now open source) Java Sphinx 4 (origin CMU, now open source) phonemes are already trained phonemes are already trained we defined the words ( = concatenation of phonemes) we defined the words ( = concatenation of phonemes) we defined the grammar ( = allowed sentences) we defined the grammar ( = allowed sentences) natural language output provides the user with audio information provides the user with audio information user can be mobile user can be mobile FreeTTS 1.2 (sourceforge) FreeTTS 1.2 (sourceforge)

Software architecture Dispatcher multi agent framework

Conclusion Advantages using sensors additional and more exact context knowledge additional and more exact context knowledge unobtrusive system unobtrusive system Multi agent framework distributed and scalable system distributed and scalable system simply extensible to further scenarios simply extensible to further scenarios Overall semantic semantic agent communication semantic agent communication central aggregation of semantic context knowledge central aggregation of semantic context knowledge Leads to more comprehensive SIP information more comprehensive SIP information seamless integration of SIP information seamless integration of SIP information intuitive HCI intuitive HCI

Thank you!

Setup & Benefit sensors for detection of SIP context: cameras cameras microphones microphones laser-range-sensors laser-range-sensors pressure-sensors, … pressure-sensors, … sensors provide knowledge about the SIP context situation dependant services situation dependant services intuitive HCI (human computer interface) intuitive HCI (human computer interface) application scenarios: support in meetings and presentations support in meetings and presentations intelligent House intelligent House external robot control external robot control

Our Test Bed Sensors: Cameras, Microphones, Laser-Range-Sensors Actuators: Monitor, Speaker, Video-Wall Scenarios: automatic login automatic login meeting reminder meeting reminder individualized gesture interaction individualized gesture interaction intuitive robot control intuitive robot control person localization person localization

Sensors person recognition person recognition (Bild) (Bild) gesture recognition gesture recognition (Bild)

Knowledgebase Web Ontology Language (W3C) Web Ontology Language (W3C)