Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Questions? What is the role of emotions in the information seeking process? Do they correspond to any form of relevance feedback? How can we effectively employ them in information retrieval scenarios? 02/03/2009Affective Feedback2

Relevance Feedback Relevance assessments can contribute in the disambiguation of the user’s information need This is achieved through the application of various feedback techniques  02/03/20093Affective Feedback

Explicit Relevance Feedback Feedback which is obtained through the explicit and intended indication of documents as relevant (positive feedback) or irrelevant (negative feedback)  02/03/20094Affective Feedback

Explicit Relevance Feedback BenefitsDrawbacks Robust method for inferring relevance feedback Interrupts the flow of the search process Better query reformulations Introduces the cognitive burden of explicit relevance judgments Improves considerably the retrieval performance of a system Trade-off between the users perusing documents because the system expects them to do so and because they are genuinely interested  02/03/20095Affective Feedback

Implicit Relevance Feedback Implicit Feedback: a passive form of feedback, which is applied in an intelligent and unobtrusive manner Can be used to individualize a system’s responses or develop user models (UM)  02/03/20096Affective Feedback

Implicit Relevance Feedback BenefitsDrawbacks Disengages users from the cognitive burden of document rating and relevance judgments Difficult to interpret Large amount of data can be obtained very easily Unreliable (compared to explicit feedback techniques) Does not account for the individual differences of users  02/03/20097Affective Feedback

Common aspects Both categories of feedback techniques determine relevance by considering what occurs on the cognitive and situational level of interaction However, they do not account for the affective dimension of the conversational interplay between the user and the system  02/03/20098Affective Feedback

Affective Computing Affective computing aims in the development of more natural and flexible systems. Human-machine interactive systems capable of sensing affect states (stress, inattention, etc) and capable of adapting and responding appropriately to these are likely to be perceived as more natural, efficient and trustworthy (Pantic, Sebe, Cohn, Huang, 2005). Can we build a multimodal retrieval system that exploits more than one modality? 02/03/2009Affective Feedback9

Can affective feedback be of any value to IR? Likely yes, since it is considered a qualitatively rich source of human affect indications, which can be potentially exploited to enhance the information retrieval process. Affective feedback can be defined as the sum of all the human affective expression/indications, which are communicated implicitly to (or identified by) a computer system and can be therefore used to facilitate a more natural, effective and robust interaction. 02/03/200910Affective Feedback

Affective Interaction Users interact with intentions, motivations and feelings besides real-life problems and information objects… Intentions, motivations and emotions are all critical aspects of cognition and decision-making  02/03/200911Affective Feedback

Affective Interaction Information systems equipped with the ability to detect and respond to user emotions could potentially: 1. Improve the naturalness of human-computer interaction 2. Progressively optimize their retrieval strategy 3. Offer a more personalized experience 4. Determine more accurately the relevancy of an information object  02/03/200912Affective Feedback

Affective Interaction What are the possible reasons of emotion? 1. System? 2. Search strategy & search results? 3. Content design and aesthetics? 4. Other  02/03/200913Affective Feedback

Emotion in IR – Some Conclusions The co-occurrence of emotions during an information seeking process, among other physiological, psychological and cognitive processes Patterns of emotional variance, which reveal a progressive transition from positive to negative valence as the degree of task difficulty increases Depending on their frequency of occurrence the value of the conveyed affective information may potentially vary?  02/03/200914Affective Feedback

Test Collection For the indexing we used TREC 9 (2000) Web Track 1.69 million document subset of the VLC2 collection We retained the original content of the TREC topics, but presented them using the structural framework of the simulated information need situations Introduce a layer of realism, while preserving well-defined relevance criteria  02/03/200915Affective Feedback

Search Tasks  02/03/200916Affective Feedback

Facial Expression Analysis Facial expression analysis was applied on the video recordings of each session For each key-frame of the video eMotion calculated the probability of the detected facial expression (assuming there was one) corresponding to any of the seven detectable emotion categories (Neutral, Happiness, Surprise, Anger, Disgust, Fear, Sadness)  02/03/200917Affective Feedback

eMotion eMotion is an automatic facial expression recognition system Developed by Nicu Sebe’s group in Amsterdam/Trento It follows a model-based approach, in which a 3- dimensional wireframe model of the face is constructed, once certain facial landmark features are detected Head motion of facial deformation can then tracked and measured in terms of motion-units (MU’s), which are eventually classified into one (or more) of the seven detectable emotion categories  02/03/200918Affective Feedback

eMotion  02/03/200919Affective Feedback

Classifier eMotion has been trained using a generic static classifier The classifier has been developed from a subset of the Cohn-Kanade database It performs reasonably well across all individuals, independently of ethnicity-specific features  02/03/200920Affective Feedback

Tools & Modalities Tools: 1) eMotion (Facial Expression Recognition System) + 2d camera 2) Pasion (Facial Expression Recognition System) + 3d camera 3) Polar RS800 Heart Rate Monitor 4) BodyMedia SenseWear Pro3 Armband Modalities: Facial Expressions (emotion categories) 1 Facial Expressions (motion units) 1 HR 3 GSR 4 Heat Flux 4 Skin Temperature 4 Acceleration 4

Facial Expressions- 02/03/2009Affective Feedback22

Facial Expressions- 02/03/2009Affective Feedback23

Biometrics 02/03/2009Affective Feedback24

Findings users' affective responses will vary across the relevance of perused information items. the results also indicate that prediction of topical relevance is possible and to a certain extent models can benefit from taking into account user affective behaviour. 02/03/2009Affective Feedback25

Open Questions How to select different modalities? Large-scale body movements; Hand-gesture recognition; Gaze-detection; Speech/voice analysis How to integrate multiple modalities? Modelling challenge? How to develop a practical system that respond to users emotional behaviour? 02/03/2009Affective Feedback26