1. Multimedia Concepts and Applications Multimedia Concepts and Applications Affect Sensing in Speech: Studying Fusion of Linguistic and Acoustic Features Alexander Osherenko, Elisabeth André, Thurid Vogt University of Augsburg

2. Multimedia Concepts and Applications Multimedia Concepts and Applications Affect Sensing Acoustic information Linguistic information (lexical, stylometric, deictic)

3. Multimedia Concepts and Applications Multimedia Concepts and Applications Fusion Decision-level Feature-level

4. Multimedia Concepts and Applications Multimedia Concepts and Applications Research Questions Fusion Context Decision-level vs. feature-level

5. Multimedia Concepts and Applications Multimedia Concepts and Applications Experimental Setting SAL corpus, 574 turns, 5 classes Decision-level using majority, feature-level – fusing features Data: 2 stages (history 0 and history 7) – Acoustic modality - 2 (discrete/continuous) acoustic datasets (A) – Lingustic modality - 29 lexical (L), 31 stylometric (S), 63 deictic datasets (D)

6. Multimedia Concepts and Applications Multimedia Concepts and Applications Tree – Nodes – features from particular modalities (A, L, S, D) – Values Maximal recall value Maximal multimodality value Dotted arcs Results‘ representation

7. Multimedia Concepts and Applications Multimedia Concepts and Applications Best results: 64.2% (history 7) and 44.2% (history 0) Significant improvement through context Insignificant improvement through fusion (about 2%) Maximal multimodality value (76.5%) Decision-level Fusion Before Discretization

8. Multimedia Concepts and Applications Multimedia Concepts and Applications Best results: 66.0% (history 7) and 49.0% (history 0) Significant improvement through context Insignificant improvement through fusion (about 2%) Maximal multimodality value (77.8%) Decision-level Fusion After Discretization

9. Multimedia Concepts and Applications Multimedia Concepts and Applications Best results: 62.8% vs. 64.2% (history 7) and 46.7% vs. 44.2% (history 0) Significant improvement through context Insignificant improvement through fusion (about 2%) Feature-level Fusion Before Discretization

10. Multimedia Concepts and Applications Multimedia Concepts and Applications Feature-level Fusion After Discretization Best results: 67.5% vs. 64.9% (history 7) and 52.8% vs. 45.9% (history 0) Significant improvement through context Insignificant improvement through fusion (about 2%)

11. Multimedia Concepts and Applications Multimedia Concepts and Applications Discussion Role of context Role of discretization Fusion?

12. Multimedia Concepts and Applications Multimedia Concepts and Applications Future work New modalities Weighting