Unsupervised Clustering in Multimodal Multiparty Meeting Analysis

The aim: „to investigate the role of participant nonverbal signals exchanged in shaping the content of agreement”

Challenges in multimodal meeting analysis No „clear and explicit definitions, particularly for functional categorizations” A lot more annotations need to be made than in the case of spoken dialogues

The experiment Six people A – F sitting around a table, face to face; they have to reach consensus on a given subject „No pre-determined roles were imposed. None of the members had professional knowledge about the topic.” Taking notes on a sheet of paper, no computers

Head movements are calculated by looking at differences between a given frame and the previous frame Faces indicated by circles, head directions by lines

2 methods Method 1 classifies the amplitude of the optical flow into 3 classes and, excluding the class with the smallest amplitude, classifies the direction of the face into 3 classes. Method 2 decomposes the optical flow into horizontal and vertical amplitude, and classifies these data with 2 dimensional features into 4 classes.

„ Speech utterances, including verbal backchannels, are frequent except from the clerk A. Main speaker utterances are accompanied by nonverbal behaviors. Nonverbal responses given by B, E, and F appear even without accompanying verbal utterances. After D yields the main speaker role to C, D doesn’t produce nonverbal behaviors.”

C produced a proposal B, D, F – nonverbal responses, E – almost no nonverbal responses B, D, F – strong support of the proposal „E didn’t get the merit of the idea and directed to C a refinement question.”

Conclusions: „By combining statistical and qualitative analysis, it was possible to obtain a clear picture of the interrelationships between types of movements, gaze shifts and head movements, and interaction organization functions of listener responses, degree of engagement and support.”