EyesWeb XMI Multimodal data recording, playing and analysis M. Mancini, Università di Genova (Italy)

Goal 1.record multimodal data: video (rgb or silhouette) audio sensor 2.play multimodal data 3.analyze multimodal data in real-time silhouette (if available) Contraction Index Quantity of Motion sensor energy smoothness

1.Recording media file writer N-th video frame clock csv file writer tsv file writer N M-th sensor frame K-th audio buffer current time

2.Player media file reader N-th video frame csv file reader tsv file reader N current time M-th sensor frame

Global measures depending on full body movement (e.g., body orientation, overall motion direction). Measures from psychological research, e.g., Boone & Cunningham’s amount of upward movement. Cues from Rudolf Laban’s Theory of Effort, e.g., directness, impulsiveness. Cues derived from analogies with audio analysis, e.g., Inter Onset Intervals, frequency analysis. Kinematic measures such as velocity, acceleration, average and peak velocity and acceleration. 3.Analysis: Expressive Features

SMIs (Silhouette Motion Images) carry information on variations of a blob (usually the silhouette of a user) in the last few frames. Silhouette Motion Images

SMIs are different with respect to MHIs, since they do not include the last silhouette, i.e., the current posture. Thus, SMIs carry information about the movement detected by the video-camera in the last n frames. The SMI area can be therefore considered as a measure of the detected amount of motion. The SMI area, normalized by the silhouette area, is called Motion Index (or Quantity of Motion). Motion_Index[t] = Area(SMI[t, n]) / Area(Silhouette[t]) Note that this is an approximated measure: e.g., movement against the video-camera is not detected. SMIs and Motion Index

Motion Index can also be computed by differently weighting pixels in the input blob. So, it is possible to compute a Motion Index where pixels near to the centre of the blob weight more than pixels near to the contour (i.e., something more similar to the physical concept of momentum). Or it is possible to compute a Motion Index where pixels near to the contour weight more than pixels near to the centre of the blob (i.e., a more perceptual measure where limbs have a stronger impact on perception of movement). Weighted Motion Index

Contraction Index is a measure of how the user’s body uses the space surrounding it. The simplest way to compute it is the ratio between the area of the blob and the area of the bounding rectangle. CI = Area(Blob) / Area(Bounding rectangle) Contraction Index

Silhouette Features Extraction media file reader N-th silhouette frame blob extractor QoM extractor CI extractor csv file writer current time

Sensor Features Extraction tsv file reader acceleration differentiate jerk (1/fluidity) integrate energy csv file writer current time