On the parameterization of clapping Herwin van Welbergen Zsófia Ruttkay Human Media Interaction, University of Twente

Content Context and goals Related work Experiment setup Results Conclusion Questions

Context: Reactive Virtual Trainer An ECA acting out exercises a user is supposed to do Perceives the movement of the user Reactive Gives feedback Using speech, gestures and motion (Re)schedules and adapts exercises Tempo changes Subtle timing and lifelikeness of motion is important

Goal The generation of believable, adaptable exercise motion in real time How can we parameterize motion? What parameters? Tempo, amplitude (for accentuation?), … How do the parameters relate? How do they affect movement? How is speech synchronized with exercise motion?

Related work: biomechanics Typical biomechanical research setup Very obtrusive Measuring one movement characteristic Gaining ‘deep’ knowledge Our setup Unobtrusive Measuring a wider range of characteristics Less depth, measure on an abstraction level that gains us parameters for movement generation

Related work: finding animation parameters Statistical methods (Egges et al), machine learning (Brand et al) Finds independed parameters Not intuitive Highly depended on analyzed data set Laban Movement Analysis Effort can automatically be found from movement data (Zhao et al) Shape? Our parameters (tempo, amplitude) can be mapped to LMA parameters

Related work: parameterized animation Rule based (EMOTE, Neff et al, Hartmann et al,..) Uses movement models Typically does not deal with dependence between parameters Lack of detail Example based (Wiley et al, Kovar et al) By blending examples Nr of examples needed grows exponentially with nr of parameters

Related work: model based gesture synthesis Kopp et al: Uses biomechanical rules of thumb to generate movement Real time Domain: speech accompanying gestures We plan to extend on this work Use in rhythmic movement domain Providing parameterization Providing whole body movement Introducing movement variability

Focus Analysis of a clapping exercise Analyzed aspects: Synchronization of speech and motion How does a change of tempo affect movement? Time distribution Movement path Amplitude Left-right hand symmetry Whole body involvement

Clapping experiment: setup Mocap analysis of two subjects Instructions: ‘Free clap’: Clap and count from 21 to 31 ‘Metronome driven clap’: Clap and count to the metronome

Synchronization of clap and speech Phases from movement in gestures The phonological synchrony rule holds for clapping

Time distribution in phases Free clap was executed consistently at ≈ 60 bpm One subject made use of a pre-stroke hold at 30 bpm The relative duration of the phases does not change with tempo The standard deviation of the relative duration decreased with one subject

Movement path of the hands

Amplitude: how to measure Maximum distance between hands Path is curved Max distance between hands alone does not display the amount of motion Distance along path

Amplitude: observations Path distance and max. hand distance decrease with tempo Average speed is constant at different tempos There is a linear relation between period and path distance Pathdistance = a + b ▪ period Amplitude of free clap is higher Average speed of free clap is higher

Amplitude

Period vs path length

Left-right hand symmetry: How to measure Model: self oscillating systems Closed orbit between position (x) and speed (v) x is the normalized angle x^ θ is the phase angle Relative phase angle: Φ = θ left -θ right Negative Φ means right hand leads

Left-right hand symmetry at 90 bpm

Left-right hand symmetry: Theory Right handed subjects lead a rhythmic task with their right hand (Treffner et al) But such asymmetry can disappear when the task is metronome driven Stability of Φ depends on the tempo and mass imbalance (Treffner et al, Fitzpatrick et al) Higher tempo => higher | Φ| Higher tempo => higher variability in Φ

Left-right hand symmetry: Findings Mean Φ is consistently negative for our right-handed subjects No difference between metronome driven and free clap in mean Φ The standard deviation of Φ increases with tempo No significant relation between mean Φ and tempo was found

Whole body involvement By annotating if markers move in the same tempo as the hands Movement was found on the head and torso for all tempos For low tempos movement was even observed up to the thighs and knees

Conclusions The phonological synchrony rule was validated for clapping Clapping can be sped up by making the path distance smaller A pre-stroke hold can be used to slow down Clapping is clearly a whole body motion At a faster tempo, fewer body parts are perceivably involved Left-right hand movement variability increases with tempo For both right-handed subjects, the right hand was leading The metronome did not diminish this lead

Further work Ultimately: generate clapping motion given tempo + personal characteristics More recordings Free clapping without counting Tempo transitions How do personal characteristics affect movement? Deeper analysis How does variability affect the movement path? Generation Can movement on the rest of the body be generated given movement on the arms (as in Egges, Pullen)? Blending clap animation at different tempos to gain animation at a new tempo (as in Kovar)?

Questions

Easter eggs

Why use gesture-like phases for clapping? The stroke of a speech accompanying gestures (SAG) is at an energy peak in the movement and expresses meaning (McNeill) Claps have such a clear peak But this peak does not express meaning Why compare SAG and clapping? The form of clap movement and SAG is similar Excursions: start in rest, end in rest Peak structure Well bounded But not symmetric May find information on the nature of the phonological synchrony rule Does it depend on form or meaning?

Precision No significant correlation between metronome period and avg clap ‘error’ or variability of clap tempo was found Measured both absolute and relative to the metronome period

Left-right hand position at 90 bpm

3D hand & elbow positions

3D hand positions at different tempos

Free clap amplitude vs metronome drive clap amplitude