1. Towards a Reactive Virtual Trainer Zsófia Ruttkay, Job Zwiers, Herwin van Welbergen, Dennis Reidsma HMI, Dept. of CS, University of Twente Amsterdam, The Netherlands zsofi@cs.utwente.nl

2. page 2 Overview n RVT usage n Related work n RVT technological challenges –Architecture –Integration of reactive and proactive actions –Multi-modal sync n A close look at clapping - demos

3. page 3 RVT usage n RVT = IVA with expert and psychological knowledge of a real physiotherapist, to be used e. g. to: –prevent RSI for computer workers –preserve/restore weight and physical condition as (personal) trainer –act as physiotherapist to cure illnesses affecting motion n RVT is medium and emphatic consultant n Relevance for society –ageing population, unhealthy life-style, –human experts: low number, expensive, at certain locations n RVT usage context –PC + 1-2 camera in normal setting (homes, offices) –‘instructed’ by authorized person (may be the user, as well as developer) –can be adapted/extended

4. page 4 Related work Trainer calibrationMedium/consultantInputFeedbackMotiondemo/correctionExercise revisionAuthoring J. Davis, A. Bobick: Virtual PAT, MIT, 1998. 1moviesplit2 cam.assess ment --script S-P.Chao et al: Tai Chi synthesizer, 2004. 1m----nl script W. IJsselsteijn et al (Philips): Fun and Sports: Enhancing the Home Fitness Experience, Proc. of ICEC 2004. 1cheart- rate assess ment --? Sony’s EyeToy: Kinetic ‘game’, 2005 2m/c1 cam.gener al, well- place d d-By User from pre- set choice/ty pes T. Bickmore: Laura & FitTrack1cdata to be typed assess ment --closed?

5. page 5 Own related work – Virtual Rap Dancer

6. page 6 Own related work – Virtual Conductor

7. page 7 RVT technological challenges n Vision-based perception, may be extended with biosignals n Reactive on exercise performance, physical state, overall performance n Smalltalk, exercise correction, plan revision n VRT body and motion parameters adaptable/calibrated n Authoring by human n Extensible by expert (new exercises) n Motion with music, speech or clapping (also as input for tempo) n Playground for multi-modal output generation n “Exercise motion intelligence”: timing, concatenation, idle poses, …

8. page 8 RVT architecture Calibration of user Multi-sensor integration Authoring scenario Exercise sce- nario revision Optical motion tracking Motion interpretation Motion specification Biosensing module(s) Acoustic beat tracking VT Monitoring the user Multi-modal feedback Motion demonstratio n Presentation of feedback of VT Planning action of VT Human expert User Interfaces

9. page 9 Multi-modal sync n Exercises are executed using several modalities –Body movement –Speech –Music –Sound (clap, foot tap) n Challenges –Synchronization –Monitoring user => real time (re)planning Exaggeration to point out details Speed up / slow down Feedback/correction …

10. page 10 Synchronization: related work n Classic approach in speech/gesture synchronization: –Speech leads, gesture follows n MURML (Kopp et al.) –No leading modality –Planning in sequential chunks containing one piece of speech and one aligned gesture –Co-articulation at the border of chunks n BML (Kopp, Krenn, Marsella, Marshall, Pelachaud, Pirker, Thórisson, Vilhjalmsson) –No leading modality –Synchronized alignment points in behavior phases –For now, aimed mainly at speech/gesture synchronization –In development

11. page 11 Synchronization: own previous work n Virtual Dancer Synchronization between music (beats) and dance animation Dance move selection by user interaction n Virtual Presenter Synchronization between speech, gesture, posture and sheet display Leading modality can change over time n GESTYLE markup language with par/seq and wait constructs

12. page 12 Close look at clapping stroke (hold) retraction (hold)

13. page 13 Clapping Exercise

14. page 14 Close look at clapping n Start with a simple clap exercise and see what we run into n The clap exercise: –Clap for the tempo of the beat of a metronome (later: of music) –When the palms touch, a clap sound is heard –Count while clapping, using speech synthesis Possible alignment at: word start/end, phonological peak start/center/end For now, we pick the center of the phonological peak, but we do generate the other alignment points for easy adaptation

15. page 15 Two examples for multi-modal sync n Specification in BML T n Planning in real-time – under/overspecification!

16. page 16 What if we speed up the tempo? n The clapping animation should be faster n Possibilities: –Lower amplitude? –Linear speedup? –Speedup of stroke? –Speedup of retraction? –A combination of above?

17. page 17 What if we slow down the metronome? n Slower clapping? (movies here) –Linear slowdown? –Slowdown of stroke? –Slowdown of retraction? –Hold at end of retraction (hands open)? –Hold after stroke (clap)? –A combination of above? n Back to idle position?

18. page 18 Open issues on planning n What do real humans do? n Do the semantics of a motion (clap) change if we change its amplitude or velocity profile? E.g. emotions, individual features n Smooth tempo changes n Automatic concatenation and inserted idle poses n Appropriate high-level parameters –Related (e.g. amplitude/speed)? –Different of parameters for communicative gestures (e.g. by Pelachaud)? n Amplitude and motion path specification n Is our synchronization system capable to re-plan in real time?