1. Robot Dance based on Online Automatic Rhythmic Perception João M. Lobato D. S. Oliveira In cooperation with: LIACC Laboratório de Inteligência Artificial e Ciência de Computadores INESC Porto Instituto de Engenharia de Sistemas e Computadores Coordinators: - Prof. Dr. Luis Paulo Reis - Prof. Dr. Fabien Gouyon Web Page: paginas.fe.up.pt/~ee03123 Mestrado Integrado em Engenharia Electrotécnica e de Computadores

2. Index Introduction Objectives Tools Components System Architecture Experiments and Results Demonstration Conclusions Future Work 2

3. Autonomous Robotic Systems Robot Dancing Rhythmic-Interactive Systems MusicRhythm Cause - Effect Body Movement Autonomy – Spontaneity Agent Environment Introduction SynchronismDynamismEntrainmentFlexibilityAdaptability 3

4. 4 Introduction

5.  Considered Limitations – Pre-programmed movements; – Little flexibility; – Poor adaptability; – Inexistent or little human control; – Direct applications of rhythmic perception models. 4 Introduction

6. Objectives Develop a flexible dancing robotic (entertainment) system where a humanoid robot performs seemingly autonomous dance movements in synchrony to musical rhythm, without former knowledge of the music. 5 Synchronism Realism Dynamism Reactive Behavioural-Based Approach Effect Cause Rhythm Dance

7. Tools – 32-bits microprocessor; – 8-bits 4MHz microcontroller; – RAM and Flash memory; – Monitor LCD 100x64; – 16kHz Speakers; – Connectivity - USB 2.0 and Bluetooth; – Supports up to 3 servo-motors; – Supports up to 4 ad-hoc sensors: colour, luminosity, sound, touch, ultrasonic, compass, between others. 6  Hardware – Lego Mindstorms NXT Kit

8. Tools Software – Used Applications – Marsyas (Music Analysis, Retrieval and Synthesis for Audio Signals) – open-source framework constituted by various computational audio analysis blocks based on the most prominent published algorithms. – MFC (Microsoft Foundation Class) to build the user interface; – MATLAB – numeric computation and programming environment. – NXT Remote API – C++ library to remotely control Lego NXT, via Bluetooth. 7

9. Tools Methodology – Interconnection of Tools 8 Visual C++ (MFC) yy Marsyas Music Analysis Module Experimentation/Simulation MATLAB Engine MATLAB Engine Robot Control Module NXT Remote API Lego Mindstorms NXT Robot Dancing TCP/IP Sockets Bluetooth Human Control Module Windows OS GUI

10. Components Dancing Humanoid Robot 9 RArm Motor LArm Motor Fan (spins with Head) Head Motor Body Motor NXT Bricks Colour Sensor Ultrasonic Sensor RLeg Motor LLeg Motor

11. Components Dancing Environment 10

12. System Architecture System Framework Operation 11 Music Analysis Module Robot Control Module Human Control Module Rhythmic Events Audio Parameters Dance Composition Dance Movements D a n c e M o v e m e n t s Sensorial Events Sensorial Events Audio Signals WAV File TCP/IP Sockets

13. System Architecture 12

14. Thread 4: Dance Movements (Output Processing) Thread 4: Dance Movements (Output Processing) Thread 5: Brick 1 BT Communication Thread 5: Brick 1 BT Communication Thread 6: Brick 2 BT Communication Thread 6: Brick 2 BT Communication System Architecture 12  Multithreading Architecture Thread 3: GUI + Asynchronous TCP/IP Server Thread 3: GUI + Asynchronous TCP/IP Server Robot Control Module Music Analysis Module Human Control Module Thread 1: Rhythmic Perception Model Thread 2: Synchronous TCP/IP Client

15. Thread 4: Dance Movements (Output Processing) Thread 4: Dance Movements (Output Processing) Thread 5: Brick 1 BT Communication Thread 5: Brick 1 BT Communication Thread 6: Brick 2 BT Communication Thread 6: Brick 2 BT Communication Thread 3: GUI + Asynchronous TCP/IP Server Thread 3: GUI + Asynchronous TCP/IP Server Robot Control Module Human Control Module Thread 1: Rhythmic Perception Model Thread 2: Synchronous TCP/IP Client Music Analysis Module  Multithreading Architecture Music Analysis Module 13 System Architecture

16. – Low-level rhythmic perception model based on onset detection. Onset Attack Decay Transient Music Analysis Module System Architecture 13

17. – Low-level rhythmic perception model based on onset detection. Onset Attack Decay Transient Pre-ProcessingReduction “Peak-Picking” Onsets Location Detection Function Audio Signal (Pre-Processed) Original Signal Music Analysis Module System Architecture 13,

18. Thread 4: Dance Movements (Output Processing) Thread 4: Dance Movements (Output Processing) Thread 5: Brick 1 BT Communication Thread 5: Brick 1 BT Communication Thread 6: Brick 2 BT Communication Thread 6: Brick 2 BT Communication Thread 3: GUI + Asynchronous TCP/IP Server Thread 3: GUI + Asynchronous TCP/IP Server Robot Control Module Human Control Module Thread 1: Rhythmic Perception Model Thread 2: Synchronous TCP/IP Client Music Analysis Module  Multithreading Architecture Robot Control Module 15 System Architecture

19. – Dance movements embodiment based on a conjunction of rhythmic and sensorial events. Too Close OK Colour Event Red Green Yellow Blue Undef Yellow Red Green Blue Undefined Proximity Defined Dance Movement Rhythmic Event Medium Soft Strong Walk Waiting Events... STOP (Turn) 15 System Architecture Robot Control Module

20. Thread 4: Dance Movements (Output Processing) Thread 4: Dance Movements (Output Processing) Thread 5: Brick 1 BT Communication Thread 5: Brick 1 BT Communication Thread 6: Brick 2 BT Communication Thread 6: Brick 2 BT Communication Thread 3: GUI + Asynchronous TCP/IP Server Thread 3: GUI + Asynchronous TCP/IP Server Robot Control Module Human Control Module Thread 1: Rhythmic Perception Model Thread 2: Synchronous TCP/IP Client Music Analysis Module  Multithreading Architecture Human Control Module 16 System Architecture

21. Robot Parameters Audio Input (14) Dance Movements List  Dance Composition  Control Panel Audio Parameters Event Visualization Connection Status (4) Velocity List 16 System Architecture Human Control Module

22. Experiments and Results Rhythmic Perception Model Regulation – Adaptive thresholding parameters analysis a) PN (Pitched Non-Percussion) excerpt using thres1 = 0.30; thres2 = 0.50; thres3 = 0.75. b) PP (Pitched Percussion) excerpt using thres1 = 0.35; thres2 = 0.50; thres3 = 0.75. d) CM (Complex Mixtures) excerpt using thres1 = 0.25; thres2 = 0.45; thres3 = 0.60. c) NP (Non-Pitched Percussion) excerpt using thres1 = 0.30; thres2 = 0.40; thres3 = 0.75. 17

23. Experiments and Results Robot Dancing Performance Synchronism Realism Multi-threading architecture in a single processor. Lacks in the order of 1s Complexity of the output decision process, in the form of dance movements. Bluetooth communication overflow. Autonomous and reactive behaviour. The robot, through its dance motion, experiences the world directly, and this world directly influences its behaviour, through sensation Dynamism Great variety of possible dance style compositions, in a total of 15 12 -1 choreographies. Robot’s perambulation around the dance environment. System behaviour conducted by human decision. 18

24. Demonstrations 19

25. Conclusions Limitations: – Synchronism limited by the Bluetooth communication system and due to the multithreading architecture in a single processor; – Dynamism limited by the robot’s degrees of freedom, dependent of the number of supported motors (a maximum of 3 for each NXT Brick); – Realism dependent from the former two factors: Reactivity dependent of synchronism Autonomy enforced by dynamism. 20 Flexibility Merits Interactivity Reactivity Autonomy Adaptability Easiness

26. Future Work 22 High-Level Rhythmic Perception Models Spatial Rhythmic Perception Models Multi-Robot Cooperative Dancing Individual dance Low-Level Rhythmic Perception Models Flexible Interactive GUI Cross-Modulation

27. Future Work 23

28. 28