IST MEGA Multisensory Expressive Gesture Applications Concertation Meeting Brussels, 3-4 October 2000 Antonio Camurri (DIST-University of Genoa) MEGA Project Coordinator

Summary MEGA: Concepts, Objectives and Mission The Consortium Project Overview: Workplan, WPs, and Deliverables

MEGA: Concepts, Objectives and Mission Concepts : –Multisensory –Expressive (emotion, kansei, affect) –Gesture –Applications

MEGA: Concepts, Objectives and Mission Objectives: –modelling and communication of expressive and emotional content in non-verbal interaction by multi-sensory interfaces –design of novel paradigms of interaction –analysis, synthesis and communication of expressive content in networked environments –implementation of the MEGA System Environment –development of artistic public events showing the usefulness, flexibility and power of the MEGA System Environment

MEGA: Concepts, Objectives and Mission Mission: –Consolidating a third original European route in affective computing (USA) and KANSEI (Japan). –Raise the competitiveness of European Industry on Digital Music Instruments Entertainment and Edutainment Multimedia: e.g. VEs, e-commerce, intelligent interfaces,... by building the MEGA System Environment and showing its effectiveness in public events.

Partic ipant no. Participant nameParticipant short name CountryStatus* 1DIST - University of Genoa Lab. on Musical Informatics DISTIC 2DEI - University of PadovaDEIIP 3IPEM – University of GentIPEMBP 4KTH - Royal Institute of Technology, Stockholm KTHSP 5Uppsala University, Department of Psychology UUPSP 6TelenorTLNNOP 7GeneralmusicGEMIA 8Consorzio Pisa RicercheCPRIP 9EidomediaEMIA Participants

Workplan: Workpackages WP1 Project management WP2 Requirements WP3 Analysis in human movement and gesture WP4 Analysis in musical gesture performance WP5 Synthesis in visual media and movement WP6 Synthesis in music performance WP7 MEGA System Environment WP8 MEGA Multimedia Performance WP9 Dissemination and implementation

Workplan: gantt - Spiral approach - Three phases, each concluded at a project milestone (13, 25, 36 months).

WP2 Requirements WP Leader: IPEM State of the art, share of results among partners, Requirements and setup for the research, the MEGA System Environment and the MEGA Multimedia Performance products. Definition of a standard communication protocol for expressive data. The protocol is defined in this WP and is implemented in the WP7 (MEGA System Environment). Contributes from all the partners. D3 (month 6) Requirements D9 (month 14) First requirement revision D12(month 26) Final requirements

WP3-6: Research on analysis and synthesis of expressivity WP3 Analysis in human movement and gesture (DIST) WP4 Analysis in musical gesture performance (DEI) WP5 Synthesis in visual media and movement (EM) WP6 Synthesis in music performance (KTH) Results: a selection of techniques and algorithms, integrated as s/w modules in the MEGA System Environment (WP7). These developed tools and modules will be continuously integrated and updated in the EyesWeb architecture, i.e., WP3-6 will deliver tested library modules and associated manual pages using the standard adopted in the project Deliverables: Reports (D5, D13), S/w tools (D6, D14)

WP3 Analysis in human movement and gesture WP Leader: DIST To extract expressive content from movement patterns. To evaluate and compare the results and performance of the obtained methods and algorithms (DIST and KTH) with the results obtained by psychological analysis of movement (UUP) with selected groups of users, both non-expert and expert (dancers, choreographers). To develop a project reference archive of microdances and movement patterns (DIST, KTH, UUP), for both psychological analysis and automated analysis, using the standard adopted in the project (audiovisual, mocap data).

WP4 Analysis in musical gesture performance WP Leader: DEI  Analysis of the different cues characterising an expressive musical performance (UUP, DEI);  Analysis of the correlation between cues and the instrument played by the performer (DEI);  Development of a model to extract automatically an expressive content of a musical performance (IPEM);  Development of tools implementing the model (DEI, IPEM);  Evaluation and test of the effectiveness of the model by means of performances analysis with the implemented tools (KTH, UUP);

WP5 Synthesis in visual media and movement WP Leader: EM Techniques inspired to the audiovisual language in cinema (EM) and their utilisation in virtual environments (TNR) and in visual and robot-human interaction (DIST). Development of models and methods to synthesise expressive visual media outputs (EM). Expressive synthesis of movements of synthetic characters and expressive modulation of VEs and Mixed Reality envs (TNR), depending on the expressive communication goals and content. Expressive movement of small robots in concrete apps on robot- human interaction and mobile scenery (DIST). Integration with Dovre VE architecture (TNR).

WP6 Synthesis in music performance WP Leader: KTH Development of tools both in MIDI and in digital audio format (KTH, DEI), to be used for the project products (WP7-8). Synthesis on MIDI format based on Director Musices (KTH) Adapting and integrating audio electronics for real-time expressive processing (GEM)

WP7 MEGA System Environment WP Leader: DIST Deliverable D4: EyesWeb software release, including tutorial and reference manuals (month 6) Deliverable D7: 1st version of MEGA Sys Env. (12) Deliverable D10: 2nd version of the MEGA Sys Env. (24) Deliverable D15: Final product - MEGA Sys Env (35) DIST: EyesWeb and architectureCPR: Wireless Sensors TNR: DovreIPEM: Mapping Strategies, Eval EM: Wireless Sensors

MEGA System Environment (WP7) Main components : -the EyesWeb software architecture (from DIST), -the (open) libraries of software modules concerning analysis and synthesis of expressive gesture (WP3-6), -the implementation of the protocol for expressive communication (WP2), -the h/w electronics and sensor systems for wireless non- invasive real-time motion capture (adapted from existing systems by CPR, DIST, EM), integration of commercial sensor systems (e.g., radio baton) and audio electronics systems (by GEM).

WP8 MEGA Multimedia Performance WP Leader: IPEM Main components: -the specification of the artistic use of the MEGA System Environment (WP7) in terms of the expressive content analysis and synthesis (IPEM, DEI); -the support to specific multimedia artistic events, based on the MEGA System Environment (DIST, KTH); -the evaluation of the MEGA System Environment with respect to audience acceptance, effectiveness, usability, and flexibility (IPEM).

WP9 Dissemination and implementation WP Leader: TNR To set up the Liason Board To disseminate products (WP7-8) and research results (Workshops, book, Frankfurt and other exhibits, etc). Internet-based dissemination and exploitation strategy by a free distribution to users of the software (under certain constraints and on-line contract forms), To enable external users to develop new “plugins” of the MEGA System Environment. Evaluate (UUP) the results from the project from WP7-8 and the internet dissemination strategy, with respect to the planned outputs. D2: Dissemination and Use plan (6) D17: Technological Implementation Plan (36)

Dissemination (WP8 and WP9) via public performances (WP8) An internet site providing the software and support to users (WP9): –s/w: the full MEGA System Environment, incl. a Wizard enabling users to develop new system plugins –library of plugins deriving from WP3-6 (research) –full documentation in the standard adopted Workshops, special tracks and sessions at IEEE CS, ACM Conf, at CEC Workshops, at Frankfurt Messe 2001 and/or 2002 (WP9)