Music and Audio Computing I 216-306A Prof. Marcelo M. Wanderley mwanderley@acm.org Weeks 12 and 13

306A - Music and Audio Computing Interactive systems This week's topic: Gestural control of real-time multi-parametric systems. New digital musical instruments (where the computer generates sound) Interactive sound/video installations Extrapolation to other types of systems (not musical) Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Interactive Systems Interactive systems becoming increasingly popular (installations, music and dance) Computers provide high computational power -> real-time processing Transducer technologies easily available Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Terminology Interaction: mutual or reciprocal action or influence. v. Interact: act upon one another. Interactive System: A system that allows mutual action or influence between a user and a computer. Nov 29 01 - week 13 306A - Music and Audio Computing

Terminology - Music applications Music - some examples: Interactive Musical Systems: Interactive computer music systems are those whose behavior changes in response to musical input. Interactive vs. Reactive systems Level of interaction Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Instrumental Model A computer-based musical instrument model Nov 29 01 - week 13 306A - Music and Audio Computing

A Technological Analysis Basic features from a technological point of view: Types of sensors - existing ones or new designs; Sensor characteristics Analysis/Conversion techniques for sensors outputs; Input Device Design Mapping strategies between control variables and sound generation; Sound synthesis methods. Nov 29 01 - week 13 306A - Music and Audio Computing

A Technological Analysis Sensors and transducers: transducer (principle); sensor (device) Sensor characteristics: sensitivity, stability and repeatability Analog-digital conversion (usually MIDI, but not necessarily) Various analog-to-MIDI interfaces Inputs, outputs, speed, portability/size, price, etc. Nov 29 01 - week 13 306A - Music and Audio Computing

A Technological Analysis Mapping of gesture parameters to sound parameters What is mapping? - Context dependent Mathematics: to assign to every element (of a set) an element of the same set or another set. Mapping Strategies? Simple (one-to-one) mappings? Neural networks? Complex strategies? Nov 29 01 - week 13 306A - Music and Audio Computing

A Technological Analysis Sound Synthesis: Signal models: try to reproduce the sound (perceptual) characteristics or are based on abstract models (mathematics, etc.) Physical models: try to reproduce the physical characteristics of the mechanisms that produced the sound. Nov 29 01 - week 13 306A - Music and Audio Computing

Instrumental Interaction in Music Focus on computer-based musical instruments (digital musical instruments) Based or not on traditional instruments Expert interaction Manipulation (mostly) - tactile/force feedback Expressive intent Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Instrumental Model Feedback Primary/passive feedback Visual, auditory and tactile-kinesthetic (tactual) Secondary/active feedback Auditory/force feedback Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Performer Gestures Analysis of performer gestures in three levels: Effective gesture - necessary to produce the sound Accompanist gesture - associated body movements Figurative gesture - without direct correspondence to a physical movement Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Gestural Acquisition In instrumental music: Direct acquisition: one captures the actual physical gesture or movement through the use of sensors Indirect acquisition: information obtained from the analysis of the instrument’s sound Nov 29 01 - week 13 306A - Music and Audio Computing

Interfaces Analog - MIDI Different models (Sensorlab, i-Cube, AtoMIC Pro, etc.) Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Gestural Controllers Instrument-like controllers: profit from highly developed (expert) gestures. Instrument-inspired controllers - SuperPolm Alternate controllers: design emphasis on capturing “natural” gestures - opposed to a learned gestural vocabulary - Hands Hybrid controllers  Intermediate options! Extended instruments - hyperinstruments Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Controllers Instrument-like controller: Saxophone MIDI - YamahaWX7 Nov 29 01 - week 13 306A - Music and Audio Computing

Instrument-inspired controller Goto - Superpolm “Violin-inspired” controller that intends to capture part of the gestural vocabulary of a violin player in order to control other sounds (e.g. granular synthesis) Nov 29 01 - week 13 306A - Music and Audio Computing

Alternate controllers Example of an alternate controller: Graphic drawing tablet - WACOM Nov 29 01 - week 13 306A - Music and Audio Computing

Alternate Controllers Example: The Hands (M. Waisvisz - STEIM) Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Input Device Design Choice of input device for sound synthesis control Design principles: idiosyncratic (musical !?) design methodology (engineering !?) Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Mapping Different points of view: Mapping is part of a composition - idiosyncratic Mapping is part of the instrument - affects instrument effectiveness Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Mapping Electronic musical instruments Input device is independent from the sound synthesis device. No implicit mapping of one to the other Too often: one single control device corresponding to a single synthesis parameter.  one-to-one mapping. Nov 29 01 - week 13 306A - Music and Audio Computing

What do I mean by mapping ? An electronic musical instrument model: Mapping is the liaison or correspondence between control parameters (derived from performer actions) and synthesis parameters. Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Mapping Mapping between controller outputs and synthesis inputs One-to-one; Convergent (many-to-one); Divergent (one-to-many);  Any mixture of the above (many acoustic instruments) Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing Mapping Acoustic instruments: Sound generation device is inseparable from the human-control device. Yielding complex control relationships between human performers and their instruments Nov 29 01 - week 13 306A - Music and Audio Computing

Simulation of traditional instrument performance Clarinet performance using a MIDI controller - Yamaha WX7 and additive synthesis. Clarinet  Reed behavior  Gestures are not independent  Embouchure is always needed Nov 29 01 - week 13 306A - Music and Audio Computing

Simulation of traditional instrument performance WX7: independent outputs - fingering - breath pressure - lip pressure Clarinet reed behavior modeled in jMax to simulate the interdependence of controller parameters. Nov 29 01 - week 13 306A - Music and Audio Computing

Simulation of traditional instrument performance Simple and complex mappings - pedagogy Nov 29 01 - week 13 306A - Music and Audio Computing

Simulation of the effect of performer movements Work with Philippe Depalle here at McGill! Clarinet movements may influence the sound recorded by a close microphone. Modeling of this effect -> Phasing! Real-time model in Max/MSP. Nov 29 01 - week 13 306A - Music and Audio Computing

Simulation of the effect of performer movements Example of phasing (flanging) effect: http://www.harmony-central.com/Effects/effects-explained.html Nov 29 01 - week 13 306A - Music and Audio Computing

Simulation of the effect of performer movements Example of the real-time simulation Nov 29 01 - week 13 306A - Music and Audio Computing

306A - Music and Audio Computing General Conclusions Different aspects of interactive systems Input from other fields - HCI, ergonomics, experimental psychology, etc. Need of a general corpus of knowledge: Trends in Gestural Control of Music ! ICMA/EMF working group/Gesture Research in Music Homepage Nov 29 01 - week 13 306A - Music and Audio Computing

Web addresses Gesture Research in Music homepage: http://www.ircam.fr/gesture Working Group ICMA/EMF www.kcng.org/wg/