1. Music and Audio Computing I 216-306A
Prof. Marcelo M. Wanderley
Weeks 12 and 13

2. 306A - Music and Audio Computing
Interactive systems
This week's topic:
Gestural control of real-time multi-parametric systems.
New digital musical instruments (where the computer generates sound)
Interactive sound/video installations
Extrapolation to other types of systems (not musical)
Nov week 13
306A - Music and Audio Computing

3. 306A - Music and Audio Computing
Interactive Systems
Interactive systems becoming increasingly popular (installations, music and dance)
Computers provide high computational power -> real-time processing
Transducer technologies easily available
Nov week 13
306A - Music and Audio Computing

4. 306A - Music and Audio Computing
Terminology
Interaction: mutual or reciprocal action or influence.
v. Interact: act upon one another.
Interactive System:
A system that allows mutual action or influence between a user and a computer.
Nov week 13
306A - Music and Audio Computing

5. Terminology - Music applications
Music - some examples:
Interactive Musical Systems: Interactive computer music systems are those whose behavior changes in response to musical input.
Interactive vs. Reactive systems Level of interaction
Nov week 13
306A - Music and Audio Computing

6. 306A - Music and Audio Computing
Instrumental Model
A computer-based musical instrument model
Nov week 13
306A - Music and Audio Computing

7. A Technological Analysis
Basic features from a technological point of view:
Types of sensors - existing ones or new designs;
Sensor characteristics
Analysis/Conversion techniques for sensors outputs;
Input Device Design
Mapping strategies between control variables and sound generation;
Sound synthesis methods.
Nov week 13
306A - Music and Audio Computing

8. A Technological Analysis
Sensors and transducers:
transducer (principle); sensor (device)
Sensor characteristics:
sensitivity, stability and repeatability
Analog-digital conversion (usually MIDI, but not necessarily)
Various analog-to-MIDI interfaces
Inputs, outputs, speed, portability/size, price, etc.
Nov week 13
306A - Music and Audio Computing

9. A Technological Analysis
Mapping of gesture parameters to sound parameters
What is mapping? - Context dependent
Mathematics: to assign to every element (of a set) an element of the same set or another set.
Mapping Strategies?
Simple (one-to-one) mappings? Neural networks? Complex strategies?
Nov week 13
306A - Music and Audio Computing

10. A Technological Analysis
Sound Synthesis:
Signal models: try to reproduce the sound (perceptual) characteristics or are based on abstract models (mathematics, etc.)
Physical models: try to reproduce the physical characteristics of the mechanisms that produced the sound.
Nov week 13
306A - Music and Audio Computing

11. Instrumental Interaction in Music
Focus on computer-based musical instruments (digital musical instruments)
Based or not on traditional instruments
Expert interaction
Manipulation (mostly) - tactile/force feedback
Expressive intent
Nov week 13
306A - Music and Audio Computing

12. 306A - Music and Audio Computing
Instrumental Model
Feedback
Primary/passive feedback
Visual, auditory and tactile-kinesthetic (tactual)
Secondary/active feedback
Auditory/force feedback
Nov week 13
306A - Music and Audio Computing

13. 306A - Music and Audio Computing
Performer Gestures
Analysis of performer gestures in three levels:
Effective gesture - necessary to produce the sound
Accompanist gesture - associated body movements
Figurative gesture - without direct correspondence to a physical movement
Nov week 13
306A - Music and Audio Computing

14. 306A - Music and Audio Computing
Gestural Acquisition
In instrumental music:
Direct acquisition: one captures the actual physical gesture or movement through the use of sensors
Indirect acquisition: information obtained from the analysis of the instrument’s sound
Nov week 13
306A - Music and Audio Computing

15. Interfaces Analog - MIDI
Different models (Sensorlab, i-Cube, AtoMIC Pro, etc.)
Nov week 13
306A - Music and Audio Computing

16. 306A - Music and Audio Computing
Gestural Controllers
Instrument-like controllers: profit from highly developed (expert) gestures.
Instrument-inspired controllers - SuperPolm
Alternate controllers: design emphasis on capturing “natural” gestures - opposed to a learned gestural vocabulary - Hands
Hybrid controllers  Intermediate options! Extended instruments - hyperinstruments
Nov week 13
306A - Music and Audio Computing

17. 306A - Music and Audio Computing
Controllers
Instrument-like controller:
Saxophone MIDI - YamahaWX7
Nov week 13
306A - Music and Audio Computing

18. Instrument-inspired controller
Goto - Superpolm
“Violin-inspired” controller that intends to capture part of the gestural vocabulary of a violin player in order to control other sounds (e.g. granular synthesis)
Nov week 13
306A - Music and Audio Computing

19. Alternate controllers
Example of an alternate controller:
Graphic drawing tablet - WACOM
Nov week 13
306A - Music and Audio Computing

20. Alternate Controllers
Example:
The Hands (M. Waisvisz - STEIM)
Nov week 13
306A - Music and Audio Computing

21. 306A - Music and Audio Computing
Input Device Design
Choice of input device for sound synthesis control
Design principles:
idiosyncratic (musical !?)
design methodology (engineering !?)
Nov week 13
306A - Music and Audio Computing

22. 306A - Music and Audio Computing
Mapping
Different points of view:
Mapping is part of a composition
- idiosyncratic
Mapping is part of the instrument
- affects instrument effectiveness
Nov week 13
306A - Music and Audio Computing

23. 306A - Music and Audio Computing
Mapping
Electronic musical instruments
Input device is independent from the sound synthesis device.
No implicit mapping of one to the other
Too often: one single control device corresponding to a single synthesis parameter.
 one-to-one mapping.
Nov week 13
306A - Music and Audio Computing

24. What do I mean by mapping ?
An electronic musical instrument model:
Mapping is the liaison or correspondence between control parameters (derived from performer actions) and synthesis parameters.
Nov week 13
306A - Music and Audio Computing

25. 306A - Music and Audio Computing
Mapping
Mapping between controller outputs and synthesis inputs
One-to-one;
Convergent (many-to-one);
Divergent (one-to-many);
 Any mixture of the above (many acoustic instruments)
Nov week 13
306A - Music and Audio Computing

26. 306A - Music and Audio Computing
Mapping
Acoustic instruments:
Sound generation device is inseparable from the human-control device.
Yielding complex control relationships between human performers and their instruments
Nov week 13
306A - Music and Audio Computing

27. Simulation of traditional instrument performance
Clarinet performance using a MIDI controller - Yamaha WX7 and additive synthesis.
Clarinet
 Reed behavior
 Gestures are not independent
 Embouchure is always needed
Nov week 13
306A - Music and Audio Computing

28. Simulation of traditional instrument performance
WX7: independent outputs
- fingering
- breath pressure
- lip pressure
Clarinet reed behavior modeled in jMax to simulate the interdependence of controller parameters.
Nov week 13
306A - Music and Audio Computing

29. Simulation of traditional instrument performance
Simple and complex mappings - pedagogy
Nov week 13
306A - Music and Audio Computing

30. Simulation of the effect of performer movements
Work with Philippe Depalle here at McGill!
Clarinet movements may influence the sound recorded by a close microphone.
Modeling of this effect -> Phasing!
Real-time model in Max/MSP.
Nov week 13
306A - Music and Audio Computing

31. Simulation of the effect of performer movements
Example of phasing (flanging) effect:
Nov week 13
306A - Music and Audio Computing

32. Simulation of the effect of performer movements
Example of the real-time simulation
Nov week 13
306A - Music and Audio Computing

33. 306A - Music and Audio Computing
General Conclusions
Different aspects of interactive systems
Input from other fields - HCI, ergonomics, experimental psychology, etc.
Need of a general corpus of knowledge:
Trends in Gestural Control of Music !
ICMA/EMF working group/Gesture Research in Music Homepage
Nov week 13
306A - Music and Audio Computing

34. Web addresses Gesture Research in Music homepage:
Working Group ICMA/EMF