Musical Toys: Interactive Audio for Non-Musicians Oded Ben-Tal Musical Toys: Interactive Audio for Non-Musicians

Background I Leverhulme Trust's artist in residence programme Bringing an artist into educational institutions where art isn't a normal part of curriculum. Encouraging collaborations between artists and... Main project: interactive installations to be placed in public areas of school buildings. Interactive Multimedia performance.

Aims Showcasing creative use of technology to students. Opportunity for staff/students to explore novel sounds. Encourage creative exchange of ideas. Communicate aspects of research at the school outside formal teaching.

Background II Previous work with Pd Anemoi for flute and live electronics; Still Life for trumpet and live electronics Creating a decision making system that listens to the input and responds by making simple discriminations about the it (e.g. high/low long/short)

Still Life Trumpet; soundfiles; live responses Sound files (~30'-300') created in CLM and triggered manually during the performance Pd patch discriminates between 3 types of playing by the trumpet: long notes, rapid passages, and even pulse. Responding in kind with generated sounds (i.e. not by transforming the input)

Context Audience of non-musicians (Engineering school). Each installation running for 2-3 weeks in a public space (hallway, corridor, etc.). Most people would spend <5 minutes. Using hardware available at the school.

Technology Infusion Systems sensors: distance, light for installations; accelerometers, light with wireless blueMIDI transmitters in performance. Installations based on Mac(Intel) computer. PlanetCCRMA FC5 (ThinkPad) used in performance and some of the development. PD/Gem + CM/CLM

Setup

“on location“ Speakers Sensor

Responsive system Easy to learn correlation between input and output. In a short time. Robust. Producing interesting results from simple inputs. A toy more than a musical instrument.

Using single sensor Because I'm a newbie to both sensors and audio-visual installations. Simple for intended users to master in a short time. Allows for a focused investigation. Distance sensor was most reliable and effective.

Mapping Mapping full range (linear or not) Dividing range (~2 sensors in one) Calculating change in input: magnitude (variance) or rate (derivative). Event detection or trigger points.

Installation #1 Photography from 4 cities around the globe (by Dr. Vanja Garaj) with sounds recorded in London. Sequence of photographs each rendered in 3 layer, with partial transparency creating a 3D effect. Short soundfiles of street noises filtered through a set of bandpass filters. In rest mode photographs running in a loop with no sound.

Installation #1 Full range mapped to (1) amplitude (2) tilt angle of photos. Range is divided such that top/bottom rotate photo (Y axis) counter/clockwise. Range is also divided into 4 possible filter settings derived from vowel approximations. Large movement triggers random soundfile, and advances to next photograph.

First lesson The sensor concept wasn't clear enough. Many people thought the slide sequence was the whole installations and didn't try to interact. Even though sound was only activated occasionally (when the sensor was activated) people in neighbouring offices complained about the noise.

Installation #2 Distance sensor activating: FM-synthesis instrument. Bandpass filter with centre frequency set to an overtone of the carrier frequency. Audio parameters (amp, freq,...) are visualized. While: Bird-like sounds activated by light sensor or clock, which also control: Visual illustration of distance sensor.

Installation #2 Direct mapping of range to amplitude, modulation index, and filter bandwidth. Range divided into categories for determining filter frequency and modulation frequency. Derivative (DPCM) mapped to carrier frequency change (i.e. Interval up/down from current note).

Second lesson Good placement right inside building entrance: plenty of people walking by but away from offices. Light sensor isn't very effective in a typical British winter. When screen was blank most people assumed nothing was working. Nevertheless many students found the installation entertaining.

Conclusions Pd is robust and reliable. (Gem somewhat less so) A responsive interface is important, but getting people to interact requires more thought about the overall design and context. Using hand movement (velocity, acceleration) rather than position seems lead to a more responsive interface. (feedback through internal monitoring is better? Inherent correlation between movement and music?)

Multimedia performance Collaboration with Dans Sans Joux (team of 14 people led by Johannes Birringer). Based on Film (and Novel) 'Woman in the Dunes'. Dance, video, animation, set design, music Wireless sensors incorporated into garments designed by Michele Danjoux.

Concept Using 'intelligent garments' in performance to allow improvising dancers to interact with their digital environment. Produce an innovative, artistic performance of the highest quality we can.

Development Bi-weekly work- meetings during winter/spring 2007. Followed by intense 2- week workshop in Göttelborn (Internationales Interaktionslabor)

Technology Infusion system sensors with wireless transmitters, using BlueMIDI to send data into PD/Gem Isadora. Projecting video & animation clips onto set through 3 projectors. Electronic music.

Challenges Technological: wireless version of sensor system proved highly unstable. Personal: integrating individual contributions into a coherent whole. Musical: composing music in an uncertain (and constantly changing) context.

Uncertainty? What happens on stage (dance/video) with the music? How long will a section be? What comes before? After? Create independent modules which can than be linked (transitions?).

Elements Pd modules generating texture, with possibility to manipulate some aspect(s) e.g. density, volume, etc. Composed sections (using CLM), 1-5 minutes long, triggered during the performance in coordination with dancers.

Module: 'phrase builder' Melody based on {amplitude, duration} pairs generated through a markov-like process. Other parameters are derived from {amp, dur} through decision trees and mapping (but also some randomness): amplitude envelope, vibrato frequency, periodic and random vibrato amplitude. All applied to a simple [~osc].