Music Coloring for Biofeedback Presented By Eyal Balla

The Demo System EKG Monitor PC with A/D and music coloring software Patient Original Music Colored Music

Music Coloring What is it? Music coloring is changing of music in different aspects (such as pitch, rhythm, timber, volume, reverberation and other aspects of the music) in order to embed data into the music.

Music Perception in Humans Recent research shows that several distinct neural functions exist in music perception:  Volume  Pitch  Rhythm  Timber  Direction of source of music This introduces the possibility of multidimensional display with audio. Moreover music has been shown to have effect on human breathing and heart rate

Music Perception in Humans Auditory DisplayVisual Display Able to percept many dimensions such as pitch, rhythm, timber etc. Number of dimensions is limited to 3D Uses partial attention. Other tasks can be done simultaneously. Uses full attention of human using it.

Previous Works in Auditory Display Sonification Mappings:  sMax: A Multimodel Toolkit for Stock Market Data Sonification. Maps stock market data into multi- streams of audio. User has control of mapping. Requires training. Mapping is through pitch, pulse speed, brightness and attack time of different instruments  MarketBuzz. A sonification of real-time financial data. 4 behaviors were identified and sonified in beeps and pitch sonifications. Requires training.

Previous Works in Auditory Display  Polyrhythm in the Human Brain. Sonification of EEG brain waves in four different streams according to EEG wave types.  Listening To the Mind listening. Sonification of ECG heart rate and two streams of EEG with difference between the two temporal sections (above ears) were used as stereo positioning.

Previous Works in Auditory Display Feedback Systems:  Auditory Feedback of Human EEG for Direct Brain Computer Communication. A spelling machine for the handicapped was built. User would use SCP EEG brain waves to control it. Learning was through auditory instruction. They also found other EEG waves controlable using multi-stream audio teaching.

Previous Works in Auditory Display

 Data Analysis through Auditory Display: Applications in Heart Rate Variability. The work deals with creating a auditory system for heart rate variability analysis. The auditory display created is made of several streams each representing a different aspect of heart condition. The work mentions the importance of using pitch and volume.

Conclusion of Current Works Several orthogonal features of music have been proved as distinguishable by humans in experiments Sonification is a good way of displaying multidimensional data. Works usually create new music according to data streams extracted from the raw data.

Current Directions Last year’s workshop created an interface which inserts samples of EKG into MAX/MSP environment using a A/D sampler. The workshop showed that online real- time analysis of this data is possible using the MAX/MSP development environment.

Current Directions We are looking for auditory “Axis” which can be precept orthogonally. We are using already build components which color music to learn the reaction graph of humans to change in different music features.

References The structural Components of music perception. Herve´ Platel, Cathy Price, Jean-Claude Baron, Richard Wise, Jany Lambert, Richard S. J. Frackowiak, Bernard Lechevalier and Francis Eustache Max/Msp 4.5 Documentation. sMax: A Multimodel Toolkit for Stock Market Data Sonification. Fabio Cifariello Ciardi MARKETBUZZ: Sonification of real-time financial data. Petr Janata and Edward Childs

References Polyrhythm in the Human Brain. Thomas Hermann, Gerold Baier, Markus Muller. Listening To the Mind listening. Hans van Raaij Listening to the mind listening concert. Auditory Feedback of Human EEG for Direct Brain Computer Communication. Thilo Hinterberger, Gerold Baier, Jürgen Mellinger, Niels Birbaumer Data Analysis through Auditory Display: Applications in Heart Rate Variability. Mark Ballora

MAX/MSP Introduction MAX/MSP is an object oriented development environment for music synthesis and creation. The MAX part of the environment handles MIDI format manipulation. The MSP part of the environment handles signal processing and manipulation.

MAX/MSP Example The following example outputs via audio several 2 dimension data items using 2 streams of data for pitch and volume.