Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices ATTEMPTS to REPRODUCE a PIANIST’S EXPRESSIVE TIMING with DIRECTOR MUSICES PERFORMANCE RULES by JOHAN SUNDBERG, ANDERS FRIBERG and ROBERTO BRESIN - BERAN PACACI - SPRING 2004 ISE 599: Spring 2004

Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Johan Sundberg Ph.D. in musicology at Uppsala University. was the head of the music acoustics research group from 1970 to 2001. a music performer as a singer. Current research areas include Singing Voice Rules for Musical Performance Breathing and Phonation http://www.speech.kth.se/~pjohan He has a case study called: Why is ugly ugly? Singers may have ugly voices. He analysis different singers’ voices. What is the problem with that? ISE 599: Spring 2004

Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Anders Friberg M.S. in Applied Physics. Ph.D. in ? a music performer as a pianist. Current research areas include Random fluctuations of timing in music performance. Swing ratios and ensemble timing in jazz. Combining the Radio Baton and Director Musices. Performance synthesis in Director Musices. http://www.speech.kth.se/~andersfr/ Combining the Radio Baton and Director Musices. The Radio Baton was developed by Max Mathews, Stanford University and is an controller device with two sticks that are sensitive to 3D position over a bottom plate. A midi score can be "conducted" by  beating the pulse with one stick and controlling dynamics with the other stick. The midi score has been preprocessed with Director Musices in order to model the musicians contribution to the performance. Since the Radio Baton controls the overall tempo and dynamics, Director Musices is used for the micro-level control. See a performance at the PEVOC 2001 opening session with  Malena Ernman, voice and Jan Risberg, Radio Baton Video (5Mb). Together with Johan Sundberg, Max Mathews, Craig Sapp and Gerald Bennett. Performance synthesis in Director Musices. Once in a while I am improving the Director Musices program with a new rule or feature. ISE 599: Spring 2004

Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Roberto Bresin Ph.D. in Music Acoustics. Current research areas include Analysis and synthesis of emotional content in music. Articulation strategies in expressive music performance. Emotional melodies for mobile phones. Synthesis of Emotional Expression in Music Performance. http://www.speech.kth.se/~roberto ISE 599: Spring 2004

KTH Music Acoustics Group Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices KTH Music Acoustics Group runs in four main streams: singing (solo and choral) stringed instruments music performance computer music composition While running in parallel for periods, they often merge and interact. An experimental approach has long been a signature of the research. In particular, experimental work with "analysis-by-synthesis" has proved to be rewarding. Today, a strong trend can be seen toward more sophisticated synthesis strategies, based on a direct mathematical description of the instrument/voice. This strategy, commonly referred to as numerical or "physical modeling," would be generally adopted within the next five years or so at most major synthesizer laboratories. We have started physical modeling of the singing voice in terms of articulator synthesis, controlled by physiologic control parameters such as jaw opening and tongue shape. The increased interest for numerical modeling has revealed a lack of reliable and detailed experimental data in many areas. This applies in particular to the function of the control systems used in actual performances, for example breathing behavior in singers and wind instrumentalists, or bow movement in string playing. The prevailing lack of thorough knowledge of such control systems will force the experimenter to explore an overwhelmingly rich parameter space, and the chance of real success in reasonable time is small. Our research on musical performance by computer modeling is a long term project. Its focus is presently to compare characteristics of locomotion in running with timing in music performance. Furthermore, our generative music performance grammar and music examples illustrating the effects on synthesized performances, has been made accessible on the internet. Our overall aim is to continue the established experimental tradition in the study of the voice and the stringed instruments (now needed more than ever, it seems), while striving to take part in the development of the new synthesis methods. As before, the parallel work on musical performance bridges several of these tasks. ISE 599: Spring 2004

Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Music Performance is an important research area today. The deviation from nominal inter-onset-interval (IOI). Every musician plays differently. Musicians play differently. How can you distinguish a good musician’s performance from others? You can express your emotions in music. Fear, anger, happiness, sadness, tenderness, solemnity can be expressed with music. The deviation from nominal IOI gives an idea about the music performance. If we define expressive performance as “deviation from the score”, then different performances differ in the way and extent the artist “deviates” from the score, i.e., from a purely mechanical/flat rendition of the piece, in terms of timing, dynamics, and articulation. In order to be able to compare performances of pieces or sections of different length, we need to define features that characterize and quantify these deviations at a global level, i.e., without reference to individual notes and how these were played. ISE 599: Spring 2004

Music Performance Rules Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Music Performance Rules The rules are divided into three main classes: Differentiation Rules: enhance the differences between tone categories. Grouping Rules: show what tones belong together. Emphasis Rules: emphasize unexpected notes. Ensemble Rules: synchronize the various voices in an ensemble. Differentiation Rules: increase the differences between tone categories such as pitch classes, intervals, and note values. Grouping Rules: mark which tones belong together and where the structural boundaries are. Think about Phrase boundaries. Emphasis Rules: emphasize unexpected notes. Ensemble Rules: synchronize the various voices in an ensemble. I’ve found this in “AI and Music” article. Quiet interestingly, these rules are applied in speech also. For example: The slowing down towards the end of a phrase. ISE 599: Spring 2004

Music Performance Rules Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Music Performance Rules Differentiation Rules Duration Contrast Melodic Charge High Sharp Grouping Rules Punctuation Double Duration Tuning Phrase Arch Inégales (or swing) Ensemble Swing Final Ritard Harmonic Charge ISE 599: Spring 2004

Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Director Musices (DM) a software for Automatic Music Performance. a rule system for Musical Performance. implementing all previously defined rules. features includes polyphony, midi I/O, performance variable graphs and user rule definition. Automatically introduces expressive deviation in performances of input score files. DM system incorporates rules for tempo, dynamic, and articulation transformations constrained to MIDI. These rules are inferred both from theoretical musical knowledge and experimentally from training, especially using analysis-by-synthesis approach. The DM system has been found to be capable also of adding various emotional colors to a performance. This result was obtained by varying the quantity parameter of various rules. The DM system has obviously some limitations. If you apply a combination of rules to a piece, it only improves the musical quality of that performance. The general applicability of the rules is doubtful. ISE 599: Spring 2004

Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Director Musices (DM) DM rules contain two elements: Context Quantity Context: defines in what context the rule should be applied. Quantity: degree of effect on the performance. (states how great effects it should produce in the performance.) The quantity parameter k determines how much duration should be transferred. By choosing different quantity values, different performances of the same piece are obtained. ISE 599: Spring 2004

Director Musices (DM)

Research Strategy Analysis-by-synthesis Analysis-by-measurement

Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Experiment Outline Comparison between the real pianist’s performance with different performances produced by the DM system. MIDI data consists of score time in beats, MIDI note numbers, IOI, sounded duration and dynamics in MIDI velocity. They have two MIDI data. One is provided by a professional pianist and the other one is downloaded from a MIDI archive web site. The downloaded MIDI is edited such that correct relations between note values were obtained. As the analysis was limited to departures from nominal IOI, only rules affecting this performance parameter were tested. ISE 599: Spring 2004

Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Experiment Outline Initial Run: Each rule is tested one by one. Duration Contrast, Melodic Charge, Punctuation, Double Duration, Phrase Arch, Harmonic Charge, Faster-uphill and Leap-tone-duration. Second Run: Combinations of rules are tested. First Phrase Arch Rule, then combining with Harmonic Charge and Duration Contrast. They have two MIDI data. One is provided by a professional pianist and the other one is downloaded from a MIDI archive web site. The downloaded MIDI is edited such that correct relations between note values were obtained. As the analysis was limited to departures from nominal IOI, only rules affecting this performance parameter were tested. ISE 599: Spring 2004

Limitation of Correlation Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Limitation of Correlation The sign of the overall deviations is considered, not their quantity. The correlation is highly sensitive to extreme values. The correlation is much more sensitive to the agreement for single notes when the number of notes compared is small as compared to when it is large. They used the correlation coefficient as a measure of the agreement between the pianist’s deviations from nominal and those produced by the DM system. At the end of the paper, they propose that the rule quantity should change depending on the musical character of the composition. ISE 599: Spring 2004

Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices Conclusion The Phrase Arch rule is the winner in the experimented performance. Rule combination has to change between sections. Time varying rule palettes would be a worthwhile target. DM limitation: The general applicability of the rule combination. Phrase Arch rules alone produced high correlation in the experiment. Phrasing was a prominent principle in the experimented performance. Rule combination has to change between sections in order to match this pianist’s deviations. ISE 599: Spring 2004

Presented by Beran Pacaci Sundberg, Friberg & Bresin: Director Musices References Ramon Lopez de Mantaras & Josep Lluis Arcos (2002). AI and Music From Composition to Expressive Performance. Efstathios Stamatatos & Gerhard Widmer (2002). Music Performer Recognition Using an Ensemble of Simple Classifiers. The Science of Music Performance http://www.speech.kth.se/music/performance/ ISE 599: Spring 2004