1. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
ATTEMPTS to REPRODUCE a PIANIST’S EXPRESSIVE TIMING with DIRECTOR MUSICES PERFORMANCE RULES
by JOHAN SUNDBERG, ANDERS FRIBERG and ROBERTO BRESIN
- BERAN PACACI -
SPRING 2004
ISE 599: Spring 2004

2. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Johan Sundberg
Ph.D. in musicology at Uppsala University.
was the head of the music acoustics research group from 1970 to 2001.
a music performer as a singer.
Current research areas include
Singing Voice
Rules for Musical Performance
Breathing and Phonation
He has a case study called: Why is ugly ugly? Singers may have ugly voices. He analysis different singers’ voices. What is the problem with that?
ISE 599: Spring 2004

3. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Anders Friberg
M.S. in Applied Physics. Ph.D. in ?
a music performer as a pianist.
Current research areas include
Random fluctuations of timing in music performance.
Swing ratios and ensemble timing in jazz.
Combining the Radio Baton and Director Musices.
Performance synthesis in Director Musices.
Combining the Radio Baton and Director Musices. The Radio Baton was developed by Max Mathews, Stanford University and is an controller device with two sticks that are sensitive to 3D position over a bottom plate. A midi score can be "conducted" by  beating the pulse with one stick and controlling dynamics with the other stick. The midi score has been preprocessed with Director Musices in order to model the musicians contribution to the performance. Since the Radio Baton controls the overall tempo and dynamics, Director Musices is used for the micro-level control. See a performance at the PEVOC 2001 opening session with  Malena Ernman, voice and Jan Risberg, Radio Baton Video (5Mb). Together with Johan Sundberg, Max Mathews, Craig Sapp and Gerald Bennett.
Performance synthesis in Director Musices. Once in a while I am improving the Director Musices program with a new rule or feature.
ISE 599: Spring 2004

4. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Roberto Bresin
Ph.D. in Music Acoustics.
Current research areas include
Analysis and synthesis of emotional content in music.
Articulation strategies in expressive music performance.
Emotional melodies for mobile phones.
Synthesis of Emotional Expression in Music Performance.
ISE 599: Spring 2004

5. KTH Music Acoustics Group
Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
KTH Music Acoustics Group
runs in four main streams:
singing (solo and choral)
stringed instruments
music performance
computer music composition
While running in parallel for periods, they often merge and interact. An experimental approach has long been a signature of the research. In particular, experimental work with "analysis-by-synthesis" has proved to be rewarding. Today, a strong trend can be seen toward more sophisticated synthesis strategies, based on a direct mathematical description of the instrument/voice. This strategy, commonly referred to as numerical or "physical modeling," would be generally adopted within the next five years or so at most major synthesizer laboratories. We have started physical modeling of the singing voice in terms of articulator synthesis, controlled by physiologic control parameters such as jaw opening and tongue shape. The increased interest for numerical modeling has revealed a lack of reliable and detailed experimental data in many areas. This applies in particular to the function of the control systems used in actual performances, for example breathing behavior in singers and wind instrumentalists, or bow movement in string playing. The prevailing lack of thorough knowledge of such control systems will force the experimenter to explore an overwhelmingly rich parameter space, and the chance of real success in reasonable time is small. Our research on musical performance by computer modeling is a long term project. Its focus is presently to compare characteristics of locomotion in running with timing in music performance. Furthermore, our generative music performance grammar and music examples illustrating the effects on synthesized performances, has been made accessible on the internet. Our overall aim is to continue the established experimental tradition in the study of the voice and the stringed instruments (now needed more than ever, it seems), while striving to take part in the development of the new synthesis methods. As before, the parallel work on musical performance bridges several of these tasks.
ISE 599: Spring 2004

6. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Music Performance
is an important research area today.
The deviation from nominal inter-onset-interval (IOI).
Every musician plays differently.
Musicians play differently. How can you distinguish a good musician’s performance from others?
You can express your emotions in music. Fear, anger, happiness, sadness, tenderness, solemnity can be expressed with music.
The deviation from nominal IOI gives an idea about the music performance.
If we define expressive performance as “deviation from the score”, then different performances differ in the way and extent the artist “deviates” from the score, i.e., from a purely mechanical/flat rendition of the piece, in terms of timing, dynamics, and articulation. In order to be able to compare performances of pieces or sections of different length, we need to define features that characterize and quantify these deviations at a global level, i.e., without reference to individual notes and how these were played.
ISE 599: Spring 2004

7. Music Performance Rules
Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Music Performance Rules
The rules are divided into three main classes:
Differentiation Rules: enhance the differences between tone categories.
Grouping Rules: show what tones belong together.
Emphasis Rules: emphasize unexpected notes.
Ensemble Rules: synchronize the various voices in an ensemble.
Differentiation Rules: increase the differences between tone categories such as pitch classes, intervals, and note values.
Grouping Rules: mark which tones belong together and where the structural boundaries are. Think about Phrase boundaries.
Emphasis Rules: emphasize unexpected notes.
Ensemble Rules: synchronize the various voices in an ensemble. I’ve found this in “AI and Music” article.
Quiet interestingly, these rules are applied in speech also. For example: The slowing down towards the end of a phrase.
ISE 599: Spring 2004

8. Music Performance Rules
Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Music Performance Rules
Differentiation Rules
Duration Contrast
Melodic Charge
High Sharp
Grouping Rules
Punctuation
Double Duration
Tuning
Phrase Arch
Inégales (or swing)
Ensemble Swing
Final Ritard
Harmonic Charge
ISE 599: Spring 2004

9. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Director Musices (DM)
a software for Automatic Music Performance.
a rule system for Musical Performance.
implementing all previously defined rules.
features includes polyphony, midi I/O, performance variable graphs and user rule definition.
Automatically introduces expressive deviation in performances of input score files.
DM system incorporates rules for tempo, dynamic, and articulation transformations constrained to MIDI. These rules are inferred both from theoretical musical knowledge and experimentally from training, especially using analysis-by-synthesis approach.
The DM system has been found to be capable also of adding various emotional colors to a performance. This result was obtained by varying the quantity parameter of various rules.
The DM system has obviously some limitations. If you apply a combination of rules to a piece, it only improves the musical quality of that performance. The general applicability of the rules is doubtful.
ISE 599: Spring 2004

10. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Director Musices (DM)
DM rules contain two elements:
Context
Quantity
Context: defines in what context the rule should be applied.
Quantity: degree of effect on the performance. (states how great effects it should produce in the performance.)
The quantity parameter k determines how much duration should be transferred. By choosing different quantity values, different performances of the same piece are obtained.
ISE 599: Spring 2004

11. Director Musices (DM)

12. Research Strategy
Analysis-by-synthesis
Analysis-by-measurement

13. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Experiment Outline
Comparison between the real pianist’s performance with different performances produced by the DM system.
MIDI data consists of score time in beats, MIDI note numbers, IOI, sounded duration and dynamics in MIDI velocity.
They have two MIDI data. One is provided by a professional pianist and the other one is downloaded from a MIDI archive web site. The downloaded MIDI is edited such that correct relations between note values were obtained.
As the analysis was limited to departures from nominal IOI, only rules affecting this performance parameter were tested.
ISE 599: Spring 2004

14. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Experiment Outline
Initial Run: Each rule is tested one by one. Duration Contrast, Melodic Charge, Punctuation, Double Duration, Phrase Arch, Harmonic Charge, Faster-uphill and Leap-tone-duration.
Second Run: Combinations of rules are tested. First Phrase Arch Rule, then combining with Harmonic Charge and Duration Contrast.
They have two MIDI data. One is provided by a professional pianist and the other one is downloaded from a MIDI archive web site. The downloaded MIDI is edited such that correct relations between note values were obtained.
As the analysis was limited to departures from nominal IOI, only rules affecting this performance parameter were tested.
ISE 599: Spring 2004

15. Limitation of Correlation
Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Limitation of Correlation
The sign of the overall deviations is considered, not their quantity.
The correlation is highly sensitive to extreme values.
The correlation is much more sensitive to the agreement for single notes when the number of notes compared is small as compared to when it is large.
They used the correlation coefficient as a measure of the agreement between the pianist’s deviations from nominal and those produced by the DM system.
At the end of the paper, they propose that the rule quantity should change depending on the musical character of the composition.
ISE 599: Spring 2004

16. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
Conclusion
The Phrase Arch rule is the winner in the experimented performance.
Rule combination has to change between sections.
Time varying rule palettes would be a worthwhile target.
DM limitation: The general applicability of the rule combination.
Phrase Arch rules alone produced high correlation in the experiment.
Phrasing was a prominent principle in the experimented performance.
Rule combination has to change between sections in order to match this pianist’s deviations.
ISE 599: Spring 2004

17. Presented by Beran Pacaci
Sundberg, Friberg & Bresin: Director Musices
References
Ramon Lopez de Mantaras & Josep Lluis Arcos (2002). AI and Music From Composition to Expressive Performance.
Efstathios Stamatatos & Gerhard Widmer (2002). Music Performer Recognition Using an Ensemble of Simple Classifiers.
The Science of Music Performance
ISE 599: Spring 2004