10/13/98Acoustical Society of America1 Synchronization of musical sound and visual images: Issues of empirical and practical significance in multimedia development Scott D. Lipscomb, Ph.D. Institute for Music Research, University of Texas at San Antonio

10/13/98Acoustical Society of America2 Star Trek IV: The Voyage Home music by L. Rosenman

10/13/98Acoustical Society of America3

10/13/98Acoustical Society of America4

10/13/98Acoustical Society of America5

10/13/98Acoustical Society of America6

10/13/98Acoustical Society of America7

10/13/98Acoustical Society of America8 Best Fit?

10/13/98Acoustical Society of America9 Model of Film Music Perception Lipscomb & Kendall (1994)

10/13/98Acoustical Society of America10 Association Judgment Referential Meaning

10/13/98Acoustical Society of America11 Subject Judgments of “Best Fit”

10/13/98Acoustical Society of America12 Semantic Differential Scales

10/13/98Acoustical Society of America13 Influential Musical Parameters: u clarity of tonal center u harmonic complexity u dynamic variation u tempo absolute rate & fluctuation u phrase structure u melodic activity

10/13/98Acoustical Society of America14 Accent Structure Alignment Syntactical Meaning

10/13/98Acoustical Society of America15 Research Questions u Is it necessary for accents in the musical soundtrack to align precisely with points of emphasis in the visual modality in order for the combination to be considered effective? u What are the determinants of “accent” in the visual & auditory fields?

10/13/98Acoustical Society of America16 Sources of Perceived Accent u Musical pitch height, loudness, timbre u Visual spatial location, shape, color

10/13/98Acoustical Society of America17 Alignment Possibilities Consonant Out-of-Phase Dissonant

10/13/98Acoustical Society of America18 Method u Design post-test only, repeated measures factorial design converging methods: VAME & similarity judgments u Subjects 3 experiments, two converging methods (N = 120; n = 20) 3 levels of musical training (untrained, moderate, and highly trained)

10/13/98Acoustical Society of America19 Response Mechanism (Music Experiment Development System, Roger A. Kendall)

10/13/98Acoustical Society of America20 Method - AV Stimuli u Experiment One simple single-object animations and pitch sequences (5-sec); created by author u Experiment Two excerpts from experimental animations by Norman McLaren (8-sec); “Dots” (1940), “Canon” (1964), & “Synchromy” (1971) u Experiment Three excerpts from “Obsession” (25-sec) with musical score by Bernard Herrmann

10/13/98Acoustical Society of America21 Experiment One original single-object animations & isochronous pitch sequences Examples

10/13/98Acoustical Society of America22

10/13/98Acoustical Society of America23

10/13/98Acoustical Society of America24 Control & Reliability vs. Ecological Validity Experiments Two & Three

10/13/98Acoustical Society of America25 Experiment Two “Dots” by Norman McLaren

10/13/98Acoustical Society of America26 Experiment Two “Synchromy” by Norman McLaren

10/13/98Acoustical Society of America27 Experiment Two - VAME (r =.94)

10/13/98Acoustical Society of America28 Experiment Two - VAME Collapsed Across Alignment Condition

10/13/98Acoustical Society of America29 Experiment Three “Obsession”--music by B. Herrmann

10/13/98Acoustical Society of America30 Experiment Three - Synchronization Collapsed Across Alignment Condition

10/13/98Acoustical Society of America31 Experiment Three - Effectiveness Collapsed Across Alignment Condition

10/13/98Acoustical Society of America32 Experiment Three Effectiveness - Combined Groups

10/13/98Acoustical Society of America33 Comparison of VAME Results Across All Three Experiments

10/13/98Acoustical Society of America34 VAME - Synchronization Across All Experiments

10/13/98Acoustical Society of America35 VAME - Effectiveness Across All Experiments

10/13/98Acoustical Society of America36 Film Music Perception Paradigm (revised) Perception A udio-Visual Congruence Association Judgment Accent Structure Alignment No Shift of Attentional Focus Shift of Attentional Focus Aural Stimulus Visual Stimulus Implicit Processes No Yes

10/13/98Acoustical Society of America37 Suggestions for Further Research u Resolution of the variable confound alignment condition vs. composer intent u Determine relative importance of associational and syntactical aspects dynamic relationship u Determine j.n.d. for AV alignment u Develop a means of quantifying accent musical, visual, or AV composite u Investigate other grouping variables of interest e.g., visual literacy

10/13/98Acoustical Society of America38 Enhancing Instructional Materials with Multimedia CD-ROM Companion to the Handbook of Music Psychology (D. Hodges, Ed.; IMR Press)

10/13/98Acoustical Society of America39 Hyperlinks to Sound Files “If a pianist plays a low A (110 Hz) on the piano, the resulting vibration does not consist of only a single, periodic vibration at the rate of 110 times per second. Rather, the string vibrates as a whole (110 Hz), in halves (220 Hz), in thirds (330 Hz), in fourths (440 Hz), and so on. Therefore, there is vibrational energy not only at the fundamental frequency of 110 Hz but also at each of these integer multiples [audio examples: a) complex tone made up of 8 partials & b) complex tone built one partial at a time …”110 Hz 220 Hz330 Hz440 Hzcomplex tone made up of 8 partialscomplex tone built one partial at a time Lipscomb & Hodges, 1996, p. 97

10/13/98Acoustical Society of America40 Sounds to Attract Attention “If a pianist plays a low A (110 Hz) on the piano, the resulting vibration does not consist of only a single, periodic vibration at the rate of 110 times per second. Rather, the string vibrates as a whole (110 Hz), in halves (220 Hz), in thirds (330 Hz), in fourths (440 Hz), and so on. Therefore, there is vibrational energy not only at the fundamental frequency of 110 Hz but also at each of these integer multiples [audio examples: a) complex tone made up of 8 partials & b) complex tone built one partial at a time …”110 Hz 220 Hz330 Hz440 Hzcomplex tone made up of 8 partialscomplex tone built one partial at a time Lipscomb & Hodges, 1996, p. 97

10/13/98Acoustical Society of America41 Exploratory Environments u Beating—experimentation with constructive & destructive interference and the critical bandwidth Beating u WaveMix—combining partials to create a complex tone WaveMix u Signal—allows student to create a complex signal by manipulating partial amplitudes & phase relationships, hear the result, and view an animation approximating molecular motion Signal

10/13/98Acoustical Society of America42 Sound as a Cue u Computer Guts Computer Guts instructional package designed by Dr. David Sebald (UTSA Division of Music) –more examples available online at:

10/13/98Acoustical Society of America43 Conclusions: Use of Multimedia to Enhance the Learning Experience u provide additional information through illustration audio examples or animations u focus user attention u explicit association judgments u create exploratory environments allowing students the opportunity to incorporate the highest levels of knowledge (Bloom, 1956) a work in progress

10/13/98Acoustical Society of America44 Contact Info : Dr. Scott D. Lipscomb Institute for Music Research Division of Music The University of Texas at San Antonio 6900 N. Loop 1604 West San Antonio, TX phone: (210) FAX: (210)