March 18, 1999Acoustical Society of America - Berlin '99 1 Cross-modal integration: Synchronization of auditory and visual components in simple and complex media Dr. Scott D. Lipscomb Institute for Music Research University of Texas at San Antonio

March 18, 1999Acoustical Society of America - Berlin '99 2 Film Music Literature past studies deal almost exclusively with the referential aspect of musical sound –“cognitive congruency” (Marshall & Cohen, 1988) –selected others: Tannenbaum (1956), Thayer & Levenson (1984) –special issue of Psychomusicology (vol. 13, 1994) vs. accent structure alignment –i.e., how often important events in the music coincide with important events in the visual scene

March 18, 1999Acoustical Society of America - Berlin '99 3 Film Music Perception Paradigm (Lipscomb & Kendall, 1995)

March 18, 1999Acoustical Society of America - Berlin '99 4 Alignment Conditions after Yeston (1975) Consonant Out-of-phase Dissonant

March 18, 1999Acoustical Society of America - Berlin '99 5 Subject Ratings synchronization: “… how often important events in the music coincide with important events in the visual scene” effectiveness: “… simply concerns [the subject’s] subjective evaluation of how well the two go together”

March 18, 1999Acoustical Society of America - Berlin '99 6 Results of Previous Study (1997) presented at Penn State ASA Conference

March 18, 1999Acoustical Society of America - Berlin '99 7 Problems with Dissonant Stimuli Solution Found

March 18, 1999Acoustical Society of America - Berlin '99 8 Revised Stimulus Preparation A-V alignment conditions were created using Media 100 software on a Macintosh G3 computer sound files were manipulated in Sonic Foundry’s Sound Forge 4.0, using the Time Compress/Expand Sonic Foundry plug-in completed A-V composites were recorded directly from Media 100 onto VHS tape

March 18, 1999Acoustical Society of America - Berlin '99 9 Consonant Alignment Condition aligned as intended by the composer

March 18, 1999Acoustical Society of America - Berlin '99 10 Out-of-phase Alignment Condition alignment conditions based on Preliminary Study

March 18, 1999Acoustical Society of America - Berlin '99 11 Dissonant Alignment Conditions audio tracks for the McLaren animations were “time expanded” by 115% audio tracks for the “Obsession” excerpts were “time expanded” by 110% –since these excerpts were longer, the gradual misalignment could occur at a slower pace

March 18, 1999Acoustical Society of America - Berlin '99 12 Video excerpts

March 18, 1999Acoustical Society of America - Berlin '99 13 Subject Info & Method N = 20 UTSA students taking music classes –stimuli presented in groups of 4 to 6 –VAME ratings were provided on a continuous line response anchored by either “not synchronized-synchronized” or “ineffective-effective” –each subject was assigned to one of three random stimulus presentation orders –response forms were generated so that order of VAME responses was also randomized –stimuli were presented to subjects using a Samsung VR 5855 video cassette recorder and an RCA F27676BC 24” television not synchronized synchronized

March 18, 1999Acoustical Society of America - Berlin '99 14 Statistical Analysis (MANOVA) repeated measures fully-factorial analysis of variance –repeated measures: two VAME ratings for each of 18 A-V combinations no significant between-groups variation –musical training: p = 0.512; f (2,17) = –gender: p = 0.508; f (1, 18) = within-groups, ratings were significantly different depending upon the A-V combinations –p <.0005; f (17) =

March 18, 1999Acoustical Society of America - Berlin '99 15 Experimental Results

March 18, 1999Acoustical Society of America - Berlin '99 16 Experimental Results

March 18, 1999Acoustical Society of America - Berlin '99 17 Experimental Results Subject Ratings Collapsed (n=3)

March 18, 1999Acoustical Society of America - Berlin '99 18 Contact Info Dr. Scott D. Lipscomb UTSA Division of Music 6900 N. Loop 1604 West San Antonio, TX (210) (210) FAX