1. Pitch, tonality, and the missing fundamentals of music cognition Pitch, tonality, and the missing fundamentals of music cognition Richard Parncutt University of Graz, Austria BRAMS, Université de Montréal 31 May 2012 This file has been revised after discussion and questions SysMus Graz

2. Abstract What are the psychological foundations of major-minor tonality? Psychologists have explored how modern listeners perceive its pitch structures, but the psychohistorical origins of those structures remain unclear. A plausible theory should be able to predict tonal styles as probability distributions of pitch-time patterns on the basis of a limited number of psychologically and historically plausible axioms. From a psychological viewpoint, such axioms should refer to pitches that are perceived (experienced) by audiences and performer - not pitches notated in scores. Non-notated pitches may include prominent partials, missing fundamentals, or pitches expected on the basis of short- or long-term experience (e.g. melodic continuations). Consider a simple example that ignores octave register and tuning. A C-major triad may have a missing fundamental at A, because E corresponds to the 3rd harmonic of A and G to the 7th. Other possible missing fundamentals are F and D. The same chord may have a prominent partial at B, if the 3rd harmonic of E and the 5th harmonic of G coincide; another prominent partial may be at D. A systematic approach should consider all such possibilities in a chord’s spectrum, weighting them relative to each other. Predicted pitches and weights should be consistent with empirical data. But the psychological reality of non-notated pitches remains unclear because “nature” (predictions based on psychoacoustics or physiology) and “nurture” (predictions based on musical experience) are often quantitatively similar. I will present recent data and plans for future work to separate nature from nurture by systematically manipulating musical expertise, cultural background, sound type, tone type, onset synchrony, duration, tuning and background noise. Further strategies include separation of “fundamental listeners” (sensitive to missing fundamentals) from “spectral listeners” (sensitive to prominent partials), and modeling musical experience by statistical analysis of symbolic music databases.

3. Origin of major-minor tonal system Scientific approach: Psychologically predetermined?   Underlying principles?   Why those pc-sets? voicings? progressions?   Can we model frequency of occurrence? Humanities approach: Historical accident? If so:   Why so widespread?   Why so stable?

4. Assumption The major-minor system is based on pitch as subjective experience   not as physical measurement (frequency)   not as physiological correlate   not as notation in musical scores.Thesis To understand the major-minor system, we must systematically investigate pitch as experienced by musicians and listeners in musical contexts.

5. Does experience exist? Visual experience is quite different from   physical world   info on the retina (upside down, moving)   neurophysiology of the visual cortex Visual experience is constructed   available info is generally incomplete   focus on affordances (survival and reproduction) Correlates of the color red ≠ red itself   light wavelengths   physiology of retina   physiology of visual cortex To study “red”, we must separate experience & physics

6. Is everything physical? Modern science is atheist - ok   Good arguments against existence of gods and spirits Conscious experience is something else!   Different from gods/spirits AND brain substrates   Emerges in infancy, disappears when we die   Foundation of arts and aesthetics The solution: Epiphenomenalism   Experience is a byproduct of neural substrates   Both experience and its substrates exist   Two sides of the same coin, paradoxically inseparable   Consistent with both neuroscience and philosophy

7. What is “more real”? Objective answer: The physical world It exists without experience - but not vice-versa Existence of experience depends on physical world Subjective answer: Experience Without it we would know nothing (not be human) Existence of physical world depends on experience (“Objective”: subject ≠ object, “Subjective”: s=o) Conclusion No idea. Can’t compare totally different things

8. Why scientists reject experience Why scientists reject experience and why some humanities scholars reject it too Scientific belief system   Success of modern physics   In inherent superiority of objectivity   Reductionism (belief in simple explanations)   Grouping of mind-body dualism with theism Humanities-science conflict   “Othering” humanities to construct own identity   Refusal to accept own subjectivity (fear?)   Competitive neoliberal research structures   Scientists too arrogant, insecure or busy for philosophy

9. Three musical representations and aspects of musical pitch structure you can explain with them 1.Physical: Frequencies and amplitudes  Room and instrument acoustics, roughness 2. Experiential: Pitches and their salience  Timbre, fusion, chord roots, harmonic function, harmonic tonality 3. Abstract: Notes in musical scores  Performance, composition

10. The “three worlds” of Karl Popper The “three worlds” of Karl Popper The broader context of music representations (not “worlds”) 1. Physical  environment, body, brain 2. Experiential  sensations, emotions 3. Abstract  knowledge, info, culture Assumption: A clear separation of 3 representations can clarify discussions of nature and origin of musical structure human consciousness

11. Literature on ecological and evolutionary psychology versus consciousness & subjective experience Gallagher, S. & Zahavi, D. (2010). Phenomenological Approaches to Self-Consciousness. Stanford Encyclopedia of Philosophy (online) Gulick, R. van (2004). Consciousness. Stanford Encyclopedia of Philosophy (online) Miller, G. (2007). Reconciling Evolutionary Psychology and Ecological Psychology: How to Perceive Fitness Affordances. Acta Psychologica Sinica 39, 546-555.

12. What I mean by “pitch”   Subjective experience – like the color red   One-dimensional   Property of pure/complex tones, noise (+tinnitus)   May be ambiguous and multiple   Depends on listener, temporal context Here: pitch = perceived pitch In music theory: pitch = notated pitch What I mean by “chroma”  Octave-generalised perceived pitch  not D4 or D5 - just D  Like pitch class, but experienced – not notated

13. Tone types  Pure tone sinusoidal function of air pressure against time  Complex tone simultaneity of pure tones in any frequency relationship  Harmonic complex tone (HCT) Complex tone whose frequencies correspond to a harmonic series

14. The harmonic series equally spaced on a linear frequency scale (e.g in Hz) unequally on a log frequency scale (e.g. in semitones) Compared to 12-tone equal temperament: 7th harmonic is 1/3 semitone flatter than a m7 above 4th 11th harmonic is midway between P4 and TT above 8th

15. Spectral versus virtual pitch Pitch perception according to Terhardt Spectral pitch (SP) pitch of a pure tone pitch of a pure tone pitch of an audible partial of a complex tone pitch of an audible partial of a complex tone hum tone of a church bell (1s after hammer) hum tone of a church bell (1s after hammer) Virtual pitch (VP) pitch of a complex tone pitch of a complex tone most consciously noticed pitches in everyday life most consciously noticed pitches in everyday life strike tone of a church bell (hammer hitting bell) strike tone of a church bell (hammer hitting bell) pitch at missing fundamental (e.g. voice on telephone) pitch at missing fundamental (e.g. voice on telephone) youtube church bells

16. Spectral versus virtual pitch This distinction is   ecological based on interaction with the environment   not physiological based on peripheral and central processing The ultimate aim is psychophysical : understand the relationship between sound and experience

17. What about neurophysiology? We don’t know the functional relationship between neural states and processes and conscious experience   Unique nature of this problem! Never solved (or did I miss the news?)   Enormous no. of neurons and connections! Which states/events correspond to experience ?

18. Spectral vs temporal processing Along auditory pathways, we find both   temporal representations (phase locking)   spectral representations (tonotopic structures) Assumptions   Both are used by neural networks   Both are inextricable in hidden layers Conclusion   Doesn’t help us understand pitch as experience

19. Neural processing of pitch in music and speech The same neural net can process… spectral and temporal patterns pitch in speech and music Bharucha, 1987

20. Virtual object (Kanizsa, 1955) Incomplete triangle Completed by virtual contours Auditory image (Bregman, 1990; McAdams, 1984) Incomplete harmonic series Completed by virtual pitch missing fundamental (f 0 ) overtones frequency Virtual objects in vision and hearing Gestalt principle of closure – filling the gaps in a familiar pattern SPL

21. Pitch perception: Experimental method   Listener adjusts frequency of pure tone until the two sounds have the same pitch   Frequency of pure tone is a measure of pitch of test sound   Results must be consistent within and between listeners Pitch salience = probability of matching

22. Pitch ambiguity Assumption: The pitch of a pure tone is unambiguous corresponds to frequency (if SPL constant) Result: The pitch of a complex tone is ambiguous = different pitch in different presentations and/or multiple = several pitches perceived simultaneously  Can explain a lot about musical structure

23. Pitch salience   In musical practice: Pitched versus unpitched percussion How clear is pitch on a continuous scale?   In experimental data: Probability of noticing a pitch Subjective clarity of a pitch   Depends on: Stimulus (esp. spectrum) Listener (“spectral” vs “fundamental”) Temporal context (proximity  expectation) high pitch salience low pitch salience

24. Analytic versus holistic perception You can consciously switch between two modes o analytic (strange black shapes) o holistic (“FLY” in white letters) Similarly for pitch?

25. Individual differences in pitch perception Auditory ambiguity test (Seither-Preisler) Individual differences  “fundamental listeners” and “spectral listeners”

26. Auditory Ambiguity Test (AAT) Seither-Preisler et al. (2007) You will hear 10 tone pairs In each pair, does the pitch rise or fall? Write your answers as arrows: ↑ pitch rises ↓ pitch falls

27. If you wrote this, you are a “fundamental listener“ If the opposite, you are an “overtone listener” You may also be a “mixed listener”

28. Schneider et al., NY Acad Sci, Vol. 1060, p. 387-395 (2005) overtone listeners fundamental listeners Finding: Listening strategy depends on music experience and instrument Research idea: Study relation to amusia?

29. Pitch dominance regions Octave register (piano keyboard) 12345678 Salient spectral pitch (spectral dominance) Salient virtual pitch (musical practice) Spectral pitch According to experimental data, SP salience is h ighest at F5 (C4-C8).  speech intelligibility & formants: f 1 ~ 500 Hz ~ C5, f 2 ~ 1500 Hz ~ G6 Virtual pitch According to model predictions, VP salience is highest at D 4 (C2-C6).  f 0 range of voice and music f1f1 f2f2 middle C

30. Dominance region of spectral pitch origin: speech perception centre at 700 Hz, central band at 300-2000 Hz after Terhardt et al., 1982

31. Calculated VP salience distribution f 0 range of speech and music After Huron & Parncutt (unpublished)

32. Origin of virtual pitch Origin of virtual pitch a bit of history Before the 1970s many assumed... low pitch = combination tone = distortion product low pitch = combination tone = distortion product peripheral origin (basilar membrane) peripheral origin (basilar membrane) In the 1970s it became clear... pitch perception = pattern recognition pitch perception = pattern recognition mixture of spectral and temporal processing mixture of spectral and temporal processing central origin (brain) central origin (brain)

33. Perception of complex tones Perception of complex tones Two separable stages 1. Auditory spectral analysis  c. 16 audible* or 8 resolvable* harmonics 2. Holistic perception  (Virtual) pitch, timbre, loudness *Audible: If you change it, the listener hears something *Resolvable: Listener can focus attention on it 1 2

34. Did you hear a bee buzzing in your ear? trials and tribulations of recorder ensemble performance ? Combination tones become audible: high frequencies, high amplitudes little low-frequency masking Origin: Non-linear distortion in inner ear

35. Perceptual fusion of HCTs Perceptual fusion of HCTs depends on:  Tuning of partials Mistuning of <1 semitone from harmonic series  Relative amplitude of partials Is spectral envelope like a typical environmental sound?  Temporal context Preceding/following tones can attract attention  Listener Fusion more likely for “holistic” or “fundamental” listeners Fusion more likely for “holistic” or “fundamental” listeners

36. Pitch at the missing fundamental Houtsma, Rossing, Wagenaars), track 37 Pitch at the missing fundamental ASA Auditory Demonstrations CD (Houtsma, Rossing, Wagenaars), track 37 Conclusions: 1. Pitch does not necessarily correspond to a partial 2. Pitch is multiple/ambiguous VP at missing fundamental SP at lowest partial 1 2 345

37. Sound demo: Masking SP and VP Houtsma, Rossing, Wagenaars) Sound demo: Masking SP and VP ASA Auditory Demonstrations CD (Houtsma, Rossing, Wagenaars) track 40 41 42 Conclusion : Masking and pitch pattern recognition happen in different places Masking is peripheral Pitch pattern recognition is central 1st tone in pair 2 nd tone in pair

38. Relation between VP and SP pattern Houtsma, Rossing, Wagenaars), Track 39 Relation between VP and SP pattern ASA Auditory Demonstrations CD (Houtsma, Rossing, Wagenaars), Track 39 VP corresponds to:  best-fit subharmonic of all partials  NOT frequency difference  small mistuning is no problem Demo no. SP1 (Hz) SP2 (Hz) SP3 (Hz) VP (Hz) 180010001200200 285010501250210

39. General relation between SP and VP 1. 1.VP lies at fundamental of audible harmonic pattern 2. 2.VP salience depends on SPs at harmonic positions how many there are (the more the better) their salience (the greater the better) their tuning (mistuning up to a semitone) their effective harmonic numbers (the lower the better)

40. Prevalence of individual tones (scale steps) in chant Source: Liber Usualis  1,900 pages; most versions of ordinary chants for the catholic mass  first edited in 1896 by Solesmes abbot Dom André Mocquereau  first edited in 1896 by Solesmes abbot Dom André Mocquereau  Online search by CIRMMT: DDMAL (Ichiro Fujinaga and team) A B C D E F G no. of notes counted

41. Prevalence of individual tones (scale steps) in chant Source: Liber Usualis  1,900 pages; most versions of ordinary chants for the catholic mass  first edited in 1896 by Solesmes abbot Dom André Mocquereau  first edited in 1896 by Solesmes abbot Dom André Mocquereau  Online search by CIRMMT: DDMAL (Ichiro Fujinaga and team)  Accidentals are ignored, but less than 1% of Bs are B=-flats A B C D E F G no. of notes counted

42. Prevalence of individual tones (scale steps) in chant How can we explain the distribution?   Musical structure depends on non-notated chroma This is just one example   Listeners have a “feel” for pitches of harmonics Or at least spectral listeners do   Tones are preferred if consonant with context An example of pitches in common (“pitch commonality”)   Up to ten harmonics are audible (resolvable?) Almost no masking from other sounds

43. Prevalence of individual tones (scale steps) in chant 1. “Octave-generalise” the harmonic series 2. How many “octave-generalised overtones” correspond to diatonic scale? Harmonic no.1, 2, 4, 83, 65, 1079 IntervalP1, P8…P5, P12…M3…m7…M2, M9… Scale stepABCDEFG No. of harmonics3133234

44. Prevalence of individual tones (scale steps) in chant A B C D E F G Data A B C D E F G Model df = 5, r = 0.90, p<.01 cf. Parncutt, R. & Prem, D. (2008). The relative prevalence of Medieval modes and the origin of the leading tone (poster). International Conference on Music Perception and Cognition (ICMPC10), Sapporo, Japan, 25-29 August.

45. Guillaume de Machaut (1300-1377) Rondeau Ma fin est mon commencement What is the origin of (rising) leading tones? Why do rising semitones “tonicize”?

46. This is not a popular theory! Music psychologists:   No “cognitive structures”   Empirical evidence is unclear (BUT: consistent with statistical learning) Psychoacousticians and neuroscientists:   Focuses on subjective experience   Avoids temporal-spectral debate Music theorists:   Challenges primacy of musical score   Focuses on tonality (not “modernist”) Music historians:   Not based on historic sources   Ignores historic mode classification Contradicts… physical monism established research paradigms in sciences and humanities

47. Non-notated chroma in triads Non-notated chroma in triads An example of looking carefully at the stimulus (for a change) 1. Spectral synthesis Build a C major triad from   first 10 harmonics of C4 (up to E7)   harmonics of E4 and G4 (up to F#7) Assume chromatic categorical perception 2. Masking Assume all partials are equally audible except inside a chromatic cluster 3. Pitch pattern recognition At each chromatic scale step:   Which harmonics are present in chord?   Synthesize that tone using “SFS Esynth”

48. C4E4G4 C4E4G4 C4C#4D4D#4E4F4F#4G4G#4A4A#4B4      C7             C6             C5             C4  

49. E4G4 C4E4G4 C3C#3D3D#3E3F3F#3G3G#3A3A#3B3      C7             C6              C5             C4             C3  

50. Estimating virtual pitch salience Compromise between  simplicity (parsimony, falsifiability)  accuracy (accounting for all factors) First approximation  Count the audible harmonics above any pitch (next slide) Second approximation  Weight each harmonic 1/n, then add weights (slide after that) Closer approximations  Estimate audibility of partial, normalise salience (Parncutt, 1989)  Consider tuning of partials (Terhardt et al., 1982)  Consider spectral dominance region (Terhardt et al., 1982)

51. Estimating virtual pitch salience Estimating virtual pitch salience of pitches within triad C4E4G4 First approximation: number of audible partials Predictions C3 > E3 and G3 In both registers, D > C# & D# In register 3, A # > B C3 D3 E3 F3 G3 A3 B3 C4 D4 E4 F4 G4 A4 B4 Note: Here, C4 > E4 > G4 is an artefact of a simple model

52. Estimating virtual pitch salience Estimating virtual pitch salience of pitches within triad C4E4G4 2 nd approx: Weight each partial 1/n, add weights C3 D3 E3 F3 G3 A3 B3 C4 D4 E4 F4 G4 A4 B4 Predictions In both registers, C > E and G, D > C# & D#, F>F#, A>G# B versus A #: different depending on register

53. Estimating virtual pitch salience (i) physical representation (ii) experiential representation Estimating virtual pitch salience of pitches within triad C4Eb4G4 (C minor) 3 rd approximation (Parncutt, 1989) (i) physical representation (ii) experiential representation audible partials

54. Experimental data Parncutt, 1993 Stimuli in one trial: A chord of OCTs, then a single OCT Listeners rate how well tone follows chord Diamonds: Mean ratings Squares : Theoretical predictions (masking + pattern rec.)

55. Gottfried Reichweger Gottfried Reichweger Diplomarbeit Uni Graz 2010 Participants 20 active musicians Sounds Test sounds: chords of natural piano tones Reference tones: octave-complex (Shepard) Task How well does the tone go with the chord? 7-point scale

56. Gottfried Reichweger Gottfried Reichweger Diplomarbeit Uni Graz 2010 Major triad Minor triad 1st inversion Root position 2nd inversion

57. Similarity judgments of successive tones Similarity judgments of successive tones (Parncutt, 1989) Effect at octave is greater: …for complex tones  Evidence for “nature” …for musicians  Evidence for “nurture” …for rising complex tones and falling pure tones  Consistent with prediction that upper/lower octave more salient for complex/pure tones  Consistent with implication-realisation model

58. Future experiments Future experiments to separate “nature” from “nurture” Listeners   Spectral versus fundamental listeners   Western versus non-Western musicians Predictions   Psychoacoustic model   Statistical analysis of symbolic music databases Stimuli   Synchronous versus asynchronous   Pure versus complex tones   With/without background noise

59. Ideas for future research Ideas for future research PhD students? Postdocs?  Further experiments to separate nature from nurture  Modeling of empirical data of Krumhansl and others

60. Are major and minor triads special? Especially consonant A combination of: 1. 1.high harmonicity/fusion (include P5/P4) 2. 2.low roughness (no 2nds) Part of culture - not “nature” The result of centuries of experimentation 3. 3.familiarity  3 psychological components of consonance

61. Origins of major-minor tonality Open triangles: chroma stability profile of MmT 1 Full squares: chroma salience profile of tonic triad 2 1 Krumhansl, C. L., & Kessler, E. J. (1982). Tracing the dynamic changes in perceived tonal organization in a spatial representation of musical keys. Psychological Review 2 Parncutt, R. (1988). Revision of Terhardt's psychoacoustical model of the root(s) of a musical chord. Music Perception From Parncutt (2011, Music Perception)

62. Analysis of C4 E4 G4 Analysis of C4 E4 G4 Using pitch algorithm of Parncutt (1989) Each tone is assumed to have many harmonics Yellow: The notes 1. Spectral pitch saliences Register 0: - - - - - - - - - - - - Register 1: - - - - - - - - - - - - Register 2: - - - - - - - - - - - - Register 3: - - - - - - - - - - - - Register 4: 0.08 - - - 0.06 - - 0.07 - - - - Register 5: 0.08 - - - 0.08 - - 0.08 - - - 0.04 Register 6: 0.02 - 0.05 - 0.06 - - 0.05 0.01 - - 0.03 Register 7: - - 0.06 - 0.02 - - - - - - - Register 8: - - - - - - - 0.01 - - - - Register 9: - - - - - - - - - - - -

63. Analysis of C4 E4 G4 Analysis of C4 E4 G4 Using pitch algorithm of Parncutt (1989) Yellow: The notes 2. Virtual pitch saliences Reg. 0: - - - - - - - - - - - - Reg. 1: 0.01 - 0.01 - - 0.01 - - 0.02 0.01 - - Reg. 2: 0.10 - 0.01 0.01 0.02 0.05 - 0.03 0.01 0.06 - - Reg. 3: 0.29 - 0.01 0.01 0.12 0.03 0.01 0.14 - 0.05 0.02 0.01 Reg. 4: 0.35 - 0.02 - 0.30 - - 0.28 - 0.02 - 0.02 Reg. 5: 0.10 - 0.03 - 0.14 - - 0.13 - - - 0.05 Reg. 6: 0.01 - 0.05 - 0.05 - - 0.02 - - - 0.02 Reg. 7: - - 0.02 - 0.01 - - - - - - - Reg. 8: - - - - - - - - - - - - Reg. 9: - - - - - - - - - - - - 3. Chroma saliences 0.87 0.01 0.19 0.03 0.66 0.09 0.01 0.64 0.05 0.15 0.03 0.12

64. Analysing different voicings of CEG Analysing different voicings of CEG Using pitch algorithm of Parncutt (1989) Which non-notated chromas are implied by CEG? Procedure: Consider a wide variety of voicings In each voicing, study non-notated chromas chroma is not C, E or G predicted salience > 0.05 (predicted probability of noticing) Root position First inversion Second inversion closeC4 E4 G4E4 G4 C5G4 C5 E5 openC3 G3 E4E3 C4 G4G3 E4 C5 skewedC3 E4 G4E3 G4 C5G3 C5 E5 very openC3 E4 G5E3 G4 C6G3 C5 E6

65. Analysing different voicings of CEG Analysing different voicings of CEG Pitches whose predicted salience are > 0.05 (Parncutt, 1989) Root pos.1st inv.2 nd inv. Close position A2, A3 F2 D6 (D7) B5 A2 A3 (D7) F3 Open position A2 F1 D5 B5 (B5) A1 D6 Skewed position A2 A3 B5 A1 D6 A3 Very open position A2 D6 B5 A1 D6 A2 B7 All pitches are virtual unless in brackets (spectral) Result: More common voicings have more salient non-notated chromas

66. Octave generalisation of the harmonic series template (Parncutt, 1988) m7 Five “root-support intervals” P1 M2 M3 P5 As vector relative to chromatic scale: 10 0 1 0 3 0 0 5 0 0 2 0

67. Perception of a C-minor triad Perception of a C-minor triad Experiential representation for extreme “overtone listeners” CDEFGAB C 1001030050020 Eb 0201001030050 G 0050020 0103 tot 1026 330180173 Implications for music theory High-register voicings: best tone to double: G best tones to add: D, Bb (  m add9, m 7 )

68. Perception of a C-minor triad Perception of a C-minor triad Experiential representation for extreme “fundamental listeners” CDEFGAB C 1002005003010 Eb 0101002005003 G 5003010 0200 tot 151213080108213 Implications for music theory Low-register voicing: best tone to double: C (  theory of the root) best tones to add: F, Ab (  7, M7)

69. Are non-notated chromas real? Are non-notated chromas real? The evidence   Many people can’t hear notated chromas! Some music students study “ear training” for years! Why should non-notated chromas be less “real”?   Consider Renaissance vocal polyphony in a church: Ear has no prior information on which partial belongs to which tone No easy way to distinguish notated from non-notated   It’s easy to model perception of non-notated chromas But hard to extract notation from signal (MIR transcription problem)   We can experience non-notated chromas directly But not “cognitive structures”   Predictions can explain basic musical structures modal and major-minor tonality