Mathematical Theory Mathematical Theory of Gestures in Music Guerino Mazzola U Minnesota & Zürich DANS CES MURS VOUÉS AUX MERVEILLES J’ACCUEILLE ET GARDE LES OUVRAGES DE LA MAIN PRODIGIEUSE DE L’ARTISTE ÉGALE ET RIVALE DE SA PENSÉE L’UNE N’EST RIEN SANS L’AUTRE (Paul Valéry, Palais Chaillot) DANS CES MURS VOUÉS AUX MERVEILLES J’ACCUEILLE ET GARDE LES OUVRAGES DE LA MAIN PRODIGIEUSE DE L’ARTISTE ÉGALE ET RIVALE DE SA PENSÉE L’UNE N’EST RIEN SANS L’AUTRE (Paul Valéry, Palais Chaillot)

LA VÉRITÉ DU BEAU DANS LA MUSIQUE Guerino Mazzola

Motivation - performance - and music theory - gestural music and painting - French philosophy - Embodied AI Motivation - performance - and music theory - gestural music and painting - French philosophy - Embodied AI Speculum - the musical oniontology - classification of global compositions and networks Speculum - the musical oniontology - classification of global compositions and networks Gestures - categories of gestures - hypergestures - the Escher theorem and free jazz Gestures - categories of gestures - hypergestures - the Escher theorem and free jazz Symbols - homotopy - gestoids - finitely generated abelian groups and networks Symbols - homotopy - gestoids - finitely generated abelian groups and networks

Motivation - performance - and music theory - gestural music and painting - French philosophy - Embodied AI Motivation - performance - and music theory - gestural music and painting - French philosophy - Embodied AI Speculum - the musical oniontology - classification of global compositions and networks Speculum - the musical oniontology - classification of global compositions and networks Gestures - categories of gestures - hypergestures - the Escher theorem and free jazz Gestures - categories of gestures - hypergestures - the Escher theorem and free jazz Symbols - homotopy - gestoids - finitely generated abelian groups and networks Symbols - homotopy - gestoids - finitely generated abelian groups and networks

Theodor W. Adorno („Zu einer Theorie der musikalischen Reproduktion“ 1946): Danach wäre die Aufgabe des Interpreten, Noten so zu betrachen, bis sie dem insistenten Blick in Originalmanuskripte sich verwandeln; nicht aber als Bilder der Seelenregung des Autors — sie sind auch dies, aber nur akzidentiell — sondern als die seismographischen Kurven, die der Körper der Musik selber in seinen gestischen Erschütterungen hinterlassen hat. Gestures in Performance Theory

Jürgen Uhde & Renate Wieland („Forschendes Üben“ 2002): Affekte waren ursprünglich ja Handlungen, bezogen auf ein Objekt draussen, im Prozess der Verinnerlichung haben sie sich von ihrem Gegenstand gelöst, aber immer noch sind sie bestimmt von den Koordinaten des Raumes. (...) Es gibt mithin etwas wie gestische (Raum-)Koordinaten.

David Lewin („Generalized Musical Intervals and Transformations“ 1987): If I am at s and wish to get to t, what characteristic gesture should I perform in order to arrive there? Musical Transformational Theory

Robert S. Hatten („Interpreting Musical Gestures, Topics, and Tropes“ 2004) Given the importance of gesture to interpretation, why do we not have a comprehensive theory of gesture in music? Music Theory

Cecil Taylor The body is in no way supposed to get involved in Western music I try to imitate on the piano the leaps in space a dancer makes. The body is in no way supposed to get involved in Western music. I try to imitate on the piano the leaps in space a dancer makes. Jazz

l h e sound events score analysis instrumental- interface √ thaw instrumentalizeinstrumentalize position pitch timegestures Musical theory/ notation Instruments/ playing action Musical sound Tellef Kvifte: Instruments and the electronic Age. Solum forlag, Oslo, 1988

The marks are made, and you survey the thing like you would a sort of graph. And you see within this graph the possibilities of all types of fact being planted.. David Sylvester: Interview with Francis Bacon: The Brutality of Fact. Thames and Hudson, New York 1975 Francis Bacon Painting

Charles Alunni (1951 -): Ce n‘est pas la règle qui gouverne l‘action diagrammatique, mais l‘action qui fait émerger la règle. Jean Cavaillès ( ): Comprendre est attraper le geste et pouvoir continuer. Gilles Deleuze ( ): Francis Bacon. La logique de la sensation. Editions de la Différence, Paris 1981 „graph“  „diagramme“  „geste“

Stumpy: AI Lab, U Zurich Embodied AI „Cheap design“

Motivation - performance - and music theory - gestural music and painting - French philosophy - Embodied AI Motivation - performance - and music theory - gestural music and painting - French philosophy - Embodied AI Speculum - the musical oniontology - classification of global compositions and networks Speculum - the musical oniontology - classification of global compositions and networks Gestures - categories of gestures - hypergestures - the Escher theorem and free jazz Gestures - categories of gestures - hypergestures - the Escher theorem and free jazz Symbols - homotopy - gestoids - finitely generated abelian groups and networks Symbols - homotopy - gestoids - finitely generated abelian groups and networks

The Oniontology of Music Facts signs realities communication Processes Gestures

Classify! The category GloCom A of global objective A-addressed compositions has objects K I, i.e., coverings of sets K by atlases I of local objective A-addressed compositions with manifold gluing conditions and manifold morphisms f  : K I  L J, including and compatible with atlas morphisms  : I  J I IV II VI V III VII facts

Theorem (global addressed geometric classification) Let A = locally free of finite rank over commutative ring R Consider the objective global compositions K I at A with (*): the chart modules R.K i are locally free of finite rank the function modules  (K i ) are projective the function modules  (K i ) are projective There is a subscheme J n* of a projective R-scheme of finite type whose points  : Spec(S)  J n* parametrize the isomorphism classes of objective global compositions at address S  R A with (*). Theorem (global addressed geometric classification) Let A = locally free of finite rank over commutative ring R Consider the objective global compositions K I at A with (*): the chart modules R.K i are locally free of finite rank the function modules  (K i ) are projective the function modules  (K i ) are projective There is a subscheme J n* of a projective R-scheme of finite type whose points  : Spec(S)  J n* parametrize the isomorphism classes of objective global compositions at address S  R A with (*).

K I can be reconstructed from the coefficient system of retracted functions on free global compositions res*n  (K I )  n  ( A  n* ) Fact: This construction is a special case of a local network. Global compositions are classified by limits of powerset denotators. K I can be reconstructed from the coefficient system of retracted functions on free global compositions res*n  (K I )  n  ( A  n* ) Fact: This construction is a special case of a local network. Global compositions are classified by limits of powerset denotators. Ÿ 12 T4T4T4T4 T2T2T2T2 T 5.-1 T D     processes

A global network

Local and Global Limit Denotators and the Classification of Global Compositions. COLLOQUIUM ON MATHEMATHICAL MUSIC THEORY H. Fripertinger, L. Reich (Eds.) Grazer Math. Ber., ISSN 1016–7692 Bericht Nr. 347 (2005), Global Networks in Computer Science? Theorem Given address A in Mod, we have a verification functor |?|: A Lf Mod red  A Glob from the category A Lf Mod red of reduced, A-addressed locally flat global networks to the category A Glob of A-addressed global compositions. Theorem Given address A in Mod, we have a verification functor |?|: A Lf Mod red  A Glob from the category A Lf Mod red of reduced, A-addressed locally flat global networks to the category A Glob of A-addressed global compositions. Corollary There are non-interpretable global networks in A Lf Mod red Corollary

Gesture Theory in Computer Music Research: Frédéric Bevilacqua Claude Cadoz Claude Cadoz Antonio Camurri Antonio Camurri Rolf Inge Godøy Rolf Inge Godøy Stefan Müller Norbert Schnell Koji Shibuya McAgnus Todd Marcelo Wanderley Marcelo Wanderley etc. etc. Gesture Theory in Computer Music Research: Frédéric Bevilacqua Claude Cadoz Claude Cadoz Antonio Camurri Antonio Camurri Rolf Inge Godøy Rolf Inge Godøy Stefan Müller Norbert Schnell Koji Shibuya McAgnus Todd Marcelo Wanderley Marcelo Wanderley etc. etc.

Ryukoku Koji Shibuya‘s Ryukoku violin robot

facts: complete classification of addressed global compositions facts: complete classification of addressed global compositions processes: relative classification of global networks via a functor to global compositions processes: relative classification of global networks via a functor to global compositions gestures: no mathematical theory gestures: no mathematical theory Mathematical Music Theory

Motivation - performance - and music theory - gestural music and painting - French philosophy - Embodied AI Motivation - performance - and music theory - gestural music and painting - French philosophy - Embodied AI Speculum - the musical oniontology - classification of global compositions and networks Speculum - the musical oniontology - classification of global compositions and networks Gestures - categories of gestures - hypergestures - the Escher theorem and free jazz Gestures - categories of gestures - hypergestures - the Escher theorem and free jazz Symbols - homotopy - gestoids - finitely generated abelian groups and networks Symbols - homotopy - gestoids - finitely generated abelian groups and networks

a 11 x+a 12 y+a 13 z = a a 21 x+a 22 y+a 23 z = b a 31 x+a 32 y+a 33 z = c a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 xyz abc = rotation matrix equation algebra compactifies gestures to symbolic formulas

„attempt at resuscitation“ Peter Gabriel: Symbolic formulas via digraphs = „quiver algebras“ SP T Q K T X RKRKRKRK R K = R[X] polynomial algebra mathematics of transformational theory

Graphs are only the „skeleton“ of gestures, the „flesh“ is missing. ?

(Local) Gesture = morphism g: D  of digraphs with values in a spatial digraph of a topological space X (= digraph of continuous curves in X) XX  D position pitch time X g body skeleton

A gesture morphism u: g  h is a digraph morphism u, such that there is a continuous map f: X  Y which defines a commutative diagram: f  D E X  Y  g h u G (g, h) category G of (local) gestures Advantage: Digraphs have an inherent (intuitionistic) logic, because the category of digraphs is a topos. This is not only cheap, but free design! Advantage: Digraphs have an inherent (intuitionistic) logic, because the category of digraphs is a topos. This is not only cheap, but free design!

A global gesture (only bodies shown)

x z y   1 1 (t ) )   6 6 ) )   2 2 ) )   3 3 ) )   4 4 ) )   5 5 ) ) One hand  product  =  1  2  3  4  5  6 of 6 gestural curves in space-time (x,y,z;e) of piano j = 1, 2,... 5: tips of fingers j = 6: the carpus e = time  1 = Ÿ

Stefan Müller

D p real forms? tip space positionpitchonset Renate Wieland & Jürgen Uhde: Forschendes Üben Die Klangberührung ist das Ziel der zusammenfassenden Geste, der Anschlag ist sozusagen die Geste in der Geste.

circle knot „loop of loops “ Hypergestures! Digraph ( F, ) = topological space of (local) gestures of of digraph F with values in a spatial digraph. Notation: X   X 

space space time ET-dance gesture

E g h hypergesture impossible! g h morphism exists!

Gestural maps are particular continuous maps (u,v):  canonically induced by a pair of maps u: G  F (digraphs) v: X  Y (continuous) The category HG = HG 1 of hypergestures has 1) hypergestures as objects and 2) gestural maps as continuous maps.  The category HG n of n-fold hypergestures has 1) objects: n-fold hypergestures g: F n  F F X 2) (n-1)-fold gestural maps as continuous maps. 

Have chain of successively refined gestural categories G  HG = HG 1  HG 2 ... HG n  HG n+1 ... which represent the granularity of gestural relations, much as in differential geometry, where the categories of n-times differentiable manifolds do. E.g. gluing local gestures to global gestures in quasi-anatomic joints

Proposition (Escher Theorem) Given a topological space X, a sequence of digraphs F 1, F 2,... F n and a permutation  of 1, 2,... n. Then there is a homeomorphism F F  F  F   

g h k Comprendre est attraper le geste et pouvoir continuer.

Motivation - performance - and music theory - gestural music and painting - French philosophy - Embodied AI Motivation - performance - and music theory - gestural music and painting - French philosophy - Embodied AI Speculum - the musical oniontology - classification of global compositions and networks Speculum - the musical oniontology - classification of global compositions and networks Gestures - categories of gestures - hypergestures - the Escher theorem and free jazz Gestures - categories of gestures - hypergestures - the Escher theorem and free jazz Symbols - homotopy - gestoids - finitely generated abelian groups and networks Symbols - homotopy - gestoids - finitely generated abelian groups and networks

homotopiccurves X Gestoids: From Gestures to Symbols composition of homotopic curves is associative X Algebraic Topology

The homotopy classes of curves of a gesture g define the Gestoid G g of a gesture g. This consists of the linear combinations  n a n c n of homotopy classes c n of curves between given points x, y of gesture g. y x

e i2  t — i—i—i—i—1 i X = S 1 G g  ¬  1 (S 1 ) fundamental group  1 (S 1 )   Ÿ e i2  nt ~ n ~ Fourier formula f(t) =  n a n e i2  nt ~ Fourier formula f(t) =  n a n e i2  nt  n a n e i2  nt g:  1 (X)   Ÿ n ? finitely generated abelian groups?  1 (X)   Ÿ n ? finitely generated abelian groups?

F 2F 3F 4F Fourier ballet QED

ZnZnZnZn L n,1 S3S3S3S3 1 action of Ÿ n  1 (L n,1 )  Ÿ n  1 (S 3 )  0

gestures = ? string theory of music gestures = ? string theory of music