Introducing Multimodality Jewitt,C., Bezemer, J., O’Halloran, K., 2016, London-New York: Routledge, pp. 14-29.

Multimodality The first assumption is : Multimodality does not just refer to the use of a different semiotic resources BUT ‘Multimodality also means recognition of the different semiotic resources and of the ways in which they are combined in actual instances of meaning making’ (p.14)

Language Vs Mode By Language the authors mean speech and writing By Mode (or semiotic resource) they mean ‘a set of resources, shaped over time by socially and culturally organized communities for making meaning’ (p.15)

Language Vs Mode Two contentious issues: Language is the most resourceful, important and widely used of all modes Language can be studied in isolation

Is Language the most resourceful, important and widely used of all modes? Saussure stated that linguistics is a ‘branch’ of a more general science he called ‘semiology’ (the study of sign systems) However, early 20th century semioticians suggested that language represents the most important, central and powerful of all sign system Read citation by Chandler p. 15-16.

Is Language the most resourceful, important and widely used of all modes? For whom? For the hearing community, maybe yes, but signing community uses other resources to serve all social and communication needs In what context of use? In many occasions/contexts gesture, facial expression and other modes fulfil all the social needs of a situation (e.g. drivers and pedestrians, football players, assembling instructions such as Lego, Ikea..) The status of language varies across communities and context of use

Is Language the most resourceful, important and widely used of all modes? Transcript 2.1 (p.19) Chief Surgeon: I think you’re fine there now M. I’d jus I’d just take it all Surgical Trainee: Clip ‘em? Chief Surgeon: Yeah. Because you can see if you push up on the gall bladder – with your hook just push up the gall bledder – go through there – and then you can see you’ve got a nice big window there

Is Language the most resourceful, important and widely used of all modes? In order to identify the referents of the deictic markers (now, here, there etc..) we need to visualize the context, the situation, where pointing gestures indivate these referents. ‘A multimodal orientation puts the significance of certain modes in relation to other modes, and enables us to see that the status and functions served by language relative to other modes vary significantly’. (p 20)

Many ‘linguistic’ principles are actually general semiotic principles. Is Language the most resourceful, important and widely used of all modes? Many ‘linguistic’ principles are actually general semiotic principles. “Just as grammars of language describe how words combine in clauses, sentences and texts, so our visual ‘grammar’ will describe the way in which depicted elements – people, places and things – combine in visual ‘statements’ of greater or lesser complexity and extantion” (Kress and van Leeuwen. 2006:1))

Is Language the most resourceful, important and widely used of all modes? The combination of elements and rules belonging to modes, such as image and object design, fulfils 4 main social functions: To construct our experience of the world (experential/ideational metafunction) To make logical connections in that world (logical/ideational metafunction) To enact social relations (interpersonal metafunction) To organize the message (textual metafunction)

Is Language the most resourceful, important and widely used of all modes? Each mode offers distinct possibilities and limitations [≠Principle of effability (Katz, 1972)] For a multimodal perspective language is not ‘more’ resourceful than other modes but differently resourceful Intensity: Speech>loudness Writing>visual prominence (bold, capitalizing) Gesture> speed of movement Colour> saturation

Can language really be studied in isolation? Michelle Phan’s make-up tutorial (p.24), see https://www.youtube.com/watch?v=OB8nfJCOIeE

Can language really be studied in isolation? In a video, where speech and writing are used, there are also gesture, gaze, postures and objects. Then, there are the resources of film: camera angle, lighting, set design + post-production resources, music, voice-over added. Language is almost always part of a whole, a text, made with a number of modes Multimodal analysis: transcription grids (tools) + the interpretation of the relations between the multimodal constituent elements across different modes.

Can language really be studied in isolation? Job interview (p. 28)

Can language really be studied in isolation? Tapescript 2.3 (p. 26) Pippa: the walks have doubled Daniel: some Language: No Agent in action (the leader of the company with new policy) and no Affected Participants ( who does the double walking)>agentless form Paralinguistic features: ‘Have’ is lenghthened (hesitation) ‘doubled’ (rising intonation) (asking for response)

Can language really be studied in isolation? Modes: Gaze Pippa vs Daniel (attention grabber) Facial expression Rising intonation = Pippa’s eyebrows rise+ smiling (hedging) Hand gestures Rising intonation= Eyebrows rise= flat hands moving outward and ‘landing’ on the table