1. Introducing Multimodality
Jewitt,C., Bezemer, J., O’Halloran, K.,
2016, London-New York: Routledge, pp

2. Multimodality The first assumption is :
Multimodality does not just refer to the use of a different semiotic resources
BUT
‘Multimodality also means recognition of the different semiotic resources and of the ways in which they are combined in actual instances of meaning making’ (p.14)

3. Language Vs Mode By Language the authors mean speech and writing
By Mode (or semiotic resource) they mean ‘a set of resources, shaped over time by socially and culturally organized communities for making meaning’ (p.15)

4. Language Vs Mode Two contentious issues:
Language is the most resourceful, important and widely used of all modes
Language can be studied in isolation

5. Is Language the most resourceful, important and widely used of all modes?
Saussure stated that linguistics is a ‘branch’ of a more general science he called ‘semiology’ (the study of sign systems)
However, early 20th century semioticians suggested that language represents the most important, central and powerful of all sign system
Read citation by Chandler p

6. Is Language the most resourceful, important and widely used of all modes?
For whom?
For the hearing community, maybe yes, but signing community uses other resources to serve all social and communication needs
In what context of use?
In many occasions/contexts gesture, facial expression and other modes fulfil all the social needs of a situation (e.g. drivers and pedestrians, football players, assembling instructions such as Lego, Ikea..)
The status of language varies across communities and context of use

7. Is Language the most resourceful, important and widely used of all modes?
Transcript 2.1 (p.19) Chief Surgeon: I think you’re fine there now M. I’d jus I’d just take it all Surgical Trainee: Clip ‘em? Chief Surgeon: Yeah. Because you can see if you push up on the gall bladder – with your hook just push up the gall bledder – go through there – and then you can see you’ve got a nice big window there

8. Is Language the most resourceful, important and widely used of all modes?
In order to identify the referents of the deictic markers (now, here, there etc..) we need to visualize the context, the situation, where pointing gestures indivate these referents.
‘A multimodal orientation puts the significance of certain modes in relation to other modes, and enables us to see that the status and functions served by language relative to other modes vary significantly’. (p 20)

9. Many ‘linguistic’ principles are actually general semiotic principles.
Is Language the most resourceful, important and widely used of all modes?
Many ‘linguistic’ principles are actually general semiotic principles.
“Just as grammars of language describe how words combine in clauses, sentences and texts, so our visual ‘grammar’ will describe the way in which depicted elements – people, places and things – combine in visual ‘statements’ of greater or lesser complexity and extantion” (Kress and van Leeuwen. 2006:1))

10. Is Language the most resourceful, important and widely used of all modes?
The combination of elements and rules belonging to modes, such as image and object design, fulfils 4 main social functions:
To construct our experience of the world (experential/ideational metafunction)
To make logical connections in that world (logical/ideational metafunction)
To enact social relations (interpersonal metafunction)
To organize the message (textual metafunction)

11. Is Language the most resourceful, important and widely used of all modes?
Each mode offers distinct possibilities and limitations [≠Principle of effability (Katz, 1972)]
For a multimodal perspective language is not ‘more’ resourceful than other modes but differently resourceful
Intensity:
Speech>loudness
Writing>visual prominence (bold, capitalizing)
Gesture> speed of movement
Colour> saturation

12. Can language really be studied in isolation?
Michelle Phan’s make-up tutorial (p.24), see

13. Can language really be studied in isolation?
In a video, where speech and writing are used, there are also gesture, gaze, postures and objects. Then, there are the resources of film: camera angle, lighting, set design + post-production resources, music, voice-over added.
Language is almost always part of a whole, a text, made with a number of modes
Multimodal analysis: transcription grids (tools) + the interpretation of the relations between the multimodal constituent elements across different modes.

14. Can language really be studied in isolation?
Job interview (p. 28)

15. Can language really be studied in isolation?
Tapescript 2.3 (p. 26)
Pippa: the walks have doubled
Daniel: some
Language:
No Agent in action (the leader of the company with new policy) and no Affected Participants ( who does the double walking)>agentless form
Paralinguistic features:
‘Have’ is lenghthened (hesitation) ‘doubled’ (rising intonation) (asking for response)

16. Can language really be studied in isolation?
Modes:
Gaze
Pippa vs Daniel (attention grabber)
Facial expression
Rising intonation = Pippa’s eyebrows rise+ smiling (hedging)
Hand gestures
Rising intonation= Eyebrows rise= flat hands moving outward and ‘landing’ on the table