Discourse and Pragmatics Lecture 11 Multimodal Discourse Analysis

The changing face of texts

Multimodal Discourse Analysis Two Strands Analysis of Interaction Influence of Conversation Analysis, Interactional Sociolinguistics, Mediated Discourse Analysis Kendon, Goodwin, Norris Analysis of Static Texts Influence of ‘Textual Analysis’ (Halliday-- Systemic Functional Grammar) Kress and Van Leeuwen Reading Images

MULTIMODAL ANALYSIS OF STATIC TEXTS Strand I: MULTIMODAL ANALYSIS OF STATIC TEXTS

Reading Images http://personal.cityu.edu.hk/%7Eenrodney/LCC/readingfour.htm Social Semiotics Halliday’s three ‘metafunctions’ of language Ideational What’s happening? (processes and participants) Interpersonal Relationship between text and reader Textual How the text is organized

Typography

Kinds of Images Ideational Narrative Image Conceptual Image “presenting unfolding actions and events, processes of change, transiting spatial arrangement” (Kress & van Leeuwen 1995, p.79) Conceptual Image “representing participants in terms of their more generalized and more or less stable and timeless essence” (ibid.) Analytical Process “relate participants in terms of a part-whole structure” (p.89) Classificational Process “relate participants to each other in terms of a ‘kind of’ relation, a taxonomy” (p. 81)

Narrative Image Ideational “what in language is realized by words of the category ‘action verbs’ is in pictures realized by elements that can be formally defined as ‘vectors’. (p. 44)

Ideational Analytical Image Parts whole: the earth

Classificational Image Ideational Classificational Image

Classification + Narrative

Perspective/ Point of View Ideational Perspective/ Point of View

Interpersonal (Gaze)

Interpersonal Demand “the participant’s gaze (and the gesture, if present) demands something from the viewer, demands that the viewer enter into some kind of imaginary relation with him or her.” (p.122)

“The viewer’s role is that of an invisible onlooker” Interpersonal “The viewer’s role is that of an invisible onlooker” (p.124) “All images which do not contain human or quasi-human participants looking directly at the viewer are of this kind.” “it ‘offers’ the represented participants to the viewer as items of information, objects of contemplation, impersonally, as though they were specimens in a display case.”

Distance

Textual Visual Space

Left-Right L Given R New

Top-Bottom T Ideal B Real

Intertextuality

Questions to Ask Rhetorical Purposes Where does the image appear? What is the image's purpose? Ideational Function Does it document a  situation, event or condition? Is it conceptual? If so, what is its point? Who is portrayed? Describe your inferences from each feature of the person(s) - age, details of dress, gender, ethnicity, class, posture and stance, portions of the body shown, tilt of head, facial expression, gesture of hands. What is the person looking at? Follow her gaze or eyeline. Does she look toward something else in the image? or out of the picture? What do you make of the direction of the gaze? .

Questions to Ask If there are two or more people, what features suggest their relationship to each other? If there are two or more people, does one seem dominant? How is this expressed? If the image has a distinct background, describe it. How does it relate to the dominant focus of the images, especially people, if any? What is the story being told in the image? Consider the people and objects in the image and their relationships to each other, the viewer, the setting and the text. Who or what is excluded from the image? Why? What time and place does the image suggest? What is the effect of this setting? Is anything "out of place" in the image? What do you make of the incongruity? Are there items or features in the image that might mean more than themselves? Consider connotations and associations of particular objects or features in the image. Relate them to the rest of the image

Questions to Ask Interpersonal Function From what angle are the people shown? Do you seem to look down on them, as if they were below the viewer? Look up at them? Look right at them? Are people shown close up or far away? What emotional effect does this have? What do you consider to be your relationship as viewer to the person or people shown? Do you empathize with them or not? Explain why. If there is no one represented, imagine what sort of person would be at home in this image. Explain why?

Questions Is the image realistic, like a photograph or more stylized, like a cartoon or caricature? If the image is realistic, do you detect any types of distortion? Describe any features that may be distorted. How polished or "professional" is the image?What tone does the image project? How seriously are we to take this image? Explain why. Who do you think are the intended viewers of the image? What features or context suggest this audience? Who do you think produced the image? Is the creator/photographer stated? What would you say your relationship is to the producer or producers? Do you think they understand you, as a viewer? What attitudes - - social, political, economic, cultural - - are suggested in this image? Who benefits from the attitudes shown? Who does not? Who can relate to the story that the image tells? Who may not find it believable or interesting?

Questions to Ask Textual Function: Overall Design What draws your eye first? What does the dominant part of the image portray? What is in the center of the image? What is shown in front and larger? What is behind and smaller? What is shown in the upper half? the lower half? Are portions more blurred? Are there very distinct parts in sharp focus? Is there "empty" space? What does the empty space frame?Are some areas or shapes very large? Are others very small? Describe the major shapes and lines created. Consider what effect the shapes and lines create. Describe the overall arrangement of parts. Are they ordered symmetrically or otherwise balanced against each other?

Questions to Ask Color Describe the colors, or absence of color, in the image. Where is color applied? Is the color realistic, in your view? If not, describe why you think it is not. How does color, or its absence, make you feel about the image? What previous associations do you have with the colors used? How do those affect your understanding of the image? Text If the image includes text, such as headlines, labels, captions, or paragraphs of explanation, relate the text to the image. In what ways does the text help you make sense of the image? Does it answer questions about the image, or only raise more questions? What is the personality and tone of the typography, or the fonts that the text uses?

MULTIMODAL ANALYSIS OF INTERACTIONS Strand II: MULTIMODAL ANALYSIS OF INTERACTIONS

Multimodality Proxemics and Posture Written Text Gaze Gesture Object Handling

Multimodal Interaction Analysis (Norris 2004, Norris and Jones 2005) All communicative actions are mediated though one or more semiotic modes Affordances Mediational Means Action Actor Semiotic modes It’s important to note that semiotic modes also place affordances and constraints on the other semiotic modes that might be used with it Reading gaze Semiotic modes come with different sets of affordances and constraints Constraints Modal Density Modal Complexity

Modal Density: Leena Body/Head Position Spoken Language Body/Head Gaze Gaze Written Text Written Text Object Handling

Modal Density: Derek Body/Head Position Body/Head Position Gaze Gaze Spoken Language Body/Head Gesture Gesture

Modal Density: Leena Body/Head Position Spoken Language Gaze Body/Head Gesture Gaze Object Handling Written Text Written Text

Modal Complexity Gesture Media Gaze Rhythm Object and Handling Timing The use of one semiotic mode affects all the other modes that are used along with it Gesture Media Gaze Rhythm and Timing Object Handling

Gaze Relationship to Speech Cultural Practices of Gaze Kendon (1990) Hearers gaze at speakers more than speakers gaze at hearers Speakers typically look towards hearers at the end of phrases Goodwin (1981) Mutual gaze occurs more typically at turn beginnings Cultural Practices of Gaze Gaze and power among East Asians Gaze and Attention/Awareness Modal Density (Norris 2004) Kendon 1967 Hearers gaze at speaker more than speaker gazes at hearers Speakers typically look towards hearers at the end of phrases Goodwin 1981 Mutual gaze at turn beginnings.

Relationship to Media

Leena and Derek

Mutual Gaze

Conversational Synchrony Birdwhistle, 1970 Postural Echo

Asynchrony

Parallel and Sequential Synchrony Gill, Kawamori, Katagiri, Shimojima, 2000; Gill, 2001; Gill, 2002; Gill and Borchers, 2003; Gill, 2003a; Gill, 2003b Sequential Synchrony: action-reaction Parallel Synchrony: parallel motion Interaction tends to move from sequential to parallel synchrony Sequential Synchrony Serves to maintain the interaction Parallel Synchrony Serves to ‘transform’ the interaction ‘Grounding’ rhythmic synchrony (Birdwhistle, 1970) Gill: Sequential Body Moves have the structure of action-reaction motion, whilst Parallel Coordinated Moves have the structure of parallel motion (). They have different priorities in their functionality. Sequential moves serve to maintain the communication, whilst parallel moves serve to transform the communication. There is a pulsation in the movement from sequential to parallel action that facilitates the process of grounding. 
 When a designer is making contact with the surface to act upon it, whilst the other person is doing so too, there is an attempt to engage with the body field of the other person, as found in the case of landscape architects drawing up conceptual designs together. It also happens in the example below (Figure 1) of design activity using a whiteboard. The designer on the right side, closest to us (E), enters the body field of the other one (F) who is currently drawing (action), and uses his index finger to trace out a shape to indicate a bed. He is proposing this idea to (F) who is drawing, to get his opinion (negotiation). Both action and negotiation are operating at the surface. 
The body field of the person drawing (F) is not disturbed, and as we know from the discussion of the engagement space, and overlapping space, this indicates a high degree of contact and is identifiable as a Parallel Coordinated Move (PCM). Space is considered as the space between bodies, and it is a resonating space. When parallel coordinated action occurs, the resonance is at a peak and this expands the space, giving it salience over time. In sequential Body Moves, resonance between bodies is variable depending on the degree of contact and commitment of the persons to each other, and time is more salient than space. Hence, imagine that I am drawing something with you on a shared sheet of paper, the sequential Body Moves that we perform of action-reaction (to show we acknowledge, agree, disagree, accept, etc), are the dynamics within the overlap of our physical communicative space, but do not overlap at a metacommunicative level. The parallel Body Moves we perform of, say, drawing different parts of the design at the same time on this sheet of paper, do not overlap in physical communicative space, but completely overlap in metacommunicative space

Leena

so...ummm

is there

anything parTICular

YOU think

you want some more

help with

mmmmm

I'd like to uh... 1 2

do you know the meaning

of this paragraph

yeah I understand

Leena ‘Can you feel it?’

and they can do anything

that they LIKE

and

that's a stop

can you feel it

that's a stop

and then you start a NEW sentence

Conversational Synchrony Derek (0.5 sec per frame)

is this is the introduc

tion

like the

yes introduction

the intro

duction

ehh

this to

this this

yes chapter

Gesture Associated with clarity and immediacy Different kinds of gestures Iconic/Metaphorical Deictic (pointing) Holding Facial Touching Body Adjusting Iconic gestures Pictoral conent Metaphorical gestures Abstract ideas Deictic gestures Beat gestures

‘Work Space’ Gestures define the ‘work space’ Relationship to higher order and lower order concerns

Leena and Derek

Type of Gesture

Leena’s Pointing

Leena’s Pointing

you're making a comparison Derek you're making a comparison

Derek’s Hand Hold

Workspace Derek

in this paragraph

I

want to

critique

the

incorrect

impression

by

this group of

people

Object Handling Pens Texts Books (Dictionaries, etc.) Mouse/Keyboard ‘Keep the pen or keyboard and mouse in the writer’s hands -- not in yours.”

Text alignment

1 min. ADD TIMESTAMPS

3 min

5 min

7 min

9 min

11 min

14:30

Leena and Derek

Interactional space = Text Leena Interactional space = Text

Leena and her Pen Pointing Writing Holding Gesturing Conducting

‘Do you have anything else…’ 1 sec Leena ‘Do you have anything else…’ 1 sec

is there

anything else

okay

yeah

Derek offers the pen

they don't know

then maybe

you can put that in

Tutor Feedback Sessions Leena ‘I can’t believe it. I thought I was really being interactive…I tried to look at him more…but it seems not…scary!’ Derek ‘I really just wanted the client to explain more…um…his idea…it got easier after I gave him the pen.’

Leena 2

Conclusions Relationship among modes Effect of media Gaze Conversational Synchrony Gestures Object Handling