What is multimodality? Bella Dicks School of Social Sciences Cardiff University

‘all the modes employed contribute to meaning…’ science centre exhibition hall

modes and media Modes - abstract, non-material resources of meaning-making (e.g. the ‘languages’ of image, sound, gesture, colour, texture, narrative) Modes - abstract, non-material resources of meaning-making (e.g. the ‘languages’ of image, sound, gesture, colour, texture, narrative) Media - specific material forms in which modes are realised (e.g. a photograph, a computer screen, an audio recording, paint, a wall, clothing, a chisel, a poem, a block of wood) Media - specific material forms in which modes are realised (e.g. a photograph, a computer screen, an audio recording, paint, a wall, clothing, a chisel, a poem, a block of wood) Kress, G.and Van Leeuwen, T. (2001). Multi-modal Discourse. London: Arnold.

multimodality –people communicate through: speech, accent, gesture, facial expression, gaze- direction, shape, weight, size, body posture, position & movement, dress (colour, shape, texture, etc.), hairstyle, etc. –environments/objects communicate through: colour, texture, shape, position, opaqueness/light, weight, movement, sound, etc.

multimodality in the research process 1. observations/elicitations of the field (=data) 2. recording of data 3. analysis of data records 4. representation/dissemination of research findings N.B. are data recorded, represented, reproduced, reduced, transformed?

1. Multimedia in the field Material environments, e.g. Material environments, e.g. –Relative position of objects –Walls and corridors –Colours, shapes and sizes Social interaction, e.g. Social interaction, e.g. –Relations of proximity –Sequencing of events –Turn-taking in conversation

modalities of the field setting Exhibits HallScience Theatre

framing: a multimodal principle framing works across diverse media: e.g. in interior design; in sound compositions, in paintings. introduces breaks and discontinuities; contrasts and continuities; uses framelines and empty space

framing In writing; e.g. punctuations, prepositions and paragraphs In writing; e.g. punctuations, prepositions and paragraphs In images; e.g. lines and borders In images; e.g. lines and borders In sound; e.g. pacing, intonation and rhythm In sound; e.g. pacing, intonation and rhythm In material environments; e.g. object positioning, corridors and walls In material environments; e.g. object positioning, corridors and walls In movement; e.g. jumps, gestures and flows In movement; e.g. jumps, gestures and flows In web-pages; e.g. pop-up boxes, graphic layout and hyperlinks In web-pages; e.g. pop-up boxes, graphic layout and hyperlinks

‘Framing parcels up the conceptual world (…) It marks off, but in doing so it establishes, at the same time, the elements which may be joined. (…) “Without framing, no meaning”, we might say.’ Gunther Kress (2003) Literacy in the new media age, p. 123/4/5.

audio recordings: framing what’s said PausesHesitations Ordering of items Soundscapes

2. recording multimodality What are the affordances of different recording media? What are the affordances of different recording media? –photographs v. writing –photographs v. video –writing v. sound

multimodal data-records still images deploy: still images deploy: colour shapelight directionality perspective position borders size voice recordings deploy: voice recordings deploy: accentexpression tonepitch languagevolumecadencerhythm

framing children’s movement

recording media E.g. cameras embody their own framing devices: –edge v. centre of photograph –composition of frame –angle and position of camera –choice of what to record –decision about when to press shutter –Etc.etc.

data records 1. photographs record spatial framing e.g. boundaries & transitsrecord spatial framing e.g. boundaries & transits 2. audio recordings –record temporal framing e.g. sequencing of items; pauses, etc. 3. video recordings –record spatial and temporal framings 4. fieldnotes –record spatial and temporal framings

fieldnotes ‘When you enter [the Science Centre], you see a huge, well-lit hall in front of you with high ceilings and a gallery above. It is a white space with big white pillars and vast windows, but there's also lots of colour, noise and movement. On entering, immediately ahead of you is a plastic ball, suspended seemingly by magic in front of a bright yellow solid pyramid. Beyond are more yellows and reds and greens and blues. All around are a variety of different brightly-coloured devices or machines, housed in brightly coloured casings and consoles. Many of them move, produce sounds, create visual effects, and so forth when activated by a user… … There are lots of children moving around excitedly, flitting from exhibit to exhibit, sometimes bumping into each other.