Jewitt, C. (2014). The Routledge Handbook of Multimodal Analysis Jewitt, C. (2014). The Routledge Handbook of Multimodal Analysis. London: Routledge. pp. 22-28 and pp. 31-43. Corroborated by Jewitt, Bezemer, O’Halloran, 2016, pp. 155-161.

CORE CONCEPTS FOR MULTIMODAL ANALYSIS Meaning making Artefact Mode & semiotic resource Semiotic principle Metafunctions (will be presented in the workshops) Modal affordances and meaning potential Motivated sign

Meaning Making Meaning making is a general term used to recognize that meaning always involves a social actor. ‘Expressing’ and ‘interpreting’ or ‘understanding’ is an act of making meaning. Meaning making is a more generic term than communication, which is usually defined as involving ‘expression’ by one actor and ‘interpretation’ by another.

Artefact Artefact refers to anything that bears the traces of semiotic work: a building, an inscription, a video recording or film, a landscape etc. We use the term specifically to refer to artefacts produced by the people whose practices we want to study.

Mode Mode is a term that is used within systemic functional linguistics and social semiotics to refer to a socially organized set of semiotic resources for making meaning. Examples of modes include image, writing, layout and speech, among others. A mode needs to have a set of semiotic resources and organizing priniciples that are recognized within a community as realizing meaning. For example, the resource of gesture have been semiotically shaped into communicative modes to serve a diverse range of communities (e.g. hearing impaired communities, visually and hearing impaired people, ballet dancers).

Semiotic Resource Semiotic resource is a term used to refer to the meaning potential of material resources, which developed and accumulated over time through their use in a particular community and in response to certain social requirements of that community

Semiotic Principle Semiotic principle is a term that is used to refer to principles for and features of meaning making that apply across modes. For instance, all modes have resources for producing Intensity. However, Mode How to realize Intensity: Speech loudness or lexically by adding ‘very’ Writing visual prominence (bold, capitalizing) Gesture speed or the extent of movement Colour saturation

Modal Affordances and Meaning Potential Affordance is a social semiotic term used to refer to the idea that different modes offer different potentials for making meaning. Modal affordances affect the kinds of semiotic work a mode can be used for, the ease with which it can be done and the different ways in which modes can be used to achieve broadly similar semiotic work. Modal affordances are connected to both a mode’s material and its social histories, that is, the social purposes that it has been used for in a specific context.

Motivated Sign Motivated sign is a social semiotic term used to reference that meaning (signified) and form (signifier) are brought together in a relation motivated by the aptness of fit (appropriateness) between the interest (intentionality) of the sign maker and the affordances of a semiotic resources.

Multimodal analysis – first steps After contextualizing the artefact, analyse: The modes employed The function of each mode in the meaning making process Meanings derived from the orchestration among modes

Multimodal analysis – first steps After contextualizing the artefact, analyse: The modes employed The function of each mode in the meaning making process Meanings derived from the orchestration among modes

Intertextuality

DIFFERENT APPROACHES TO MULTIMODALITY Systemic functional multimodal discourse analysis (SF-MDA) Social semiotic multimodality (Social semiotics & Critical Discourse Analysis) Multimodal interactional analysis (Conversation Analysis ) Others: Cultural Studies, Usability, etc..

Systemic functional multimodal discourse analysis (SF-MDA) Systemic Functional Linguistics (SFL) is the theory of language developed by Michael Halliday. Language is viewed as a social system, i.e. a meaning making process. The functions that language has evolved to serve in society are reflected in its underlying organization. Then, SFL has developed a systemic functional grammar (SFG) to represent the meaning making potential of language (as a mode) and to provide a model thanks to which we can understand actual language use (texts). The same two dimensions are used in systemic functional multimodal discourse analysis.

Social semiotic multimodality Social semiotics is an approach concerned with how the processes of meaning making (communication > signification, interpretations > semiosis) shape and are shaped by individuals and societies to realize power and ideologies. It stresses the relationship between modes and their affordances and the social needs they are used to serve, the agency of the sign maker and the context of meaning making. It is related to three main strands of influence : SFL, Semiotics, and Critical linguistics and Critical Discourse Analysis (CDA)

Multimodal interactional analysis This approach moves from Conversation Analysis (CA) that is the study of social interaction. In its early days, the focus was on analysing ‘talk’ and ‘conversation’. The analytical scope has since been expanded to include attention to a much broader range of semiotic resources used in interaction. Central to a CA approach is the systemic and in-depth analysis of video recordings of multimodal interaction, with a focus on the sequential organizational of action and the coordination of action. Multimodal interactional analysis investigates how people co-deploy language, gesture, gaze posture, movement, space and objects to mediate interaction in a given contexts

Different Approaches to Multimodality (cf Different Approaches to Multimodality (cf. Jewitt, Bezemer, O’Halloran, 2016:11)