1. MULTIMODALITY AND GENRE : CLUSTERING WORDS AND IMAGES IN TEXTS
Fauziah Taib
Faculty of Languages and Linguistics
University of Malaya
Kuala Lumpur, Malaysia.

2. Multimodality and Genre :
Clustering Words and Images in Texts
Five parts of the presentation :
Introduction : Multimodality and genre
The cluster analysis framework
An example of cluster analysis
A system network for the genre of business-to-business brochures
Summary

3. Multimodal Texts
Texts that use multi-semiotic modes such as language, images, layout, colours, typography and these modes are
integrated in the meaning making of the texts.
All meaning making practices like early spelling, classroom interaction and scientific papers are multimodal for they
deploy more than one mode of communication.
(Kress, 1997; Kress and van Leeuwen,2001; Lemke 1998).

4. Images have meaning potential
contact
subjectivity
social distance

5. The meaning of composition / layout
Margin
Ideal
Given
Margin
Ideal
New
centre
Margin
Real
Given
Margin
Real
New
(Kress & van Leeuwen, 1996)

6. Weight, expansion, slope, curvature, connectivity, orientation etc

7. Colours have meaning potential
Warm colours
Cold colours
Brightness, saturation, purity, modulation, hue, luminosity

8. Genre in Systemic Functional Linguistics (SFL)
Genre, in SFL, is seen as a
staged, goal-oriented social process (Martin, 1992) meaning each
stage in the genre contributes with one part of the overall that must be
made for the genre to accomplish its aims successfully.
Research in this area sought to
explicate the distinctive stages, moves of genres together with patterns
of lexical, grammatical, and cohesive choices which construct the
function of the stages of the genre (Rothery, 1996).

9. Why the SFL approach to genre theory?
Despite focusing on language, the principles of SFL reflect the awareness of the fact that
…there are other modes of meaning, in any culture, which are outside the realm of language.
(Halliday and Hasan, 1985)
It has a systematic model of grammar which makes extensive use of a stratified, metafunctional grammar.
(Hyland, 2002)

10. Example of genre analysis
Text : magazine advice column
Functional label
I lied on my cv
Revealing a problem (confession)
Should I come clean with my boss
Appealing for help (question)
Yes
Providing a solution (answer)
But prepared for the possibility of losing your job if you have a scrupulous Boss
Issuing a caveat (warning)
The bright side: you will gain her respect if you speak up and accept your mistake
Predicting the result (prediction 1)
and having got this burden off your chest will help you focus better on your work
(prediction 2)
Multimodal
realisations
Language T y p o g r a h
Overall communicative purpose : Solution to a problem
(van Leeuwen, 2005)

11. Clusters are functionally related to each other.
Cluster Analysis in Multimodal Texts
(Baldry and Thibault, 2006)
Analyzing clusters which refer to a local grouping of items which may be verbal and/or visual in nature and which are spatially proximate thereby defining a specific region or subregion of the page as a whole.
Clusters are functionally related to each other.

12. Gestalt Psychology The Law of Figure-Ground Discrimination
The Law of Closure
The Law of Proximity
The Law of Similarity
& many more…
Gestalt Psychology

14. Business-to-business texts
texts produced to advertise products or services to other businesses.
texts are all about disseminating information for “content is king for business-to-business marketing” (Murphy, 2007) unlike business-to-consumer brochures which appeal to the emotions of the individual customers (Kotler et al. 2002).

15. 6 - step procedure in analysing large sample
of multimodal texts.
1.Clustering
2.Refining the clusters
3.Identifying functions of clusters
4.Labelling clusters
5.Grouping the clusters
6.Identifying major clusters

16. Step 2 : Refining the title cluster
new cluster

17. Other parameters of clustering ?
Local groupings based on spatial proximity.
Local groupings based on lexical connectivity.
Local groupings based on contents of items.

18. Step 3 : Labelling cluster No. Clusters Function(s)
Step 4 : Identifying function(s)
No.
Clusters
Function(s)
Multimodal realisations 1.
Detail
state date, venue and/or time and/or name of facilitator
-Language
-typography
-layout etc 2.
Course highlights
state contents of programme 3.
Training provider information
i. identify the company organizing the programme
ii. identify corporate identity 4.
Call and contact
provide contact details
-Typography

19. Step 5 : Grouping clusters Step 6 : Identifying major cluster
Title
Detail
Methodology
resource material
Content
Focus
Target audience
training provider information
Registration
Call and contact
testimony
Fee
Introduction
Purpose
Objective
Benefit
Illustration
Focuser
catch phrase
complimentary item
certificate of completion
discounts and savings
claimable fee
facilitator’s profile
training company’s profile
accreditation
Groups (9)
Major clusters
Detail
Methodology
Resource material
Content
Focus
Target audience
Registration
Call and contact
Testimony
Fee
Introduction
Purpose
Objective
Benefit
Programme information
major cluster
Response solicitation major cluster
Value major cluster
Justifier major cluster

20. Major clusters of business-to-business texts
1. Title
2. Programme information
3. Signature
4. Value
5. Justifier
6. Attention- getter
7. Incentive
8. Credibility
9. Response solicitation

21. System network for the generic structure of Malaysian business-to-business brochures
This would be one of the earliest attempts to establish a system network for a genre.
The syntagmatic and the paradigmatic relations of elements can be presented in what is called as the system (Eggins, 2004).

22. SYSTEM NETWORK FOR THE GENERIC STRUCTURE OF BUSINESS-TO-BUSINESS TEXTS

23. Detail + Content + Methodology
Realisations of each
cluster type
Syntagmatic relations
of major clusters
Paradigmatic
choices
Title
One-cluster
Title
Value
One-cluster
Testimony
Fee
Registration
Call and contact
One-cluster
Solicit response
Two-cluster
Registration + Call and contact
Detail + Content
Two-cluster
Programme
information
Detail + Content + Target audience
Three-cluster
Detail + Content + Methodology
Four-cluster
Detail + Content + Target audience
+ Course highlight

24. Summary Cluster analysis + SF Multimodal discourse analysis
Additional parameters to be considered in cluster analysis framework
A 6 – step procedure in cluster analysis to analyse large corpus of multimodal texts
A system network for the generic structure of business advertising texts.

25. References
Baldry, A. and P. Thibault Multimodal Transcription and
Text Analysis. London/Connecticut: Equinox Publishing Ltd.
Eggins, S An Introduction to Systemic Functional Linguistics.
(2nd Edition). London/New York: Continuum.
Halliday, M.A.K. and R. Hasan Language, Context and Text :
Aspects of Language in a Social-Semiotic Perspective. Geelong,
Australia: Deakin University Press.
Kress, G. and T. van Leeuwen Reading Images : The Grammar of
Visual Design. London/New York: Routledge.
Van Leeuwen, T Introducing Social Semiotics. London/New York: Routledge.

26. Faculty of Languages and Linguistics
THANK YOU
Fauziah Taib
Faculty of Languages and Linguistics
University Malaya
50603 Kuala Lumpur Malaysia.