Designing Multimodal Discourses

1
Designing Multimodal Discourses
Bai Hongai
College of Foreign Languages, YBU
2
monomodality Language as the central and
only full means for representation and
communication
3
means of expression (material and other
technologies)
multimodality
When humans converse with one another, we
utilize a wide array of media to interact,
including spoken language, gestures, and
drawings. We exploit multiple human sensory
systems or modes of communication, including
vision, audition, and taction.
the manner of expression
4
Some media and modes of communication are more
efficient or effective than others for certain
tasks, users, or contexts e.g. the use of speech
--- control devices in hand and
eyes- busy contexts,
the use of maps ---
convey terrain and cartographic
information
5
Communication is defined as a process in which a
semiotic product or event is both articulated or
produced and interpreted or used, and a semiotic
product or event may involve multiple modes and
media.
The design process of this multimodality semiotic
product or event, is to be explored in the
present paper, i.e. the generation of multimodal
discourses
6
I. What is multimodal discourse
Multimodal discourse is defined as the use of
several semiotic modes in the design of a
semiotic product or event, together with the
particular way in which these modes are combined
they may for instance reinforce each other
(say the same thing in different ways), fulfil
complementary roles, or be hierarchically
ordered, as in action films, where action is
dominant, with music adding a touch of emotive
colour and sync sound a touch of realistic
presence (Kress G. Van Leeuwen, T, 1996,
2001 Batesman et al., 2002)
7
This suggests that we need to expand our
conception of mode to embrace multimodal
discourse analysis (hereafter MDA).
MDA draws upon recent work in the field of social
semiotics in an attempt to theorise how
discourses are generated in a variety of modes
through a variety of media, and to investigate
empirically the design decisions employed and how
these have been distributed across the
presentation modes available.
8
II. How to design a multimodal discourse
STEP 1 The Process of Designing a multimodal
discourse

• Content planning determining what information
  should be presented
• Technique selection selecting appropriate media
  or modalities which will be used for realization,
  and coordinating the selections for specific
  information content
• Presentation design determining how media or
  modalities will be used to communicate their
  selected information content
• Coordination composing, organizing, resolving
  conflicts and maintaining presentation
  consistency.

9
Figure 1 Feedback and Multiple Knowledge Sources
involved in the Processes
Presentation Design
Content Planning
Technique Selection
Coordination
Data Task
Context Discourse Characteristics
Characteristics State
10
STEP 2 Determine what and how media or modes to
be used
Myers (1999 91)
Different media convey different information
Different media go with different practices
Spoken and written words can limit, link and
extend images
Words (including typography) define voice and
audience
11
STEP 3 Establishing the right modality
Naturalistic modality, Abstract modality and
Sensory modality
Naturalistic modality produced with sound,
images, audio-visual texts or the combination of
all these.
Abstract modality with regard to images
Sensory modality based on the sensory pleasure
or displeasure represented by an audio-visual
text
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
larger-than-life or more-than-real images or
sounds
16
III. Advertisements typical
multimodal discourses
17
Medium with sound only--- Radio
MAN I I I love you. WOMAN Awwww. MAN No, I
mean I I mean. I dont just WOMAN What
(GIGGLES), go ahead MAN I just dont think
Id be anything without you WOMAN Aw thats
so nice MAN I, uh, made these for you
(Baking pan and foil sounds.) WOMAN Aw you
didnt have to do this MAN Theyre just
brownies. WOMAN Ooh, and theyre still warm too.
18
MAN I was gonna get you something good. WOMAN
Aww, dont be silly, could I have one? MAN
(Laughs.) Well, yeah sure go ahead WOMAN
(Mouth full.) You know ? MAN What you dont
like brownies? WOMAN No no (Slight gag.)
miggkkk MAN Im sorry. I cant understand
you. WOMAN Neet dmiggkk. MAN You need
Mick? WOMAN Miccliggkk! MAN A guy named Mick! I
cant believe this WOMAN Pleeeease
miiiggggggllkk
19
MAN Yeah, yeah, okay you dont have to scream
his name in ecstasy. I get the
message. All right I got it, Im gone. WOMAN No,
umbhum, no, miiiiiiggglllkk! MAN Look, I gotta
go, Im sorry Im gonna need my CDs back and
WOMAN No, no, miiiggllk. MAN and Im sorry
for this whole thing WOMAN No, miiigggllkk
MAN No, its Scott remember ? WOMAN No,
miigggkkkffff ANNOUNCER True love means never
having to say, Got Milk?
20
Medium with image, sound, colour, words, etc.
--- internet banner ads, TV ads
21
(No Transcript)
22
(No Transcript)
23
V. Implications to Language teaching
There is a need to re-conceptualise the term
literacy as a multimodal process. As language
is beginning to be de-centred and image and other
modes are increasingly foregrounded, the
traditional conceptions of learning and
literacy as entirely language based are
inadequate.
24
Multimodal discourse with language de-centred
25
Literacy, as a complex process, is multimodally
mediated, in which the modes of speech and
writing are only ever one part of the multimodal
modes, just the same as image, colour, sound and
animated movement on screen.
26
We need to pay attention to the substance of
learning including new learning materials. In
recent years, educationalists have focused upon
learning environments as social and personal
settings and forms of interaction. It is time to
take note of the fact that learning is about
something and that something is communicated
through a range of media from books and
photocopies, through audiovisual media to
computer media and in the near future perhaps
also mobile media. Only with the advent of a
range of new media it is perhaps made clear to us
that learning and media were always intimately
increasingly intertwined.
27
Thank you for your attention!