Новости

Multimodal     transcription     as     a     tool of understanding visual metaphors

Natalia Kryukova1*, Elena Aleksandrova2, and Elena Isakova1

1 Tver State University, Zhelyabova str., 33, Tver, 170001, Russia

2 Murmansk Arctic State University, Yegorova street, 15, Murmansk, 183038, Russia

Abstract. The articles presents an ongoing study of visual metaphors in terms of cognitive approach. Visual metaphor is viewed by the authors as a cognitive tool that structures the perception of the world. The perception of visual metaphors is analysed by means of multimodal transcription that allows to decipher the semiotic codes that produce the meaning. Multimodal transcript is made applying the system of 14 semiotic codes represented by verbal and non-verbal elements of the visual metaphor. The metaphors are divided into motionless and motion ones depending on the type of channel that is engaged for perceiving and processing the information. Depending on the type of the visual metaphor different semiotic codes are transcribed in the structure of the metaphor.

1   Introduction

Sight has always been one of the essential senses of the human being. It is considered that about 80 % of the information people get and process is obtained by means of sight. This can be proved by the fact that nowadays most of the information the modern society perceives is provided via visual aids: pictures, memes, gif-images, short videos, motivational videos in linear presentationformat, vlogs etc. Hence, it is natural that in order to enhance the processes of perception and cognition visual metaphor is used both for educational purposes and for those of entertaining, advertising, informing and (in some cases even) manipulating people.

Visual metaphor has been in the focus of research in recent years due to the so called ‘visual turn’ in terms of which the metaphor in general is viewed not as a rhetorical device but as a fundamental tool that structures how people think about abstract ideas.

Charles Forceville divides visual metaphors into monomodal and multimodal ones. The modality in Forceville’s interpretation is defined by the source of information rather than by the mode of perception (audial, visual, etc). Thus, monomodal metaphors are represented in the same modality whereas in multimodal metaphors the source and the target are represented in different modalities [1, 2].

The above mentioned modalities, according to Forceville, are represented by eight sources of information:

• oral communication;

• Corresponding author: Адрес электронной почты защищен от спам-ботов. Для просмотра адреса в браузере должен быть включен Javascript.

© The Authors, published by EDP Sciences. This is an open access article distributed under the terms of the Creative

Commons Attribution License 4.0 (http://creativecommons.org/licenses/by/4.0/).

• written communication;
• images;
• sounds;
• non-verbal sounds;
• smell;
• taste;
• and touch [2].

Monomodal metaphors are divided by Forceville into four types:

• hybrid pictorial metaphor where one object is made up of two parts belonging to different spheres, and the meaning is perceived by interpreting one part in terms of the other;
• contextual pictorial metaphor in which the target domain is not represented in the visual image but is interpreted with the reference to the visual context;
• pictorial similes, images combining the target and source domains presented close to each other or interlaced;
• integrated metaphor where the target domain is represented in a posture or form similar to the source domain [1].

The variety of multimodal visual metaphors is difficult to classify as they can presented different mixes of modalities [3-8].

Speaking about audiovisual communication, Yves Gambier [9] sums up 14 semiotic codes that produce the meaning of a multimodal discourse element. The same classification of semiotic codes can be applied to visual metaphors in all their variety. These codes are active to different degrees in different types of metaphors (monomodal or multimodal ones). They include verbal and non-verbal elements and can be described as follows:

Table 1. The semiotic codes in the production of meaning

As is seen from the table, eight out of fourteen semiotic codes refer to visual ones, that is the information is perceived via the visual channel. These codes apply to all the graphic elements of the visual metaphor, colours, perspective, facial features and postures, the use of space and distance, visual environment signs, the font and style of the inscriptions, etc. The relationship between these codes in all their complexity, as defined by Yves Gambier [9], can be summarized as follows:

• redundancy (where one element of the visual metaphor repeats or emphasizes another one);
• complementarity;
• autonomy (in cases when one object is zoomed);

• contradiction (one element of the metaphor is opposed to another one or to the whole meaning of the metaphor);
• distance;
• criticism;
• help (one element of the visual metaphor aid in understanding of why the meaning is conveyed in a given way).

Different methods have been applied to studying visual metaphor in different fields of its application varying from market studies to sociology and psychology [10-12]. One of the common academic practices for research is transcription – turning a verbal text into writing for further analysis. It is widely used in a range of disciplines, such as discourse analysis, sociolinguistics, pragmatics, linguistic anthropology, etc. In recent years the traditional transcription has undergone certain changes. The emergence of multimodal transcripts was prompted by increasing availability of digital video technology bringing about the necessity to construct multimodal transcripts [13]. The use of multimodal transcription in the research of different types of visual metaphors can allow for the different modalities or semiotic codes to be dissected and analysed separately as parts of meaning production.

2   Materials and methods

The study of visual metaphors was carried out with the application of multimodal transcription – a new academic method applicable to elements referring to several modalities, or several semiotic codes.

The method of multimodal transcription was applied separately to monomodal and multimodal visual metaphors with reference to the semiotic codes as described by Yves Gambier.

The material for the study was chosen using the basic conceptual metaphor TIME IS MONEY represented in different sources as monomodal and multimodal visual metaphors.

3   Results

3.1  Hybrid pictorial metaphor

The table below transcribes the verbal and non-verbal elements of the visual metaphor representing the basic conceptual metaphor TIME IS MONEY.

Table 2. The verbal and non-verbal elements of the visual metaphor representing the basic conceptual metaphor TIME IS MONEY

The combination of verbal and non-verbal elements is enhanced by the use of the green colour that is typically associated with the colour of money by means of reference to dollars as one of the most frequently used currencies. Another element enhancing the idea of money is the symbol of the American dollar substituting the letter ‘i’ in the word ‘time’. The dollar sign is specifically accentuated by the use of the green colour that stands out on the background of the rest of the letters in the word. The contrast of the black and green colours draws the recipient’s attention to the idea of money. The verbal element that is in some studies considered a tool of identifying the visual metaphor [14] presents the basic conceptual metaphor itself.

3.2  Pictorial simile

The given table presents a multimodal transcript of a monomodal visual metaphor – a pictorial simile representing the basic conceptual metaphor TIME IS MONEY.

Table 3. A multimodal transcript of a monomodal visual metaphor

The metaphor is visualized by means of such non-verbal elements as a red alarm-clock and a golden dollar symbol on a teeterboard. An unstable cylinder serves as a foundation of the teeterboard, yet, the board is absolutely stabilized by the alarm-clock and the dollar symbol which emphasizes the idea of the equation TIME IS MONEY.

Table 4. Pictorial simile (verbal elements interlacing with the image)

In this visual metaphor the verbal elements are interlaced with the image: the inscription ‘time is money’ is position on the face of the alarm-clock that is ringing. Black lines on both sides of the alarm-clock serve as a substitution of the alarm going off and emphasizing the idea of time running quickly.

4   Discussion

As is seen from the three examples of multimodal transcription, the iconographic code used in different types of visual metaphors is visualized by means of an image of the clock, with the image of the alarm-clock in two out of three cases. Visual metaphors may or may not include verbal elements of the graphic code, but this fact does not prevent interpretation of the meaning due to the abundance of such non-verbal elements as colour, position, background, etc.

Motionless visual metaphors, as described above, are interpreted with reference to the following semiotic codes:

graphic (numbers, verbal inscriptions);

iconographic (images inducing cognitive processes); photographic (colours);

scenographic (organizing the elements of the metaphor in the picture); proxemic (positioning and distancing the elements of the metaphor).

Motion visual metaphors (represented by gif-images, boomerang pictures, videos, etc) will also apply such semiotic codes as film and kinesic ones. If the motion visual metaphor is followed by an audio channel, all possible audio semiotic codes may be applicable to both structuring the metaphor and interpreting it.

Multimodal transcription can be used as an independent tool for analyzing visual metaphors, but can also be used in combination with advance methods applied in electrophysiological studies that can help uncover the principles of brain electrical activity while processing metaphoric images [15].

The reported study was funded by RFBR and FWF according to the research project № 25-512- 14002

References

1. Forceville, Styles of Communication, 9, 26-41 (2017)
2. S. Shilyaev, E. A. Shlotgauer, Vestnik Tomskogo gosudarstvennogo universiteta. Filologiya – Tomsk State University Journal of Philology, 67, 154–173 (2020) DOI: 10.17223/19986645/67/8
3. A. Maalej, Language Sciences, 47, Part A, 1-17 (2015) https://doi.org/10.1016/j.langsci.2014.07.005.
4. Sobrino, Multimodal Metaphor and Metonymy in Advertising (Amsterdam, 2017)
5. Silaški, T. Đurović, Discourse, Context & Media, 31, 100318 (2018) https://doi.org/10.1016/j.dcm.2019.100318.
6. Piata, Journal of Pragmatics, 106, 39-56 (2016) https://doi.org/10.1016/j.pragma.2016.10.003.
7. J. Ortiz, Journal of Pragmatics, 43 (6), 1568-1580 (2011) https://doi.org/10.1016/j.pragma.2010.12.003
8. Domınguez, Metaphor and Symbol 30(3), 240-255 (2015)
9. Gambier, The position of audiovisual translation studies, The Routledge handbook of translation studies, 45–59 (2013)
10. Josephson, J. Kelly, K. Smith, (ed.) Handbook of Visual Communication: Theory, Methods, and Media (2020)
11. Refaie, Analysing metaphors in multimodal texts. In: Semino, E. & Demjén, Z. (eds) The Routledge Handbook of Metaphor and Language, 148–160 (2017)
12. Jewitt, T. van Leeuwen, Handbook of Visual Analysis (London: SAGE, 2013)
13. Bezemer, D. Mavers, Multimodal transcription as academic practice: a social semiotic perspective, Intern. J. of Social Research Methodology, 14:3, 191-206 (2011)
14. Negro Alousque, Procedia - Social and Behavioral Sciences, 212, 309-315 (2015) https://doi.org/10.1016/j.sbspro.2015.11.379
15. J. Ortiz, M.D. Grima Murcia, E. Fernandez, Brain and Cognition, 113, 117-124 (2017) https://doi.org/10.1016/j.bandc.2017.01.005