A cogntive analysis of gesture and gaze direction in multimodal narratives of Brazilian Portuguese
DOI:
https://doi.org/10.54221/rdtdppglinuesb.2021.v9i1.216Keywords:
Cognitive Linguistics. Gestures. Gaze Direction. Multimodal Narratives.Abstract
In this master’s thesis, I intend to analyse the performance of gesture and gaze direction in multimodal narratives of Brazilian Portuguese. I assume the hypothesis that two multimodal articulators, especifically gesture and gaze direction, are able to be located independently in different Mental Spaces. To do so, I selected a personal narrative told by the Brazilian actress Marisa Orth, in the weekly TV show “Que História é Essa, Porchat?”, from the TV channel “GNT”. Methodologically, I divided the narrative into 11 occurrences 3 narrative excerpts: 3 occurrences present in the “Exhibition” excerpt, the presentation of the initial facts; 5 occurrences of the “Climax” excerpt, the conflict itself, the culminating moment of the story and 3 occurrences of the “Outcome” excerpt, the conclusion of the conflict. First, I analysed the linguistic choices of each occurrence, and then, we initiated the multimodal analysis of the occurrences, with support in the identification and description of the gestural strokes and, later, we identified the form of the gesture, taking into account the parameters of the Linguistic Annotation System for Gestures (LASG). With regard to the marking of Mental Spaces by gestures, I used the parameters that relate the gestures to the segment of the narrative in which a given interaction is allocated, whether it is part of an interaction in the Base Space (Ground) or part of an interaction in the narrative context (Story Space). Regarding the marking of Mental Spaces by gaze direction, I used the categorisations that verify whether the narrator gaze represents a character gaze (in the Story Space) or the narrator own gaze in the interaction (in the Ground). Only in the cases where the gestures marked the Story, I analysed, right after, if they are configured as gestures from the observer viewpoint (O-VPT) or as gestures from character viewpoint (C-VPT). Still in these cases, I analysed the gestures regarding their dominant function within the narrative: representation, expression or appeal. Likewise, only in cases where the direction of the gaze marked the Story Space, I also analysed its function within the narrative: i) character enaction and ii) narratorial gaze as itself. The results have shown that the predominant marking of Mental Spaces by gestures was the Space Story and that the predominant marking of Mental Spaces by the direction of the eye was the marking of the Ground.
Downloads
References
BAVELAS, J; COATES, L; JOHNSON, T. Listener responses as a collaborative process: The role of gaze. Journal of Communication, 2002.
BRESSEM, J. A linguistic perspective on the notation of form features in gestures. In: MÜLLER, C.; CIENKI, A.; FRICKE, E.; LADEWIG, S.; MCNEILL, D.; TESSENDORF, S. (Eds.), Body - Language – Communication.v. 1. Berlin/Amsterdam/New York:De Gruyter Mouton, 2013, p. 1079-1098.
BRESSEM, J.; Ladewig, S. H.; Müller, C. Ways of expressing action in multimodal narrations – The semantic complexity of character viewpoint depictions. In A. Hübl and M. Steinbach, Linguistic Foundations of Narration in Spoken and Sign Languages (p. 223-249). Amsterdam/Philadelphia: John Benjamins Publishing Company, 2018.
BRESSEM, J.; Müller, C.; Ladewig, S. H.. Linguistic Annotation System for Gestures. In: MÜLLER, C.; CIENKI, A.; FRICKE, E.; LADEWIG, S.; MCNEILL, D.; TESSENDORF, S. (Eds.), Body - Language – Communication.v. 1. Berlin/Amsterdam/New York:De Gruyter Mouton, 2013, p. 1079-1098.
BRYANT, D. J.; Tversky, B. Mental representations of perspective and spatial relations from diagrams and models. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25 (1), 137–156, 1999.
BÜHLER, K. Sprachtheorie: Die Darstellungsfunktion der Sprache. Stuttgart: Fischer. First published, 1982.
CASSELL, J.; McNeill, D. Gesture and Ground. Proceedings of the Sixteenth Annual Meeting of the Berkeley Linguistics Society, pp. 57-68, 1990.
CIENKI, A. Why study metaphor and gesture? In: CIENKI, Alan; MÜLLER, Cornelia (eds.), Metaphor and Gesture. Amsterdam/Philadelphia: John Benjamins. p. 5 – 25, 2008..
CIENKI, Alan. Cognitive Linguistics, gesture studies, and multimodal communication. Cognitive Linguistics, p. 603-618, 2016.
CIENKI, A. Analysing metaphor in gesture: A set of metaphor identification guidelines for gesture (MIG-G). In: E. Semino, & Z. Demjén (Eds.), The Routledge handbook of metaphor and language (pp. 131-147). London: Routledge, pp. 131-147, 2017.
CIENKI, A.; Mittelberg, I. Creativity in the forms and functions of spontaneous gesture with speech’, in T. Veale, K. Feyaerts and C. Forceville (eds) The Agile Mind: a multi-disciplinary study of a multi-faceted phenomenon. Berlin: De Gruyter Mouton. pp.231-252, 2013.
DANCYGIER, B. The Language of Stories: A cognitive Approach. Cambridge University Press, 2011.
DANCYGIER, B; Sweetser, E. Viewpoint in Language: A Multimodal Perspective. Cambridge: Cambridge University Press, 2012.
DUDIS, Paul. Body partitioning and real-space blends. Cognitive Linguistics 15(2), 223, 2004.
EMMOREY, K. Tversky, B; Taylor, H. A. Using space to describe space: perspective in speech, sign, and gesture. Spatial Cognition and Computation, 26, 157–180, 2000.
FAUCONNIER, G. Mental Spaces. Cambridge: Cambridge University Press, 2003 [1985].
FERRARI, L. Introdução à Linguística Cognitiva. São Paulo: Contexto, 2011.
FRICKE, E. Geste und Raum – Probleme der Analyse und Notation. Talk held at the lecture series “Analyse und Notation von Korperbewegungen“. Technical University Berlin, 2005.
GANCHO, C. V. Como Analisar Narrativas. São Paulo: Ática, 2006.
GIBSON J.; PICK, A. Perception of another person’s looking behavior. Am. J. Psychol, 1963.
GOODWIN, C. Conversational organization interaction between speakers and hearers. New York, NY: Academic Press, 1981.
GOFFMAN, E.. The neglected situation. American Anthropologist, 66(6, Pt. 2), 133–136, 1963.
HOUGAARD, A.; OAKLEY, T. Mental spaces in discourse and interaction: an introduction. In.: T. Oakley & A. Hougaard (Eds.), Pragmatics & beyond new series: Vol. 170. Mental spaces in discourse and interaction (p. 27–50). John Benjamins Publishing Company, 2008.
KENDON, A. Gesticulation and speech: two aspects of the process of utterance, in M. R. Key (ed.). The Relation between Verbal and Nonverbal Communication. The Hague: Mouton. pp.207-227, 1980d.
KENDON, A. Language and gesture: Unity or duality. In: D. McNeill (ed.), Language and Gesture, Cambridge: Cambridge University Press, 47-63, 1986.
KENDON, A. How gestures can become like words. In.: F. Poyatos (Ed.), Cross-cultural perspectives in nonverbal communication, 131-141. Toronto: Hogrefe, 1988a.
KENDON, A. Conducting Interaction: Patterns of Behavior in Focused Encounters. Cambridge: Cambridge University Press, 1990.
KENDON, A. Gesture: visible action as utterance. Cambridge: Cambridge University Press, 2004.
KIDWELL, M. The Role of Gaze in Conversational Interaction. In Body – Language – Communication: An International Handbook on Multimodality in Human Interaction. Mouton de Gruyter, 2014.
KITA, S.; Özyürek, A. What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory & Language, 48 (1), 16–32, 2003.
LABOV, W.; Waletzky, J. Narrative analysis. In J. Helm (ed.), Essays on the Verbal and Visual Arts. Seattle: U. of Washington Press. Pp. 12-4, 1967.
LANGACKER, R. W. Foundations of Cognitive Grammar. 2 vols. Stanford, 1960a.
LANGACKER, R. W. Subjectification. Cognitive Linguistics 1, 5–38, 1990a.
MCNEILL, D. The conceptual basis of Language. Hillsdale, NJ: Erlbaum, 1979.
MCNEILL, D. Hand and mind: What gestures reveal about thought. University of Chicago Press, 1995 [1992].
MCNEILL, D. Gesture and Thought. Chicago: University of Chicago Press, 2005.
MCNEILL, D.; Duncan, S.; Cole, J., Gallagher, S.; Bertenthal, B. Growth points
from the very beginning. Interaction Studies, 9(1), 117-132, 2008
MCNEILL, D. So you think that gestures are non-verbal? In: Psychological Review 92(3):
pp. 350-371, 1985.
MCNEILL, D. Psycholinguistics: a new approach. New York: Harper & Row, 1987.
MÜLLER, C. Forms and uses of the Palm Up Open Hand: a case of a gesture family?’, in C. Müller and R. Posner (eds) The Semantics and Pragmatics of Everyday Gestures: the Berlin conference. Berlin: Weidler Buchverlag. pp.233-256, 2005.
MÜLLER, C. Gestures as a medium of expression: The linguistic potential of gestures. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body – Language – Communication. An International Handbook on Multimodality in Human Interaction, Volume 1 (pp. 202-217). Berlin: De Gruyter Mouton, 2013.
MÜLLER, C., Ladewig, S., H., & Bressem, J. Gestures and speech from a linguistic perspective: A new field and its history. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & J. Bressem (Eds.), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction (pp. 55-81). Berlin: De Gruyter Mouton, 2013.
MÜLLER, C. Gesture and sign: cataclysmic break or dynamic relations? Frontiers in Psychology. 2018;9:1651, 2018.
PARRILL, F. Dual viewpoint gestures. Gesture 9(3). 271–289, 2009.
PARRILL, F. Interactions between discourse status and viewpoint in co-speech gesture. In Barbara Dancygier & Eve Sweetser (eds.), Viewpoint in language: A multimodal Perspective, 2012.
ROSSANO, M. J. The essential role of ritual in the transmission and reinforcement of social norms. Psychological Bulletin, 138(3), 529–549, 2012.
SCHRÖDER, Ulrike. et al. Um sistema para transcrever a fala-em-interação: GAT 2. Veredas Atemática. Vol. 20 n. 2, 2016
SIDNELL, J. Coordinating gesture, gaze and talk in reenactments. Research on Language and Social Interaction 39(4), 2006.
SLOETDJES, H.; WITTENBURGH, P. ELAN. ELAN (Version 5.9) [Computer software]. (2020). Nijmegen: Max Planck Institute for Psycholinguistics, The Language Archive. Retrieved from https://archive.mpi.nl/tla/elan", 2020.
STEC, K; SWEETSER, E. Managing multiple viewpoints: Coordinating embedded perspective in multimodal narrative. 12th International Cognitive Linguistics Conference, in Edmonton, Alberta, 2013.
SWEETSER, E.; STEC, K. Maintaining multiple viewpoints with gaze. In B. Dancygier, W. Lu and A. Verhagen (eds.), Viewpoint and the Fabric of Meaning. Mouton de Gruyter, 2016.
SWEETSER, E. Introduction: Viewpoint and perspective in language and gesture, from the Ground down. In Barbara Dancygier & Eve Sweetser (eds.), Viewpoint in language: A multimodal perspective, 1–2. Cambridge: Cambridge University Press. University Press, vol. I, 2012.
TOMASELLO, M. Why don’t apes point? In N. J. Enfield & S. C. Levinson (Eds.), Roots of human sociality: Culture, cognition, and interaction (pp. 506 – 524). Oxford, UK: Berg, 2007.
VERHAGEN, A. Constructions of intersubjectivity: Discourse, syntax and cognition. Oxford: Oxford University Press, 2005.
