The relationship between synchrony of gestural and intonational patterns for dubbing intelligibility

Abstract

The growth of the internet and digital media in recent times has revolutionized how we consume content, with a notable emphasis on the use of streaming services, with Netflix taking center stage. These platforms have brought a vast amount of foreign content; however, many people face linguistic barriers to appreciate it. Not everyone is fluent in a foreign language, making it crucial to consider the diversity of linguistic and cultural challenges in the production and distribution of content on streaming platforms to reach a broader audience. However, despite the intensified efforts of production companies, there are still constant complaints about the quality of dubbing reaching the end viewers. These complaints point out flaws and issues, even leading to the abandonment of projects halfway through. To address these issues, the research analyzed dubbed and subtitled content, aiming to determine whether problems and flaws during dubbing can significantly impact the general audience. Additionally, the research provided a brief overview of translation in Brazil, along with audiovisual translation techniques, referencing prosodic elements. The central question of the research sought to understand how synchronization occurs between the actor's gestures on screen and the dubber's speech in the studio, evaluating to what extent the presence or absence of this synchrony affects the intelligibility (the ability to understand and comprehend a message clearly) of the consumer. Our hypothesis was that a lack of synchrony between visual input (gesture) and auditory input (sound) would compromise comprehension. The research objectives included describing the main flaws in dubbing, as identified by the audience, and analyzing the differences in melodic patterns in both languages during the dubbing process. For this purpose, we analyzed five selected films from the genres of comedy and romance, from which one scene was taken from each for the analysis of gestural and melodic patterns. The analysis involved annotating prosodic elements perceived in each scene, an acoustic analysis of the fundamental frequency in the source and target languages, the classification of gestures made by the actors, and the administration of a comprehensive questionnaire about Brazilian dubbing to 52 judges from our target group. Our results allowed us to understand the main issues highlighted by the public regarding Brazilian dubbing and how changes and/or flaws during the dubbing process affect the viewer's experience in the final product.