Session 3F

Over the last decades, foreign-language (L2) subtitles have increasingly been recognised as a fruitful means to improve FLL (for a recent review, see Montero Perez, 2022). However, little is known about how learners engage with such written script as it unfolds on screen, and how they integrate information from multiple sources (e.g. sound, moving images, subtitles) in order to achieve comprehension of such multimodal content.
Reading text in dynamic viewing situations differs from reading static text in several ways, which is likely to have specific implications in the context of watching subtitled videos for both incidental and deliberate L2 learning – practices that are becoming very popular in the age of VOD (video-on-demand) and streaming services. Therefore, understanding how learners process L2 subtitles is paramount to a fuller understanding of the learning mechanisms that might ensue from that processing. 
Subtitle reading is bound to be affected by both moving images and sound, which so far has made it impossible to examine the distinct contribution of L2 subtitles to overall comprehension of video content, as virtually no studies have tried to isolate the effect of these separate components. To complicate things further, with the aforementioned increase of VOD services (e.g., Netflix, Amazon Prime), we are witnessing an increase of subtitle speeds on the market (Szarkowska and Gerber-Moron, 2018), which is also bound to affect the viewing experience in ways that are still for the most part unknown. 
A notable exception is an eye-tracking study by Liao et al. (2020), where the authors assessed the effects of manipulating video (presence/absence of the moving images) and subtitle speed (12, 20, 28 characters per second) on both comprehension and subtitle processing in English native speakers watching English video. Our study replicates their experiment in the context of EFL (English as a foreign language) learning. Our main goal is to disentangle the impact of video presence and increasing subtitle speed on the processing of English L2 subtitles by EFL learners and on their video comprehension. To this end, 42 Polish native speakers watched six videos with English L2 subtitles whilst their gaze was monitored by an EyeLink 1000 Plus eye tracker. Both on-line (eye movements) and off-line (comprehension questions) measures were analysed using (generalised) linear mixed-effect models. To assess processing, several eye movements (e.g. fixation count, saccade length, skipping rates) were computed both globally (i.e., on whole subtitles) and locally (on individual subtitle words). In this talk, we will review relevant literature, describe in detail the methodology adopted in the study and present preliminary results of the data analysis. 


Liao et al. (2020).Using Eye Movements to Study the Reading of Subtitles in Video. Scientific Studies of Reading, pp. 1–19.
Montero-Perez, M. (2022). Second or foreign language learning through watching
audio-visual input and the role of on-screen text. Language Teaching, pp. 1–30.
Szarkowska A. & Gerber-Moron, O. (2018). Viewers can keep up with fast subtitles: Evidence from eye movements. PLoS ONE 13(6), pp. 1–30.

TV series (with or without on-screen text) are robust tools for language learning because they expose the learner to a large amount of rich, authentic input. The presence of images provides an additional semantic support that aids information processing (e.g., Rodgers, 2018a), while the addition of on-screen text may facilitate comprehension for those learners whose L2 language skills are not high enough (Webb & Rodgers, 2009a). In fact, Studies on L2 learning from audio-visual materials have consistently shown a significant advantage of captioned over non-captioned conditions for content comprehension (e.g. Chung, 1999; Guillory, 1998; Huang & Eskey, 1999; Li, 2014; Montero-Pérez, Peters & Desmet, 2014), but the differences in comprehension accuracy between the two conditions varies widely from study to study (Pujadas, 2019), and some studies also report overall non-significant differences (e.g. Montero-Perez, Peters, Clarebout & Desmet, 2014; Rodgers; 2013; Rodgers & Webb, 2017). 
While most research in this area has been conducted with intermediate L2 learners, it is possible, however, that the advantage of captions over non-captions conditions may fade when participants are more advanced (and less dependent on the on-screen text). The question is, then: are they necessary for more advanced learners, who can already process input automatically (Vandergrift, 2007)? The addition of text alters the way viewers process information, as attention is divided among multiple channels. While several studies have found that the automatic reading of the text does not prevent viewers from processing the audio (Bird & Williams, 2002; Garza, 1991), nor the image (Bisson, Van Heuven, Conklin & Tunney, 2014), others suggest that lack of attention to audio could hinder the development of listening skills (Borras & Lafayette, 1994), and that captions might actually be distracting for more advanced students (Baristow & Lavaur, 2011). 
The aim of the present study is to explore whether there is a proficiency threshold for satisfactory viewing comprehension without captions. It also seeks to investigate whether a measure of general proficiency or vocabulary size would be more adequate to set said threshold, if the lexical profile of the episodes viewed affect comprehension rates. Participants were 270 university students, who took part in a 3-month intervention in which they watched 9 episodes of a TV series. They were distributed in 8 classes, of which 4 watched the episodes with captions and 4 without. Comprehension was assessed after each episode through multiple-choice and true-false items, which included a combination of textually explicit and inferential items. Preliminary results showed that, overall, captions groups outperformed the non-captions groups, but further analysis will show to what extent learner-related variables (i.e. proficiency and vocabulary size) and video-related variables (i.e. lexical coverage) affect comprehension, and determine the effect of these variables on caption and non-caption viewing.

Primary school learners' still developing L1 reading skills and their lack of a solid L2 English Linguistic Infrastructure (Birch & Fulop, 2021) result in a struggle to process print with ease. Therefore, an increasing number of investigations has explored the effects of multimodal input to support L2-reading instruction. Overall, the evidence has shown that the use of aural and written input simultaneously (with and without imagery) may facilitate the reading process and increase motivation. However, the outcomes of studies on reading-while-listening cannot not be entirely extrapolated to the effects of reading audiovisual input, where captions stay on screen for a limited amount of time. Likewise, even when the evidence on the use of captions with young learners has turned out to be beneficial as regards L1-reading skills development (Linebarger, Taylor & Greenwood, 2010), the outcomes may well change with participants from input-limited EFL contexts. Hence, this investigation intended to explore the development of L2 reading efficacy (Llanes, 2018) through captioned-video viewing (11 episodes) in a group of 92 L1-Spanish primary school learners of English in years 4 and 5 (9-11 years old). In addition, it assessed the influence of cognitive factors (complex working memory, phonological short-term memory and visual processing speed), as well as L1 and L2-related factors (e.g. L2 vocabulary knowledge, listening skills, L1 reading efficacy) on the outcomes. The results of a series of GLMMs and GLMs showed learners' significant improvement as regards English reading efficacy, supporting the use of captioned videos to strengthen L2-reading instruction. Another important finding was that learners' complex WM capacity did not predict learners' performance as a result of the treatment. It can thus be suggested that during the intervention, WM capacity was not overloaded with the simultaneous processing of audio, print and imagery to extract meaning. Instead, the results indicated that learners' outcomes relied on visual processing speed and their ability to decode and hold key information in the phonological loop (PSTM) to foster comprehension. However, the effects of PSTM lost statistical significance when the model included language-related factors. L2 vocabulary knowledge and listening skills were found to be key performance predictors. Finally, the significant main effect for Spanish reading efficacy may provide further support for the hypothesis that learners assimilate and accommodate their L1 reading strategies and mechanisms to deal with L2 print. 




Alderson, J.C., Huhta, A., Nieminen, L. (2016). Characteristics of weak and strong readers in a foreign language. The Modern Language Journal, 100(4), 1-28.

Birch, B., & Fulop, S. (2021). English L2 reading: Getting to the bottom. Routledge. 

Linebarger, D., Taylor Piotrowski, J., & Greenwood, C. (2010). On-screen print: the role of captions as a supplemental literacy tool. Journal of Research in Reading, 33(2), 148-167. 

Llanes, A. (2018). Reading in English as a foreign language: Examining differences in reading speed, comprehension, efficacy and L1 cross-linguistic influence across grades. Investigaciones Sobre Lectura, 9, 1–49.