Sendebar. Revista de Traducción e Interpretación https://revistaseug.ugr.es/index.php/sendebar Publisher: Universidad de Granada Original Articles · Artículos originales Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities in the ImAc Project Accesibilidad en vídeos de 360º: aspectos metodológicos y resultados principales de las actividades de evaluación del proyecto ImAc Anna Matamala orcid 0000-0002-1607-9011 Universitat Autònoma de Barcelona ABSTRACT This article presents a global overview of the results of the evaluation activities performed in the European project Immersive Accessibility (ImAc), which investigated how access services can be integrated into 360º videos. More specifically, the paper presents the methodological approach in ImAc testing, and reports on the results of the evaluation activities on the tools (a subtitling editor, an audio description editor, a sign language editor, and an accessibility content manager), on the media player, and on the access services (subtitling, audio description, audio subtitling, and sign language interpreting). Keywords: accessibility, virtual reality, 360º videos, subtitling, audio description, sign language interpreting RESUMEN El artículo presenta de modo global los resultados obtenidos en las actividades de evaluación del proyecto eu- ropeo Immersive Accessibility (ImAc). El objetivo de dicho proyecto era investigar cómo los servicios de acce- sibilidad pueden integrarse en vídeos de 360º. Más concretamente, el artículo describe la metodología seguida en el proyecto ImAc y da cuenta de los resultados obtenidos en la evaluación de las herramientas desarrolladas, que incluyen un editor de subtítulos, un editor de audiodescripciones, un editor de lengua de signos, un gestor de contenidos y un reproductor de vídeos de 360º. El artículo también presenta los resultados de la evaluación de distintas estrategias aplicadas a los servicios de accesibilidad siguientes: subtitulado, audiodescripción, au- diosubtitulado e interpretación en lengua de signos. Palabras clave: accesibilidad, realidad virtual, vídeos 360º, subtitulado, audiodescripción, interpretación en lengua de signos Article info Corresponding author: Conflict of interest: Anna Matamala None. Anna.Matamala@uab.cat Funding: Publication history: Este trabajo ha recibido financiación del programa Horizon 2020 de la Unión Europea (761974) y del programa SGR de la Received: 22.11.2020 Generalitat de Catalunya (2017SGR113). Reviewed: 17.04.2021 Accepted: 16.06.2021 How to cite: Matamala, A. (2021). Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities in the ImAc Project. Sendebar, 32, 65-89. creative-commons 4.0 BY-NC https://doi.org/10.30827/sendebar.v32.16881 ISSN-e 2340-2415 SENDEBAR (2021), 32, 65-89 65 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 1. Introduction 360º videos are an example of virtual reality (VR) content that allows audiences to immerse themselves in different stories and be exposed to new experiences. According to a report on the use of VR by public broadcasters in Europe (EBU, 2017), almost half of the respondents offer 360º video content. However, it remains to be seen how this content can be made ac- cessible for all. In other words, further research is needed on how access services offered in traditional 2D media, such as audio description (Maszerowska, Matamala and Orero, 2014), subtitling (Matamala and Orero, 2010), or sign language interpreting (McDonald, 2012), may be adapted to the new medium to suit end user needs. This was one of the research questions that the Immersive Accessibility (ImAc) project aimed to answer. ImAc was a 30-month H2020 European project led by i2Cat which started in 2017. The project was developed by an interdisciplinary team of engineers, audiovisual translation and accessibility experts, broadcasters, and end user representatives. Partners included universi- ties (Universitat Autònoma de Barcelona-UAB, University of Salford), broadcasters (CCMA, RBB) and broadcaster’s research centres (IRT), user associations (RNIB), and SMEs (An- glatècnic, Motion Spell). The project aimed to research how access services could be integrat- ed into 360º videos both from the perspective of professionals producing accessible content and from the perspective of end users consuming accessible content. To that end, a series of prototypes were developed and tested in order to come up with the most suitable solutions for 360º accessible video creation and consumption. The specific aim of this article is to present the rationale behind the evaluation plan and the methodological choices made in the ImAc project and summarise the main results. Although some of the results have been partially reported in other publications—and this is duly acknowledged where relevant—, the innova- tion in this article lies in the presentation of a global overview of how accessibility could be approached in 360º videos. Section 2 reports on the methodological approach in ImAc and summarises the evaluation plan. Section 3 reports on the evaluation activities with professional tools. Section 4 presents the evaluation activities on the media player interface which gives access to the accessible content. Section 5 discusses the evaluation activities on the different access services presenta- tion modes. The article ends with a conclusion and suggestions for future research. 2. ImAc’s Methodological Approach: Defining Users, Measures and Methods When designing ImAc’s methodology, the first step was to define what evaluation activities needed to be performed, with whom and how. However, this question could only be answered in the framework of the aims our evaluation wanted to achieve. 2.1. Aims of the evaluation activities The ImAc project aimed at developing tools to create access services and producing accessible 360º videos following a human-centric methodology, i.e. by involving the users throughout the design and evaluation process. Therefore, evaluation activities to inform the technological development needed to be planned. In terms of production, the aim was to identify the needs Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 66 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 of professionals when creating access services for 360º videos and assess different prototype tools developed taking into account their requirements. In terms of reception, the aim was to identify the needs of end users when consuming access services for 360º videos, to produce requirements, and to assess different solutions. Beyond the specific solutions tested in the different evaluation activities, there was an underlying broader aim: to define how access ser- vices could be integrated in the new medium of 360º videos in a moment in which these new formats are being developed (Catapult Company, 2018; EBU, 2017). In other words, the aim was not to research and integrate accessibility as an afterthought, but to find the means to pro- duce 360º videos which are “born accessible”, to use Orero’s (2020) terminology, and which comply with the EU current legal framework for accessibility. 2.2. Defining users and metrics Since we are concerned with the production and consumption of access services, two broad categories of users were identified at the beginning of the project: the so-called “professional users” and “home users”. Professional users, which is the term used in the project to refer to producers or creators of access services, would be involved in evaluation activities concerning the professional tools. Home users, which is how the end users of the access services were referred to in ImAc, would be involved in evaluation activities to assess the access services features. The challenge lay in the fact that accessible 360º video content is non-existent and, therefore, most home users are not familiar with this environment. This is why, in the first stages of the project, users with technological expertise were prioritised in a series of focus groups (Agulló and Matamala, 2019; Agulló, Matamala and Orero, 2018) that laid the founda- tions for the evaluation activities that are described in this article. Concerning the methodological approach, it was decided that a combination of standard metrics and open questions would allow us to triangulate results from the evaluation activities. The evaluation approach was based on three core concepts: usability, presence, and preferenc- es. When using a tool to professionally produce accessible content, or when using a tool to access content, usability is fundamental. We understand usability as the “extent to which a product can be used by specific users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (ISO 9241-11:1998, definition 3.1.). In order to measure the usability of the different ImAc tools, the System Usability Scale (SUS) was considered a fast and reliable metric. SUS includes 10 items, and it is considered easy to ad- minister and reliable with small sample sizes (Brookes, 2013). When developing solutions to integrate access services in virtual environments, it was considered critical to develop solutions that would not impact negatively on the user’s pres- ence. Presence, also referred to as immersion, can be understood as the sense of “being there” (Reeves, 1991), the “perceptual illusion of non-mediation” (Lombard and Ditton, 1997), the “psychological sense of immersion in any mediated environments” (Fryer and Freeman, 2012), or “the experience of being engaged by the representation of a virtual world” (Jacob- son, 2002). Presence is a multiconstruct concept with many dimensions that can be measured by various means. In our case, the Igroup Presence Questionnaire (IPQ) (Schubert, Friedmann and Renbrecht, 2001) was chosen for three main reasons: it combines previous questionnaires Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 67 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 and differentiates for the first time between spatial presence, involvement, and experienced realism; it has been validated in virtual environments, and its length (14 questions) is adequate for experimental purposes. In both cases, when developing solutions for creation and for reception, subjective feed- back and general preferences of participants were also considered to be fundamental. Hence, open fields and general preference questions were included in the different evaluation activi- ties. The results of the different evaluation activities allowed ImAc to identify the best approach to produce and consume access services in 360º. These solutions were then implemented in two open final pilots, led by German broadcaster RBB and Catalan broadcaster CCMA, in which the evaluation measures chosen were mostly related to quality of service and quality of experience metrics. These final pilots will not be reported in this article, as our interest lies in the evaluation process prior to them. 2.3. Evaluation protocol Evaluation activities in ImAc were carried out by different partners. To guarantee a unified approach, an evaluation protocol to be followed in any evaluation activity was developed, with the following steps. First, participants are welcomed, they are informed about the project and its ethical procedures, as approved by the UAB’s ethical committee. Ethical consent forms are signed. Secondly, participants are asked to fill in a short demographic questionnaire. This questionnaire, with slight variations for professional and end users, was consistent across all evaluation activities. Thirdly, participants are asked to perform certain tasks and reply to the selected questionnaires. Finally, participants are thanked and are offered the possibility to ob- tain more information about the project. 2.4. Evaluation plan ImAc evaluation activities can be categorised into three blocks: a) activities to evaluate the tools that professionals will use to create the services, b) activities to evaluate the media pla- yer, which home users will use to reproduce the content, and c) activities to evaluate different access services features. The selected access services were subtitling (ST)—including a hybrid service: easy-to-read subtitles—, audio description (AD), audio subtitling (AST), and sign language (SL). Table 1 provides an overview of all evaluation activities. Table 1. Evaluation activities in ImAc Professional tools Accessibility content manager (ACM) Subtitling editor Audio description editor Sign language editor Player Traditional menu in head-mounted display (HMD) versus traditional menu on a tablet Voice interaction menu versus enhanced accessibility menu Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 68 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Subtitles Guiding mechanisms Subtitle position Easy-to-read subtitles Audio description Static, Classic, Dynamic presentation modes Classic, Radio, Extended presentation modes Audio subtitling Classic versus Dynamic presentation modes AD with AST versus AST only presentation modes Sign Language Continuous display versus non-continuous display of signer video Speaker representation: textual versus emojis Sign language only versus subtitles and sign language 3. Evaluating professional tools ImAc developed four tools: a subtitling editor (Figure 1), an audio description editor (Figure 2), a sign language editor (Figure 3), and an accessibility content manager (Figure 4). An accessibility content manager is a type of software that can be used by a broadcaster to keep track of the creation of access services for certain content. For example, the content manager can be used to assign video content to professionals and keep track of what access services have already been produced for each video asset. Initial development for ImAc’s tools was informed by a series of focus groups (Agulló and Matamala, 2019; Fidyka and Matamala, 2018, 2021) which gathered initial user requirements. A first prototype was then developed and tested for the tools (except for the SL editor). This evaluation round allowed the consortium to collect more user requirements and suggestions and develop a second prototype, which was again tested (see picture on the right on figures 1, 2 and 4). Figure 1. Subtitling editor (first and second iteration) Figure 2. AD editor (first and second iteration) Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 69 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Figure 3. SL editor Figure 4. ACM tested (first and second iteration) 3.1. Methodology Evaluation activities on the tools focused on two aspects: usability, by means of SUS, and user preferences, through an open questionnaire which read: • What did you like most about the (tool)? • What did you like less about the (tool)? • What do you think could be improved, and how? • Did you miss any functionalities? If so, can you tell us which ones? • (for editors) Do you find the features of setting the angle easy to use? Explain why. • (for editors) Were the preview modes useful for you? Explain why. • (for editors) Do you think it will take longer to subtitle/audio describe/interpret videos in 360º? Why? • (for editors) Do you think 360º videos will impact your work as a subtitler? • (for ACM) Was it intuitive? If not, please write why. • Other comments (open field). The protocol was as follows: first, participants were informed about ImAc and the specific test, and informed consent forms were signed. Then, participants were asked to fill in an online demographic questionnaire and to perform a series of tasks included on an instructions sheet. Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 70 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 After finishing these tasks, they were asked to fill in the SUS questionnaire and a preference questionnaire, both online, and were thanked for their participation. 3.2. Participants The number of participants per tool and iteration was: • ACM: 7 participants (1st iteration), 27 participants (2nd iteration). • ST editor: 27 participants (1st), 16 participants (2nd). • AD editor: 24 participants (1st), 24 participants (2nd). • SL editor: 6 participants (one single iteration). The number of participants was 131 professionals, who provided objective and subjective data on the different ImAc tools. Sociodemographic information about these participants was obtained through a standardised questionnaire, including: age, sex, main language, current job, previous experience with 360º content and with access services, level of studies and spe- cific training, software use, technological use (devices used on a daily basis, virtual reality consumption), and attitudes towards virtual reality. A detailed description of this demographic information is beyond the scope of this article but can be found in Matamala (2020). It should be stressed that involvement of participants was more challenging for some tools, such as the SL editor, because current professional sign language interpreters do not generally use this type of software, hence the difference in numbers. 3.3. Results SUS results are presented in Table 2, for both the first and second iterations, where applicable. Table 2. SUS scores for ImAc tools Tool SUS (1st iteration) SUS (2nd iteration) ACM 54.6 72.31 ST 59.5 65 AD 55.9 60.52 SL 39.17 -- In all cases an improvement is found from the first to the second iteration, showing the impact of user feedback on tool development. However, results are not always above aver- age—68 or more is considered above average in SUS—and the values for the SL editor are low. Possible reasons that explain these results are the fact that many audiovisual translation professionals, especially audio describers and sign language interpreters, do not always use tools to produce their access services, contrary to content and accessibility managers using the ACM. For instance, many describers taking part in the test stated that they are normally in charge of writing the audio description using text processing software, but they are not actual- ly involved in the recording, which was one of the tasks performed in the test. This was even more striking in the SL editor test, where participants stated that they do not usually deal with these processes and acknowledged the challenge of familiarising themselves with a new tool. This highlights the need for further training in technological tools for future professionals in the field, together with an even more thorough analysis of the specific needs of professionals. Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 71 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Beyond quantitative results, what was more interesting for ImAc was to gather qualitative data from users on the specific tools, but also on how professionals envision the impact of 360º videos in their processes. The results presented next do not deal with specific functionalities (see Agulló, 2020, and Fidyka and Matamala, 2019, for details) but provide a broad overview of what professionals consider important in professional tools. The analysis shows that professionals are willing to work online, with cloud-based tools, with a clear user-friendly interface and visible icons. Visual design is critical but at the same time very much influenced by participants’ previous experience with other tools. Professionals want to be able to customise the user interface but also the different shortcuts to make their work more efficient, and they are interested in editors that integrate additional tools such as spellcheckers or quality assurance tools. They want the tool to provide automatic features (for instance, subtitle segmentation) but, at the same time, they want to be empowered to adapt these automatic tools to their needs and ignore them if necessary. Time-coding is fundamental, as well as the presence of a clear sound wave and multiple options to edit the services and to export the final result to different formats. Overall, ImAc prototypes were defined very often as “intuitive” and “easy to use”, but room for improvement was also identified and implement- ed as the project progressed. Some of the most praised functions were related to navigating in a 360º environment and setting the angle. In fact, setting the angle is a specific feature that needed to be developed for 360º videos. In 360º content, speakers may not be located within the audience’s field of view, but somewhere else in the sphere. This is why the position of the speaker needs to be indicated for each subtitle or sign language interpreted section. To do so, the professional needs to navigate in the video and select the angle where the speaker is positioned in order to anchor the access service to the speaker’s position. This metadata will then make it possible to implement different features, such as positioning the subtitle/sign language interpreter close to the speaker or activating a guiding mechanism such as an arrow or a radar that will show the viewer where to find the speaker. Similarly, in audio description settings the angle allows professionals to link the audio description to the place in the sphere where the action being de- scribed is taking place, generating metadata that will make it possible to produce audio coming from that specific region. Beyond the specific features of the editors, what is interesting from a global perspective are the attitudes of the professionals towards a new technological development such as 360º vid- eos. When asked if they thought that producing subtitles, audio descriptions, or sign language interpretations would take longer in 360º, the replies were: in the first iteration, 61% thought that subtitling would take longer due to the need to set the angle, 26% of the subtitlers were not sure, and 11% stated that it would not take longer with the right tools. In the second iteration, the percentages were 50% for those who considered it would take longer, 16% for those who thought it would not take longer and 34% for those who stated that it would take more time only at the beginning, when getting used to this new process. Concerning AD, 79% of the pro- fessionals in the first iteration and 70% of the participants in the second iteration saw AD 360º content as more time-consuming due to the level of visual details available and the need to set the angle. As for sign language interpreters, 50% thought it would take longer, 25% consid- ered it would depend on the content, and 25% thought that, with the right tools, it should not Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 72 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 have an impact. Overall, it seems that professionals acknowledge the challenges of producing accessible 360º content but also see the need for the right tools to make this process easier. When asked about the expected impact of 360º videos on their professional work, their attitudes were the following: 42% of subtitlers in the first iteration and 50% of the subtitlers in the second iteration think 360º videos will have an impact, although maybe not immediately. 19% in the first iteration and 17% in the second iteration do not think 360º videos will impact their professional work, and the remaining participants do not have a clear idea about it and state that it is highly dependent on how 360º videos evolve in the market. Regarding audio de- scribers, 58% in the first iteration and 39% in the second iteration think it can have an impact, even if maybe not in the short term, but 21% and 35% do not agree with this view. The rest are not sure and express concerns. With regards to sign language interpreting, 33.32% think it will have an impact, 49.98% indicate that it will not, at least yet, and 16.6% express uncertainty. Overall, there is some uncertainty about the impact of this new format on the professional work of access service professionals and a feeling that this may happen sometime in the future. 4. Evaluating the player ImAc developed a player (Hughes and Montagud, 2020) to enjoy the 360º accessible content and tested different versions. On the one hand, two tests with users with hearing loss were performed in Spain and Germany. The aim was to compare the performance of a standard menu—what in the project was called “traditional menu”—when used on an HMD and on a tablet. The first iteration took place in September-October 2018, and the second iteration in May-June 2019 (Figure 5). Figure 5. Traditional menu (1st and 2nd iteration) On the other hand, a test with users with sight loss, both in Spain and the United Kingdom, evaluated an enhanced accessibility menu (Figure 6) and a voice interaction system (Figure 7) with Amazon Echo. The enhanced accessibility menu was developed taking into account the needs of participants with partial sight loss, who may require bigger icons. This test was preceded by a pilot test in which the methodology was tested. Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 73 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Figure 6. Enhanced accessibility interface Figure 7. Voice interaction activation menu 4.1. Methods Testing the player interface focused on two aspects: usability of the tools, by means of SUS, and user preferences, which were gathered by means of a set of questions adapted to each evaluation action and by some additional open questions: • What did you like most about the ImAc player/menu interaction? • What did you like less about the ImAc player/menu interaction? • What do you think could be improved, and how? • Other comments. The protocol was the same as in previous tests: first, participants are informed about ImAc and the specific test, and informed consent forms are signed. Then, participants are asked to fill in an online demographic questionnaire and to perform a series of tasks: play and pause the video, change the volume, activate access services, personalise them, etc. After finishing these tasks, they are asked to fill in the SUS questionnaire and a preference questionnaire, and are thanked for their participation. 4.2. Participants The number of participants in the traditional menu test in Germany was 12 participants in the first iteration, in 2018, and 12 participants in the second iteration, in 2019. In Spain, the num- ber was 13 in the first iteration, in 2018, and 12 in the second iteration, in 2019. In respect of the enhanced accessibility menu versus voice interaction test, data from 23 users was obtained in Spain in 2020 (19 voice interaction/9 enhanced accessibility) and data from 28 participants Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 74 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 was gathered in the UK in 2019-2020 (15 voice interaction/9 enhanced accessibility). Pilot tests were performed in all cases, but they are not reported here. Detailed demographic infor- mation about the participants is available in Matamala (2020). 4.3. Results Table 3 provides SUS scores for the different tests concerning the player. Table 3. SUS scores for the player Germany (RBB) 1st iteration Spain (CCMA) 1st iteration HMD 77.3 68.8 Tablet 75.4 82.9 Germany (RBB) 2 iteration nd Spain (CCMA) 2nd iteration HMD 74.79 70.62 Tablet 63.96 86.25 UK (RNIB) Spain (UAB) Enhanced accessibility 93.30 86.39 Voice interaction 82.67 77.37 The player interface evolved during the project, but most versions assessed obtained above average SUS values. A strict comparison of the results across countries and iterations is not possible, as the participant profile was not homogeneous (for instance, mean age in the Ger- man second iteration was higher than in the first one) and the user interface was localised into the different languages. In the tests with users with hearing loss, the extensive personalisation and the clear design were considered useful features. Users provided positive feedback on the fact that they could choose a guiding mechanism, be it an arrow or a radar, to locate the speakers in the 360º sphere when the speakers were not visible. Most users were able to perform the requested tasks. However, it was observed that some of the personalisation features were named on the menu with terms that were not clear enough and needed to be changed. Similarly, the universal icons that were used in the interface were not clearly identified by participants, who suggested adding an abbreviation to clarify their meaning. Further suggestions for improvement and specific bugs were also identified during the tests. For instance, when users activated the subtitles in tablet mode, the changes were not immediately visible but hidden behind the main menu, which caused some confusion. This and other aspects were improved in the final version of the player. Moving beyond specific details linked to the specific user interface versions tested, it was observed that many participants found the HMD uncomfortable if they were wearing glasses, cochlear implants, or hearing aids. The users wearing the latter two devices asked whether it was possible to stream the audio directly to their devices. Overall, most users reported not having had any experience with 360º content before, because accessible 360º content is almost not available, but they were amazed by the experience and saw the potential for documentaries and concerts, for instance. Regarding users with sight loss, they tested either the enhanced accessibility menu or the voice interaction, depending on their needs. Concerning the enhanced accessibility menu, they Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 75 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 provided specific advice which was very much related to their sight condition and to the size, colour, and bordering of different menu items, putting forward the need for user customisa- tion. Regarding the voice interaction, most participants found that interaction with the ImAc player through Amazon Echo was easy to use, but also expressed frustration with the specific phrasing that needs to be used. They thought more conversational language should be used, an aspect to be further developed by voice assistants. The differences in assessments across countries were maybe due to other factors such as the development of the artificial voice used and previous experiences with Amazon Echo. It should also be highlighted that blind users complained about using an HMD, which they found cumbersome. 5. Evaluating the access services Different presentation modes were tested for each of the access services. They are reported independently in the following subsections. 5.1. Subtitling The main focus when testing subtitles in 360º was on two central topics: where to position the subtitles (subtitle positioning), and how to guide the viewers to the speaker in the 360º sphere (guiding mechanisms). When users do not have access to the audio, and the speaker is not in their field of view, mechanisms to guide them to the speaker’s position are needed. Otherwise, the user may lose part of the action. In ImAc, an arrow and a radar were developed and tested in different itera- tions (Figure 8). Figure 8. Arrow (1st iteration, top left; 2nd iteration top right) and radar (1st iteration, bottom left; 2nd iteration, bottom right) Subtitling positioning is also critical to avoid losing dialogue. In ImAc, different options were tested: first of all, the so-called always visible with arrow (Figure 9) was tested against fixed positioned with arrow (Figure 10). Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 76 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Figure 9. Always visible with arrow Figure 10. Fixed positioned with arrow Fixed positioned subtitles are attached to the speaker, which means that if the speaker is not on the field of view, the subtitles will not be shown. This is why a guiding mechanism is added, in this case an arrow (Figure 10). On the contrary, always visible subtitles in ImAc are subtitles displayed in front of the viewer, following the viewer’s movements. They are equiv- alent to static-follow subtitles by Brown et al. (2017, 2018) and to static subtitles by Rothe et al. (2018). Secondly, the always visible ImAc subtitles were tested against the subtitling po- sitioning mechanisms used in other platforms such as the BBC (n.d.) or The New York Times, namely subtitles equally spaced in a fixed position, in this case every 120º (Figure 11) (Agulló and Matamala, 2019, 2020). Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 77 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Figure 11. Fixed positioned every 120º Additionally, the innovative concept of easy-to-read subtitles was put to the test with ImAc content. Easy-to-read subtitles (Bernabé and Orero, 2019) are simplified subtitles which follow Easy-to-Read Language standards (Bredel and Maaß, 2016; IFLA, 2010; Inclusion Europe, 2009) and are validated by users (Bernabé et al., 2020). Other minor tests related to the use of emojis or textual information and related to the comfort field of view were also performed but will not be reported here in detail. 5.1.1. Methods Subtitling testing focused on presence and preferences. The aim was to make sure the ImAc solutions did not have a negative impact on presence whilst considering user preferences. The protocol was in line with the other ImAc protocols: welcoming participants and providing in- formation, signing informed consent forms, and watching two randomised clips where each of the presentation modes were shown. After watching each clip, participants were administered the IPQ questionnaire and at the end were asked to reply to a general questionnaire with slight variations. In all cases there was a question about what system they preferred and a question in which they had to explain why. They were also asked what could be improved, and how. Some additional questions were included. For the guiding mechanism test, these questions were: • Would you implement another system to guide you to the speaker? • How easy was it to identify who was speaking in the clip with the arrow system? (Likert scale) • How easy was it to identify who was speaking in the clip with the radar system? (Likert scale) • Do you think you will be able to enjoy 360º videos with this type of subtitles? Explain your answer. For positioning, the additional questions were: • How easy was it to read the subtitles in (option a/b)? (Likert scale) • How easy was it to find the subtitles in (option a/b)? (Likert scale) • How easy was it find the speaker in (option a/b)? (Likert scale) • Did you like the blinking effect on the arrow? Yes/No. For easy-to-read subtitles, the additional questions were: • How easy was it to read the subtitles in (option a/b)? (Likert scale) Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 78 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 • How easy was it to understand the subtitles in (option a/b)? (Likert scale) In all cases, participants were also invited to provide additional comments. Pilot tests were performed to test the methodology but are not reported here. 5.1.2. Participants The number of participants was as follows: • Test on guiding mechanisms: 12 participants (Germany) and 13 participants (Spain) in 2018. All were persons with hearing loss. The material used was an excerpt of the short film I, Philip. • Tests on positioning: • Always visible with arrow versus fixed positioned with arrow: 12 participants (Germa- ny) and 12 participants (Spain) in 2019. All participants were persons with hearing loss. The material used was an excerpt of the short film I, Philip. • Always visible versus equally spaced in a fixed position: test in Spain, in 2018, with 40 participants: 20 persons with hearing loss and 20 hearing persons, as it was considered that subtitles were useful for all user profiles regardless of their hearing loss, in line with Agulló, Matamala and Orero (2018). The material used was the documentary series Holy Land. • Test on easy-to-read subtitles versus standard subtitles: 36 participants, in Spain, in 2019. The targeted participants were elderly and their mean age was 69.4 (Oncins et al., 2020). The material used was Romeo and Juliette, a 360º opera in French subtitled into Spanish. 5.1.3. Results Results were gathered on presence and preferences for both guiding mechanisms. The ImAc solutions did not present a statistically significant difference in terms of presence, implying that both options have a place in 360º and their usage can actually be a matter of preference. Table 4 summarises all median values on a 7-point Likert scale and provides a summary of the statistical tests performed. A Wilcoxon Signed-Rank test was chosen for evaluation activities where we wanted to assess whether the distribution of two paired variables in two related sam- ples was the same. This is a common non-parametric test for data that may not follow a normal distribution. A planned comparison test (T-test) was chosen to compare preferences between easy-to-read subtitles versus standard subtitles by elderly participants (Spain 2019b), because of the number of participants. Table 4. Subtitling test results Presentation General Spatial Involvement Experienced mode presence presence realism Germany, 2018 Arrow 7 4.70 3.37 3.62 Radar 7 5.30 2.62 3.87 Test: Wilcoxon Signed-Rank test shows no statistical differences for spatial presence (Z=36.5, p=.094), involvement (Z=22, p=.952) and experienced realism (Z=28.5, p=.918). Spain, 2018a Arrow 6 5.60 4.00 3.50 Radar 6 5.80 4.75 3.50 Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 79 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Presentation General Spatial Involvement Experienced mode presence presence realism Test: Wilcoxon Signed-Rank test shows no statistical differences for spatial pres- ence (Z=21, p=.858), involvement (Z=3.5, p=.276) and experienced realism (Z=12.5, p=.799). Germany, 2019 Always visible 6 3.60 4.25 3.50 Fixed position 5 3.44 4 3.25 Test: Wilcoxon Signed-Rank test shows no statistical differences for general pres- ence (Z=-1.186, p=.236), spatial presence (Z=-358, p=.721), involvement (Z=-1.162, p=.245) and experienced realism (Z=-848, p=.396). Spain, 2019a Always visible 6 5.4 3.88 3.50 Fixed position 5.50 5.2 3.88 3.50 Test: Wilcoxon Signed-Rank test shows no statistical differences for general presence (Z=-.535, p=.592), spatial presence (Z=-.479, p=.632), involvement (Z=-724, p=.469) and experienced realism (Z=-178, p=.859). Spain, 2019b E2R 4.71 4.91 4.26 3.90 SDH 4.74 3.87 4.41 3.90 Test: Planned comparison (T-test) shows no statistical differences for general pres- ence (t(34)=.138, p=.891), spatial presence (t(34)=.361, p=.720), involvement (t(34)=- .803, p=.428) and experienced realism (t(34)=.000, p=1). An effect was only observed when comparing the always visible ImAc solution and the fixed positioned every 120º solution which is found on other websites. Table 5 presents mean values and a summary of the statistical analysis. Table 5. ImAc solution versus fixed positioned subtitles Presentation General Spatial Involvement Experienced mode presence presence realism Spain, 2018b Always visible 4.70 3.74 3.31 2.36 Fixed posi- 3.95 3.43 3.38 2.49 tioned every 120º Test: Wilcoxon Signed-Rank test shows no statistical differences for spatial presence (Z=-1.791, p=.073), involvement (Z=-1.229, p=.219) and experienced realism (Z=- 0.064, p=.949). For general presence, results were statistically significant (Z=-2.694, p=.007), showing a higher presence in the ImAc always visible solution. Arrow 4.6 3.66 3.5 2.49 Radar 4.27 3.49 3.27 2.47 Test: Wilcoxon Signed-Rank test shows no statistical differences for presence scale (Z=-1.852, p=.064), spatial presence (Z=-1.000, p=.317) and experienced realism (Z=-1.430, p=.153). For involvement, results were statistically significant (Z=-2.138, p=.033), showing a better performance of the arrow system. In terms of guiding mechanisms, arrows performed better than the radar system, and were also the preferred system in all tests (Table 6). The reasons for preferring the arrows were re- lated to the fact that the arrow was easier to identify and understand and less distracting. The radar was considered more visually obstructing and less intuitive. However, for some users, the radar was better when there were many speakers at the same time, as it gave more accurate spatial information. Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 80 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Table 6. Preferences on guiding mechanisms: arrow versus radar Arrow Radar Spain, 2018a 69.2% 30.8% Germany, 2018 72.7% 27.3% Spain, 2018b 82.5% 17.5% Concerning subtitle positioning, the always visible solution was better accepted than the ImAc fixed positioned solution or the fixed positioned found in other broadcasters such as the BBC. The reasons expressed by participants relate to the fact that in this mode they do not miss the content and subtitles are easier to find. Fixed positioned subtitles force the user to move their head to follow the information and are considered more challenging. However, there are some challenges related to a possible dizziness effect that some participants choosing the fixed positioned every 120º reported for the always visible mode. These participants also stated that fixed positioned subtitles were easier to read (see Matamala, 2020, for details on the 2018 Spanish tests). Table 7. Preferences on subtitling positioning Always visible Fixed positioned Spain, 2019a 83.3% 16.7% Germany, 2019 75% 25% Always visible Fixed positioned every 120º Spain, 2018b 82.5% 17.5% Finally, in the easy-to-read (E2R) subtitles versus standard subtitles for the deaf and hard of hearing (SDH) (Table 8), data indicate that preferences are distributed equally, which shows the potential of this new mode, especially with the elderly. When asked whether the subtitles were easy to read and easy to understand, easy-to-read subtitles always obtained better values. It remains to be seen what the results would be in other context beyond opera, where the lan- guage may be literary and the focus is put on the musical performance. Table 8. Preferences: Easy-to-read subtitles versus standard SDH (Spain, 2019) E2R SDH Both 52.8% 44.4% 2.8% 5.2. Audio description Audio description provides an auditory explanation of what is seen on screen, in other words, it is an intersemiotic translation of images into words. There were two AD tests: in a first round, a pilot test was performed with a reduced sample of participants (5 in the UK and 6 in Spain) to test three AD modes based on sound treatment: Classic, Static, and Dynamic. In the Classic mode, the ambisonic sound is placed above the user’s head. In the Static mode, the sound is located on the user’s side, as if someone were sitting close to the user, telling the story. In the Dynamic mode, the sound is placed at different angles, depending on where the main action being audio described takes place. The rationale behind this mode was that it would allow users to locate the events within the 360º sphere more easily, hence feeling more immersed in the story. The results of this test were not as expected and showed us that partici- Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 81 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 pants felt confused with this approach (Fidyka et al., 2021). They were more interested in the script features, and some of them actually required more extended audio descriptions. Taking into account the feedback from the pilot test, we revisited the AD modes and suggested three new approaches: Classic, Radio and Extended. In all three modes the sound treatment stays the same—it is placed above the user’s head—but the scripting style changes. Classic AD offers a standard approach to AD. Radio AD features an unconventional approach, with more engaging and informal language. Extended AD allows users to pause the video and listen to an additional description, activated at their will. 5.2.1. Methods Testing focused on presence and preferences. The protocol followed was the same as in other ImAc tests: welcoming participants and providing information, signing informed consent for- ms, filling in a demographic questionnaire, and watching clips. In this case, they listened to one general audio introduction to the content and then watched three episodes of the series Holy Land with randomised audio presentation modes. After each video they replied to the IPQ questionnaire and, at the end, they were given a general questionnaire with preference questions. This questionnaire included the following questions: • Which AD type do you prefer in 360º? Rank the AD modes according to your preferences. • Please explain why you ranked the AD modes in this way. • How could AD be improved in this medium? • Other comments. At the end of this test, participants stayed for one additional test on audio subtitling, which is reported later in this article. 5.2.2. Participants All participants were persons with sight loss recruited through user associations. 31 partici- pants took part in the UK in 2019-2020 and 30 participants took part in Spain at the beginning of 2020. 5.2.3. Results Results on presence are summarised in Table 9 and results on preferences are presented in Table 10. Table 9. Median values for IPQ (UK and Spain) General Spatial Involvement Experienced presence presence realism Classic UK 4.32 4.10 3.90 3.83 Spain 4.73 4.62 4.97 3.40 Radio UK 4.61 4.32 4.22 3.92 Spain 4.70 4.55 5.04 3.54 Extended UK 4.61 4.22 4.22 3.92 Spain 4.73 4.64 4.86 3.63 Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 82 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Paired samples statistics T-tests did not present any statistically significant difference (all p>.05), showing it is a matter of preferences. In fact, regarding preferences, Table 10 sum- marises the main results. Table 10. AD preferences Classic Radio Extended 1st preferred mode UK 24% 43% 33% Spain 33% 27% 40% 2 preferred mode nd UK 52% 14% 34% Spain 33% 33% 33 % 3 preferred mode rd UK 24% 43% 33% Spain 34% 40% 27% The two innovative approaches offered by ImAc were the preferred ones by participants, although with differences: whereas participants in the UK preferred the radio version, partic- ipants in Spain preferred the extended option. UK participants choosing this option reported being frequent users of radio dramas and audiobooks, which may adopt a similar approach. It is remarkable that 76% of the participants in the UK and 66% of the participants in Spain chose the classic mode either as their first or second option. This does not come as a surprise, as this was the format that was more similar to what is currently offered on standard television in both countries. 5.3. Audio subtitling Audio subtitles (AST) (Braun and Orero, 2010) are written subtitles which are read aloud either by a human professional or a text-to-speech system. This access service is especially useful for those who cannot see the subtitles (for instance, persons with sight loss) and for those who cannot read them (for instance, persons with reading difficulties). It can be offered independently or it can be integrated in the audio description. To gather data on this access service, participants in the AD tests (see above) were asked to perform one task in which they had to state their preference between AST only or AST with AD in a very specific type of content: opera. 5.3.1. Methods Testing audio subtitling was part of the same experimental session in which the AD was tes- ted. To avoid participants’ fatigue, the test focused only on preferences, with the following questions: • Which type of AST do you prefer in 360º? AST/AST-AD. • Explain in your own words why you prefer the option chosen in the first question. • What system would you prefer? AST with no subtitles on screen./AST with subtitles on screen that can be enlarged./I would like to select when I want subtitles on screen and when not. • How could AST be improved in this medium? Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 83 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 • In what excerpt did you feel more present? Only AST./AST-AD./I did not feel present in any of them./ I felt equally present in both. • Other comments. The content used for the test were two clips from the opera Romeo and Juliette, with a length of approximately 4 minutes each. An audio introduction was offered before the test to contextualise the clip. 5.3.2. Participants The same participants as in the AD test took part in this test (see section above). 5.3.3. Results Table 11 provides a summary of the results both in the UK and in Spain. Table 11. AST results UK Spain Preference AST 8 16 (25.8%) (55.2%) AST-AD 23 13 (74.2%) (44.8%) More present in... AST 8 10 (25.8%) (34.5%) AST-AD 11 8 (35.5%) (27.6%) Equally in both 3 9 (9.7%) (31%) In none 9 2 (29%) (6.9%) How to visualise AST with no subtitles on screen 14 13 subtitles on screen? (45.2%) (44.8%) Possibility to choose 4 16 (12.9%) (55.2%) Subtitles on screen that can be 13 0 enlarged (41.9%) (0%) Results indicate that preferences and self-reported presence differ in the two countries under study. It should be highlighted that the stimuli were translated and voiced by different talents, which can have an impact on the results. It is also possible that cultural factors and different habits in terms of accessible opera consumption may be behind the choices of the participants. Opera audio subtitling and audio description is a reality in Spain, but users in the UK are not very familiar with this practice, hence this may explain why Spanish participants were more willing to accept AST than British participants. 5.4. Sign language interpreting The aim of the SL interpreting tests was to address three main aspects: a) display of signer vi- deo, which could be continuous—meaning the signer window would be visible all throughout the content—or non-continuous—meaning the signer window would only be visible when interpreting was needed; b) presentation of sign language only versus presentation of sign Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 84 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 language plus subtitles; c) speaker representation, which could be performed through an emoji representing the speaker or a textual description. 5.4.1. Methods The protocol followed was the same as in other ImAc evaluation activities. In this case, three tasks were performed, addressing each of the items above. For each of the tasks, one clip per condition was prepared and their presentation was randomised across participants. The stimuli for the tasks on signer video display and for the tasks on speaker representation were extracted from I, Philip, whereas the videos for the second task came from the opera Romeo and Juliette. The test was developed at RBB in Germany, and content was translated into German and Ger- man Sign Language. For the first task, presence was measured through IPQ and preferences were assessed using the following questionnaire: • What system do you prefer? • Please indicate why you prefer the above-indicated option. • Please indicate why you did not choose the other option. • Other comments. When participants were administered the SL with subtitles versus SL only task, only pref- erences and self-reported values were gathered, by means of three questions: • How easy was it to read the information with SL and subtitles/SL only? (7-point Likert scale) • How easy was it to understand the information with SL and subtitles/SL only? (7-point Likert scale) In the last test, and to avoid participants’ fatigue, a single question with the same scale was put to participants: How easy was it to know who was speaking with the textual representa- tion/emoji representation? 5.4.2. Participants 10 Deaf participants took part in the test. They were recruited through German broadcaster RBB. Their main languages were German (2) and German Sign Language (8). 5.4.3. Results As far as the presentation of continuous or non-continuous sign language interpreting is con- cerned, median scores for each subscale on the IPQ questionnaire are shown in Table 12. Table 12. Sign language interpreting results General Spatial Presence Involvement Experienced Realism Presence Continuous 4 3.60 4.13 2.5 Non-Continuous 5 3.60 3.75 2.5 The comparison of each of the subscales of the IPQ questionnaire through a Wilcoxon Signed-Rank test does not show any significant difference between the two conditions (Gener- al presence Z=-.604, p= .546, ties=3; Spatial presence Z=-1.532, p=.125, ties=3; Involvement Z=-.341, p=.733, ties=3; Experienced realism Z=-.421, p=.674, ties=2). Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 85 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 In terms of preferences, data from one user could not be retrieved due to technical prob- lems. 6 out of 9 users prefer a non-continuous display because it is easier to concentrate on the video content. Those participants who preferred the continuous display felt reassured by this option and thought it reproduced a real-life situation, where the interpreter is always present. The results of the comparison of SL with subtitles versus SL were unexpected by the re- searchers, as Deaf communities often express a strong preference for SL and are reluctant to use subtitles. When participants were recruited, they were asked whether they would like to test subtitles or SL, and the participants on this specific test chose SL. Still, when confronted with signed content that also displays subtitles, four users indicated that they preferred to use SL only and six users indicated that they preferred to use both services. Some of the advan- tages offered by the combination of access services, according to the test participants, are: the colour of the subtitles helps them to identify the speaker and, especially in the opera content, the subtitles transport more information than the SL interpreting. When asked how easy it is to “read” the video with SL and subtitles and the video with SL only, 80% of the responses were on the 5 to 7 range on a 7-point Likert scale, where 7 equals “very easy”. When asked how easy it is to understand it, the video with SL and subtitles received values between 5 and 7 in 80% of the responses, whereas the video with SL only had 60% of the replies in the same range. Finally, when asked to indicate their level of agreement with the statement “I think reading SL and subtitling simultaneously is overwhelming for me”, only 30% selected the maximum value. Regarding the last task, 7 users preferred the textual speaker representation and only 3 chose the emojis. They reported that the text enables an easier identification of the speaker. This is undoubtedly a topic that still needs further research, as this was only assessed with one question at the end of the test. 6. Conclusions ImAc has taken one of the first steps in researching accessibility in virtual environments. Evaluation activities have been performed when VR is still not widely spread and when both professional and home users are not used to this technology. This implies many challenges but, at the same time, many opportunities. The different evaluation activities presented in this paper have allowed us to map the initial needs and requirements of professionals producing accessible content and gather interesting input on their views. There is still much uncertainty about the impact 360º videos will have on their professional lives, but many of them envisage this technology positively, albeit acknowledging that producing access services in VR will be more time-consuming. In respect of home users, tests with end users have allowed us to evaluate different proto- type versions of the player. The player has received positive usability assessments and an in- novative voice interaction feature has been welcomed by participants with sight loss, although they would like to interact with more conversational language. Regarding access services, subtitling tests have put the emphasis on positioning and guiding mechanisms, showing that ImAc solutions perform better than existing solutions, and also showing a user preference for the arrow as an indicator. However, it remains to be seen if this behaviour is kept when users Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 86 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 are confronted with narratologically complex scenes in which different speakers are found at different places in the 360º sphere. Additionally, the tests have put forward the potential of applying easy-to-read principles to subtitling with the elderly, although there is a need to expand this testing to other user groups and content typologies. Regarding audio description and audio subtitling, two access modes generally addressed to persons with sight loss, script-based evaluation has shown the potential of new approaches beyond the classic mode. Again, replication with other types of content and future tests with improved sound features are still much needed. Finally, sign language testing has identified user preferences in the presentation of the in- terpreter video and has put forward an unexpected result, i.e. that a significant percentage of users prefer to have both SL and subtitling, even if they are native SL speakers. Overall, tests have not shown statistically significant results. Although the sample size may have had an impact and further tests with wider samples are needed, it seems the ImAc solu- tions have a similar effect on presence, which makes us think users should be given different choices according to their needs and preferences. Customisation is undoubtedly the future of access services in media consumption and more research is needed in the field. References Agulló, B. (2020). Technology for subtitling: a 360-degree turn. Hermeneus, 22, 11-40. Agulló, B., & Matamala, A. (2019). Subtitling for the deaf and hard-of-hearing in immersive environ- ments: results from a focus group. Jostrans. The Journal of Specialised Translation, 33, 217-235. Agulló, B., & Matamala, A. (2020). Subtitles in virtual reality: guidelines for the integration of subti- tles in 360º content. Íkala, 25(3), 643-661. Agulló, B., Matamala, A., & Orero, P. (2018). From disabilities to capabilities: testing subtitles in im- mersive environments with end users. Hikma, 17, 195-220. Agulló, B., Montagud, M., & Fraile, I. (2019). Making interaction with virtual reality accessible: ren- dering and guiding methods for subtitles. Artificial Intelligence for Engineering Design, Analysis and Manufacturing, 33(4), 416-428. https://doi.org/10.1017/S0890060419000362 BBC (n.d.). Reel World. 360° Videos from the BBC. https://www.bbc.com/reel/playlist/reel-world-360- videos-from-the-bbc Bernabé, R., & Orero, P. (2019). Easy to read as multimodal accessibility service. Hermeneus, 21, 53- 74. Bernabé, R., Orero, P., García, O., & Oncins, E. (2020). Validation of easy-to-read subtitles. In D. De- jica, C. Eugeni, A. Dejica-Cartis (Eds.), Translation Studies and Information Technology – New Pathways for Researchers, Teachers and Professionals (pp. 168-181). Editura Politehnica. Braun, S., & Orero, P. (2010). Audio description with audio subtitling. Perspectives: Studies in Trans- latology, 18(3), 173-188. Brookes, J. (2013). SUS: a retrospective. Journal of Usability Studies, 8(2), 29-40. Bredel, U., & Maass, Ch. (2016). Leichte Sprache. Theoretische Grundlagen, Orientierung für die Praxis. Duden. Brown, A., Turner, J., Patterson, J., Schmitz, A., Armstrong, M., & Glancy, M. (2017). Subtitles in 360-de- gree video. Proceedings of the 2017 ACM International Conference on Interactive Experiences for TV and Online Video, Hilversum, Netherlands, 3-8. https://doi.org/10.1145/3084289.3089915 Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 87 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Brown, A., Turner, J., Patterson, J., Schmitz, A., Armstrong, M., & Glancy, M. (2018). Exploring sub- title behaviour for 360° video (White Paper WHP 330) https://www.bbc.co.uk/rd/publications/ whitepaper330 Catapult Digital (2018). Growing VR/AR companies in the UK: A business and legal handbook. https:// www.pwc.co.uk/intelligent-digital/vr/growing-vr-ar-companies-in-the-uk.pdf European Broadcasting Union (EBU). (2017). Virtual reality: How are public broadcasters using it? https://www.ebu.ch/publications/virtual-reality-how-are-public-broadcasters-using-it Fidyka, A., & Matamala, A. (2018). Audio description in 360º videos: results from focus groups in Barcelona and Kraków. Translation Spaces, 7(2), 285-303. Fidyka, A., & Matamala, A. (2019). Production of access services in immersive content: understanding the needs of audio describers. Hikma, 18(2), 277-300. Fidyka, A., & Matamala, A. (2021). Retelling narrative in 360º videos: implications for audio descrip- tion. Translation Studies. https://doi.org/10.1080/14781700.2021.1888783 Fidyka, A., Matamala, A., Soler-Vilageliu, O., & Arias-Badia, B. (2021). Audio description in 360º content: results from a reception study. Skase, 14(1), 14-32. Fryer, L., & Freeman, J. (2012). Presence in those with and without sight: Audio description and its potential for virtual reality applications. Journal of Cyber Therapy & Rehabilitation, 5(1), 15-23. Hughes, Ch., & Montagud, M. (2020). Accessibility in 360º video players. Multimedia tools and appli- cations. https://doi.org/10.1007/s11042-020-10088-0 IFLA (2010). Guidelines for easy-to-read materials. https://www.ifla.org/files/assets/hq/publications/ professional-report/120.pdf Inclusion Europe (2014). Information for all: European standards for making information easy to read and understand. https://inclusion-europe.eu/wp-content/uploads/2015/03/2113-Informa- tion-for-all-16.pdf ISO (1998). ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability. Lombard, L., & Ditton, T. At the heart of it all: The concept of presence. Journal of Computer-Mediat- ed Communication, 3(2). https://doi.org/10.1111/j.1083-6101.1997.tb00072.x Maszerowska, A., Matamala, A., & Orero, P. (Eds.) (2014). Audio description. New perspectives illus- trated. John Benjamins. Matamala, A. (2020). Deliverable 5.4. Pilot evaluation report. Final deliverable, March 2020. Matamala, A., & Orero, P. (Eds.) (2010). Listening to subtitles. Subtitling for the Deaf and Hard-of- Hearing. Peter Lang. McDonald, A. (2012). The in-vision sign language interpreter in British television drama. In A. Re- mael, P. Orero, & M. Carroll (Eds.), Audiovisual translation and media accessibility at the cross- roads (pp. 189-206). Rodopi. Oncins, E., Bernabé, R., Montagud, M., & Arnáiz Uzquiza, V. (2020). Accessible scenic arts and Vir- tual Reality. A pilot study with aged people about user preferences when reading subtitles in im- mersive environments. MonTI, 12, 214-241. Orero, P. (2020). Born accessible: Beyond raising awareness. https://ddd.uab.cat/record/222130 Reeves, B.R. (1991). Being there: Television as symbolic versus natural experience. Unpublished man- uscript, Stanford University, Institute for Communication Research, Stanford, CA. Rothe, S., Tran, K., & Hussmann, H. (2018). Dynamic subtitles in cinematic virtual reality. ACM TVX 2018, Seoul, Republic of Korea, 209–214. https://doi. org/10.1145/3210825.3213556 Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 88 SENDEBAR (2021), 32, 65-89. https://doi.org/10.30827/sendebar.v32.16881 Schubert, T., Friedmann, F., & Regenbrecht, H. (2001). The experience of presence: Factor analytic insights. Presence: Teleoperators and Virtual Environments, 10(3), 266-281. Acknowledgements This work was supported by the European Union’s Horizon 2020 research and innovation programme under Grant 761974 and by Generalitat de Catalunya under Grant 2017SGR113. I would like to thank all the ImAc colleagues and participants who took part in the tests, as well as Olga Soler-Vilageliu for her support in the statistical analysis. Matamala, A. Accessibility in 360º Videos: Methodological Aspects and Main Results of Evaluation Activities… 89