OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi Journal of Deaf Studies and Deaf Education, 2019, 1–13 doi: 10.1093/deafed/enz021 Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 Theory/Review Manuscript T H E O RY / R E V I E W M A N U S C R I P T Toward a Sign Language-Friendly Questionnaire Design Marta Bosch-Baliarda*,† , Olga Soler Vilageliu and Pilar Orero Universitat Autònoma de Barcelona *Correspondence should be sent to Marta Bosch-Baliarda, Edifici MRA 126—Campus UAB, 08193 Bellaterra, Barcelona, Catalonia (EU) (e-mail: Marta.Bosch@uab.cat) Abstract The United Nations Convention on the Rights of Persons with Disabilities requests “Nothing about us without us.” User-centered methodological research is the way to comply with this convention. Interaction with the deaf community must be in their language; hence sign language questionnaires are one of the tools to gather data. While in the past interacting with an online video questionnaire was out of the question, today it is a reality. This article focuses on the design of an interactive video questionnaire for sign language users. From a historical review of the existing literature on research methods and previous sign language questionnaire, the article examines the design features affected in the process of making accessible questionnaires with sign language videos: format and layout. The article finishes with the solution developed toward mainstreaming sign language questionnaires in order to contribute to a diverse and inclusive society for all citizens. Questionnaires are a common instrument used for research. A of information and communication technologies (ICTs) allows set of questions extracts relevant information given to a sample for deaf-friendlier design and also to implement translation of respondents to complete (Harkness, 2008). They are used in protocols toward access to information for all users (Fontaine, research over other data collection methods and frequently have 2012; Young & Hunt, 2011; Young & Temple, 2014). Moreover, standardized answers that provide easy to compile and analyze designs are crucial when creating a fully accessible question- data. They are relatively cheap to implement, compared to other naire to develop robust instruments to collect data on sign surveying or assessment instruments, such as interviews or language from sign language users, within the sign language focus groups (Survey Research Center, 2016). communities (De Meulder, Krausneker, Turner, & Conama, 2018; As with any research instrument involving deaf participants, Harris, Holmes, & Mertens, 2009). questionnaires need to incorporate a deaf-friendly perspective In this article we review the evolution of methods in toward equity for the deaf population in both surveys and testing filmed-based approaches that include sign language to make research. Questionnaires are frequently administered through questionnaires accessible. Specifically, we look into the language the official, written languages and rarely present accessibility accessibility features, design features, and administration features, such as using Easy to Read (Bernabé & Orero, 2019). modes implemented in previous approaches. We present our Cultural and linguistic lack of perspective in instrument design own methodological approach to create a sign language- has a negative impact on both the appropriateness of the friendly questionnaire design, taking into consideration time- instrument itself as well as the validity of the results (Freeman, efficient and cost-efficient limitations. The implementation 1989). More recently, accessibility through computerized video is a computerized video questionnaire fully accessible in sign technologies in questionnaire designs has promoted the language that can robustly deploy a small-scale, research- possibility of incorporating sign languages. The application oriented, sign language recall and comprehension test. † MartaBosch-Baliarda, http://orcid.org/0000-0002-8778-2820 Received September 17, 2018; revisions received April 19, 2019; accepted April 27, 2019 © The Author(s) 2019. Published by Oxford University Press. All rights reserved. For permissions, please email: journals.permissions@oup.com 1 OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi 2 Journal of Deaf Studies and Deaf Education Toward a Deaf-Friendly Questionnaire Design promote cultural and linguistically adequate research methods. This means involving members of the Deaf community in the Fully accessible questionnaire designs in sign language are Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 research process in meaningful ways following emancipatory important to guarantee the reliability of the research and the or transformative paradigm approaches, not only as research validity of the results. They are also relevant both to protect participants (Harris et al., 2009; Mertens, 2005, 2010; Munger ethical standards in human subject research within the deaf & Mertens, 2011; Mertens, Sullivan, & Stace, 2011). The par- population and to promote accessibility to comply with the ticipation of deaf native signers and deaf organizations in new, inclusive “human right” paradigm (Berghs, Atkin, Graham, research teams is subsequently regarded as a key concern. Hatton, & Thomas, 2016; Ewart & Snowden, 2012). This is not Research teams should include different roles such as deaf only to control the bias that may arise from the lack of linguistic researchers, deaf research assistants, deaf research facilitators, and cultural concordance between the researchers and the deaf interpreters or support communicators, and deaf-signing signing deaf participants (McKee, Schlehofer, & Thew, 2013) models (Ladd, 2003; McKee et al., 2013; Pollard, 2002; Singleton but also to promote inclusive research that undertakes and et al., 2012, 2015). integrates the diversity inherent within the Deaf communities, On the other hand, developing deaf-friendly research especially regarding language accessibility: language choice and methods involves merging accessibility and communication literacy (Guardino & Cannon, 2016). best practices with different research materials, documents, and the dissemination of research projects and results. The potential of eAccessibility through ICT for social inclusion, Deaf-Friendly Research Methods and participation of all citizens, is increasingly allowing full The literature for sign language research methods tackles integration in everyday life for all (Orero, 2017). The use specific knowledge areas in both theoretical and applied sign of video protocols in computer-based technologies is an language linguistics (Meurant, Sinte, Vermeerbergen, & Van opportunity to represent the modality-specific visual features Herreweghen, 2013) such as sign language acquisition (Baker, of sign languages (Haug, 2011, p. 46). Video technologies have van den Bogaerde, & Woll, 2006), sociolinguistics (Lucas, Mirus, also had an impact on the type of data and data collection Palmer, Roessler, & Frost, 2013), and sign language assessment methods for research involving sign language users, such as (Haug, 2011). Both hearing and deaf researchers have focused face-to-face video chatting or user-generated video sharing on ethical considerations when conducting research within the (Lucas et al., 2013, p. 548). deaf communities, particularly on communication accessibility, truly informed consent, and anonymity (Baker-Shenk & Kyle, 1990; Benedict & Sass-Lehrer, 2007; Gutman, 2005; Harris et Language Accessibility in Questionnaire Designs al., 2009; McKee et al., 2013; Pollard, 2002; Singleton, Jones, & Hanumantha, 2012; Singleton, Martin, & Morgan, 2015; Sign The language and culture of the participants can affect their Language Linguistics Society, 2016). Accessible communication perception of questions and therefore their answers (Choi & Pak, and informed consent should be a priority for researchers. 2005). Ethnic minority groups have a higher rate of nonrespon- Additionally, privacy issues may arise from the fact that it is dence (Erens, 2013). This can be related to “language barriers, almost impossible to separate the video-recorded sign language lack of trust, wariness of government authorities, perception data from the participants’ identity; hence signed, accessible that the research is unimportant or that their contribution is image release forms should be provided. unimportant, reluctance to have their information written down, Some of these ethical considerations and guidelines on deaf- and a feeling that they have been over-researched” (Erens, 2013, friendly research have been compiled by Orfanidou, Woll, & p. 59). Similar factors have been described for signing deaf par- Morgan (2015) in their handbook Research Methods in Sign Lan- ticipants, though their low participation may be partially linked guage Studies: A Practical Guide. This text is a landmark on sign to participant fatigue (Cawthon & Garberoglio, 2017). language research devoted to data collection methods and new In this article we examine traditional monolingual-written instruments and tools for sign language research in different text questionnaires using signing participants, because they are fields of linguistics, paying special attention to ethical consider- widely used in many research fields with sign language users. ations. Unfortunately, deaf-friendly questionnaire design using Hearing researchers may be unaware of the cultural and lin- sign language as a mode of administration is not discussed. More guistic characteristics of the deaf community and vice versa. recently, Cawthon & Garberoglio (2017) edited the text Research in Understanding questions will depend on the respondent’s lit- Deaf Education: Context, Challenges, and Considerations, featuring an eracy skills that can lead to uncontrolled variation in ques- array of diverse research methodologies for the Deaf community. tion misunderstanding bias and participant cognitive burden. This volume includes a chapter on accessible, large-scale sur- Adequate language options and question wording are impor- vey design in education (Cawthon, 2017). The chapter describes tant toward the validity of the questionnaire design. Questions some considerations when developing large-scale surveys in must be phrased appropriately for the target audience with deaf education, for example, how to control the heterogeneous the information acquired avoiding ambiguity and connotative demographics within the Deaf communities or how to enhance meaning (Boynton & Greenhalgh, 2004). Even though it is still the recruitment process. In the section on accessible survey not common practice, our literature review includes several designs, the author argues that dual-language designs not only examples on explicit explanations, or overt discussion, on how provide better access but also promote engagement within the data collection instruments and materials are made available to deaf community. Although she advocates the use of technology signing deaf participants and the language choices in which data platforms that fully support dual-language/dual-modality sur- and questions are presented. veys is a requirement, technical design features are not further discussed. Translation procedures in questionnaire design. Most research A number of authors have included deaf-friendly method- is still developed in written languages. Translation should be ological approaches and good practices in their research to performed according to standards to guarantee the quality OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi Sign Language-Friendly Questionnaire Design 3 of the instrument and to control the effects of translation The goal of adaptation in cross-cultural questionnaire design (Harkness, 2008). The process of preparing, performing, review- is to match the needs of the target population, also known Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 ing, and assessing translated questionnaires is thus quite as “localization” in translation studies, as is the case in Pol- complex and costly (Erens, 2013). lard, Dean, O’Hearn, and Haynes (2009) for health education There are different translation procedures and protocols material for deaf audiences.The modifications “may be made devised to control translation quality and minimize translation to the content, format, response scale, or visual presentation of bias that may jeopardize research results. One of the early doc- any part of the question, questionnaire, or instrument” (Survey umented translation approaches in sign-translated question- Research Center, 2016). Sign language linguists have outlined naires is back translation. Back translation was a recommended several phases in the adaptation process of sign language assess- standard for translating survey instruments (Erens, 2013; Survey ment tests to ensure the standardization of the developed tools. Research Center, 2016). This procedure involves a specialist These adaptation protocols aim to guarantee the validity and translating the text into the target language, which is then reliability of the test instrument (Enns & Herman, 2011; Haug, independently translated back into the source language. The 2011; Hauser et al., 2016; Mann, Roy, & Morgan, 2016) original and back translated source language texts are compared to check if there are any translation issues in the target text. Even New language approaches in sign language questionnaire designs. today, back translation is one of the most common translation More recently, Cawthon (2017) suggests the inclusion of techniques. Some of the reviewed questionnaires for deaf bimodal–bilingual for the development of research materials. populations included professional sign language interpreters The author favors a dual-language development process rather and/or signing deaf research team members carrying out the than designing survey items in the written form and the imple- role of back translators in the development of the target video mentation of translation or adaption procedures. “Beginning sign language questionnaires (VSL questionnaire; Brauer, 1993; the development of survey items with an understanding of Cohen & Jones, 1990; Goldstein, Eckhardt, & Joyner, 2004; Lipton, how they may be represented in different linguistic formats Goldstein, Fahnbulleh, & Gertz, 1996; Rojba, 2016). holds promise in making large-scale surveys more accessible Nowadays, best practices and guidelines recommend team by design” (Cawthon, 2017, p.31). Dual-language development translation approaches for survey instrument production. Cur- of survey items would only include concepts that are equally rently, the common best practice in survey translation is the five- easy to express in both languages. They would control potential phase iterative team approach model called translation, review, sources of bias and also provide a more accessible design for all adjudication, pretesting, and documentation (TRAPD; Harkness, participants. 2008). The TRAPD method involves several translators, survey A different innovative language approach to survey item experts, and consultants and is pretested with a small-scale development has recently been designed and implemented target population group before producing the final translation. by Napier, Lloyd, Skinner, Turner, and Wheatly (2018), tar- The team members involved may take on one or more roles geting multilingual deaf signers using multiple national sign as translators, reviewers, or adjudicators. During the first stage, languages from various European countries. Their goal is to translators produce initial parallel translations independently. implement a large-scale survey to engage a larger number of Then an expanded team including the translators and a reviewer international participants from a broader range of linguisti- go through the draft translations, discuss the versions, and agree cally diverse signing backgrounds. Because of time and cost on a review translation. restrictions, the authors choose to use International Sign Adjudicators decide if the reviewed version moves to pretest- Language for those users who lack literacy or prefer to use ing. Adjudication may take place at the review session, or at a a signed language as opposed to a multicultural multilingual different meeting between the reviewer and the adjudicator. All approach including many national sign languages. This solution steps are fully documented. Documentation is a monitoring tool also avoids prioritizing certain national sign languages over for quality assurance that provides information for further anal- others, potentially excluding users of minority sign languages ysis. For example, team members take notes on compromised (Napier, et al., 2018, p. 104). decisions, unresolved issues, or consultant feedback. The proce- On balance, time and cost constraints will greatly influence dures are partially iterative. For example, pretesting may trigger the choice of language approach, translation, and adaptation further modifications of the translation before the adjudicator protocols and procedures, as will study specifications such as decides on the final version for fielding (Harkness, 2008; Survey sample size and standardization (Kappelhof, 2015). Adaptation Research Center, 2016). and translation procedures may be determined according to the nature of the material being implemented and the project Adaptation in sign language questionnaire design. Sign linguists characteristics. Lastly, a successful questionnaire translation is in the field of sign language tests have addressed methodolog- expected to keep the content of the questions semantically sim- ical issues concerning translation and adaptation procedures ilar or maintain the same stimulus and measurement options. to develop reliable and valid assessment instruments (see, e.g., It should also keep the question format similar between the Enns & Herman, 2011; Haug, 2011, 2015; Haug & Mann, 2008; source and target questionnaires, within the bounds of the target Hermans, Knoors, & Verhoeven, 2010). Within the field, the term language (Survey Research Center, 2016). For sign languages that adaptation is used to emphasize that the goal is to create a test differ greatly in modality and have no established written form, in the target language that parallels the source test not only much more research is required. in its language contents but also in its functionality. Although the translation process is assumed to be centered in faithfully rendering the source contents into the language of the target A Review of the Literature on Deaf-Friendly test, the adaptation process involves more flexibility in test construction and may introduce further alterations according to Questionnaires the cultural and social needs of the target language users (Haug Computerized sign language implementations for questionnaire & Mann, 2008, p. 139). research have rapidly changed, thanks to the advances of new OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi 4 Journal of Deaf Studies and Deaf Education technologies especially in the development of video technology (Lipton et al., 1996); animated questionnaire for deaf and hard of and online survey platforms (Lucas et al., 2013, p. 54). When hearing (DHH) people, or ANIMAQU (Gerich & Lehner, 2006); Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 reviewing early research methods in sign language studies dual-mode (CATI/CAPI) survey-computer-assisted telephone linguists often relied on written-only questionnaires for data interview/computer-assisted personal interview (Sloan, Wright, collection on sign language structure and use. To present sign & Barrett, 2012); virtual video survey (Lucas et al., 2013); and language stimuli, written forms of the signs or sign structures dual-language, dual-modality survey (Cawthon & Garberoglio, being analyzed were used in paper format questionnaires (see, 2017). e.g., Woodward, 1973, and Woodward & DeSantis, 1977). During In this article, the term used is VSL questionnaire to the following decade, video protocols were introduced using the refer to the computerized deaf-friendly questionnaire designs different video formats available. Early video implementations implementing full access in sign language using video protocols. involved a written questionnaire with a video-taped sign- When it is necessary to distinguish between the modes of translated version that was played alongside (Brauer, 1993). administration we use the short forms VSL-CASI and VSL-CAPI, Although this solution would give access to the content, the for convenience. signed and written versions differed largely in format and layout. In the 21st century, sign language assessment tests have been The first attempt of a self-administered sign language ques- another important source of advances in computer- and web- tionnaire that did not require literacy was the Interactive Video based video sign language implementations (Enns et al., 2016; Questionnaire, developed to question deaf people about drug Haug, 2011; Haug, Herman, & Woll, 2015; Haug & Mann, 2008; abuse (Lipton et al., 1996). The questionnaire was implemented Hauser et al., 2016; Herman, Holmes, & Woll, 1999; Hermans et using a Laserdisc player on a standard television monitor dis- al., 2010; Lara-Escudero, 2017; Mann et al., 2016). New method- playing questions in American Sign Language. In Phase I of the ologies have started to adopt VSL questionnaires as a mode study, the participants had a bar code reader that served to scan of administration, such as sign language comprehension and the responses and add them on the paper questionnaire answer video quality evaluations on technological devices (Cavender, sheet. In Phase II, touchscreen technology for automatic data Ladner, & Riskin, 2006; Tran, 2014; Tran, Kim, Riskin, Ladner, & capture and storage was used, which proved to be a simple mode Jacob, 2011; Tran, Riskin, Ladner, & Wobbrock, 2015) including of administration. Responses were recorded directly to the hard synthesized sign language played by signing avatars or anima- disk and then downloaded to a cassette tape. tions (Huenerfauth & Kacorri, 2015). Other VSL questionnaires For interviewer-administered tests, Haug (2015, p. 45), in his are described in some research-led surveys and tests in different exploratory study based on an international survey on the usage fields, such as Deaf education and sign language teaching of ICT for sign language test delivery, listed different test modes applications (Hansen et al., 2018; Higgins, Famularo, Bownman, of administration including noncomputerized and computer- & Hall, 2015; Hussein & Al-Bayati, 2009), cross-modal language ized formats. Noncomputerized formats included, for instance, user tests (Lucas et al., 2013), cross-cultural social surveys videotape (VHS and other formats) or DVD either with or without (Fontaine, 2012), sign language teaching surveys (Pyfers, 2017), an accompanying booklet. Haug (2015) also listed some comput- sign language interpretation (De Wit, 2011; Lang, 2015), or erized modes of administration, for example, web-based tests accessibility services and use of technology (Maiorana-Basas or sign language tests implemented into existing learning tools & Pagliaro, 2014; Napier et al., 2018), among others. Even though such as Moodle. all these researchers acknowledged the use of a computer-based There are many deaf-friendly designs implementing video VSL questionnaire, the technical design of the implementations protocols in order to give full access to signing participants. The varies greatly, ranging from specially created, interactive term VSL questionnaire is used to refer to all instrument designs applications to the use of existing online survey platforms that and modes of administration using video protocols that grant play video sign language content alongside it. accessibility in sign language to a certain extent. This includes Moreover, for self-administered interviews, participants both computerized and noncomputerized modes of administra- should go back-and-forth between the two versions, which tion that have used different video protocols and technologies would increase both time burden and cognitive burden, and available since the 1980s. therefore affect response time and nonresponse rate. Together There are three types of computerized survey instruments with participant burden, this mode of administration still classified according to their mode of administration: required some degree of literacy to match the sign language • Computer-assisted surveys using video clips are called video to the written in order to submit the responses. The written source questionnaire was laid out in a clear manner, containing computer-assisted personal interviews, or video-CAPI. This is not only the source language but also the main language of when a physically present interviewer administers the ques- interaction with the test platform. The sign language version tionnaire and enters the answers on a computer or similar contained an additional target language with a less prominent device. • A video computer-assisted self-administered interview, or or lower position, thus subordinating sign language to the official written language. video-CASI, is when the respondent enters the answers on Nowadays, new technologies have allowed sign language a computer provided by an interviewer who is physically interlocutors to interact virtually using their own language present. • Computer-assisted video interviewing, or CAVI, is when the (Lucas et al., 2013). Deaf community members have rapidly become extensive users of mobile and computer applications communication between the interviewer and the respondent that implement video technologies, such as video telephones, is established remotely via video chat. video chats, video calls, video blogs, video messages, or In addition there are other terms that have been used video sharing. At the same time, computerized, web-based in the literature to refer to video questionnaires specifically instruments containing sign language have started to become designed to include the use of sign languages as the language of more frequent. New technologies not only introduce the interview administration. These video questionnaires are listed possibility of embedding video clips to develop web-based here in chronological order: interactive video-questionnaire questionnaire application, but also existing online survey OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi Sign Language-Friendly Questionnaire Design 5 platforms (such as MonkeySurvey or Google forms), which require written language literacy. Multiclip VSL questionnaires introduce the possibility of inserting videos from video-sharing can implement monolingual questionnaires in the signed Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 platforms (such as YouTube) and upload user video files in modality. Thus, they can provide improved reliability and their features. These platforms and applications allow sign validity of the survey or research instrument for sign language language on-screen setups following different technical designs users, especially in the case of language assessment tests. that directly impact accessibility features, language status, Furthermore, multiclip formats are a way of implementing fully and usability. translated bilingual questionnaires, where sign languages can be laid out in a prominent on-screen position, removing the Accessible technical designs: question format and layout. Most hegemonic subordination to the written formats. studies including a computer-based VSL questionnaire do Similarly, developing multilingual cross-modal question- not provide detailed descriptions, if any at all, particularly naires allows one to implement visual communication systems in regard to technical design, delivery format, and usability toward universal design. They grant access to a greater number of the instrument. Some exceptions are Gerich and Lehner and a wider range of potential deaf respondents within the (2006), Haug et al. (2015), Lipton et al. (1996), Napier et al. diverse deaf population and their language choices in a number (2018), and Samar, Barnett, Oyzon, Mowl, and Sutter (2012). of ways (Graybill et al., 2010; Napier, et al. 2018; Tran, 2014). After conducting a survey among sign language test developers, Multiclip formats present not only advantages but also limita- Haug (2015, p.37) acknowledges that none of the studies that tions. Among the former they can reduce burden and nonre- had a computerized format addressed the usage of ICTs for sponse in questionnaires targeting deaf populations by offering sign language test development. According to the language choice of language and mode of questionnaire completion that of on-screen presentation, we distinguish two types of VSL might suit groups with differing communication skills. They may questionnaires: One-clip VSL questionnaires and multiclip help to solve the issue of underrepresentation of the signing Deaf VSL questionnaires community in general population surveys. Finally, they provide better representation of languages in the signed modality and One-clip VSL questionnaires. One-clip VSL questionnaires show a promote equality. Regarding limitations, they commonly display written online survey with one video screen embedded, showing only one item—one question and the set of responses—at a time. a video clip of the sign-translated version (see Figure 1). Video Thus, they cannot provide an equivalent format for standardized clips contain both questions/stimuli and answers in a single questionnaires that present a whole set of questions (Dillman video sequence. Answers can be identified by using numbers, & Christian, 2005, as cited in Gerich & Lehner, 2006, p.271). Be letters, and vertical or horizontal lists. In order to make the video that as it may, using computer-assisted VSL questionnaires has clips and question sequence more usable in the signed version, the potential to create better questionnaire adaptations and some authors use one clip per questionnaire item; only one item enhances accessibility for sign language users.1 is shown on screen at a given time (Barnett et al., 2017; Goldstein et al., 2010; Pyfers, 2017; Tran, 2014; cf. Huenerfauth & Kacorri, 2015, simultaneously display four questions and their response Implementing Accessibility and Usability items on one screen). The common layout shows the video at the top of the screen Features in Sign Language-Friendly with the text version at the bottom or on the left. Typically, Questionnaires responses have to be submitted using the written format. For Our multiclip video sign language questionnaires were first closed-ended questions the answer has to be selected from developed withn the HBB4ALL project funded by the European a set of predefined written responses, for example, yes/no or Commission (CIP-ICT-621014; 2013-2016) as a web-based data multiple choice. For open-ended questions, frequently, a text box collection tool for sign language interpretation pilot tests. The requires the participant to write the answer (see, e.g., Fontaine, tests aimed to collect data about perception and processing 2012; Goldstein et al., 2010; Huenerfauth & Kacorri, 2015; Pyfers, of information of the contents, usability, and user preferences 2017). As with the noncomputerized video-taped questionnaire regarding size and position of the sign language interpreter on in Brauer (1993), this is an accessibility option that adds too much TV in four different screen configurations. The detailed design strain to the signing participant, as it only translates the linguis- and procedures, results, and findings of the experimental pilot tic content but not the format of the questionnaire. Additionally, tests are further presented in Bosch-Baliarda, Soler-Vilageliu, it requires literacy and has limited interaction features with the and Orero (2019). For the purposes of this paper we will first signed video version. This might affect usability in an attempt briefly describe the experimental procedure to provide a context to remove written language literacy requirements and enhance to the questionnaire and later focus on the description of the accessibility. design and development of the tool itself. Some studies have used color codes and symbols instead of written options to submit responses in multiple choice and Likert scale answers (Gerich & Lehner, 2006; Kipp, Nguyen, Heloir, Experimental Settings & Matthes, 2011; Napier et al., 2018; Pardo-Guijarro et al., 2015). Participants in the pilot test were 32 deaf users (16 men/16 Multiclip VSL questionnaires. The second type of VSL question- women) from the metropolitan area of Barcelona in Catalonia naire format is multiclip VSL questionnaire, which displays a (Spain). Their ages ranged from 17 to 76 years (mean, 40 years). combination of more than one set of videos simultaneously to They were recruited through the Catalan Federation of the present both the question and the selected answers in separate Deaf (Federació de Persones Sordes de Catalunya, FESOCA). All video sequences. Multiclip layout designs aim to reproduce stan- of them use Catalan Sign Language (LSC) to communicate in dard, computerized written instruments, both in content and in their everyday lives and have self-identified as being proficient layout to provide better adaptations. signers. One of the most prominent advantages over other designs The experimental task consisted of watching four clips is that they are fully accessible in sign language and do not extracted from the documentary “Joining the Dots” OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi 6 Journal of Deaf Studies and Deaf Education Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 Figure 1 One-clip layout for cross-modal bilingual VSL questionnaire. (Romero-Fresco, 2012). Each clip was displayed in a different Questionnaire Design screen configuration and an Eye tracker Tobii 60, controlled by a Toshiba Portable personal computer recording participant’s eye The design approach to develop the target questionnaires was movements. a mixed mode. The source questionnaires were first created Users were individually tested in different local Deaf asso- in written Spanish and were later translated into LSC for the ciation offices. After being welcomed, they had to fill out the target questionnaire. The initial approach in question design consent and image release forms and answer the demographics involved asking the same questions and translating, or the ASQT questionnaire. Following these preliminary steps, the experi- approach. The questionnaires developed for captioning user mental procedure began. Each participant watched one clip and tests were taken as the source question sets. Later some ques- answered three questionnaires immediately after a visual recall tions were adapted to the visual culture of the Deaf community test, a language recall test, and a user experience test to tackle in Catalonia and others had to be newly created to address usability and user preferences for each screen configuration. specific features of the sign language interpreter on screen We did not include any questions to gather data regarding the presentation (asking different questions, or the ADQ approach; usability of questionnaire itself. All tests were administered and Harkness, 2008; Survey Research Center, 2016). recorded on a MacBook Air personal computer. This process was The translation/adaptation was dealt with in a one-team repeated four times, one for every clip. framework by a small team comprised of three members: one OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi Sign Language-Friendly Questionnaire Design 7 Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 Figure 2 Multiclip layout monolingual VSL questionnaire. deaf native signer and two hearing nonnative signers. The deaf The team translated both the video clips for the stimulus and team member was a highly skilled sign language teacher for sign all the questionnaire items. The signing model for the trans- language interpreters and communication support workers. She lated documentary clips was down to the input of the pro- has a vast experience as a deaf consultant in research and in cre- fessional hearing interpreter, providing a parallel input to the ating and adapting educational materials into LSC. The second most frequent use of sign language on TV, whereas the sign- team member was a certified hearing sign language interpreter ing model for the questionnaire items was the deaf signer, in with more than 15 years of intensive working experience in order to provide a native input. Special care was taken when many areas and skilled areas, such as TV. The third member creating the linguistic recall questions to use the same lexi- of the translation team was the bilingual bimodal researcher cal items in both the interpreted clips and the questionnaire assistant, trained in deaf studies, sign language linguistics, and items, questions and responses. Thus, the linguistic input was sign language interpretation. controlled in both dialect and phonological variation of the The translation procedures included two translations from targeted signs. Failing to do so would make the evaluation of the hearing team members reviewed by the deaf consultant. the participants’ performance impossible to evaluate and would She fine-tuned the translations and also pointed out which have compromised the validity of the results. Finally, all ques- questions should be further adapted. The final translation was tionnaire items were video-recorded using a chroma key. A video thus collaborative and agreed on by the three team members. protocol was the selected format so that the questionnaire input, OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi 8 Journal of Deaf Studies and Deaf Education accuracy, and consistency could be guaranteed throughout the pilot test. Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 Technical design features implemented in HBB4ALL questionnaires. Even though the test was designed to be conducted with a small sample of 32 participants, the research team wanted to create a technical instrument design that allowed full accessibility of the questionnaire contents in sign language. The researchers wanted, on the one hand, to create a sign language-friendly data collection tool that could include video protocols to enhance validity and reliability of the instrument, especially for the sign language recall test. On the other, they wanted to create a deaf- friendly instrument that would enhance research engagement and usability of research tools by the targeted respondents, deaf sign language users. Additionally, we aimed to design a web- based on-screen layout that would attribute the same social and linguistic status to both language modalities, LSC and the writ- Figure 3 Screenshot of the multiclip layout design from the LSC HBB4ALL cross- ten languages of the territory, namely Spanish and Catalan. This modal bilingual questionnaire. methodology was innovative because it had never been used in the country before nor in the field of accessibility research. Because no existing online survey platforms allowed multi- clip layout customization for the different types of questions (multiple choice, yes–no, etc.), the technical team members developed a multiscreen, computer-assisted data collection program that could implement the desired features. Both local and online versions were tested in the different pilot tests. Even though developing tools that allow implementations of questionnaires fully accessible in sign language was more time- and cost-consuming, it was an effort that was not to be avoided to grant research standards. For a more detailed description of the technical platform implementation, question technical designs, and technical formats see López et al. (2019). Regarding language on-screen presentation, we wanted the research tool to be fully accessible in both signed and written modalities. The aim was to improve its usability within the diverse deaf signing communities, from skilled deaf signers regardless of their literacy levels to nonnative deaf signers that may rely on the written form or that prefer the written form to access certain questionnaire items, especially in the demo- graphic questions. The platform was designed so that it could include a computerized sign language version of the informed consent and image release forms, so that these crucial aspects of the research tools were also accessible to signers who preferred Figure 4 Screenshot of a multiple choice question design with four answers this modality. Both forms were played or read before conducting selected for submission in the Spanish Sign Language HBB4ALL cross-modal the test and answering the questionnaires. bilingual questionnaire. As in most of the previous multiclip VSL questionnaire designs, only one question was displayed on-screen at a time so that no scrolling was necessary. The multiclip question click on the selected response or a bullet in line with the layout was designed to make LSC prominent on the screen response item. In our questionnaire we wanted sign language (see Figure 3). The video clips for the questions were placed top to be the language of interaction too, so that both language center of the screen, and the different possible responses were modalities could play the same central role. To select the desired displayed horizontally underneath. Each answer video clip was answers, participants could click on the video clip. Once the identified with a thumbnail showing the fingerspelling hand answer was selected, a blue frame would appear and the next shapes for A, B, C, and D, respectively—or number hand shapes button would be enabled. See Figure 4. in the Spanish Sign Language test—to make them visually more Thus, participants’ responses would not rely on their literacy, distinct from the question video clips and to provide consistency and they could avoid going back and forth between the writ- throughout the different question designs. The written versions ten and signed versions of the questionnaire items in order to were displayed under each video clip. respond. This innovation was designed not only to improve the To give the same status to both language modalities in the language status but also to reduce cognitive burden, response technical design we introduced an innovation for submitting time, and overall time, which was one of our concerns regarding the answers. As mentioned in Implementing Accessibility and test duration. Usability Features in Sign Language-Friendly Questionnaires A reproduction feature to allow “skimming” and “scanning” section, most VSL questionnaires require participants to submit the videos was designed. The function was to make the user their responses using the written items, generally by a single experience more comparable within the two languages in the OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi Sign Language-Friendly Questionnaire Design 9 fully cross-modal bilingual questionnaire. By swiping on the submit their answers through the interactive signed version available answers the video clips were played automatically. of the responses. It has shown to be a valid research tool to Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 Both the question and answer clips could be replayed, paused, implement an interviewer-administered sign language recall forwarded, and rewound by clicking on the symbol buttons in a test that can match the modality representation of the stimuli, standardized video player bar. the questions, and the response items, and thus enhances the Other features were introduced to improve the experience reliability of the results. of older users, blind or partially sighted signers. For example, However, schedule and budget limitations together with the the speed rate and size of the video clips could be adjusted scope of the study impeded the implementation of more features according to the user needs in order to improve usability and that have been described for deaf-friendly surveying instru- accessibility. Each video clip item could be played full-screen ments, such as offering choice from different signing models if selected. using a wider range of sign language styles or other accessibility features in the written versions such as including Easy to Read. Similarly, other customization features such as offering Results background color choice were discussed but could not be imple- mented in this prototype version. Still, background color is a The participants acknowledged that it was the first time they feature that could both reduce eye fatigue and improve accessi- had used a questionnaire fully accessible in sign language and bility and readability among signers with different sight status. positively valued the language choice offered in the bilingual– As already mentioned, computerized, sign-friendly question- bimodal approach. They also valued the use of deaf signers. naires are the most affordable and reliable mode of adminis- Although we did not include any formal controlled questions tration that give access to sign language as a prerequisite to to gather feedback on the strategies used, some were discussed comply with ethical standards and achieve human rights for informally at the end of the test. All participants used both sign language communities in research. Nevertheless, this mode language modalities to some extent when completing the ques- of administration holds some disadvantages when compared tionnaire. Participants with higher literacy skills skipped most to traditional, written paper-and-pencil questionnaires. Namely, of the video items, especially in the demographic and user there are at least three major drawbacks: they are more expen- experience tests, because it was much faster than going through sive to produce, they are bound to technical malfunctioning, and the response items. However, they all watched the sign language sometimes there is increased time burden, as they take longer to version for the recall tests. Strategies regarding the language be conducted. chosen to access the questionnaire items differed greatly. Some Regarding the detrimental time factor, Graybill et al. (2010) participants first read the written items and used the signed acknowledge that access to answer choices in a sign language version to double-check comprehension, whereas some used survey is necessarily sequential, unlike what occurs in a the opposite strategy. For example, some users would watch the written survey. Participants in a written survey have a near- video first and then read the responses to “learn” the words for simultaneous access to all response items. They are able to scan some of the signs they did not know previously. However, the and skim answer choices and reread them repeatedly, in any signed version was mostly used for response submission. order, before making a final choice, whereas respondents to a The “skimming” feature was used both to previsualize the VSL questionnaire have to watch and consider each answer in possible responses and to double-check the response before turn before selecting a response or replay any of the answer submitting the final choice. Younger participants exploited this video clips. These authors admit that the modality differences feature more than older subjects, which helped them going greatly affect the experience of completing a survey, although much faster through the tests. Additionally, three older partic- VSL questionnaires would be more comparable to the experience ipants asked the interviewer to help them submit the responses, of a telephone survey. because they were not confident enough to interact with the When designing our interface, we discussed some features in computerized interface. the questionnaire design to reduce the overall interview time. On The other built-in accessibility features were scarcely used. the one hand, reducing time burden was one of the arguments None of the participants used the speed adjustment, and only for choosing a cross-modal bilingual design for all the ques- one used the full-screen video feature in two questionnaire tionnaire sets over a monolingual sign language design. This items. The size and color contrast of the filmed items was option clearly reduces time burden but introduces new sources considered to grant accessibility to the signed content. of uncontrolled bias among and within participant responses. The web-based questionnaire design allowed automatically Furthermore we wanted to implement some technical inno- saving the results on the server. However due to server connec- vation that would allow simultaneous access to all the video tivity issues 5 recall test responses and 11 individual response responses. A first attempt in the design displayed all the signed items were not saved automatically. response items playing simultaneously, nonstop along with the written items. However, this version produced too much noise and was not deployed. In this same direction, we finally imple- Discussion of the Current Approach mented the “skimming” function that had to be activated by the The present questionnaire design aims to encompass pre- respondent. This feature helped manage the sign-based version vious findings in the method designs developed for sign faster. language assessment tests and in deaf-friendly surveying instruments. The resulting interface is an appropriate tool that can successfully implement a cross-modal bilingual questionnaire design. It not only is fully accessible in both sign Conclusion language and in writing, but also provides a similar status of the This article revised existing literature toward the design of a two language modalities in both their on-screen representation new sign language questionnaire. From the revision, the first and in their functionality, such as allowing participants to recommendation is that data collection from end users should OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi 10 Journal of Deaf Studies and Deaf Education be performed in their preferred language and language modality. Cross-modal bilingual designs can thus provide better language This is the safest way to avoid any polluted resulting data due accessibility and language choice and avoid or bridge literacy Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 to misunderstanding, biased replies, and lack of participation. requirements. In fact, advances and innovations on computer- Moreover, dealing with deaf sign language users means com- ized VSL questionnaires are promising and can benefit not only bining the language and culture minority paradigm of the sign sign language users and researchers. Offering the possibility language communities. Including sign language in all steps of to choose and combine one screen several language variants, research to grant full accessibility is a requirement to comply language modalities or other adapted visual communication with the human rights of this minority group. In the human systems, is a benefit not only for the diverse deaf population but rights paradigm, authors generally agree that accessibility plays also for the diverse general population. a central role, either as a human right per se or as an instrument However, there is still much need for research, collaboration, for the fulfillment of human rights (Greco, 2016). and development. Haug (2015) acknowledged that exploring ICT Therefore, removing communication barriers and granting for sign language testing is a barely researched and indeed accessibility to question content in sign language is a prereq- almost neglected area. This is still applicable to surveying and uisite in order to conduct ethical research with signing deaf general questionnaire design in other field areas too. In our individuals. Yet it is not enough to meet research standards. pilot questionnaires we implemented skimming advance and Specifically in questionnaire-based research, such as surveys or other customizations such as size and speed of the video. These tests, it is equally important to develop robust instruments in are examples of better adaptations aiming to reduce both time terms of question design, format, and mode of administration and cognitive burden. However, more research is needed to to reduce affliction and participant burden and measure effects promote new advances related to video processing and usability due to the language visual modality. Although the production of for video questionnaires. These advances should be widely avail- video sequences is time-consuming and costly (Gerich & Lehner, able through cost-efficient technological platforms that should 2006), this is a prerequisite for result validity and reliability. be developed collaboratively in research teams involving deaf We have argued that multiclip VSL questionnaires are professionals and consultants at all stages. All advances toward currently the best mode of administration for sign-friendly mainstreaming sign language-friendly and deaf-friendly ques- video-CASI. These questionnaires can deliver adapted designs tionnaires will thus contribute to a more diverse and inclusive for both signed monolingual and cross-modal bilingual tools. society not only for deaf populations but also for all citizens. Whereas instruments fully implementing a sign language are necessary when carrying out sign language testing. Further adaptations may be necessary when the questionnaire is Funding designed for surveying a wider range of deaf populations with This work was partially funded by the European Commission different communication needs and language preferences. Deaf- through the H2020 research program funded projects [HBB4ALL: friendly research instruments are crucial to any survey research 621014, ImAC GA: 761974, EasyTV GA: 761999] the Catalan Gov- within the social sciences that target a more general population. ernment funding scheme 2017SGR113. Deaf-friendly questionnaires should capture the inherent heterogeneity within deaf populations regarding the diversity of communication and literacy skills along with language choice Conf lict of Interest and preference. Failing to do so would entail underrepresenta- tion of deaf people, both due to noncontact (such as in telephone No conflicts of interest were reported. surveying) and a higher rate of nonresponse. The literature review led to the finding that most VSL ques- tionnaires happen in very specific field areas, such as health Acknowledgments surveys or sign language testing. VSL questionnaires are not We gratefully acknowledge Vanessa Beroy, Noelia Hernández, mainstreamed across any democratic citizen participation por- and Sílvia Pujol for the translation and adaptation of the signed tal or any general topic. Moreover, differences are even greater versions of the questionnaire items; RTVE, the Spanish Radio when we have a look at the sign languages that have been imple- and Television Corporation, and WEBVISUAL TV for video pro- mented. The vast majority of computerized VSL questionnaires duction; Juan Pedro López-Velasco and Carlos Alberto Martín- have been developed in American Sign Language, and just a Edo for video postproduction and developing the questionnaire few are sparsely found in other national sign languages: de Wit technical implementation and delivery system; Idoia Vallverdú- (2011) for Dutch Sign Language, Fontaine (2012) for Belgium Sign Segura for the graphic design of Figures 2 and 3; FESOCA, the Language, Gerich and Lehner (2006) for Austrian Sign Language, National Association of the Deaf in Catalonia, and CNLSE, the Kipp et al. (2011) in German Sign Language, Pardo-Guijarro et Centre for Linguistic Normalization of Spanish Sign Language, al., 2015 for Spanish Sign Language, Napier et al. (2018) and for their collaboration and support. Pyfers (2017) in International Sign System, or our questionnaires developed in LSC and Spanish Sign Language. An encouraging exception is found in the field of sign language assessment tests, where the collaboration between researchers and a well- Notes 1 documented tradition of procedures in test adaptation have See Gerich and Lehner (2006) and Haug (2015) for a review of the favored the development of tools in multiple sign languages.2 advantages and disadvantages of video-supported CASI com- Cross-modal VSL questionnaires, displaying both a written pared to traditional noncomputerized data collection methods language and a signed language, enhance the linguistic and in the context of the research in sign languages. 2 social status of both languages represented, thus raising the Visit Sign Language Assessment Instruments hosted by Tobias status of the minority language. Additionally, they introduce Haug at http://www.signlang-assessment.info for a fully com- new possibilities for universal design questionnaires that can prehensive resource; see Lara-Escudero (2017) for an updated simultaneously include different forms of communication. short list of the languages available. OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi Sign Language-Friendly Questionnaire Design 11 References De Wit, M. (2011). A sign language interpreter in inclusive education: The view of deaf persons on their quality of life (Master’s thesis). Baker-Shenk, C., & Kyle, J. G. (1990). Research with deaf people: Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 Edinburgh, Scotland: Heriot-Watt University. Issues and conflicts. Disability, Handicap & Society, 5, 65–75. Dillman, D. A. & Christian, L. M. (2005). Survey mode as a source doi:10.1080/02674649066780051 of instability in responses across surveys. Field Methods 17, Baker, A., van den Bogaerde, B., & Woll, B. (2006). Methods and 30–51. doi:10.1177/1525822X04269550. procedures in sign language acquisition studies. Sign Lan- Enns, C. J., Haug, T., Herman, R., Hoffmeister, R., Mann, W., & guage & Linguistics, 8, 7–59. doi:10.1075/sll.8.1.03bak Barnett, S. L., Matthews, K. A., Sutter, E. J., DeWindt, L. A., Pransky, McQuarrie, L. (2016). Exploring signed language assessment J. A., O’Hearn, A. M., & Pearson, T. A. (2017). Collaboration tools around the world. In M. Marschark, V. Lampropoulou & with deaf communities to conduct accessible health E. K. Skordilis (Eds.), Diversity in deaf education (pp. 171–218). surveillance. American Journal of Preventive Medicine, 52, 250– New York, NY: Oxford University Press. S254. doi:10.1016/j.amepre.2016.10.011. Link to American Enns, C. J., & Herman, R. C. (2011). Adapting the assessing Sign Language video adaptation of this article retrieved British Sign Language development: Receptive skills test from https://www.urmc.rochester.edu/ncdhr/research/ into American Sign Language. Journal of Deaf Studies and Deaf current-research-2/deaf-health-survey-2013/ajpm-video. Education, 16, 362–374. doi:10.1093/deafed/enr004 aspx Erens, B. (2013). Designing high-quality surveys of ethnic Benedict, B. S., & Sass-Lehrer, M. (2007). Deaf hearing partner- minority groups in the United Kingdom. In J. Font & M. ships: Ethical and communication considerations. American Méndez (Eds.), Surveying ethnic minorities and immigrant Annals of the Deaf , 152, 275–282. doi:10.1353/aad.2007.0023 populations. Methodological challenges and researches strate- Berghs, M., Atkin, K., Graham, H., Hatton, C., & Thomas, C. (2016). gies (pp. 45–68). Amsterdam, the Netherlands: Amsterdam Implications for public health research of models and the- University Press. ories of disability: A scoping study and evidence synthesis. Ewart, J., & Snowden, C. (2012). The media’s role in social inclu- Public Health Research, 4, 1–166. doi:10.3310/phr04080 sion and exclusion. Media International Australia, 142, 61–63. Bernabé, R., & Orero, P. (2019 forthcoming). Easy to Read as doi:10.1177/1329878X1214200108 multimode accessibility service. Hermeneus, 21. Fontaine, S. (2012). Surveying populations with disabilities. Specific Bosch-Baliarda, M., Soler-Vilageliu, O., & Orero, P. (2019). mixed-mode methodologies to include sensory disabled people TV screen exploration in sign language users. Unpublished in quantitative surveys. Paper presented at the. American manuscript. Universitat Autònoma de Barcelona, Bellaterra, Statistical Association, Survey Methods for Hard to Reach Catalonia. Populations,. New Orleans, Louisiana, October–November: Boynton, P. M., & Greenhalgh, T. (2004). Hands-on guide to International Conference on Methods for Surveying and questionnaire research: Selecting, designing, and develop- Enumerating Hard-to-Reach Populations. ing your questionnaire. BMJ, 328, 1312–1315. doi:10.1136/ Freeman, S. T. (1989). Cultural and linguistic bias in mental bmj.328.7451.1312 health evaluations of deaf people. Rehabilitation Psychology, Brauer, B. A. (1993). Adequacy of a translation of the MMPI into 34, 51–63. doi:10.1037/h0091705 American Sign Language for use with deaf individuals: Gerich, J., & Lehner, R. (2006). Video computer-assisted self- Linguistic equivalency issues. Rehabilitation Psychology, 38, administered interviews for deaf respondents. Field Meth- 247–260. doi:10.1037/h0080302 ods, 18, 267–283. doi:10.1177/1525822X06287535 Cavender, A., Ladner, R. E., & Riskin, E. A. (2006). MobileASL: Intel- Goldstein, M. E., Eckhardt, E. A., & Joyner, P. (2004). The inclusion ligibility of sign language video as constrained by mobile of deaf individuals in survey research. Paper presented at the phone technology. In Proceedings of the 8th International ACM Washington, DC, April: Federal Interagency Committee on SIGACCESS Conference on Computers and Accessibility—Assets Disability Research Conference on Best Practices for Sur- ‘06 (Vol. 71). doi:10.1145/1168987.1169001. Portland, Oregon, veying People with Disabilities. Interagency Commitee on USA Disability Research Cawthon, S. W. (2017). Large-scale survey design in deaf educa- Goldstein, M. F., Eckhardt, E. A., Joyner-Creamer, P., Berry, R., Par- tion research. In S. W. Cawthon & C. Lou Garberoglio (Eds.), adise, H., & Cleland, C. M. (2010). What do deaf high school Research in deaf education: Contexts, challenges, and consider- students know about HIV? AIDS Education and Prevention, 22, ations (pp. 1–28). New York, NY: Oxford University Press. 523–537. doi:10.1521/aeap.2010.22.6.523 doi:10.1093/oso/9780190455651.001.0001. Graybill, P., Aggas, J., Dean, R. K., Demers, S., Finigan, E. Cawthon, S. W., & Garberoglio, C. L. (2017). Research in G., & Pollard, R. Q. (2010). A community-participatory deaf education: Contexts, challenges, and considerations. approach to adapting survey items for deaf individuals New York, NY: Oxford University Press. doi:10.1093/ and American Sign Language. Field Methods, 22, 429–448. oso/9780190455651.001.0001. doi:10.1177/1525822X10379201 Choi, B. C. K., & Pak, A. W. P. (2005). A catalog of biases Greco, G. M. (2016). On accessibility as a human right, with an in questionnaires. Preventing Chronic Disease, 2, A13. application to media accessibility. In A. Matamala & P. Orero doi:A13 (Eds.), Researching audio description (pp. 11–33). London, UK: Cohen, H., & Jones, E. G. (1990). Interpreting for cross-cultural Palgrave Macmillan. research: Changing written English to American Sign Lan- Guardino, C., & Cannon, J. E. (2016). Deafness and diversity: guage. Journal of the American Deafness and Rehabilitation Reflections and directions. American Annals of the Deaf , 161, Association, 24, 41–48. 104–112. doi:10.1353/aad.2016.0016 De Meulder, M., Krausneker, V., Turner, G. H., & Conama, Gutman, V. (2005). Ethical reasoning and mental health services J. B. (2018). Sign language communities. In G. Hogan- with deaf clients. Journal of Deaf Studies and Deaf Education, Brun & B. O’Rourke (Eds.), The handbook of minority lan- 10, 171–183. doi:10.1093/deafed/eni017 guages and communities (pp. 207–232). London, UK: Palgrave Hansen, E. G., Loew, R. C., Laitusis, C. C., Kushalnagar, P., Macmillan. Pagliaro, C. M., & Kurz, C. (2018). Usability of American Sign OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi 12 Journal of Deaf Studies and Deaf Education Language videos for presenting mathematics assessment Retrieved from https://digitalcommons.unf.edu/joi/vol24/ content. Journal of Deaf Studies and Deaf Education, 23, iss1/4 Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 284–294. doi:10.1093/deafed/eny008 Lara-Escudero, C. (2017). Adapting the British Sign Language Harkness, J. A. (2008). Comparative survey research: Goal and receptive skills test into Catalan Sign Language: Preliminary challenges. In E. D. de Leeuw, J. J. Hox & D. A. Dillman (Eds.), studies (Master’s thesis). Barcelona, Catalonia: University of International handbook of survey methodology (pp. 56–77). Barcelona. London, UK: Routledge. Lipton, D. S., Goldstein, M. F., Fahnbulleh, F. W., & Gertz, Harris, R., Holmes, H. M., & Mertens, D. M. (2009). Research E. N. (1996). The interactive video-questionnaire: A new ethics in sign language communities. Sign Language Studies, technology for interviewing deaf persons. American Annals 9, 104–131. doi:10.1353/sls.0.0011 of the Deaf , 141, 370–378. Haug, T. (2011). Adaptation and evaluation of a German Sign Language López, J. P., Bosch-Baliarda, M., Martín, C. A., Menéndez, J. M., test. A computer-based receptive skills test for deaf children Orero, P., Soler-Vilageliu, O., & Álvarez, F. (2019). Design and ages 4–8 years old. Hamburg, Germany: Hamburg University development of sign language questionnaires based on video and Press. web interfaces. Manuscript submitted for publication. Haug, T. (2015). Use of information and communication Lucas, C., Mirus, G., Palmer, J. L., Roessler, N. J., & Frost, A. (2013). technologies in sign language test development: Results of The effect of new technologies on sign language research. an international survey. Deafness and Education International, Sign Language Studies, 13, 541–564. doi:10.1353/sls.2013.0018 17, 33–48. doi:10.1179/1557069X14Y.0000000041 Maiorana-Basas, M., & Pagliaro, C. M. (2014). Technology use Haug, T., Herman, R., & Woll, B. (2015). Constructing an online among adults who are deaf and hard of hearing: A national test framework, using the example of a sign language recep- survey. Journal of Deaf Studies and Deaf Education, 19, 400–410. tive skills test. Deafness and Education International, 17, 3–7. doi:10.1093/deafed/enu005 doi:10.1179/1557069X14Y.0000000035 Mann, W., Roy, P., & Morgan, G. (2016). Adaptation of a vocabulary Haug, T., & Mann, W. (2008). Adapting tests of sign language test from British Sign Language to American Sign Language. assessment for other sign languages-a review of linguistic, Language Testing, 33, 3–22. doi:10.1177/0265532215575627 cultural, and psychometric problems. Journal of Deaf Studies McKee, M., Schlehofer, D., & Thew, D. (2013). Ethical issues in con- and Deaf Education, 13, 138–147. doi:10.1093/deafed/enm027 ducting research with deaf populations. American Journal of Hauser, P. C., Paludneviciene, R., Riddle, W., Kurz, K. B., Emmorey, Public Health, 103, 2174–2178. doi:10.2105/AJPH.2013.301343 K., & Contreras, J. (2016). American Sign Language compre- Mertens, D. M. (2005). Research and evaluation in education and hension test: A tool for sign language researchers. Journal psychology: Integrating diversity with quantitative, qualitative, of Deaf Studies and Deaf Education, 21, 64–69. doi:10.1093/ and mixed methods (2nd Ed., ). Thousand Oaks, CA: Sage. deafed/env051 Mertens, D. M. (2010). Transformative mixed methods Herman, R. C., Holmes, S., & Woll, B. (1999). Assessing BSL research. Qualitative Inquiry, 16, 469–474. doi: development—Receptive skills test. Coleford, UK: The Forest 10.1177/1077800410364612 Bookshop. Mertens, D. M., Sullivan, M., & Stace, H. (2011). Disability commu- Hermans, D., Knoors, H., & Verhoeven, L. (2010). Assessment of nities: Transformative research and social justice. In N. K. sign language development: The case of deaf children in Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research the Netherlands. Journal of Deaf Studies and Deaf Education, (4th Ed., ). Thousand Oaks, CA: Sage. 15, 107–119. doi:10.1093/deafed/enp030 Meurant, L., Sinte, A., Vermeerbergen, M., & Van Herreweghe, Higgins, J., Famularo, L., Bownman, T., & Hall, R. (2015). M. (2013). Sign language research, uses and practices: Research and development of audio and American Sign A Belgian perspective. In L. Meurant, A. Sinte, M. Van Language guidelines for creating accessible computer-based Herreweghe & M. Vermeerbergen (Eds.), Sign language assessments. Retrieved from http://gaap.measuredprogress. research, uses and practices: Crossing views on theoretical and org/gaap/. applied sign language linguistics (pp. 1–14). Berlin, Germany: Huenerfauth, M., & Kacorri, H. (2015). Best practices for con- De Gruyter Mouton. ducting evaluations of sign language animation. Journal on Munger, K. M., & Mertens, D. M. (2011). Conducting research with Technology and Persons with Disabilities, 3, 20–32. Retrieved the disability community: A rights-based approach. In T. from http://scholarworks.rit.edu/article/1787 S. Rocco (Ed.), Challenging ableism, understanding disability, Hussein, K. Q., & Al-Bayati, M. A. (2009). A multiple kinds of e- including adults with disabilities in workplaces and learning exam system for the deaf & dumb: Developing & evaluating spaces: New directions for adult and continuing education (pp. “view points of experts in review”. International Journal of 23–33). New York, NY: Wiley Periodicals. doi:10.1002/ace. Computer Science and Network Security, 9, 309–317. Retrieved Napier, J., Lloyd, K., Skinner, R., Turner, G. H., & Wheatley, M. from http://paper.ijcsns.org/07_book/200905/20090541.pdf (2018). Using video technology to engage deaf sign lan- Kappelhof, J. (2015). Surveying ethnic minorities: The impact of guage users in survey research: An example from the Insign survey design on data quality. Den Haag, the Netherlands: project. The International Journal for Translation & Interpreting SCP Publications. Retrieved from http://www.scp.nl/ Research, 10, 101–121. doi:10.12807/ti.110202.2018.a08 Publicaties/Alle_publicaties/Publicaties_2015/Surveying_ Orero, P. (2017). The professional profile of the expert in media ethnic_minorities. accessibility for the scenic arts. Rivista Internazionale di Tec- Kipp, M., Nguyen, Q., Heloir, A., & Matthes, S. (2011). Assessing nica della Traduzione/International Journal of Translation, 19, the deaf user perspective on sign language avatars. Assets 143–161. doi:10.13137/2421-6763/17356 ‘11 (Xiii), 107–114. doi:10.1145/2049536.2049557 Orfanidou, E., Woll, B., & Morgan, G. (Eds.) (2015). Research methods Ladd, P. (2003). Understanding deaf culture. Clevedon, UK: Multilin- in sign language studies: A practical guide. London, UK: Wiley- gual Matters. Blackwell. Lang, C. (2015). Language use at RID conferences: A survey Pardo-Guijarro, M. J., Martinez-Andres, M., Notario-Pacheco, B., on behaviors and perceptions. Journal of Interpretation, 24. Solera-Martinez, M., Sanchez-Lopez, M., & Martinez- OUP UNCORRECTED PROOF – FIRST PROOF, 19/6/2019, SPi Sign Language-Friendly Questionnaire Design 13 Vizcaino, V. (2015). Self-reports versus parental perceptions languages. In E. Orfanidou, B. Woll & G. Morgan (Eds.), of health-related quality of life among deaf children and Research methods in sign language studies: A practical guide First Downloaded from https://academic.oup.com/jdsde/advance-article-abstract/doi/10.1093/deafed/enz021/5523870 by UNIVERSITAT AUTONOMA DE BARCELONA user on 15 August 2019 adolescents. Journal of Deaf Studies and Deaf Education, 20, Edition (pp. 7–20). London, UK: Wiley-Blackwell. 275–282. doi:10.1093/deafed/env018 Sloan, M., Wright, D., & Barrett, K. (2012). Data comparabil- Pollard, R. (2002). Ethics in mental health and deafness. In ity in a mixed mode telephone and face to face survey V. Gutman (Ed.), Ethical conduct in research involving deaf of persons with disabilities. Mathematica Policy Research people (pp. 162–178). Washington, DC: Gallaudet University Reports 51e4878b2264c7793ccb74dd, Mathematica Policy Press. Research. Retrieved from: https://ideas.repec.org/p/mpr/ Pollard, R. Q., Dean, R. K., O’Hearn, A. M., & Haynes, S. L. (2009). mprres/51e48478b2264c7793ccb74ddc73f3f3.html Adapting health education material for deaf audiences. Survey Research Center (2016). Guidelines for best practice in cross- Rehabilitation Psychology, 54, 232–238. Retrieved from https:// cultural surveys. (S. R. C. Institute for Social Research, Ed.) (4th www.northeastern.edu/cali/wp-content/uploads/2017/03/ Ed., ). Ann Arbor, MI: University of Michigan. Retrieved from Adapting-health-education-material-for-deaf-audiences. http://www.ccsg.isr.umich.edu/. pdf Tran, J. J. (2014). Human-centered optimization of mobile sign Pyfers, L. (Ed.) (2017). SignTeach: The survey. In Sign language language video communication (Doctoral dissertation). teaching in Europe. Report & recommendations. Brussels, Bel- Washington, DC: University of Washington. gium: European Union of the Deaf. Retrieved from https:// Tran, J. J., Kim, J., Riskin, A., Ladner, R. E., & Jacob, O. W. www.signteach.eu/index.php/4-signteach-the-survey. (2011). Evaluating quality and comprehension of real-time Rojba, F. (2016). Fieldwork conducted by deaf sign language users. sign language video on mobile phones. In ACM SIGACCESS In I. Lathinen & P. Rainò (Eds.), Deaf people in Albania in Conference on Computers and Accessibility (pp. 115–122). 2015. A survey study (pp. 16–20). Tirana, Albania: Finnish New York, NY: ACM. Association of the Deaf. Tran, J. J., Riskin, E. A., Ladner, R. E., & Wobbrock, J. O. (2015). Romero-Fresco, P. (2012). Joining the dots. The Journal of Evaluating intelligibility and battery drain of mobile sign Specialised Translation, 20. Retrieved from http://www. language video transmitted at low frame rates and bit jostrans.org/issue20/int_romero.php rates. ACM Transactions on Accessible Computing, 7, 1–26. Samar, V. J., Barnett, S., Oyzon, E., Mowl, C., & Sutter, E. (2012). doi:10.1145/2797142 Modality-independent survey tool: Imagine the potential. Woodward, J. C. (1973). Interrule implication in American Sign NTID Research Bulletin, 15, 1–5. Language. Sign Language Studies, 3, 47–56. doi:10.1353/ Sign Language Linguistics Society (2016). SLLS Ethics Statement for sls.1973.0010 Sign Language Research. Retrieved from https://slls.eu/slls- Woodward, J. C., & DeSantis, S. (1977). Two to one it happens: ethics-statement/. Dynamic phonology in two sign languages. Sign Language Singleton, J., Jones, G., & Hanumantha, S. (2012). Deaf friendly Studies, 17, 329–346. doi:10.1353/sls.1977.0013 research? Toward ethical practice in research involving Young, A., & Hunt, R. (2011). Research with d/Deaf sign language deaf participants. Deaf Studies Digital Journal, 3. Retrieved users. London, UK: NIHR School for Social Care Research. from http://dsdj.gallaudet.edu/index.php?view=entry& Retrieved from www.lse.ac.uk/LSEHealthAndSocialCare/ issue=4&entry_id=123 pdf/SSCR%20Methods%20Review_9_web.pdf. Singleton, J. L., Martin, A., & Morgan, G. (2015). Ethics, deaf- Young, A., & Temple, B. (2014). Approaches to social research: The friendly research, and good practice when studying sign case of deaf studies. New York, NY: Oxford University Press.