Integrated research model for web interfaces: toward a taxonomy of quantitative and qualitative analysis for digital native media Santiago Tejedor; Luis M. Romero-Rodríguez; Lluís Codina Nota: Este artículo se puede leer en español en: https://revista.profesionaldelainformacion.com/index.php/EPI/article/view/87110 Recommended citation: Tejedor, Santiago; Romero-Rodríguez, Luis M.; Codina, Lluís (2022). “Integrated research model for web interfaces: toward a taxonomy of quantitative and qualitative analysis for digital native media”. Profesional de la información, v. 31, n. 6, e310619. https://doi.org/10.3145/epi.2022.nov.19 Article received on September 8th 2022 Approved on October 14th 2022 Santiago Tejedor * Luis M. Romero-Rodríguez https://orcid.org/0000-0002-5539-9800 https://orcid.org/0000-0003-3924-1517 Universitat Autònoma de Barcelona Universidad Espíritu Santo Departamento de Periodismo y Ciencias ESAI Business School, Ecuador de la Comunicación Universidad Rey Juan Carlos Campus UAB, Edifici I Camino del Molino, 5 08193 Bellaterra (Barcelona), Spain 28943 Fuenlabrada (Madrid), Spain santiago.tejedor@uab.cat luis.romero@urjc.es Lluís Codina https://orcid.org/0000-0001-7020-1631 Universitat Pompeu Fabra Departament de Comunicació Roc Boronat, 138. 08018 Barcelona, España UPF Barcelona School of Management Balmes, 134. 08008 Barcelona, Spain lluis.codina@upf.edu Abstract In a context marked by the emergence of new platforms and interfaces, digital native media face the challenge of or- ganizing and presenting their content in an effective and eloquent way. This milestone requires ongoing work toward the identification and conceptualization of the dimensions that map the structural and formal requirements necessary for fluid communication with audiences and adequate visibility in the digital ecosystem. This study, which adopts a quantitative–qualitative methodological design, proposes, describes, and applies a taxonomy for the analysis of digital media web interfaces. Based on a review of the literature, two rounds of expert judgment, and a pilot test, the research identifies 14 thematic dimensions, 36 indicators, and 69 variables. The study focuses on the importance of renewing research about taxonomy related to structural dimensions that address narrative, quality, usability, information search, and content redissemination and socialization, among other aspects. The study of web interfaces applied to journalism demands long-term research work with a holistic approach that, at a conceptual and practical level, can offer proposals for analysis and diagnosis on the basis of an integrated method. Keywords Internet; Web; Journalism; Digital journalism; Content; Communication; Interfaces; Digital; Slow journalism; Benchmar- king; Indicators; Models; Taxonomies; Quantitative analysis; Qualitative analysis; Digital native media; Cybermedia; Na- rrative; Quality; Usability; Search. e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 1 Santiago Tejedor; Luis M. Romero-Rodríguez; Lluís Codina Funding This work is part of the project “Parameters and strategies to increase the relevance of media and digital commu- nication in society: curation, visualisation and visibility (Cuvicom),” PID2021-123579OB-I00, Spanish Ministry of Science and Innovation (Micinn), Spain. Knowledgements The authors thank the following researchers for their participation in the panel of experts and for their suggestions, comments and tips: Xosé López-García, Manuel Martínez-Nicolás, Diana Rivera-Rogel, Jesús-Miguel Flores-Vivar, Pavel Sidorenko-Bautista, Ainara Larrondo-Ureta, Carlos Arcila-Calderón, Francisco Segado-Boj, Jorge Vázquez-He- rrero, José Soengas-Pérez, Daniel Barredo-Ibáñez, and Belén Puebla-Martínez. 1. Introduction The global communication ecosystem is giving increasing importance to digital native media. The migration of legacy media to the digital scene has given way to an explosion of new journalistic projects and developments conceived “from” and “for” cyberspace (Salaverría, 2020a). In this scenario, the study of the properties of their websites, concei- ved as the nerve centers of their communication strategy, constitutes an important area of study and research. Although it has not been part of the mainstream of studies in digital journalism, understanding journalism from each era requires knowledge of its foundation, that is, the scenario in which it occurs. Thus, at various points, understanding journalism implied understanding the properties of printed publications (the written press), the spoken word (radio), or the moving image (television). Currently, the study of the digital platform (the media’s website) thus becomes essential to understand the media, especially digital native media, and their dynamics (Salaverría, 2020b). Although it lacks a dominant role at the research level, previous work in the academic literature has dealt with this subject. The study of “what are” [“qué son”] and “what are not” [“qué no son”] new media (Cabrera-Méndez; Codina; Sala- verría, 2019) has become a recurrent line of research during the last decade (López-García et al., 2022; Salaverría; Martínez-Costa, 2021; García-Orosa; López-García; Vázquez-Herrero, 2020). The classic studies of the particularities of multimedia, hypermedia with a special emphasis on links (Landow, 1995; 1997), or information architectures (Wur- man, 1997) have given way in recent years to various studies about organic positioning in search engines, innovation in formats and narratives, business models, Big Data, or trends such as slow journalism (Romero-Rodríguez, Tejedor; Castillo-Abdul, 2021; Le-Masurier, 2015; 2016; Zelizer, 2009). The study of links and thereby hypertextuality forms part of the class of works that have the digital as a focus of study and, therefore, as one of the genuine characteristics of online journalism (Stroobant, 2019). In that paper, Stroobant looks at hyperlinks as navigational cues that can guide users through the information available in the digital ecosystem. Specifically, Stroobant’s study focuses on health journalism. This aspect is important because online media are frequent- ly used and represent convenient sources of information on aspects that Google (2022, p. 11) calls “your money or your life.” This study conducts an environmental analysis of search engine results pages for the term “health news.” One of the findings highlights that native news websites have a tendency to provide more hyperlinks than legacy news websites do. A second result refers to the use of corporate hyperlinks, that is, links to sites or pages of the company or business group themselves, a trend that is more pronounced in legacy media. Although the study focuses on health news, the results about hyperlinking strategy remain significant in general. The functional use of hyperlinks to external sources was also studied by Humprecht and Esser (2018). In this case, the authors categorize hyperlinks around three main functions they identify: increasing transparency, increasing understan- ding, and increasing the capacity for deliberation. This typology of links and functions allows for the design of digital media analysis based on the important functions in each case. The study of the hypermedia component, which connects with the media ownership, has required studies on new phenomena such as the use of clickbait, among other subjects (Bravo-Araujo; Serrano-Puche; Novoa-Jaso, 2021; García-Serrano; Romero-Rodríguez; Hernando-Gómez, 2019). The relationship between website design elements and related usability elements has also been studied (Kim; Jeon; Choi, 2020). In this research, they analyze seven design elements that impact the usability of news media websites: advertisements, font, layout, volume of information, use of categories, use of color, and the way in which titles are presented, combining text and images. Although the study focuses on Korean media, the conclusions can probably be extrapolated, as their research concludes, among other things, that users prefer consistent layout designs throughout the site and at different levels, that titles are displayed next to a photo, that the depth of the category menu structure has a maximum of two levels, and that the links follow the usual blue color convention. In a similar vein, Al-Said and Al-Said (2022) study the influence of news site design on information perception and recall. They conclude that interactive sites that combine different resources –images, video, and typography– to present news improve both the reception and recall of information. A study by García-Perdomo and Magaña (2020) in which they interview managers of digital native media is focused on their perception of the drivers of their activity. The results rate the quality of the content and journalistic standards e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 2 Integrated research model for web interfaces: toward a taxonomy of quantitative and qualitative analysis for digital native media highest, higher than the technology and the digital plat- Our study proposes a quantitative-qua- forms themselves, which shows that, for some of these litative design of descriptive scope with companies, innovation is not limited to technological aspects but rather to the ability to offer journalistic pro- the aim of building, validating and tes- ducts and integrate users, although this last point is still ting a taxonomy for the analysis of digi- considered very limited. tal media web interfaces The role of SEO in newsrooms is also being progressively studied, as in the work of Prawira and Rizkiansyah (2018), who applied ethnographic methods and observed the role of organic SEO in the newsrooms of three Indonesian media outlets, which allowed them to see how the respective business cultures influence the adoption of SEO as part of jour- nalistic routines. In a similar vein, Lopezosa et al. (2020a; 2021) carried out research on newsrooms and concluded that, regarding the adoption of SEO as part of the strategies of the media, direct support from management is essential, as is the establi- shment of two tiers corresponding to the actions of the editors and those of the person in charge of the global SEO strategy. Likewise, Lopezosa et al. (2020b) propose an integrated system of indicators to analyze and measure the SEO of media that for use by scholars or practitioners. In turn, Seyser and Zeiller (2018) propose the portmanteau “scrollytelling” to designate certain longform articles in journalism. Specifically, scrollytelling, as these authors define it, is a format that uses extensive narrative texts and multimedia content, and they draw a conclusion about the essential role of infographics in these kinds of what they call complex stories. In addition, data journalism is often considered one of the best resources in digital journalism because of its well-known ability to help readers understand complex phenomena. Plazas (2019), after analyzing a sample of the best cases of data journalism from the Malofiej Awards, categorizes up to six modalities of vision, which he calls the visualization of events, of hidden problematics, of places, of narratives, of subjectivities, and of convergences. In the search for innovation, newspaper companies have opted to merge newsrooms and have set up labs for new for- mats, such as virtual reality. Vázquez-Herrero, Negreira-Rey, and López-García (2019) propose that innovation in digital media can be studied through four dimensions: the use of multimedia narratives, data journalism, virtual reality or 360° video, and fact-checking. They add a fifth dimension that involves identifying the media that have been equipped with a lab. They conclude that the models are unstable and that the search for a model that is sustainable, through a com- bination of technological and human resources, can provide everything necessary to ensure their consolidation in the digital environment. In addition, several authors, such as Pérez-Seijo (2021), have studied the use of immersive video in digital native media. The so-called i-docs, web docs, or interactive documentaries are undoubtedly a genuinely digital form of innovation in journalism. Although i-docs combine contributions from film documentaries and video games, they are an area in which the journalistic approach of taking a critical look at current affairs is fundamental. In this sense, the work of Dowling (2022) consists of the analysis of how the i-doc in US productions has evolved since 2015 to focus, above all, on the prin- ciples of transparency, social responsibility, and commitment to truthfulness, thus configuring i-docs as an empathetic alternative to traditional news coverage work. In a similar vein but focusing in this case on interactive multimedia productions, Van-der-Nat, Müller, and Bakker (2021) examine what they call the “architecture of interactivity”, focusing specifically on the analysis of the navigation systems of some specific cases of such productions. In their study, they identify structures with varying degrees of sequential and open trajectories, with the latter providing the most agency to users. An original approach to the treatment of digital news comes from the study by Kulkarni et al. (2022). In their work, these authors explore structures other than the inverted pyramid to analyze users’ reception of these structures. They conclude that audiences consider structures that are more linear and narrative, rather than the inverted pyramid, to be more attractive, as they can better convey content. The case study is applied again in the work of Peña-Ascacíbar and Álvarez-Peralta (2021). Specifically, they analyze com- mon characteristics (such as societal structure) and differential characteristics (paywalls and ethical codes), as well as the trajectory of the Spanish digital native media outlets El confidencial, ElDiario.es, and infoLibre. In their study, the authors also evaluate the main digital network metrics, although they conclude that the latter data are not a good predictor of membership as identified by the number of followers. Regarding recent protocols applied to the analysis of specific aspects of digital journalism, Santos-Hermosa, Lopezosa, and Codina (2022) analyze interactivity in the digital press, focusing their study specifically on parameters and indicators related to user experience and user interaction options related to the characteristic dimensions of digital media such as the media-user relationship, user-generated content, the user-user relationship, and content personalization options. In the work of Tejedor et al. (2020b), the business models of various Ibero-American digital native media are analyzed. The findings include a trend toward business models based on diverse and “hyper-specialized” content as well as those targe- e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 3 Santiago Tejedor; Luis M. Romero-Rodríguez; Lluís Codina ting “micro-audiences.” In addition, the research identifies a clear interest in what the authors call horizontality, participa- tion, and user engagement. The study emphasizes the need for native media to diversify their sources of income. Romero-Rodríguez, De-Casas-Moreno, and Torres-Toukoumidis (2016), in turn, ask how to evaluate and assess the informative quality of digital media. Specifically, they look for convergences between academics and practitioners, with the aim of identifying “areas and dimensions of digital media news quality in terms of objective or quantifiable values” [“áreas y dimensiones de la calidad informativa de los medios digitales en función de valores objetivos o cuantificables”] (pp. 1-2). Among other results, they present a “taxonomy of dimensions of information quality” [“taxonomía de dimen- siones de la calidad informativa”] (pp. 3-4), obtained both from a review of the scientific literature and from a three-way Delphi analysis. The research ends by proposing a structured model to analyze the information quality of digital media. Meanwhile, Romero-Rodríguez and Aguaded (2017) propose a model for analyzing the quality of media information by using structured categories and dimensions to evaluate the quality of information, also taking into account aspects of the “macro-environment of the journalistic environment” [“macroentorno del entorno periodístico”] (p. 12). The re- search concludes that political polarization and the discourse of social confrontation have a direct impact on the quality of news products. It should be noted that this research focuses on print media, but the scope of its proposals can be applied to journalism in general with few changes, which is why it is included in this literature review. The digital context should not lead us to forget that the essence of journalism, in any format, is investigative journalism, which can be approached through various formats, with the case of slow journalism being very significant. Against this background, Romero-Rodríguez, Tejedor and Castillo-Abdul (2021) conducted a case study in which 12 examples of slow journalism media in Ibero-America were reviewed. The research focuses on the thematic axes, content, esthetic elements, and funding sources, concluding that their content can be considered “elitist,” at least in comparison with conventional digital media, which reduces their ability to attract both subscriptions and advertising and increases their dependence on international cooperation. Another way of analyzing the quality of digital native media is proposed by Rodríguez-Hidalgo, Rivera-Rogel and Ro- mero-Rodríguez (2020). In their work, they use a taxonomy of evaluation dimensions of news quality that include news sources, uses of international agencies, and content review. In addition, they consider ideological plurality in their opi- nion and other pieces. The aspect of media visibility, although it tends to attract less academic interest than professional interest, also has no- table research. In the case of Lopezosa et al. (2021), the authors present a system of analysis for media visibility through the use of search engine positioning tools from a set of research studies. The authors propose a system called AVCIS (Análisis de visibilidad de cibermedios con indicadores SEO), which integrates a set of metrics for analyzing the visibility of digital media into a single protocol. On a more abstract level, researchers in the field of the study of digital resources proposed a system of analysis (Codina; Pedraza-Jiménez; Guallar, 2016) centered on a group of operational concepts presented through terminological units and integrated through a group of articulated elements that determine “the what” (parameters) and “the how” (indi- cators). This proposal, although based on developing analysis protocols for digital media that do not necessarily have to be media outlets, is a meta-protocol since its objective is to help develop particular systems of analysis for each case. Previously, part of the same group (Codina et al., 2014) proposed the SAAC (Sistema articulado de análisis de ciberme- dios) protocol that was more focused on digital media analysis, of which the SAAMD (Sistema articulado de análisis de medios digitales) was an extension. These proposals have been applied in subsequent studies focused on a wide range of topics, such as the work of Tejedor et al. (2020a) on the websites of the most important hospitals in Latin America. Although not forming a dominant research line, a wealth of studies have adopted mainly theoretical and operational approaches in the form of different sets of metrics or indicators to analyze various dimensions of digital media, in some cases, only digital natives, and in others, without making this distinction. This panorama indicates the need to address a comprehensive treatment of how the quality of digital media can be measured or evaluated, particularly in relation to the platform. In the past, both the professional and academic sectors had shared constructs and evaluation criteria for assessing the quality of print media. On the basis of these examples, a systematic study of the literature, and consultation with experts, we propose herein a model for the integrated study of web interfaces using a mixed methodology to enable a diagnostic, analytical, and descriptive evaluation of digital media content and platforms. 2. Materials and methods The study proposes a quantitative–qualitative design of descriptive scope with the objective of constructing, validating, and testing a taxonomy of analysis of digital media web interfaces. To this end, first, an exploration of the literature on methods and models for the evaluation of media and channel was carried out, providing raw material for an initial theo- retical matrix of fields, dimensions, and integrated indicators. This first stage made it possible to draw up an initial matrix of elements related to quality, usability, and the search for information, combining a total of 14 thematic dimensions, 36 indicators, and 69 variables for consideration in the construction of the proposal (Table 1). e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 4 Integrated research model for web interfaces: toward a taxonomy of quantitative and qualitative analysis for digital native media Table 1. Matrix of fields, dimensions, indicators, and variables derived from the literature review Field Dimension Indicator Variable Immersive/transmedia Conventional/pyramidal Narrative model Mixed Other Form of content and narrative Conventional videos and photographs Augmented videos and images, 360°, virtual reality Multimedia and hypermedia Hyperlinks and interactive architecture (i-docs, animations, gamifica- tion, newsgames) Other Conventional genres News item, interview, report, chronicle, review Specialized genres Literary, cultural, travel, technology Investigative genres and slow journalism Major stories, documentary, exposé Journalistic Infographics, illustrations, and data visualization (DatViz) genres and styles Data journalism Open statistical data (dataset and interoperability) Complete statistical information (sample, dates, error) Articles, columns, editorials, mastheads, political cartoons, blogs, and Opinion genres vlogs. News headlines Quality Headlines Clickbait Top title and summary Date and time of publication Name and surname(s) of author(s) Authors’ bio and email or link to their social network Metadata of the Identification of institutional or corporate authorship Metadata news stories Name and surname in credits of photo, illustration, or infographics Name and surname in credits of photo, illustration, or infographics Statistics (metrics) of the news story (unique accesses, readings, average reading time) Fact-checking systems and procedures Transparency, Clear societal structure truthfulness, Ethical codes and validity of Information and legal form Transparency and readers’ ombudsman content Sources of financing and transparency portal (subsidies, advertising, subscriptions, crowdfunding, cooperation agencies, sale of copies, etc.). Ombudsman, letters to the editor Link to content related to the informa- Related content tion inside or outside the media Search engines and collection, filtering, Content curation and selection systems Layout of advertising spaces that do not disrupt the reading or com- Ads/advertising (layout) prehension of the news stories or the home page Noninvasive reading Layout of legal notices that do not disrupt the reading or comprehen- Cookie information, subscriptions, and sion of the units terms and conditions Information or home page Blog layout Interface layout Content layout Layout of printed newspaper Transmedia integrations and interactive blocks Font (stylography) Legibility of Volume of information content Use of subheadings and dividers Usability Readers’ comments section in the feed- back for each news story Readers’ votes (like/dislike) section for Readers–me- each news story dia–authors interaction Twitter, Facebook, Instagram, LinkedIn, WhatsApp, Telegram, email, Content sharing buttons copy link Possibility of communication between Authors’ email and/or Twitter accounts readers and authors Ability to request a newsletter Content customi- Content syndication via mobile, app, or web zation Content recommendations based on users’ browsing cookies e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 5 Santiago Tejedor; Luis M. Romero-Rodríguez; Lluís Codina Field Dimension Indicator Variable Simple search, using Google or Bing Web browser Search engine Simple search, using the media outlet’s website information Search for Advanced search (filters, categories, tags) SEO metrics (domain authority [DA] and SEO and visibility page authority [PA] in Moz), metadata, taxonomies, visibility index (Sistrix) Subsequently, two rounds of expert judgment were conducted, the first for construction and content validity and the second for the assessment of the weights of each dimension in an effort to fix the taxonomy. To validate the applicability of the model, a pilot test was applied to 10 Ibero-American digital media outlets, selected for their geographic diversity as well as position in international rankings and directories. The sample, designed to be a geographically representative portion of the Ibero-American digital media community, was selected to test the analysis approach presented in Table 2. The corpus for analysis was selected by intentional or nonprobabilistic convenience sampling since a heterogeneous sample or a sample with maximum variation is required to test the usefulness of the model (applicability test) and de- termine its usefulness in different environments. In this regard, several inclusion criteria, such as geographic diversity, year of creation, and interface characteristics, were chosen. The unit of analysis has been Ibero-American digital media focused on the production of various types of journalistic con- tent that have appeared in the last 15 years, during which many new journalistic developments have occurred (Zulua- ga-Trujillo; Gómez-Montero, 2019). This study thus applied the constructed analysis matrix to 10 digital media outlets that appeared between 2009 and 2020. This deliberate or judgmental sampling enabled the test or pre-test to incorporate media from different countries, years of creation, and characteristics. For this purpose, we used the media directory made by the platform SembraMedia, which includes 1,104 Spanish-language digital media outlets in 24 countries. https://directorio.sembramedia.org This directory, created by a nonprofit organization, maps the digital media ecosystem in Ibero-America and has been cited in more than 150 articles and research papers. The test was carried out between June and August 2022. Table 2. Digital media analyzed in the pilot test Number Medio Country Creation URL 1 GK Ecuador 2011 http://gk.city 2 Eldiario.es Spain 2012 http://eldiario.es 3 El surtidor Paraguay 2016 https://elsurti.com 4 Efecto Cocuyo Venezuela 2015 http://efectococuyo.com 5 Acento Dominican Republic 2011 http://acento.com.do 6 Periodismo de barrio Cuba 2015 http://www.periodismodebarrio.org 7 Conexión migrante Mexico 2016 https://conexionmigrante.com 8 La silla vacía Colombia 2009 http://lasillavacia.com 9 La brava Bolivia 2020 http://revistalabrava.com 10 LadoB Mexico 2011 http://ladobe.com.mx The first tool was developed using theoretical constructs. That is, based on an exploratory review of the scientific litera- ture, we extracted those dimensions, indicators, and variables that were included in previously validated tools related to information quality, media transparency, and the analysis of web platforms and interfaces (Table 1). This tool went through a first round of expert judgment, understood as an intersubjective knowledge-building technique of people who, by their knowledge and expertise in a field, contribute by separately assessing and evaluating a proposal (O’Haggan, 2018). In this sense, expert judgment consists of “asking a number of people who judge an object, tool, or material, or give their opinion on a specific aspect” [“solicitar a una serie de personas que juzguen un objeto, instrumen- to o material, o que den su opinión sobre un aspecto concreto”] (Cabero-Almenara; Llorente-Cejudo, 2013, p. 14), being closely associated with the Delphi technique (López-Gómez, 2018; Mengual-Andrés; Roig-Vila; Blasco-Mira, 2016). The experts were selected on the basis of competence and expertise, as well as their self-assessment with respect to the selected objectives, a method proposed by Cabero-Almenara and Barroso-Osuna (2013) and López-Gómez (2018). The determining aspects for the selection included: - Experience as a teacher and researcher for more than 10 years in digital media, platforms, and interfaces - Recent publications (less than 3 years) on digital media, platforms, and interfaces - Members of research groups with recognized prestige and track record in the field of digital media for more than 10 years. e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 6 Integrated research model for web interfaces: toward a taxonomy of quantitative and qualitative analysis for digital native media A total of 15 experts participated in the first round, of whom only 12 were considered suitable (self-assessment) to make an objective judgment. They were sent a PDF document with the theoretical constructs tool (first version1) and also a survey consisting of Likert scales from 1 to 5 (1, strongly disagree; 5, strongly agree) for each dimension and its indica- tors within the primary model. Also, the experts had a space for each dimension in which they could make observations about that dimension or its indicators. The results of this first round are presented in Table 3. Table 3. Results of the first round of expert judgment: theoretical constructs Fleiss’ kappa Asymptotic Indicator Median (ME) p-Value (k) standard error Dimensions presented (general–articulated) 5 0.725 0.33 1.24 Form of content and narrative 5 0.628 0.35 1.22 Journalistic genres and styles 5 0.621 0.28 1.28 Headlines 5 0.700 0.36 1.26 Metadata of the news stories 4 0.538 0.39 1.22 Transparency, truthfulness, and validity of content 5 0.616 0.29 1.28 Related content 5 0.658 0.33 1.24 Content curation 5 0.525 0.29 1.28 Uninterrupted reading 5 0.608 0.28 1.21 Interface layout 5 0.856 0.19 1.03 Legibility of content 5 0.902 0.11 1.01 Reader–media–author interaction 5 0.616 0.29 1.28 Content customization 5 0.658 0.33 1.24 Web browser 5 0.608 0.28 1.21 SEO and visibility 5 0.525 0.29 1.28 The study uses Fleiss’ kappa (k) as an indicator of interobserver agreement when there are more than two evaluators. According to Landis and Koch (1977), Cohen’s kappa can be interpreted as follows: < 0.20 indicates poor agreement, 0.21-0.40 indicates fair agreement, 0.41-0.60 indicates moderate agreement, 0.61-0.80 indicates strong agreement, and 0.81-1.00 indicates very strong agreement. In this regard, as seen in Table 1, all the concordances (k) of the theoretical indicators showed moderate (0.515) or very strong (0.902) agreement, whereas the median –as a measure of the central position– of the Likert responses was between 4 (agree) and 5 (strongly agree). Table 4. Results of the second round of expert judgment: weighting by dimension and indicators Field Dimension Median (ME) Relative weight Weighted average* Value for the field 9.5 21.43% 10 Form of content and narrative 9.0 36.99% 4 Narrative Journalistic genres and styles 7.3 30.12% 3 Headlines 8.0 32.88% 3 Value for the field 8.2 18.41% 8 Metadata of the news stories 7.5 18.22% 1 Transparency 8.2 19.82% 2 Quality Truthfulness and validity of content 9.5 23.08% 2 Related content 7.8 19.02% 1 Content curation 8.2 19.82% 2 Value for the field 9.5 21.43% 10 Uninterrupted heading 8.2 23.54% 3 Usability Interface layout 8.5 24.53% 2 Legibility of content 9.7 27.87% 3 Content customization 8.3 24.04% 2 Value for the field 8.8 19.92% 9 Search for information Web browser 7.3 46.30% 5 SEO and visibility 8.5 53.69% 4 Value for the field 8.3 18.79% 8 Redissemination of content Interaction with new stories 8.3 49.40% 4 and socialization Redissemination 8.5 50.60% 4 *A correction factor is applied to the weighted average to standardize the rubric and avoid decimals e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 7 Santiago Tejedor; Luis M. Romero-Rodríguez; Lluís Codina Overall, the dimensions presented a median (ME) of 5, a k equal to 0.725 (good), an asymptotic standard error We propose a matrix to analyze narrati- of 0.33, and a p-value of 1.24. Although this confirms ve, quality, usability, search for informa- the construct and content validity of the initial model, tion, and redistribution of content and the experts made various qualitative observations on socialization the questionnaire that were considered appropriate to incorporate into a second version, while others were discarded. Among these observations, two unrelated experts ob- served that accessibility should be included in this analysis model. However, accessibility, being such a complex subject, is already part of numerous academic approaches and also has specific protocols and means of analysis, validated by the international community and international organizations. Thus, a second version of the tool2 was developed, including two new dimensions (narrative and content redissemination). In the second round of expert analysis, in which the questionnaire was sent only to the 12 experts who answered the first round, 10 experts participated. This time, the survey did not ask for the validation of dimensions and indicators –al- ready previously validated– but rather a weighting (value) of each dimension and indicator with a score from 1 (lowest) to 10 (highest). This would provide a rating of the variables of the tools, as a rubric, which would allow a posteriori quantitative and qualitative evaluation of the digital media outlets. The results of this round are presented in Table 4. 3. Results The study has yielded results at different stages or phases of the research. In this sense, Table 1, derived from the litera- ture review on taxonomies or analysis models designed for the analysis of news websites or digital media platforms, has allowed us to identify three thematic areas of interest: quality, usability, and the search for information. Each of these is broken down into a total of 14 thematic dimensions, 36 indicators, and 69 variables, which are detailed in the afore- mentioned analysis sheet (Table 1). Based on this, the incorporation of two rounds of evaluation and review by experts enabled the creation of a matrix that expanded the main thematic areas to five: narrative, quality, usability, the search for information, and content redissemination and socialization. These, in turn, include a proposal that encompasses 16 dimensions (Table 5). Table 5. Proposed taxonomy for the analysis of digital media web interfaces Field Dimension Range Diversity of formats and proposals Application of innovation in the creation Form of content and narrative of content Narrative Journalistic genres and styles Type and variety of journalistic genres and styles included in the medium Headlines Type and features of headlines (approach, wording, and use of teasers) Metadata of the news stories Use of keywords in the labeling of the content presented Identification of sources, resources, and other aspects Presentation of the Transparency organizational chart Quality Truthfulness and validity of content Support in sources: diversity of references used Related content Access to related thematic content that enable a deep dive into the subject Content curation Verification, editing, and quality control work Presentation of content and communication with users without the inclu- Uninterrupted reading sion of advertising or invasive content Interface layout Physical design of informative messages Usability Legibility of content Adaptation of messages for screen-based enquiry Content customization Option of customizing the digital media content Search for Web search engine Actions and tools designed for the (internal/external) search for content information SEO and visibility Organic positioning in search engines and content visibility strategies User–content–media communication options through different actions Redissemination of Interaction with new stories (liking, forwarding, commenting, commenting, rating, etc.) content and socialization Dynamics of content dissemination on other platforms for dialog, especially Redissemination social networks The applicability of the model was tested through a pilot test with 10 Ibero-American digital media outlets. The following are the results derived from this application, organized according to the five main thematic areas of this model (narrati- ve, quality, usability, the search for information, and content redissemination and socialization). This testing was carried out with a descriptive quantitative-qualitative design. Table 6 quantifies the presence of the different elements in the set of digital media tested using a Likert scale ranging from 0 (absent) to 5 (very present) and its reconversion to the value for each dimension. e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 8 Integrated research model for web interfaces: toward a taxonomy of quantitative and qualitative analysis for digital native media Table 6. Results of the quantitative application of the model in the pilot test (Dominican Republic) Periodismo de barrio magazine (Bolivia) Conexión migrante La brava digital (United States) Efecto Cocuyo La silla vacía (Venezuela) (Colombia) (Paraguay) Eldiario.es El surtidor (Ecuador) (Mexico) (Spain) Acento (Cuba) LadoB GK Field Dimension Form of content and narrative 2.4 2.4 1.6 3.2 1.6 1.6 0.8 2.4 3.2 2.4 Narrative Journalistic genres and styles 1.8 1.2 1.2 2.4 1.8 1.2 0.6 1.8 2.4 2.4 Headlines 2.4 1.2 1.2 1.8 1.8 1.2 1.2 1.8 1.8 1.8 Metadata of the news stories 0.6 0.8 0.6 0 1.0 0 0 0 5.0 0 Transparency 1.6 2.0 0.8 1.2 0.4 2.0 1.0 1.6 1.6 0.8 Quality Truthfulness and validity of content 1.6 2.0 1.2 1.6 1.2 1.2 0.8 1.6 1.6 1.2 Related content 0.6 0.6 0.6 0.4 0 0.2 0.2 0 0.2 0.2 Content curation 1.6 1.6 1.2 1.2 0.8 0.8 0.4 1.2 1.2 0.8 Uninterrupted reading 1.8 2.4 1.2 1.2 1.8 1.2 1.2 2.4 1.8 2.4 Interface layout 1.2 1.2 0.8 1.2 0.8 0.8 0.8 1.2 1.2 1.2 Usability Legibility of content 1.8 2.4 1.8 1.8 2.4 0.6 1.2 1.8 2.4 1.8 Content customization 0.8 0.8 0.4 0.4 0.4 0 0.4 0.4 0.4 0 Search for Web browser 2.0 3.0 2.0 1.0 1.0 3.0 2.0 2.0 1.0 1.0 information SEO and visibility 2.4 2.4 2.0 1.0 2.0 2.0 2.0 1.0 1.0 2.0 Redissemi- Interaction with new stories 0.8 0.8 2.4 0.8 3.2 3.2 3.2 0 0.8 0.8 nation of content and Redissemination 0 0.8 2.4 0.8 4.0 3.2 3.2 0 3.2 3.2 socialization TOTAL 23.4 25.6 21.4 20.0 24.2 21.4 19.0 19.2 28.8 22.0 The reading and interpretation of the results derived from the pilot test applied to the 10 selected digital media outlets was organized around the five thematic areas that structure the defined model: narrative, quality, usability, the search for information, and content redissemination and socialization (Graph 1). As can be seen in Graph 1, there is a great deal of correlational variability among the results of each media outlet for each dimension. Efecto Cocuyo (Venezuela) obtained the best rating for the narrative dimension, whereas La brava digital magazine (Bolivia) obtained the best rating for the quality dimension. In terms of usability, as well as the search for information, the Spanish Eldiario.es scored better. In the case of content redissemination, Acento (the Dominican Republic) stood out above its Ibero-American peers. 35 30 4 25 1.6 2 0.8 5.4 7.2 20 4.4 5.8 4 4.8 1.6 0 6.4 3 2 3 15 5.6 6.8 3 6.4 4 4.6 5.8 5 9.6 5.4 5.4 4 10 4.2 6 4.4 7 2.6 4.4 3 3.4 3.6 4.4 3.4 5 6.6 7.4 2.4 7.4 6.6 4.8 5.2 6 4 4 2.6 0 GK eldiario.es El surtidor Efecto Acento Periodismo Conex ión La silla vacía La Brava LadoB Cocuyo de barrio migrante Digital Magazine Narrative Quality Usability Search for Information Redissemination of content and Socialization Graph 1. Quantified results derived from the pilot test e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 9 Santiago Tejedor; Luis M. Romero-Rodríguez; Lluís Codina 3.1. Narrative The analysis of the results derived from the dimen- sions related to the narrative shows a similar com- mitment to the media as a whole, especially in re- ference to the form of content, journalistic genres, and styles. The headlines, on the other hand, take a strictly informative approach in all cases, avoiding headlines that are flashy or that could be designed as clickbait. The presence of graphic elements in the news pie- ces is very limited, as they are relegated to individual sections in most of the analyzed media, despite the fact that this is high-value content for accompanying, complementing, expanding, and enriching news pie- ces predominated by textual information. This as- pect, framed in the narrative section of the proposed analysis table, opens an important reflection on the incorporation, from a protagonistic approach, of this type of content and resources. There are, however, exceptions such as El surtidor from Paraguay, which designs graphics, tables, time- lines, and other visual resources as “paragraphs” of the developed content and not just as complemen- tary elements or additions that do not connect with the thread of the story presented by the content of the piece in which they have been inserted. Regarding journalistic genres and styles, this pilot test identified that there is no clear indication of the type of pieces presented in the digital media as a whole. In some cases, however, such as the Ecuado- rian media outlet GK, founded in 2011 and focused on in-depth journalism and social impact, it was ob- served that the content is organized from its home site or home page by means of bylines that allow the Figure 1. GK. http://gk.city reader to anticipate what type of content they will find. This is a headline element that groups several pieces of information based on the type of journalistic genre. They are short or one-word expressions: “News” [“Noti- cias”], “Reports” [“Reportajes”], “Opinion” [“Opinión”], or “Trending” [“En tendencia”]. In addition, there are sections such as “More news, explained” [“Más noticias, explicadas”]. This distribution (Figure 1) makes it easier for the reader to identify and locate the different types of messages digital media display. Figure 1. For its part, Cuba’s Periodismo de barrio organizes its content on the basis of categories that blend genre type with the particularities of the platform. Thus, at the same hierarchical level and using the same graphic elements (type- face, color, and font size), “Latest articles” [“Últimos artículos”], “Topics [“Temas”],” “Podcast,” “Illustrations” [“ilustra- Figure 2. La silla vacía. http://lasillavacia.com Figure 3. La brava. http://revistalabrava.com e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 10 Integrated research model for web interfaces: toward a taxonomy of quantitative and qualitative analysis for digital native media ciones”], “Images” [“Imágenes”], or “Gender manual” [“Manual de género”] are mentioned. In addition to the above, one can find products such as that of La si- lla vacía from Colombia, which opts for very creative section names (“The empty chair” [“La silla vacía”], “The full chair” [“La silla llena”], “The academic chair” [“La silla académica”], “The chair –courses” [“La silla cursos”], “Super friends” [“Super amigos”], or “Sunday edition” [“La dominguera”]) that, therefore, may make it difficult for the user to anticipate the type of jour- nalistic genres they contain (Figure 2), whereas other media outlets such as La brava from Bolivia opt for a more conventional structure (Figure 3). Social networks are a resource used by various media outlets to report on different topics. The inclusion of pieces derived from other platforms, especially Twitter, YouTube, or Instagram, is a recurring element in all the media studied. The application of the model of analy- sis, therefore, enables a dual diagnostic and descriptive work that, thanks to link journalism, makes it possible to identify the platform ecosystems and synergies of di- gital media on their own and other platforms. The case of the Venezuelan media outlet Efecto Cocuyo is a good Figure 4. Efecto Cocuyo. http://efectococuyo.com example of this type of material coming from both pri- mary sources and other journalistic pursuits (Figure 4). FigEl surtidor, a Paraguayan digital media outlet, stands out for its graphic design focused on the use of caricatures, mas- theads, and illustrations with a clear commitment to innovation in formats and content. This media outlet, winner of the Gabriel García Márquez Journalism Award in 2018 in the Innovation category, opts to breathe new life into stories with both their approach and their presentation, that is, the “what” and the “how” (Figure 5). 3.2. Quality The proposed taxonomy has made it possible to analyze the quality on the basis of the presence of metadata in the news stories, the support for (varied and numerous) news sources, and the content curation process at a formal and stylistic level. It is worth noting that the pilot test found that very few digital media outlets recommend or link other content, whether their own or others’, in their news pieces. This aspect is limited, in most cases, to the use of keywords or tags, not taking advantage of the navigational capabilities of hyperlinks and the possibilities they offer when it comes to delving deeper into a topic. Figure 5. El surtidor.https://elsurti.com Figure 6. Eldiario.es. http://eldiario.es e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 11 Santiago Tejedor; Luis M. Romero-Rodríguez; Lluís Codina In terms of transparency, in general, all the media out- The study of the digital platform (the lets analyzed had sections dedicated to introducing the website of the media) is essential to un- members of their respective teams, their profiles, and the means of contacting them. This information was derstand the media and its dynamics, also enriched using data related to economic balance, especially native digital media income, or potential sources of financing. The case of Eldiario.es from Spain is undoubtedly a paradigmatic element owing to the effort it makes (Figure 6) to offer detailed information about the media outlet, its editorial line, the entities that support it, its economic situation, and other infor- mation aimed at promoting transparency between the digital media outlet and society as a whole. In September 2018, on its sixth anniversary, this newspaper company published its official accounts with a detailed report of its revenues, expenses, and profits. Colombia’s La silla vacía is also an example of transparency owing to its disclosure of its structure, operation, organiza- tional chart, and economic aspects. It showed the rates for each advertising package in detail, while precisely stating its sources of income and its partners, as well as how they invest their resources. Incidentally, the Cuban media outlet Periodismo de barrio also stood out for its commitment to transparency. On its platform, together with a section entitled “Financial transparency” [“Transparencia financiera”], it includes a section ca- lled “Code of ethics” [“Código de ética”], which presents information about the principles that define the media outlet; aspects related to the “integrity, independence, responsibility, commitment and confidentiality” [“integridad, indepen- dencia, responsabilidad, compromiso y confidencialidad”] of the project; financing; coverage, research, and relations- hips with said sources; the writing, editing, and publication of content; photography and audiovisual components; the dissemination of publications and interaction with users, their use of Creative Commons licenses; and the digital media outlet’s code of ethics, among other things. 3.3. Usability Regarding the usability component, the application of the model of analysis identified that the selected media outlets are at the same level, with the exception of some that present a content layout that is somewhat invasive, as the layout makes it difficult to read some pieces on screen. Readability is, however, an element that was achieved in all cases, with nuances that differentiate them. Content personalization is one of the general weaknesses shown by the 10 digital me- dia outlets studied. In this dimension encompassed by the usability of the platforms, the Dominican media outlet Acento, winner of the Na- tional Digital Journalism Award in the category of Best Digital Native Newspaper, stood out for tagging their content very extensively, especially in association with their pieces. In this way, the reader can access other related content. However, this proposal for a hypermedia roadmap is not very sophisticated, since it uses keywords to generalize themes and topics that sometimes correspond to very specific thematic areas. The Ecuadorian media outlet GK stood out for its short-form content accompanied by images whose sizes end up im- pacting on usability and the interaction between users and content (Figure 7). This news project, which focuses on three thematic areas (gender, environment, and trans- parency), stood out for its use of bold type and links (especially external ones) –aspects that contribute to optimizing the adaptation of content to on-screen rea- ding– in all its content. Other media outlets, such as El- diario.es in Spain, Periodismo de barrio in Cuba, La silla vacía in Colombia, and El surtidor in Paraguay, hardly took advantage of hyperlinks in their news stories; at the same time, they avoid using bold type, both for the organic positioning strategy of the content and for usa- bility and readability aspects on screen. 3.4. Search for information The search for information was centered in practically all cases on a generic search tool, located in the upper frame, generally in the right-hand portion of the infor- mation hub. In this sense, particular importance was not given to the implementation of filters or advanced searches, which are crucial aspects of journalistic do- cumentation to receive results that are more selective and specific to the information requirements of the users. Figure 7. GK. http://gk.city e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 12 Integrated research model for web interfaces: toward a taxonomy of quantitative and qualitative analysis for digital native media With regard to organic positioning in search engines, simple actions, such as the use of bold font, links (internal or external), and an ad- vanced process regarding the wor- ding of the title elements, which would be of great value to enhance performance in terms of SEO posi- tioning, were neglected. The Cuban media outlet Perio- dismo de barrio is focused on the development of research about communities affected by natural disasters or found themselves in situations of great social and eco- nomic vulnerability. Together with the usual search engine, it offers a list of thematic categories that, in Figura 8. Periodismo de barrio. http://www.periodismodebarrio.org addition to being used for content tagging, plays an important role in the identification and retrieval of content when using the general website tool to search (Figure 8). 3.5. Redissemination of content and socialization The redissemination of content and the promotion of socialization is one of the dimensions that showed the greatest difference among the media outlets analyzed. Thus, the pilot test identified media outlets that did not provide any ser- vices in this regard, whereas others offered users the possibility of sharing content on social networks, reporting errors, and contributing content. In the case of the Ecuadorian media outlet GK, for example, there were no options for users to interact with the content for redissemination or socialization. The comment option is offered only after registration and is subject to a series of restrictive requirements. 4. Discussion and conclusions Research on the characteristics of digital platforms used by digital native media is an area of growing importance, as highlighted by the various approaches and perspectives adopted by Cabrera-Méndez, Codina, and Salaverría (2019), Salaverría and Martínez-Costa (2021), García-Orosa, López-García and Vázquez-Herrero (2020), and López-García et al. (2022), among others. Just as it would have been impossible to study radio journalism without considering the proper- ties of the audio medium, the study and understanding of journalism today requires ongoing and up-to-date studies on the digital media ecosystem. In this context, digital native media emerge as an object of study of great interest. Accordingly, this research shows that studies centered around the particularities of digital media have focused on speci- fic characteristics of the digital medium. Visibility or aspects linked to new formats have become increasingly important. This body of research provides valuable contributions to the necessary understanding of the digital medium, which, as this study indicates, is key. On this basis, the current research adopts a holistic method in an attempt to include all dimensions, without neglecting any significant ones. Therefore, the work presented herein offers researchers and pro- fessionals, especially digital native media entrepreneurs, a tool that, both conceptually and practically, provides a 360° view for the analysis and auditing of the quality of digital media, including aspects of visibility. The designed methodology combines learning from previous work with expert judgment, by means of a two-round system and the application of stringent measures of levels of agreement, before finally testing the taxonomy with a pilot test (application test). This study thus presented a taxonomy of elements of analysis of the quality of digital native media websites that can be applied by digital media entrepreneurs and managers to improve their chances of sustainability insofar as they depend on the quality of their website to act as a media–user interface. Moreover, the study is useful for digital journalism researchers who need to understand the digital native media of a country or a cultural area or a type of journalism, such as investigative or cultural journalism, among others, in terms of their adaptation to the digital context. This proposed ta- xonomy for the analysis of the elements of digital media To understand the journalism of each aims to provide an element, from academia, to support era is necessary to know its substrate, the sustainability of journalism thanks to the possibility of better understanding the medium on which it is ba- that is, the scenario where it develops sed. e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 13 Santiago Tejedor; Luis M. Romero-Rodríguez; Lluís Codina 5. Notes 1. Available at: https://doi.org/10.5281/zenodo.6811157 2. Available at: https://doi.org/10.5281/zenodo.6811176 6. References Al-Said, Nidal; Al-Said, Khaleel M. (2022). “The effect of visual and informational complexity of news website designs on comprehension and memorization among undergraduate students”. AI & society, v. 37, pp. 401-409. https://doi.org/10.1007/s00146-021-01164-6 Bravo-Araujo, Andrea; Serrano-Puche, Javier; Novoa-Jaso, María-Fernanda (2021). “Uso del clickbait en los medios nativos digitales españoles. Un análisis de El Confidencial, El Español, Eldiario.es y Ok Diario”. Dígitos. Revista de comu- nicación digital, n. 7, pp. 185-210. https://doi.org/10.7203/rd.v1i7.184 Cabero-Almenara, Julio; Barroso-Osuna, Julio (2013). “La utilización del juicio de experto para la evaluación de TIC: el coeficiente de competencia experta”. Bordón. Revista de pedagogía, v. 65, n. 2, pp. 25-38. https://doi.org/10.13042/brp.2013.65202 Cabero-Almenara, Julio, Llorente-Cejudo, María-del-Carmen (2013). “La aplicación del juicio de experto como técnica de evaluación de las tecnologías de la información (TIC)”. Eduweb. Revista de tecnología de información comunicación en educación, v. 7, n. 2, pp. 11-22. https://revistaeduweb.org/index.php/eduweb/article/view/206 Cabrera-Méndez, Marga; Codina, Lluís; Salaverría, Ramón (2019). “Qué son y qué no son los nuevos medios. 70 visiones de expertos hispanos”. Revista latina de comunicación social, n. 74, pp. 1506-1520. https://doi.org/10.4185/RLCS-2019-1396 Codina; Lluís; Pedraza, Rafael; Díaz-Noci, Javier; Rodríguez-Martínez, Ruth; Pérez-Montoro, Mario; Cavaller-Reyes, Víctor (2014). “Sistema articulado de análisis de cibermedios (SAAC): Una propuesta sobre el qué y el cómo para estu- diar medios de comunicación digitales”. Hipertext.net, n. 12. https://raco.cat/index.php/Hipertext/article/view/275560 Codina, Lluís; Pedraza-Jiménez, Rafael; Guallar, Javier (2016). “Características y componentes de un sistema de análisis de medios digitales: el SAAMD”. En: Codina, Lluís; Pedraza-Jiménez, Rafael; Guallar, Javier (coords.). Calidad en sitios web. Método de análisis general, ecommerce, imágenes, hemerotecas y turismo. Barcelona: UOC, pp. 15-39. ISBN: 978 84 9064 487 4 Dowling, David O. (2022). “Interactive documentary and the reinvention of digital journalism, 2015-2020”. Convergence, v. 28, n. 3, pp. 905-924. https://doi.org/10.1177/13548565211059426 García-Orosa, Berta; López-García, Xosé; Vázquez-Herrero, Jorge (2020). “Journalism in digital native media: Beyond technological determinism”. Media and communication, v. 8, n. 2. https://doi.org/10.17645/mac.v8i2.2702 García-Perdomo, Víctor; Magaña, María-Isabel (2020). “The adoption of technology and innovation among native onli- ne news media in Colombia”. International journal of communication, v. 14, pp. 20-40. https://ijoc.org/index.php/ijoc/article/view/12124 García-Serrano, Jesús; Romero-Rodríguez, Luis M.; Hernando-Gómez, Ángel (2019). “Análisis del ‘clickbaiting’ en los titulares de la prensa española contemporánea / Estudio de caso: diario ‘El País’ en Facebook”. Estudios sobre el mensaje periodístico, v. 25, n. 1, pp. 197-212. https://doi.org/10.5209/ESMP.63724 Google (2022). Search quality evaluator guidelines. https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf Humprecht, Edda; Esser, Frank (2018). “Mapping digital journalism: Comparing 48 news websites from six countries”. Journalism, v. 19, n. 4, pp. 500-518. https://doi.org/10.1177/1464884916667872 Kim, Hyun K.; Jeon, Hwaseung; Choi, Jeongwon (2020). “How does the design element of a news website influence user experience?”. ICIC express letters, v. 14, n. 3, pp. 265-271. https://doi.org/10.24507/icicel.14.03.265 e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 14 Integrated research model for web interfaces: toward a taxonomy of quantitative and qualitative analysis for digital native media Kulkarni, Shirish; Thomas, Richard; Komorowski, Marlen; Lewis, Justin (2022). “Innovating online journalism: New ways of storytelling”. Journalism practice, first online. https://doi.org/10.1080/17512786.2021.2020675 Landis, J. Richard; Koch, G. Gary (1977). “The measurement of observer agreement for categorical data”. Biometrics, v. 33, pp. 159-174. Landow, George P. (1995). Hipertexto. La convergencia de la teoría crítica contemporánea y la tecnología. Barcelona: Paidós. ISBN: 978 8 449 30186 5 Landow, George P. (comp.) (1997). Teoría del hipertexto. Barcelona: Paidós. ISBN: 844 9 30259 5 Le-Masurier, Megan (2015). “What is slow journalism?”. Journalism practice, v. 9, n. 2, pp. 138-152. https://doi.org/10.1080/17512786.2014.916471 Le-Masurier, Megan (2016). “Slow journalism”. Journalism practice, v. 10, n. 4, pp. 439-447. https://doi.org/10.1080/17512786.2016.1139902 López-García, Xosé; Canavilhas, João; Negreira-Rey, María-Cruz; Vázquez-Herrero, Jorge (2022). “Political journalism in digital native media”. In: García-Orosa, Berta (ed.). Digital political communication strategies. Multidisciplinary reflec- tions. Cham: Palgrave, pp. 59-74. ISBN: 978 3 030 81567 7 https://doi.org/10.1007/978-3-030-81568-4_4 López-Gómez, Ernesto (2018). “El método Delphi en la investigación actual en educación: una revisión teórica y meto- dológica”. Educación XX1, v. 21, n. 1, pp. 17-40. https://doi.org/10.5944/educxx1.20169 Lopezosa, Carlos; Codina, Lluís; Díaz-Noci, Javier; Ontalba, José-Antonio (2020a). “SEO and the digital news media: From the workplace to the classroom”. Comunicar, n. 63, pp. 65-75. https://doi.org/10.3916/C63-2020-06 Lopezosa, Carlos; García-Carretero, Lucía; Gonzalo-Penela, Carlos; Codina, Lluís (2020b). “Sistema de análisis (AVCIS) de la visibilidad en cibermedios mediante indicadores SEO”. In: Meso-Ayerdi, Koldobika; Peña-Fernández, Simón; La- rrondo-Ureta, Ainara. Nuevos actores y estrategias en el espacio mediático compartido: XI Congreso internacional de ciberperiodismo. Bilbao: Servicio Editorial Universidad del País Vasco, pp. 185-204. ISBN: 978 84 1319 167-6 https://dialnet.unirioja.es/servlet/articulo?codigo=7433485 Lopezosa, Carlos; Trillo-Domínguez, Magdalena; Codina, Lluís; Cabrera-Méndez, Mar (2021). “El SEO en la empresa pe- riodística: percepciones y elementos clave para su adopción en las redacciones”. Revista latina de comunicación social, n. 79, pp. 27-45. https://doi.org/10.4185/RLCS-2020-1487 Mengual-Andrés, Santiago; Roig-Vila, Rosabel; Blasco-Mira, Josefa (2016). “Delphi study for the design and validation of a questionnaire about digital competences in higher education”. International journal of educational technology in higher education, v. 13, n. 1, p. 12. https://doi.org/10.1186/s41239-016-0009-y O’Hagan, Anthony (2019). “Expert knowledge elicitation: Subjective but scientific”. The American statistician, v. 73, n. 1, pp. 69-81. https://doi.org/10.1080/00031305.2018.1518265 Peña-Ascacíbar, Gonzalo; Álvarez-Peralta, Miguel (2021). “Emergencia, innovación y consolidación de nuevos modelos para el periodismo digital: estudio de los casos de El Confidencial, elDiario.es e infoLibre”. Estudios sobre el mensaje periodístico, v. 27, n. 2, pp. 593-606. https://doi.org/10.5209/esmp.71245 Pérez-Seijo, Sara (2021). “Uso del vídeo 360º por los medios nativos digitales. Análisis exploratorio de los primeros pasos en el ecosistema periodístico español”. Profesional de la información, v. 30, n. 3, e300304. https://doi.org/10.3145/epi.2021.may.04 Plazas, Hugo-Alonso (2019). “Modalidades de la visión en dispositivos de visualización periodística online”. Revista Ke- pes, v. 16, n. 20, pp. 249-281. https://doi.org/10.17151/kepes.2019.16.20.11 Prawira, Indra; Rizkiansyah, Mariko (2018). “Search engine optimization in news production online: Marketing practice in Indonesia online news media”. Social sciences & humanities, n. 26, pp. 263-270. e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 15 Santiago Tejedor; Luis M. Romero-Rodríguez; Lluís Codina Rodríguez-Hidalgo, Claudia; Rivera-Rogel, Diana; Romero-Rodríguez, Luis M. (2020). “Information quality in Latin Ame- rican digital native media: Analysis based on structured dimensions and indicators”. Media and communication, v. 8, n. 2, pp. 135-145. https://doi.org/10.17645/mac.v8i2.2657 Romero-Rodríguez, Luis M.; Aguaded, Ignacio (2017). “Toward a taxonomy of newspaper information quality: An ex- perimental model and test applied to Venezuela dimensions found in information quality”. Journalism, v. 18, n. 10, pp. 1327-1345. https://doi.org/10.1177/1464884916663596 Romero-Rodríguez, Luis M.; De-Casas-Moreno, Patricia; Torres-Toukoumidis, Ángel (2016). “Dimensions and indicators of the information quality in digital media”. Comunicar, n. 49, pp. 91-100. https://doi.org/10.3916/C49-2016-09 Romero-Rodríguez, Luis M.; Tejedor, Santiago; Castillo-Abdul, Bárbara (2021). “From the immediacy of the cybermedia to the need for slow journalism: Experiences from Ibero-America”. Journalism practice, v. 16, n. 8, pp. 1578-1596. https://doi.org/10.1080/17512786.2020.1870530 Salaverría, Ramón (academic editor) (2020a). Digital native news media: Trends and challenges. Media and communi- cation, v. 8, n. 2. Lisboa: Cogitatio Press. https://www.cogitatiopress.com/mediaandcommunication/issue/view/174 Salaverría, Ramón (2020b). “Exploring digital native news media”. Media and communication, v. 8, n. 2. https://doi.org/10.17645/mac.v8i2.3044 Salaverría, Ramón; Martínez-Costa, María-del-Pilar (2021). Medios nativos digitales en España. Caracterización y ten- dencias. Salamanca: Comunicación Social Ediciones y Publicaciones. ISBN: 978 84 17600600 Santos-Hermosa, Gema; Lopezosa, Carlos; Codina, Lluís (2022). “Analysis of interactivity in digital journalism”. In: Freixa, Pere; Codina, Lluís; Pérez-Montoro, Mario; Guallar, Javier (eds.). Visualisations and narratives in digital media. Methods and current trends. Barcelona: DigiDoc-EPI, pp. 13-25. ISBN: 978 84 120239 9 2 https://doi.org/10.3145/indocs.2022.2 Seyser, Doris; Zeiller, Michael (2018). “Scrollytelling - An analysis of visual storytelling in online journalism”. In: 22nd international conference information visualisation, pp. 401-406. https://doi.org/10.1109/iV.2018.00075 Stroobant, Joyce (2019). “Finding the news and mapping the links: a case study of hypertextuality in Dutch-language health news websites”. Information, communication & society, v. 22, n. 14, pp. 2138-2155. https://doi.org/10.1080/1369118X.2018.1477971 Tejedor, Santiago; Pérez-Escoda, Ana; Ventín, Augusto; Tusa, Fernanda; Martínez, Fátima (2020a). “Tracking websites’ digital communication strategies in Latin American hospitals during the Covid-19 pandemic”. International journal of environmental research and public health, v. 17, n. 23, 9145. https://doi.org/10.3390/ijerph17239145 Tejedor, Santiago; Ventín, Augusto; Cervi, Laura; Pulido, Cristina; Tusa, Fernanda (2020b). “Native media and business models: Comparative study of 14 successful experiences in Latin America”. Media and communication, v. 8, n. 2, pp. 146-158. https://doi.org/10.17645/mac.v8i2.2712 Van-der-Nat, Renée; Müller, Eggo; Bakker, Piet (2021). “Navigating interactive story spaces. The architecture of interac- tive narratives in online journalism”. Digital journalism, first online. https://doi.org/10.1080/21670811.2021.1960178 Vázquez-Herrero, Jorge; Negreira-Rey, María-Cruz; López-García, Xosé (2019). “La innovación multimedia e interactiva en el ciberperiodismo argentino”. Revista de comunicación, v. 18, n. 1, pp. 191-214. https://doi.org/10.26441/RC18.1-2019-A10 Wurman, Richard-Saul (1997). Information architects. New York: Watson-Guptill Publications. ISBN: 978 1 888 0013 8 9 Zelizer, Barbie (2009). The changing faces of journalism: Tabloidization, technology and truthiness. London: Routledge. ISBN: 978 0 415 7782 5 1 Zuluaga-Trujillo, Jimena; Gómez-Montero, Silvia-Marcela (2019). “Medios nativos digitales en América Latina: agenda, sostenimiento e influencia”. Chasqui. Revista latinoamericana de comunicación, n. 141. https://doi.org/10.16921/chasqui.v0i141.3333 e310619 Profesional de la información, 2022, v. 31, n. 6. e-ISSN: 1699-2407 16