Evaluating Quality and Excellence in Translation Studies Research: Publish or Perish, the Spanish Way

Academia has embarked on a journey where credentials must be shown on a regular basis: publish or perish. Obtaining the ticket to success is beyond quantity, which can be measured; it is related to more intangible terms such as excellence and quality. Research assessment exercises are now here to stay, and they test not only those who wish to jump onboard, but also their baggage, and ultimately whether their boat stays afloat or sinks. Qualitative assessment is a much maligned exercise which has to evaluate and balance criteria measured in terms of: originality, thematic diversity, innovation, sustainability, multidisciplinary research, benefits, and impact. This article departs from a study of some Spanish evaluating agencies – also making reference when possible to their European equivalent – and their assessment criteria. After drafting a list of merits listed 2 in each agency the article poses some basic questions regarding assessment. It then goes into the nature of assessment criteria and offers concrete data on Translation and Interpreting Studies journal indexing. The paper concludes with some general comments and recommendations, which are valid across Europe. Resumé Le monde académique s'est engagé dans une voie qui situe aux candidats à y participer dans une impasse: publier ou périr. Le succès dépend non seulement de la quantité, qui peut être mesurée, mais surtout de conditions plus intangibles telles que l'excellence et la qualité. De tels exercices d'évaluation de la recherche sont maintenant là pour être testés, et ils évaluent non seulement ceux qui veulent s'embarquer dans cette expédition, mais aussi leurs bagages, donc leur trajectoire, et, finalement, si leur train arrive à destination ou s'il reste sur place. L'évaluation qualitative est un exercice très décrié qui a pour but d'évaluer et d'équilibrer les critères mesurés en termes d'originalité, de diversité thématique, d'innovation, de durabilité, de recherche multi-disciplinaire, d'avantages et d'impact. Cet article part d'une étude sur certains organismes espagnols d'évaluation et sur leurs critères d'évaluation-faisant également référence à leurs équivalents européens si possible. Après l'élaboration d'une liste de conditions requises dans chaque agence, l'article pose des questions fondamentales concernant l'évaluation. Il aborde ensuite la nature des critères d'évaluation et fournit des données concrètes sur l'indexation des revues dans le domaine de la Traduction et de l'Interprétariat. Le document se termine 3 par quelques observations générales et recommandations qui sont valables dans toute l'Europe.


Introduction
In the last 20 years most European countries have created systems to evaluate the quality and excellence of university research.Researchers in Social Sciences and Humanities have been faced during the last decade with new assessment trends and practices 3 .While teaching standards go unattended -moreover since the implementation of the EHEA 4 (European Higher Education Area) which focuses on competencies rather than content and its evaluation-research appears to be the only output at the core of all academic activities which can be measured -administrative duties have been overlooked by everyone.Research output is analysed for all purposes, that is from a job appointment, through research funding, and departmental grading for excellence (and subsequent funding), to personal career development.Some countries have a set figure for successful career progression 5 , requirements for tenure and appointments.Promotion requirements change from country to country, and sometimes within a university, so it is difficult to give an overview.The article focuses in the Catalan and Spanish assessment system, since there are commonalities with benchmarking in other countries; "publish or perish" is now a widely accepted way of life in research oriented universities across the world.
Mapping evaluation agencies offers three different scenarios: no agency, one national agency, and a mixed system.The first, where there is no agency as such, is the case of Finland 6 , South Africa or Ireland.The second, where there is only one national agency, such as France, Italy or Norway.Finally there are some countries where many agencies coexist, reflecting the political map such as Belgium, UK or Spain: each agency will have their own benchmarks and evaluating system.Spain is an interesting case, since quality is evaluated at two levels: at estate level with the ANECA and CNEAI and also in the eleven autonomic regional agencies.In this paper we will analyse and discuss the criteria. 7Analysis is produced from collecting existing published data, but also from our personal experience in two categories: users and evaluators at both Catalan and Spanish quality agencies: AQU 8 , ANECA 9 and CNEAI 10 .Other institutions, which also evaluate, such as the Spanish Ministry of Science and Innovation (MICIN), and the Catalan Agency AGAUR don't evaluate academics but research projects, the production of research groups, pre and post doctoral fellowships, and also allocate funds to organise academic conferences and seminars.Despite not being evaluating agencies stricto sensu -they do not explicitly search for quality and excellence, they have also been included in our study since they establish and publish their own specific evaluating criteria for their calls -which may vary according to their scientific strategic policies.
Who evaluates, what is evaluated and how is the evaluation carried out are three pertinent issues.What "quality" and "excellence" means, and how to achieve it in the Humanities and Social Sciences are also at stake.Research "impact" is beginning to surface as a possible measurement, in Spain as in the UK 11 , though there is no international agreement as to what it means and how it should be accounted (Collini   2009).Impact, in general, has been up to now measured (Harzing 2010) using citation metrics in terms of some data obtained from fields such as total number of papers, total number of citations, average number of citations per paper, average number of citations per author, average number of papers per author, average number of citations per year and an analysis of the number of authors per paper.
In the UK "impact" is now taking a more nuanced meaning, as explained by Collini (2009): But one very significant new element has been introduced.In this exercise, approximately 25 per cent of the rating (the exact proportion is yet to be confirmed) will be allocated for "impact".The premise is that research must "achieve demonstrable benefits to the wider economy and society".
The guidelines make clear that "impact" does not include "intellectual influence" on the work of other scholars and does not include influence on the "content" of teaching.It has to be impact which is "outside" academia, on other "research users" (and assessment panels will now include, alongside senior academics, "a wider range of users").Moreover, this impact must be the outcome of a university department's own "efforts to exploit or apply the research findings": it cannot claim credit for the ways other people may happen to have made use of those "findings".
In this paper we focus on the current situation in both the Humanities and the Social Sciences since Translation and Interpreting Studies (T&I) is interdisciplinary per se.T&I research is a relative new discipline in Spain, while in some countries it is still taught within Comparative Literature or Modern Languages departments, usually under the Linguistics umbrella, and in Arts Faculties.T&I departments, faculties and doctorate studies in Spain did not start until 1992 when four-year T&I degrees were approved by the Spanish Government 12 .T&I faculties were staffed by professional translators, since the discipline borders the academic landscape of the applied science.A common scenario -and political demand-is to consider the practice of translation as a research activity, which can have accountability in evaluations and be taken into consideration when drafting benchmarks.Initially, understanding and studying T&I theory, and how to bridge it with the practice, was one of the thorniest issues of the discipline.These days both areas appear to have found their niche and coexist with mutual respect and recognition, teaching at university side by side, but with different contracts, responsibilities and salaries.
Quite recently a new knowledge dissemination culture has been implemented in the Humanities and Social Sciences -indirectly through the introduction of new research assessment criteria and processes.Independently of the agency, publications are the object of assessment, and the locus where assessment criteria lies, in the whole evaluation process and more specifically, papers published in indexed journals.In some cases, papers published in non-indexed journals are not even taken into account for evaluation, especially if they are published by the institution one is affiliated to.Thus new trends in research quality assessment have led to a shift in the value of some publications traditionally favoured by some T&I scholars which aimed at professional studies.The introduction of new assessment benchmarking has begun in some countries -such as Catalonia or UK -through pilots whose results are still not fully accepted by the academic community, especially since the transition from the traditional system to the new and competitive method has been a hasty exercise.Fry et al (2009: 35) express their worry in the following terms: "Concern has been expressed about the effect of the research assessment on Early Career Researchers, especially in terms of the process of learning to research and publish".
To offer a panoramic view of the current situation in Spain (which could be applied to other European contexts), in this paper we have revised the existing assessment procedures and criteria.The aim is twofold: to highlight keywords related to quality and excellence, and to identify weaknesses in the system.First we look at the quality agencies and list the explicit merits which will be evaluated.Then we look at who are the evaluators, and their impact in the assessment, leading us to formulate some questions.Since articles published in journals seem to be the locus for quality, we have drawn lists of journals along the index where they are included, to facilitate the task of sending articles for publications and justifying CVs.
The present work is also the result of a period of research in the assessment procedures and outcomes in T&I Studies, which had as primary objective to offer new research and assessment data and tools to academics in order to increase both: output excellence and objectivity of assessment reports.

What in T&I research assessment
The object of assessment is the first issue analysed.The Catalan and Spanish agencies AQU and ANECA, respectively, evaluate researcher's CVs throughout their academic career: tenure-track lecturer, tenured assistant professor and full professor.Since in Spain there is no promotion available, beyond the three aforementioned categories, the only way to move upwards in the academic ladder -and financially -is to undergo a research assessment.CNEAI and AQU are responsible for this evaluation 13 which takes place every six years: assessment is exclusively on research, other merits such as teaching, technology transfer or admin are not taken into consideration.
Regarding books, originality and achievement is also at stake, and consideration is made whether there is: one author vs. multiple author, monograph of collected articles vs. volume of collected articles and published in international prestigious publishing house.
Edited monographic and miscellanea books may in some occasions be considered in the same category as articles, but guidelines do not clearly state in which cases this may take place.In any case, from an assessment point of view, no indication is given whether editing a book is equally valuable to authoring, and the trend seems to go to dismiss editorship -this fact has been observed also in countries such as Belgium, Italy and Norway.
When assessing contributions to conferences, consideration is given to their scope and international significance, if there is a scientific committee, the existence of a selection process, and also the type of contribution: 20 minutes presentation, a paper, a plenary session or a poster.In some cases, resulting publications from a conference proceedings are assessed, but only when there has been a process of selection previous to the publication.A contribution to an edited collection of conference proceedings bears no credits.
In general terms, research publications will be assessed on the basis of the type of contribution and its length.Positive or negative weight can be applied, i. e. a book of over 50 pages can have a factor greater than 1, whereas an article or book chapter of less than 10 pages or a review are worth less than 1 point.A negative factor could be for example a high number of authors of a given contribution (AQU 2007: 8).
In Spain and Catalonia research impact is taken into account, and also the degree of creativity, thematic diversity, and innovation.Impact is calculated by a coefficient of the publications and the number of citations received.Both parameters are not straight forward in T&I.
As can be seen in table 1 in Appendix 1 the four agencies of our study differ greatly in the merits considered for research quality assessment, both quantitatively and qualitatively.CNEAI seems to be the most demanding, which may be understandable since it assesses excellence for six-year research periods.As far as AQU and ANECA are concerned, attention should be paid to the fact that while assessing similar merits, AQU is more demanding than ANECA, using international projection, for example, as a positive indicator.This may explain why the same researcher might obtain a negative assessment from AQU and a positive from ANECA -when theoretically tenured for equivalent academic profiles.
In any case, assessment guidelines should contain more specific data concerning how these merits are rated and applied.To raise several examples: is editing a book equally valuable as authoring?At a conference, is presenting a poster worth as much as an oral communication?An indexed article has now been established as the most valuable item on a CV, but it is impossible to assess the value of a book or a book chapter in relation to it.Finally, the absence of the concept "translation" as a category is an important indicator in our field.Given the lack of homogeneous application of these parameters across countries, within countries, and disciplines, it only goes to show how thorny evaluation and benchmarking can be.

Who in T&I research assessment
Within the Humanities, T&I studies is by default included under the category of Linguistics and in particular that of Applied Linguistics (Desblache forthcoming), which in turn has been traditionally a part of the wider field of Philology.Linguistics is a very broad and dynamic field which deals with all aspects of any form of language as an object of study -history, variation, documentation, acquisition, etc. -and the many applications -communication, translation, engineering, etc.The issue is the degree of specialisation needed to be a competent expert evaluator in the field of Linguistics.Furthermore, it should not go unnoticed the rivalry between different areas within Linguistics, and the fact that "Peers monitor the flow of people and ideas through the various gates of the academic community (...) and some peers are given more of a voice than others, and serve as gatekeepers more often than others (Lamont 2009: 2).
Consulting the available lists of evaluators for each agency, T&I faces a lack of democratic representation (Chubin and Hackett 1990 & 2003, Crane 1976, Desblache forthcoming). 14In Spain, for example, see table 2 in Appendix 2, agencies have a pool of evaluators.The process for their selection is not public and their performance -if evaluated -is never disclosed.From our own experience as evaluators, we can say that agencies provide uneven type and specificity of benchmarking criteria.In some cases guidelines are confidential, with instructions guiding the values given to each criterion of merit and contribution, others leave it up to personal choice.The lack of common evaluation procedures and goals has been already stated by Lamont (2009: 2) who has brought to light the difference in standards across disciplines, between what may be considered quality in Sociology or Anthropology departments, and Economics.Her explanation to this decalage is the different ways in which quality is defined for each field.This gap can be made wider since "the criteria for assesing quality or excellence can be differently weighted and are the object of intense conflicts".As far as we know, evaluators do not follow any kind of specific training to become evaluators but, since they themselves might undergo evaluations of some kind, from time to time, they are supposedly familiar with the system and to a greater or lesser extent accept its rules.
Consideration should also be paid to the handicaps which evaluators are faced with: the sheer number of searchable indexes and databases, their different nature, and their consultation methodology -far from user-friendly (Harzing and Van der Wal 2007).If to these we add that data is not usually free of access, or straightforward, it leaves evaluators at the hands of the libraries in their institution, and the funds available for subscriptions which will give access to consult them.Time is a factor which is not taken into consideration, but looking up index, publications, database, impact, etc. can be a hard job and quite time-consuming.
There is a debate on the subject of direct versus indirect research quality assessment.
Those in favour of the former claim that is the only valid method to ensure fair assessments and should be adopted regardless its cost.Conversely, those in favour of indirect assessment claim contributions have already undergone direct assessment through peer-reviewing before publication and thus direct assessment would not only be too costly, but also redundant.
A final consideration in this section should go to the scarcity of resources at universities, especially in times of global economic crisis.Budgets are severed in all departments, evaluation being no exception, as pointed out by Fry et al (2009: 35) when they affirm that "the crucial dilemma of not only how to measure quality, but how to measure quality across different disciplines at as low a cost as possible in both money and time, and with as little impact as possible on the actual quality of the research process itself".
An urgent study for T&I will be to perform a comparative analysis of results according to indexes, in a similar way as that from Harzing and Van der Wal (2007), Torres-Salinas et al (2009) and Harzing (2010).It may be that Google Scholar results, even if it's not an index and data is mechanically gathered, throw a more objective and thorough results than the established and more valued indexes -with the added value of being free of access.

Objective versus subjective criteria
An interesting exercise, while understanding the new order in academic evaluation, is to find the objectiveness of some of the existing criteria in Spain.In order to do so, and according to available assessment guidelines, we have drafted a list of the most common standards used to evaluate research output, in any form or shape.The following list should be compared to that found in the introduction to this article when defining "impact": It is important to note that not all the above criteria can be easily and objectively applied by evaluators, indicators such as "adequacy" or "innovation" require further clarification, if that can be provided, never mind measured.The most objective indicator, according to agencies, is indexing. 15The downturn is that researchers in T&I Studies are informed of indexes and databases referred by agencies in a very unspecific way (see Rovira-Esteva & Orero forthcoming).This problem could be solved by elaborating journal lists per area with information on the indexes and databases that include them taken into consideration by agencies, and we return to this issue in the next section 16 .
There are three more criteria from the above list that could be objective: the number of authors, publisher's prestige and the methodological approach -analytic and comparative works versus purely descriptive.Although some of these indicators can lead to quantifiable assessment, they fail since neither evaluators nor researchers have As far as the publisher's prestige is concerned, agencies could provide evaluators and researchers with editorial rankings, as it is already granted to journals.Finally, it is often difficult for evaluators to establish clear-cut boundaries between methodological approaches, since the same article may be partly descriptive and partly analytical and/or comparative.
Apart from the above mentioned four standards, the remaining are subjective and open to debate, especially issues such as: adequacy, quality, innovation, and originality.In any case, these qualities can only be adequately assessed by a well informed researcher within the field and, in any case, through direct assessment methods, although, in some cases, it might even be subject to controversy among experts.

Journal indexes and databases
Peer-reviewed journals, especially those included in international or national journal indexes and databases, have become the most valued mode of dissemination and publication.As with the implementation of any new system, this has been greeted with some initial reticence.Some teething problems are seen as follow: Each evaluation agency mentions its own list of indexes and databases considered for assessment (see table 3 in Appendix 3), though many more exist.The indiscriminate adoption and rejection of indexes is a constant object of puzzlement amongst researchers, since they might be assessed by different agencies at different stages of their academic career.With respect to indexes, it is explicitly stated "value will be placed basically on publications in indexed journals, that is to say, publications that have passed a peer review assessment process and that appear in the SCI databanks or similar" (AQU 2008: 3, our emphasis).This leads us to the second problem faced by researchers.
Indexes and databases mentioned by agencies greatly vary in terms of scope, nature and searching methodology.In some cases journals are rated on a three or four degree scale (European Reference Index for the Humanities, CARHUS+ 17 ) while in others, each journal has a diffusion or numerical index measuring the journal impact, 18 influence or prestige (Journal Citation Reports, Revistas españolas de Ciencias Sociales y Humanas 19 ).In most cases these indexes are a mere list or database of journals without any ranking (Bibliographie Linguistique, FRANCIS 20 , Library and Information Science Abstracts).This arbitrariness raises the question of validity.For example, is an article rated higher if it appears at the SCCI or AHCI (SCI databanks) than in other indexes?
Will it suffice for a positive assessment to be included in any of these databases, and will articles rate higher if they appear in more than one index?The paradox is that lists like ERIH's are not intended to be a bibliometric tool.For this reason the ERIH Steering Committee and the Expert Panels advise against using the lists as the only basis for assessment for positions or promotions, or applicants for research grants.
Despite the warning in Spain they are actually used as such. 21e last problem with these indexes and databases is found when dealing with issues such as themes, country or language of publication, etc. Unbalanced and biased lines have paved the way for the creation of new indexes in order to compensate such asymmetries.For instance, ERIH has been gathered by the European Science Foundation and focuses in journals relevant within the context of the European Research Area; DICE has been sponsored by the Spanish CSIC (Consejo Superior de Investigaciones Científicas) and ANECA and IN-RECS by the Spanish Ministry of Education and Innovation, so both target Spanish journals; CARHUS+ has been gathered by the Catalan AGAUR and raises to higher ranks journals dealing with Catalan topics or using Catalan as a vehicular language -which would hardly ever find a place in either Spanish or European lists.Still, considerable thought should be dedicated to finding a comprehensive and fairer system of listing which may for example include Chinese journals devoted to T&I Studies, which has nothing to do with their intrinsic value or impact, but rather to the language used and diffusion channels, which highlights power relations within academia.
The need for a positive assessment sometimes leads researchers to publish in journals that are not the natural reading environment, as Fry et al (2009: 34) comment: One way in which interpretation of research assessment criteria has manifested is in the pressure experienced by researchers to publish in journals with an international audience, when a national journal might be more appropriate for the area of research and more accessible to the target audience.
It is thus understandable that Spanish agencies favour databases that they directly or indirectly promote (DICE-CINDOC, INRECS, RESH, SJR 22 ).It is not the case, though with the Catalan agency (AQU), which has recently included European (ERIH) and Spanish indexes (IN-RECS, DICE-CINDOC, Latindex) and rejects the Catalan (CARHUS+).
In their guidelines, both ANECA and AQU mention that they may considered as quality factor a journal included in indexes and databases of specific fields of knowledge different from the ones listed in table 2. It remains to be seen if evaluators actually use different indexes than those explicitly listed by the agencies and object of our analysis.

Triggering quality in research: A indefatigable quest
In the Department of Translation and Interpreting at the Universitat Autònoma de Barcelona (Spain), an active policy has been adopted to increase positive results in research exercises at all levels: pre and postdoctoral scholarships, accreditation and promotion of staff, design and draft of national and international research projects, and the setting up of a research laboratory.All these initiatives pivot on a common element: a powerful CV, which in turn depends on articles indexed in journals and their respective impact.Within this context, the need to raise awareness regarding grading publications was paramount.The first step was drafting lists of journals for our area of research.These could be used as guidelines to gather information about the indexes and databases, and be of help in understanding the new assessment system.To this aim a joint collaboration with the staff of the Library of Humanities was established.A list of 141 journals in T&I studies and related areas was created. 23is exercise was not only useful from a pragmatic perspective, but also allowed us to quantify and understand the position of T&I Studies journals in international indexes and databases.We thus found that some of these indexes are either too broad in scope, embracing so many subareas that T&I Studies can only receive a very small portion of the cake.This is the case with DICE, RESH, JCR, SCCI, Répertoire Bibliographique de Louvain, Francis, International Bibliography of the Social Sciences, AHCI and SSCI. 24her indexes are too specialised (such as IN-RECS, Bibliography of the History of Arts, Index Islamicus and International Medieval Bibliography) and consequently do not include any journal from T&I. Indexes where T&I journals are better represented by decreasing order are MIAR (which so far does not count for any agency), ERIH and Bibliographie Linguistique.From table 4 (appendix 4) it is clear how despite its multidisciplinary character, T&I is practically absent from Social Sciences indexes and databases.
Given the low presence of T&I journals in the most prestigious indexes and databases, it was decided to search again, including journals covering other "related areas"-mainly language as a form of communication.The objective was two-fold: first, to show that multidisciplinary journals, or those with a broader scope, are more likely to be indexed than those focusing exclusively in T&I Studies.Secondly, to provide T&I researchers with a wider range of indexed journals available to submit their research papers.After this broader search the results were slightly better (see figures in table 5, appendix 4).
Indexes and databases where journals in languages and communication (included T&I Studies) enjoy higher visibility are by decreasing order MIAR, ERIH, Bibliographie Linguistique, Periodicals Literature in Humanities and Social Sciences, Humanities Citation Index and Francis.Appendix 5 shows a list of T&I journals indexed in at least one of the indexes or databases considered for assessment by the Spanish agencies.As we can see, scholars in T&I have 38 journals to choose from, according to their research and assessment needs.
While production of quality and excellence publications is the main research assessment indicator -though not the only one-lately the demand for publication in highly valued journals and especially indexed ones has increased exponentially, with the effect of a delay in the publication date. 25This new policy in research quality assessment has also had the collateral effect of increasing the number of T&I Studies journals.Hence new quality journals are beginning to absorb the increasing demand for publication but, since bibliographic indexes and databases only include journals with at least a two or three year history, scholars are forced to send their manuscripts to well established journals.
Both lists (the reduced and extended one), have proven to be of use not only for those researchers in our Department, who have undergone research assessment exercises during the last few months, but also constitutes in itself a useful material for drawing conclusions regarding visibility of T&I journals, the evolution of research, tendencies, and the current state of the art in T&I Studies, among others.It is also a tool to carry out bibliometric studies concerning T&I research such as the present article.

T&I Research Assessment Survival
Looking at how to succeed, and how to comply with all the evaluation criteria beyond publishing in indexed journals, we have formulated a list of problems, which at this stage do not seem to have a clear response, and hence more research and transparency will be needed in order to ascertain a set of objective recommendations, in particular in the field of T&I Studies: • How are evaluators chosen and how is their work contrasted?
• How are interdisciplinary contributions assessed?
• How can we be sure of evaluators' competence dealing with academic journal indexes in T&I Studies?
• How is information regarding contributions' indexing and impact validated?
• How is participation in conferences assessed?
• How can evaluators assess the degree of innovation and originality of a given contribution?
• How many authors per article is considered adequate and too many seen as a crowd?
• How is the reducing factor in consideration of the number of authors being applied?
• How many pages are considered as adequate?
• How are multiplying or reducing factors applied as far as the number of pages is concerned, if layout and typographic elements vary from one journal to another?
• How are reducing factors applied as far as author ordering is concerned?

Conclusions
After drafting this article, and beyond the questions in the previous section, we have come up with three different sets of conclusions.The first is related to research quality assessment criteria.Having analysed standards in several quality agency assessments, we have come to the conclusion than there exist many dissimilarities but they agree in the importance of publishing in indexed journals.We have also found that guidelines mix objective with subjective criteria.Quality, originality and innovation are intangible parameters and their assessment and measurement are subject to the evaluator's better judgement.Indexing, number and ordering of authors, number of pages and methodological research approach can be objective parameters but nowadays are very difficult to apply given the lack of objective data available in the field of T&I Studies.
There is an urgent need for internationally agreed figures in: number of authors and pages per article, author ordering conventions, indexing, publication in miscellanies, etc.
While there is a wide number of searchable indexes and databases, these are very diverse in nature, and have different and not always straightforward or user-friendly consultation methods -to make matters worse access is not always free.Hence searching for a given journal in an index, or database, can be a hard job and quite timeconsuming.Therefore, we should be able to jointly develop an agreed set of standards, procedures and guidelines on quality assurance, and to explore ways of ensuring an adequate peer review system for accreditation agencies or bodies.For example the creation of a professional body of evaluators, who will not run in the same race, and who are trained specifically for this practice, as it is the case in many other quality agencies, will be a long term solution.
The second conclusion is the need to generate lists of journals per area of research (such as the one presented here for T&I Studies) containing information about the indexes and databases where they are included.This would constitute a very important tool for research, especially to issues related to visibility, impact and dissemination of research outcomes.We believe that if researchers and evaluators share such a tool there will be a significant improvement in the research activity, and will also optimise assessment exercises: providing a more transparent and objective enterprise.All actors will be playing with the same known rules: fair play.
The third conclusion would read better as recommendations to those who wish to pursue a successful academic career, yes: to publish rather than perish.The first recommendation is to write articles which are sent to journals included in international and prestigious indexes (i.e.those included in assessment guidelines).The second is to consider the possibility of publishing in journals not strictly falling within the field of T&I Studies, i.e. journals belonging to related areas, and thus making the most of the interdisciplinary character of our research.Unfortunately we are unable to make more specific recommendations such as language used in research articles, but according to the results obtained in another study (see Rovira-Esteva and Orero, forthcoming) we can offer some guiding data specific to T&I Studies concerning length of contributions (which have 17.7 pages on average), order or contributors' names (no convention settled, therefore it would be better to explicitly state the criterion in a footnote) and number of authors (1.18 per article on average).
Under the current state of affairs it is not enough to carry out quality research, since to obtain a positive assessment one has to be able to prove it, which demands an expert use of bibliometric tools.Consequently, one of the main contributions of this article is the list of journals in appendix 5.It is hoped they will constitute a new and useful tool both for researchers and evaluators involved in research quality assessment within T&I Adequacy of the contribution.o Indexing.Importance of articles published in national or international indexed journals.o Innovative character of the contribution.Contributions should represent a step forward in the field of knowledge, or present some kind of methodological innovation.o Methodological approach.Analytic and comparative works are preferred to descriptive.o Number of authors.A high number of authors can reduce the assessment of a contribution, unless it deals with a highly complex subject or it is justified by the length of the article.o Originality of the contribution.Thematically or conceptually reiterative or redundant works will not be considered (unless they include new and innovative elements).o Publisher's prestige.o Quality of the contribution.
concrete data and instructions how to apply them.What is considered average number of authors in T&I?Since what is considered normal and adequate may vary considerably from one discipline to another.According to Rovira-Esteva & Orero (forthcoming), the average number of authors per article within T&I Studies is 1.18.

Table 3 .
Journal indexes and databases taken into account by agencies

Table 6 .
List of T&I journals indexed in those indexes and databases taken into account by Spanish quality assessment agencies.