Reputation or peer review? The role of outliers
Grimaldo, Francisco (Universitat de València. Departament d'Informàtica)
Paolucci, Mario (Italian National Research Council. Institute of Cognitive Sciences and Technologies)
Sabater-Mir, Jordi (Institut d'Investigació en Intel·ligència Artificial (IIIA-CSIC))
Universitat Autònoma de Barcelona

Fecha:	2018
Resumen:	We present an agent-based model of paper publication and consumption that allows to study the effect of two different evaluation mechanisms, peer review and reputation, on the quality of the manuscripts accessed by a scientific community. The model was empirically calibrated on two data sets, mono- and multi-disciplinary. Our results point out that disciplinary settings differ in the rapidity with which they deal with extreme events-papers that have an extremely high quality, that we call outliers. In the mono-disciplinary case, reputation is better than traditional peer review to optimize the quality of papers read by researchers. In the multi-disciplinary case, if the quality landscape is relatively flat, a reputation system also performs better. In the presence of outliers, peer review is more effective. Our simulation suggests that a reputation system could perform better than peer review as a scientific information filter for quality except when research is multi-disciplinary and in a field where outliers exist.
Nota:	Altres ajuts: We are grateful to Aron Szekely for his support in our discussions, and to Nicolas Payette for helping us with Netlogo code. This work was partially supported by the COST Action TD1306 "New frontiers of peer review" (www.peere.org), by the FuturICT 2.0 (www.futurict2.eu) project funded by the FLAG-ERA JCT 2017, by the Spanish Ministry of Science and Innovation Project TIN2015-66972-C5-5-R and by the University of Valencia under grant UV-INV_EPDI17-548224. We acknowledge two anonymous reviewers for their generous and careful reading of our first draft and for sharing ideas on current and future work.
Derechos:	Aquest document està subjecte a una llicència d'ús Creative Commons. Es permet la reproducció total o parcial, la distribució, la comunicació pública de l'obra i la creació d'obres derivades, fins i tot amb finalitats comercials, sempre i quan es reconegui l'autoria de l'obra original.
Lengua:	Anglès
Documento:	Article ; recerca ; Versió publicada
Materia:	Peer review ; Reputation ; Agent-based simulation ; Multi-disciplinary science ; Outliers ; Information filter
Publicado en:	Scientometrics, Vol. 116 (july 2018) , p. 1421-1438

DOI: 10.1007/s11192-018-2826-3
PMID: 30147204

18 p, 812.2 KB

El registro aparece en las colecciones:
Artículos > Artículos de investigación
Artículos > Artículos publicados