Web of Science: 14 cites, Scopus: 22 cites, Google Scholar: cites,
Why Cohen's Kappa should be avoided as performance measure in classification
Delgado de la Torre, Rosario (Universitat Autònoma de Barcelona. Departament de Matemàtiques)
Tibau, Xavier Andoni (Universitat Autònoma de Barcelona. Departament de Matemàtiques)

Data: 2019
Resum: We show that Cohen's Kappa and Matthews Correlation Coefficient (MCC), both extended and contrasted measures of performance in multi-class classification, are correlated in most situations, albeit can differ in others. Indeed, although in the symmetric case both match, we consider different unbalanced situations in which Kappa exhibits an undesired behaviour, i. e. a worse classifier gets higher Kappa score, differing qualitatively from that of MCC. The debate about the incoherence in the behaviour of Kappa revolves around the convenience, or not, of using a relative metric, which makes the interpretation of its values difficult. We extend these concerns by showing that its pitfalls can go even further. Through experimentation, we present a novel approach to this topic. We carry on a comprehensive study that identifies an scenario in which the contradictory behaviour among MCC and Kappa emerges. Specifically, we find out that when there is a decrease to zero of the entropy of the elements out of the diagonal of the confusion matrix associated to a classifier, the discrepancy between Kappa and MCC rise, pointing to an anomalous performance of the former. We believe that this finding disables Kappa to be used in general as a performance measure to compare classifiers.
Nota: Número d'acord de subvenció MICIU/PGC2018-097848-B-I0
Drets: Aquest document està subjecte a una llicència d'ús Creative Commons. Es permet la reproducció total o parcial, la distribució, la comunicació pública de l'obra i la creació d'obres derivades, fins i tot amb finalitats comercials, sempre i quan es reconegui l'autoria de l'obra original. Creative Commons
Llengua: Anglès
Document: article ; recerca ; publishedVersion
Matèria: Classification ; Datasets as topic ; Models theoretical ; Reproducibility of results ; Supervised machine learning
Publicat a: PloS one, Vol. 14, Issue 9 (September 2019) , art. e0222916, ISSN 1932-6203

DOI: 10.1371/journal.pone.0222916
PMID: 31557204


26 p, 1.6 MB

El registre apareix a les col·leccions:
Articles > Articles de recerca
Articles > Articles publicats

 Registre creat el 2020-06-03, darrera modificació el 2021-04-05



   Favorit i Compartir