We address the general problem of finding suitable evaluation measures
for classification systems. To this end, we adopt an axiomatic
approach, i.e., we discuss a number of properties ("axioms") that an
evaluation measure for classification should arguably satisfy. We
start our analysis by addressing binary classification. We show that
F1, nowadays considered a standard measure for the evaluation of
binary classification systems, does not comply with a number of them,
and should thus be considered unsatisfactory. We go on to discuss an
alternative, simple evaluation measure for binary classification, that
we call K, and show that it instead satisfies all the previously
proposed axioms. We thus argue that researchers and practitioners
should replace F1 with K in their everyday binary classification
practice. We carry on our analysis by showing that K can be smoothly
extended to deal with single-label multi-class classification,
cost-sensitive classification, and ordinal classification.
Fabrizio Sebastiani has been a Principal Scientist at the Qatar
Computing Research Institute since July 2014; previously he was a
Senior Researcher at the Institute for the Science and Technologies of
Information of the National Council of Research (ISTI-CNR), Italy,
from which he is currently on leave, and an Associate Professor at the
Department of Pure and Applied Mathematics of the University of
Padova, Italy. He is a Senior Associate Editor for ACM Transactions
on Information Systems (ACM Press) and an Associate Editor for IEEE
Transactions on Affective Computing (IEEE Press) and AI Communications
(IOS Press), and a member of the Editorial Boards of Information
Retrieval (Kluwer) and Foundations and Trends in Information Retrieval
(Now Publishers); of the latter he is also a past co-Editor-in-Chief
and Founding Editor. He has been the General Chair of ECIR 2003 and
SPIRE 2011, and a Program co-Chair of ACM SIGIR 2008 and ECDL 2010; he
is the appointed General co-Chair of ACM SIGIR 2016. Fabrizio's
research interests lie at the intersection of information retrieval,
machine learning, and human language technologies, with particular
emphasis on text mining, text classification, information extraction
from text, opinion mining, quantification, and their applications in
fields such as medical informatics, market research, and customer
relationships management.