AUC: a misleading measure of the performance of predictive distribution models
Top Cited Papers
- 14 September 2007
- journal article
- Published by Wiley in Global Ecology and Biogeography
- Vol. 17 (2) , 145-151
- https://doi.org/10.1111/j.1466-8238.2007.00358.x
Abstract
The area under the receiver operating characteristic (ROC) curve, known as the AUC, is currently considered to be the standard method to assess the accuracy of predictive distribution models. It avoids the supposed subjectivity in the threshold selection process, when continuous probability derived scores are converted to a binary presence–absence variable, by summarizing overall model performance over all possible thresholds. In this manuscript we review some of the features of this measure and bring into question its reliability as a comparative measure of accuracy between model results. We do not recommend using AUC for five reasons: (1) it ignores the predicted probability values and the goodness‐of‐fit of the model; (2) it summarises the test performance over regions of the ROC space in which one would rarely operate; (3) it weights omission and commission errors equally; (4) it does not give information about the spatial distribution of model errors; and, most importantly, (5) the total extent to which models are carried out highly influences the rate of well‐predicted absences and the AUC scores.Keywords
This publication has 44 references indexed in Scilit:
- Constrain to perform: Regularization of habitat modelsEcological Modelling, 2006
- Selecting thresholds of occurrence in the prediction of species distributionsEcography, 2005
- Improving the Quality of Distribution Models for Conservation by Addressing Shortcomings in the Field Collection of Training DataConservation Biology, 2003
- Niche Modeling Perspective on Geographic Range Predictions in the Marine Environment Using a Machine-learning AlgorithmOceanography, 2003
- Receiver Operating Characteristic MethodologyJournal of the American Statistical Association, 2000
- Comparing classifiers when the misallocation costs are uncertainPattern Recognition, 1999
- Predictive Performance of the Binary Logit Model in Unbalanced SamplesJournal of the Royal Statistical Society: Series D (The Statistician), 1999
- The use of the area under the ROC curve in the evaluation of machine learning algorithmsPattern Recognition, 1997
- A review of methods for the assessment of prediction errors in conservation presence/absence modelsEnvironmental Conservation, 1997
- Verification, Validation, and Confirmation of Numerical Models in the Earth SciencesScience, 1994