A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms

André M. Carrington*, Paul W. Fieguth, Hammad Qazi, Andreas Holzinger, Helen H. Chen, Franz Mayr, Douglas G. Manuel

*Corresponding author for this work

Research output: Contribution to journalArticle

Abstract

Background: In classification and diagnostic testing, the receiver-operator characteristic (ROC) plot and the area under the ROC curve (AUC) describe how an adjustable threshold causes changes in two types of error: false positives and false negatives. Only part of the ROC curve and AUC are informative however when they are used with imbalanced data. Hence, alternatives to the AUC have been proposed, such as the partial AUC and the area under the precision-recall curve. However, these alternatives cannot be as fully interpreted as the AUC, in part because they ignore some information about actual negatives. Methods: We derive and propose a new concordant partial AUC and a new partial c statistic for ROC data - as foundational measures and methods to help understand and explain parts of the ROC plot and AUC. Our partial measures are continuous and discrete versions of the same measure, are derived from the AUC and c statistic respectively, are validated as equal to each other, and validated as equal in summation to whole measures where expected. Our partial measures are tested for validity on a classic ROC example from Fawcett, a variation thereof, and two real-life benchmark data sets in breast cancer: the Wisconsin and Ljubljana data sets. Interpretation of an example is then provided. Results: Results show the expected equalities between our new partial measures and the existing whole measures. The example interpretation illustrates the need for our newly derived partial measures. Conclusions: The concordant partial area under the ROC curve was proposed and unlike previous partial measure alternatives, it maintains the characteristics of the AUC. The first partial c statistic for ROC plots was also proposed as an unbiased interpretation for part of an ROC curve. The expected equalities among and between our newly derived partial measures and their existing full measure counterparts are confirmed. These measures may be used with any data set but this paper focuses on imbalanced data with low prevalence. Future work: Future work with our proposed measures may: demonstrate their value for imbalanced data with high prevalence, compare them to other measures not based on areas; and combine them with other ROC measures and techniques.

Original languageEnglish
Article number4
JournalBMC Medical Informatics and Decision Making
Volume20
Issue number1
DOIs
Publication statusPublished - 6 Jan 2020

Keywords

  • Area under the ROC curve
  • C statistic
  • Classification
  • Concordance
  • Diagnostic testing
  • Explainable artificial intelligence
  • Imbalanced data
  • Partial area index
  • Prevalence
  • Receiver operating characteristic

ASJC Scopus subject areas

  • Health Policy
  • Health Informatics

Fingerprint Dive into the research topics of 'A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms'. Together they form a unique fingerprint.

Cite this