Efficient evaluation of classification and recognition systems
- 25 August 2005
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 1 (10636919) , 50
- https://doi.org/10.1109/cvpr.2001.990455
Abstract
In this paper, a new framework for evaluating a variety of computer vision systems and components is introduced. This framework is particularly well suited for domains such as classification or recognition systems, where blind application of the i.i.d. assumption would reduce an evaluation's accuracy, such as with classification or recognition systems. With few exceptions, most previous work on vision system evaluation does not include confidence intervals, since they are difficult to calculate, and are often coupled with strict requirements. We show how a set of previously overlooked replicate statistics tools can be used to obtain tighter confidence intervals of evaluation estimates while simultaneously reducing the amount of data and computation required to reach such sound evaluatory conclusions. In the included application of the new methodology, the well-known FERET face recognition system evaluation is extended to incorporate standard errors and confidence intervals.Keywords
This publication has 9 references indexed in Scilit:
- Practical Methods for Design and Analysis of Complex SurveysPublished by Wiley ,2003
- The FERET evaluation methodology for face-recognition algorithmsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2000
- Performance Characterization in Computer VisionPublished by Springer Nature ,2000
- Performance assessment through bootstrapPublished by Institute of Electrical and Electronics Engineers (IEEE) ,1997
- Variance estimation for complex surveys using replication techniquesStatistical Methods in Medical Research, 1996
- Ignorance, myopia, and naiveté in computer vision systemsCVGIP: Image Understanding, 1991
- Inference From Stratified Samples: Properties of the Linearization, Jackknife and Balanced Repeated Replication MethodsThe Annals of Statistics, 1981
- Constructing Orthogonal Replications for Variance EstimationJournal of the American Statistical Association, 1975
- The Design of Optimum Multifactorial ExperimentsBiometrika, 1946