Assessing and Improving Evaluation of Aircrew Performance
- 1 July 2002
- journal article
- Published by Taylor & Francis in The International Journal of Aviation Psychology
- Vol. 12 (3) , 223-240
- https://doi.org/10.1207/s15327108ijap1203_3
Abstract
Evaluation of aircrew performance serves the critical functions of assessing the qualifications of individual pilots and, under newer proficiency-based training programs, providing data for modifying training programs. We apply psychometric methods to assessing and improving the quality of evaluation of aircrew performance. Quality evaluations require human judges to recognize and discriminate changes in performance levels (sensitivity) and map these observations onto the appropriate grade-scale values (accuracy). We define statistical measures for both of these properties. A distinction is made between referent-rater reliability (RRR) and traditional interrater reliability, and we argue that RRR more meaningfully measures evaluators' grading performance and has clearer training implications. We also discuss the implementation of training and calibration sessions that are intended to help improve evaluators' ratings of aircrew performance. We offer several practical guidelines for designing and conducting these sessions.Keywords
This publication has 9 references indexed in Scilit:
- Performance Measurement in Distributed Environments: Initial Results and Implications for TrainingMilitary Psychology, 1999
- PERFORMANCE EVALUATION IN WORK SETTINGSAnnual Review of Psychology, 1998
- Pilot Selection in the Norwegian Air Force: A Validation and Meta-Analysis of the Test BatteryThe International Journal of Aviation Psychology, 1998
- Application of Psychometrics to the Calibration of Air Carrier EvaluatorsProceedings of the Human Factors and Ergonomics Society Annual Meeting, 1997
- Large Sample Validations of Three General Predictors of Pilot Training SuccessThe International Journal of Aviation Psychology, 1997
- Analyzing Team Performance: In the Eye of the Beholder?Military Psychology, 1996
- Pilot Selection Batteries: Shortcomings and PerspectivesThe International Journal of Aviation Psychology, 1996
- On the Relation Between Personality and Job Performance of Airline PilotsThe International Journal of Aviation Psychology, 1996
- Situation Awareness in Team Performance: Implications for Measurement and TrainingHuman Factors: The Journal of the Human Factors and Ergonomics Society, 1995