Office of Aerospace Medicine Technical Reports
Civil Aerospace Medical Institute
PUBLICATIONS
AVIATION MEDICINE REPORTS
Report No: DOT/FAA/AM-00/2
Title and Subtitle: Measuring Air Traffic Controller Performance in a High-Fidelity Simulation.
Report Date: January 2000
Authors: Manning, C.A. (Editor)
Abstract: In the summer of 1997, the Air Traffic Selection and Training (AT-SAT) High Fidelity Simulation Study was conducted at the FAA Academy in Oklahoma City, OK. The purpose of the study was to test the performance of 107 operational en route controllers during 2½ days of simulations. The performance of these controllers during the high-fidelity simulations was compared with their performance on two medium-fidelity performance measures to assess the construct validity of the latter measures to serve as criteria against which to validate a set of selection tests. The reports included in this document describe the high- fidelity simulation exercise, the development of performance measures utilized during the exercise, and the interrelationships between the performance measures.
The first report describes the development of a work sample approach to capturing air traffic controller performance, and establishes that high fidelity performance measures can adequately reflect the performance of the controller. The work sample was developed in an environment that simulated as nearly as possible the actual conditions existing in the controller's job, but was conducted in a "generic" airspace. Scenario development included the most important tasks from the task-based job analysis developed for the AT-SAT project. Sufficient time was provided for participating controllers to learn the airspace and procedures and demonstrate their knowledge through 1) a multiple choice test of airspace knowledge and 2) running 8 practice scenarios. Performance was measured by 1) an over-the-shoulder (OTS) rating scale, 2) counts of mistakes, 3) counts of actions that would be required to move aircraft from the sector at the end of the scenario, and 4) statistics derived from aircraft positions and controller/pilot data entries recorded for the simulation.
The second report used measures collected during the high-fidelity simulation study to predict the overall OTS performance rating. It was found that a model that included both counts of mistakes and the computer-derived performance measures predicted the OTS rating reasonably well, while a model containing only the computer-derived measures did not. Remaining actions did not contribute to the prediction of the OTS rating in addition to the contribution provided by the other types of measures.
Key Words: Air Traffic Control, Performance Measurement, Simulation
No. of Pages: 37
