FAA Office of Aerospace Medicine
Civil Aerospace Medical Institute

Report No: DOT/FAA/AM-15/7

Title and Subtitle: Air Traffic Scenarios Test: Will Modifying the Instructions Change Performance?

Report Date: April 2015

Authors: Scarborough AL, Bleckley MK

Abstract: The Air Traffic Scenarios Test (ATST) is a low-fidelity radar simulation that is part of the Air Traffic Training and Test (AT-SAT) battery. Past Federal Aviation Administration (FAA) and military research has shown the ATST to be a strong predictor of training success for air traffic control specialists (ATCSs). Because the ATST was such a strong predictor of controller performance, the U.S. Air Force (USAF) was interested in using the ATST as part of its entrance testing selection process for air traffic. However, the current version of the ATST takes about an hour and a half to complete, and is too long for the USAF to use in its present form. In agreement with the FAA, the USAF requested that the FAA shorten (can be completed in less than an hour) the ATST without jeopardizing its reliability and validity. Two studies were conducted to examine whether it would be possible to shorten the ATST.

In the preliminary study (Study 1), we analyzed data from 391 job applicants that took the AT-SAT during a two-month period. We examined scores for efficiency of aircraft movement to destination, safety or maintenance of aircraft separation, and procedural accuracy. Overall, performance changed across trials. Because performance did not consistently increase or decrease across trials, we cannot simply reduce the number of trials without changing the validity of the test. We thought it was possible that, with a more structured approach to designing instructions, performance on the trials may become more consistent, thus allowing us to reduce the number of trials to yield a reduction in total testing time.

For Study 2, new interactive instructions were developed. Data were collected from Air Force recruits who were naive about the ATST. A final sample size of 193 cases was used in the data analysis. Participants spent 6 minutes longer on the modified instructions than on the standard ones; however, both times were substantially less than the currently allotted time of 25 minutes. ATST performance for both groups was poor. Although a profile analysis of the two instruction types showed no significant difference, the multivariate tests for trial revealed a change across trials for the three measures.

Additionally, when we compared the results of this study with our original sample of ATCS applicants, we found large differences in performance that might indicate that the USAF recruits were physically tired and had less motivation to perform well than did our applicants; accordingly, their performance was poorer. In sum, the results of these studies were inconclusive with regard to whether a shorter ATST would be viable as a controller selection instrument. Additional research will be necessary to determine if the modified instructions are beneficial and whether fewer scenarios will maintain similar or improved predictive validity for ATST.

Key Words: Air Traffic Scenarios Test, Air Traffic Selection, Improving Test Instructions

No. of Pages: 19