Aviation Safety: Research Supports Limited Use of Personal Computer Aviation Training Devices for Pilots (Letter Report, 07/12/1999, GAO/RCED-99-143). Last year, 621 people died in about 1,900 general aviation accidents. The National Transportation Safety Board estimates that 87 percent of all fatal general aviation accidents are caused by pilot error, especially when pilots who do not have appropriate instrument training fly with poor visibility, such as during bad weather. To reduce general aviation accidents, the Federal Aviation Administration (FAA) has been exploring ways to strengthen training for general aviation pilots. One possible enhancement is the use of special personal computers and software called personal computer-based aviation training devices, which can be used for up to 10 hours of instrument training. FAA's decision to allow the use of these devices has sparked debate. Some contend that pilots trained with these devices will actually be less skilled. Others argue that pilots trained with the devices are better trained at lower cost. This report (1) describes the process and information FAA used to approve the use of personal computer devices for 10 hours of instrument training and (2) discusses what is known about the training effectiveness of these devices and their long-term impact on a pilot's ability to fly safely. FAA's decision to allow the use of computer-based devices for instrument flight training took more than six years to be finalized and was based on two major research studies and input from aviation industry representatives. The two major research studies generally support the use of computer-based devices for training. Similarly, most experts GAO spoke with saw some training value from the use of devices and did not believe that they were likely to degrade aviation safety. GAO recommends that FAA gather more information on the long-term safety issues associated with computer-based devices and previously approved flight training devices. --------------------------- Indexing Terms ----------------------------- REPORTNUM: RCED-99-143 TITLE: Aviation Safety: Research Supports Limited Use of Personal Computer Aviation Training Devices for Pilots DATE: 07/12/1999 SUBJECT: Flight training Aircraft pilots Computer assisted instruction Aviation Transportation research Transportation safety Data collection Safety standards ****************************************************************** ** This file contains an ASCII representation of the text of a ** ** GAO report. This text was extracted from a PDF file. ** ** Delineations within the text indicating chapter titles, ** ** headings, and bullets have not been preserved, and in some ** ** cases heading text has been incorrectly merged into ** ** body text in the adjacent column. Graphic images have ** ** not been reproduced, but figure captions are included. ** ** Tables are included, but column deliniations have not been ** ** preserved. ** ** ** ** Please see the PDF (Portable Document Format) file, when ** ** available, for a complete electronic file of the printed ** ** document's contents. ** ** ** ** A printed copy of this report may be obtained from the GAO ** ** Document Distribution Center. For further details, please ** ** send an e-mail message to: ** ** ** ** ** ** ** ** with the message 'info' in the body. ** ****************************************************************** United States General Accounting Office GAO Report to Congressional Requesters July 1999 AVIATION SAFETY Research Supports Limited Use of Personal Computer Aviation Training Devices for Pilots GAO/RCED-99-143 GAO United States General Accounting Office Washington, D.C. 20548 Resources, Community, and Economic Development Division B-280735 July 12, 1999 The Honorable John J. Duncan Chairman, Subcommittee on Aviation Committee on Transportation and Infrastructure House of Representatives The Honorable Thomas W. Ewing House of Representatives In 1998, general aviation had 1,907 accidents, with 621 fatalities. The National Transportation Safety Board estimates that 87 percent of all fatal general aviation accidents are caused by pilot error, especially when pilots who do not have appropriate instrument training fly when visibility is poor, such as during bad weather. To reduce the occurrence of general aviation accidents, the Federal Aviation Administration (FAA) has been exploring a number of means to enhance the training of general aviation pilots. One possible enhancement is the use of new technologies for the training that occurs on the ground. For over 40 years, general aviation student pilots have used flight training devices to help them learn how to fly using an aircraft's instruments alone. These flight-training devices resemble an aircraft's cockpit and are often constructed with actual airplane instruments; they can be used by student pilots to substitute for up to 20 of the 40 hours of airplane training required by FAA to obtain an instrument rating. The instrument rating permits a pilot to fly when visibility is poor. In May 1997, FAA also approved the use of special personal computers, controls, and software called personal computer-based aviation training devices (PCATD), which can be used for up to 10 hours of instrument training.1 FAA's decision to allow the use of PCATDs has sparked debate. Some assert that pilots trained with these devices will be less skilled, thereby compromising aviation safety. Others argue that pilots trained with the devices are actually better trained at lower cost. As a result of this controversy, you asked us to (1) describe the process and information FAA used in deciding to approve the use of personal 1"Personal Computer-Based Aviation Training Device," or PCATD, is FAA's term for computer-based devices that have been approved for use, or qualified for credit hour training, by FAA officials. FAA requires PCATDs to be qualified, approved, and used in connection with an integrated ground and flight instrument training curriculum. Training using PCATDs should include the procedural tasks listed in FAA's Advisory Circular 61-126. In this report, we refer to these devices as computer-based devices before they were approved for use in training and PCATDs once the devices were approved for use by FAA. PCATDs describes one of the four FAA-approved devices. Page 1 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 computer devices for 10 hours of instrument training and (2) discuss what is known about the training effectiveness of these devices and their long-term impact on a pilot's ability to fly safely. To respond to these objectives, we interviewed federal officials and others familiar with FAA's decision and reviewed FAA's supporting documentation; conducted a comprehensive literature search on the use of these and similar computer-based devices, on instrument training, and on general aviation safety; reviewed about 700 studies and articles; analyzed and summarized the most relevant data-based literature; interviewed government, academic, and private sector flight instruction experts on the use of the devices; and identified other issues related to the devices' potential impacts on aviation safety. Results in Brief FAA's decision to allow the use of computer-based devices for instrument flight training took over 6 years to be made final and was based on two major research studies, FAA's professional judgment, and input from aviation industry representatives. Because the two studies did not address the appropriate number of hours of training on these devices, FAA's decision to allow 10 hours was based on its professional judgment and industry input. FAA had earlier relied on its professional judgment, rather than empirical studies, to support its approval of up to 20 credit hours for the devices now known as flight training devices. A University of Illinois research study is planned to assess the appropriate number of credit hours for the use of these devices in an instrument training course. The two major research studies generally support the use of computer-based devices for training. Despite some methodological limitations, these are the most complete controlled studies to date on the training effectiveness of computer-based devices. One study shows that the use of computer-based devices may modestly reduce the training time spent in an airplane, while the other shows that the training effects of computer-based devices and previously approved flight training devices may not differ greatly. Other studies we reviewed also generally supported the use of computer-based devices in training. Similarly, the majority of the experts we interviewed saw some training value from the use of the devices and did not believe that they were likely to reduce aviation safety. Although most experts did not speculate on the appropriate number of credit hours that should be granted for the devices, several disagreed with FAA's decision to allow any credit hours. We found no empirical evidence, however, on the long-term safety impact of the devices, their potential safety benefits, or the long-term safety impact or benefits of the currently Page 2 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 approved flight training devices. Moreover, FAA does not currently collect the data needed to conduct future research on a possible link between the use of the devices and pilots' long-term safety records. This report recommends that FAA gather additional information on the long-term safety issues associated with computer-based devices and previously approved flight training devices. FAA will soon revise and computerize its pilot application form. This form could collect additional information at minimal cost on the use of pilot training devices and would facilitate future research. Background To establish training requirements for pilots, FAA issues regulations for pilots, pilot schools, instructors, and pilot training equipment manufacturers, and provides guidance through advisory circulars, handbooks, and other types of informational material.2 Within FAA, several organizations regulate pilot training. For example, the Flight Standards Office is responsible for certifying and overseeing pilot schools, pilots, and pilot training equipment. Likewise, FAA's Civil Aeromedical Institute (CAMI) studies human performance factors in aviation, among other issues. Finally, FAA's National Simulator Program studies and evaluates the more sophisticated simulator equipment used in training pilots. Beginning students first learn to fly under FAA's visual flight rules, which generally require a minimum visibility of 3 miles, although most new pilots fly with much better visibility. Pilots then must obtain an instrument rating to fly in conditions of reduced visibility, such as in clouds or in poor weather. Flying when visibility is poor is more difficult because a pilot loses reference to the outside horizon and must rely completely on instruments to know the plane's spatial orientation. Instrument training is important to aviation safety because the vast majority of accidents involve poor weather and poor visibility. When a pilot without instrument training mistakenly flies into conditions of poor visibility, disorientation and loss of control can occur quickly; many fatal weather-related accidents occur when a pilot who is flying according to visual flight rules accidentally encounters poor visibility conditions that are beyond his or her abilities. According to experts, good instrument skills are critical to flying safely through bad weather. 2The regulation of pilots, pilot schools, and pilot training equipment by FAA is governed by title 14, C.F.R. parts 61, 91, 121, 135, and 141. Page 3 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 An instrument rating is considered difficult to obtain and normally requires additional training, an FAA written exam, and a practical test-called a "check ride"-with an FAA-designated examiner in an aircraft. Before taking these exams, pilots must take an additional 35 to 40 hours of training, which may include up to 20 hours in a flight simulator or flight training device, with the remaining time in an aircraft.3 However, pilots typically take about 70 hours of training in an aircraft or training device before they are proficient enough to pass the tests needed to obtain their instrument rating. In May 1997, FAA issued an advisory circular approving the use of PCATDs for training credit toward the instrument rating.4 These aviation training devices are personal computers with specialized software and add-on equipment, including separate simulated radio and navigational controls to the right of the computer monitor, a throttle, a steering yoke under the computer screen, and rudder pedals on the floor. A fully equipped FAA-approved PCATD costs from $5,000 to $10,000. In comparison, a previously approved flight training device costs from $40,000 to $90,000.5 When training with either a PCATD or a flight training device, the instructor usually sits with a student, sometimes at a separate personal computer console where the instructor can plan, monitor, and control a student's flight. Figure 1 shows an FAA-approved PCATD. 3There are two types of pilot training schools. One is an FAA-approved school (Part 141) and others are nonapproved schools that still must meet FAA training requirements (Part 61). An FAA-approved school may be authorized to give its graduates written exams and check rides. In addition, an FAA-approved school can qualify pilots with fewer hours than can a nonapproved school. Many of the nonapproved pilot schools find it impractical to qualify for FAA certificates, but 85 percent of all instrument ratings are granted for the training conducted at these schools. 4FAA Advisory Circular 61-126. 5In contrast, a full-motion simulator used by airlines to train commercial pilots can cost up to $20 million. Page 4 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 Figure 1: Personal Computer-Based Aviation Training Device Source: Aviation Supplies and Academics, Inc. Page 5 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 FAA's advisory circular spells out the conditions that an aviation training device must meet in order to be qualified by FAA as a PCATD for use in satisfying the instrument rating requirements. The advisory circular's guidelines are not mandatory and contain a voluntary reporting mechanism requesting information on students' PCATD use. FAA could have used its rule-making authority to approve PCATD use. It did not do so for several reasons, according to Flight Standards officials. First, rules are mandatory and advisory circulars are not, making any significant changes to mandatory rules more difficult and time-consuming than making any significant changes to advisory circulars. Second, in the past, the qualification of ground flight training devices in general aviation was handled by issuing advisory circulars. Finally, FAA officials wanted to facilitate the rapid introduction of the new PCATD technology without going through a lengthy rule- making process. Once FAA has qualified a PCATD-including its hardware, software, and accompanying equipment-the device can be used by a flight school or an individual flight instructor in an integrated ground and flight training curriculum for up to 10 hours of flight instruction toward an instrument rating. FAA's Decision to Allow FAA's decision to allow the use of computer-based devices and to grant the Use of Computer-Based credit hours evolved over many years. Since the 1980s, FAA had been Devices Was Based on Two evaluating various designs of PCATDs but did not consider these early Studies, FAA's Professional devices to be sophisticated enough for general use in instrument flight Judgment, and Industry courses. As computer equipment advanced and training software was refined, FAA in 1991 began to seriously consider proposals from Input manufacturers and aviation schools to allow the use of computer- based devices for instrument flight training. Initially, FAA officials considered allowing the use of the devices for general instructional purposes only and not for students to earn flight hours toward an instrument rating. In 1992, Flight Standards officials proposed developing a research project to obtain information on the use of the devices because they were concerned that they needed empirical data to support the introduction of this new training technology. According to Flight Standards officials, FAA has never sponsored a study that evaluated the performance of previously approved flight training devices, which remain in wide use. FAA relied on its professional judgment rather than empirical studies to support the approval of up to 20 credit hours for flight training devices. Later in 1992, FAA provided a grant to Embry-Riddle Aeronautical University to study the use of computer-based devices. A principle Page 6 GAO/RCED- 99-143 Personal Computer-Based Aviation Training Device B-280735 objective of the study was to measure the transfer of instrument training knowledge to an aircraft for three groups of students trained using two different computer-based training devices and one approved flight training device. According to Flight Standards officials, the Embry-Riddle study was designed to provide them with information on whether the devices were useful and effective. In June 1993, while the Embry-Riddle research was ongoing, CAMI's Human Factors Research Laboratory submitted a research proposal to expand and participate in the study. This detailed approach was not approved, according to a Flight Standards official, because FAA at the time did not have the funds to support it. For the next year, CAMI officials continued to help explore possible research methodologies for qualifying the devices in a structured training curriculum that would provide some insight into determining if credit hours should be granted and what tasks the devices are best suited for in instrument training. However, at that time, FAA was not proposing to grant flight training credit hours for training on computer-based devices. When the Embry-Riddle study was completed in 1994, some FAA officials outside of Flight Standards believed that the study had significant limitations, was not properly designed, and did not fully answer important questions on whether computer-based devices should be approved. CAMI officials stated that the failure to include a control group that received training only in the aircraft seriously limited the usefulness of the study. In addition, officials in FAA's National Simulator Program stated that no meaningful conclusion could be drawn from the research. In June 1994, FAA sponsored a meeting of manufacturers and others in the aviation training industry to obtain their views on FAA's current and future direction in approving the use of computer-based devices. In addition, starting in 1994, FAA sponsored further research through a series of grants to the University of Illinois to study the training effectiveness of the devices and to answer many of the questions and concerns about their use that remained following the Embry-Riddle study. While FAA was evaluating the qualifications of computer-based devices for instrument training without credit hours, it received requests to allow some of the training hours done on the equipment to be credited toward the minimum hours of flight time in instrument training. Some manufacturers, for example, wanted FAA to consider granting credit hours in order to make their equipment more credible to their customers. Furthermore, in May 1995, the Aircraft Owners and Pilots Association's Air Safety Foundation formally petitioned FAA to allow training on Page 7 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 computer-based devices to count for training time toward an instrument rating and proposed a syllabus illustrating the integration of the devices into an instrument flight course. Although neither the Embry-Riddle nor the University of Illinois study addressed any appropriate number of flight training credit hours, FAA officials used their experience and professional judgment to support their decision and saw it as a reasonable compromise between the various proposals. Furthermore, FAA officials decided to add a voluntary reporting mechanism to the draft advisory circular designed to gather information to validate the approval and use of the devices with feedback from users. Finally, FAA officials told us that the required check ride by an FAA-designated examiner after a student's training would ensure that all instrument students were qualified to receive the instrument rating, no matter what equipment they used during training. Several flight training experts we interviewed, however, told us that the check ride is an imperfect measure of a pilot's ability, should not be seen as assurance of proper training, and is instead a snapshot of a pilot on a single day. These experts believe assurances on the quality of training devices, a quality curriculum, capable instructors, consistent testing by FAA- designated flight examiners, and long-term research on pilot safety and training are needed. In August 1995, FAA sponsored an additional meeting of aviation industry officials and presented a draft advisory circular that proposed, for the first time, granting 10 flight training credit hours. The draft advisory circular and all significant changes to the draft were submitted to over 60 groups, organizations, and other interested parties. For the next 12 months, FAA officials received numerous comments on the use of computer-based devices. During this time, several aviation industry officials suggested that FAA delay or withhold approval of a final advisory circular pending more research and the completion of the University of Illinois study. In September 1996, FAA decided to delay approving the advisory circular until after the University of Illinois study was completed. In November 1996, the University of Illinois issued its final report and in March 1997 briefed FAA officials on the results. The study concluded, in part, that computer-based devices can provide a worthwhile training benefit and can save a small portion of the aircraft time that would otherwise be needed. In January 1997, the National Air Transportation Association, after extensive discussion, recommended that FAA approve a draft advisory circular, including flight training credit hours for computer- based devices, as well as other recommendations. Page 8 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 In May 1997, FAA issued Advisory Circular 61-126, which created a separate category of training device-PCATD-rather than integrating PCATDs into the "ground training device" category already approved for credit hours in instrument training. These existing ground training devices were renamed flight training devices (FTD) to distinguish them from PCATDs. Older models of the new FTD category were grandfathered in and automatically approved by FAA for instrument credit hours. According to Flight Standards officials, the decision to approve 10 hours of flight credit was based on (1) the general utility of computer-based devices in instrument training, as shown by the Embry-Riddle and University of Illinois studies; (2) a wide variety of proposals, ranging from zero hours of credit to 20 hours; and (3) the input from manufacturers that they needed credit hours to sell their equipment. Two additional research studies are planned: One will try to assess the appropriate number of credit hours for PCATD use in an instrument course; the second will investigate the use of PCATDs to maintain instrument skills.6 Over the next several months, FAA officials qualified four PCATD devices, which included three software programs and several other manufacturers of controls, while finding an equal number unacceptable. As of June 1999, no information had been sent to FAA under the advisory circular's voluntary reporting mechanism on the use of PCATDs. An official of the National Association of Flight Instructors estimated that very few of the 200 Part 141 flight schools now use PCATDs for instrument instruction. 6This University of Illinois research study is likely to be funded by the National Aeronautics and Space Administration's Ames Research Laboratory, which investigates human factors issues and funds research investigating general and commercial aviation safety. The study proposes to investigate, over 3 years, the progress of three groups of instrument students using PCATDs for 5, 10, and 15 hours of training in an instrument training course. A control group would receive all training in an airplane. The use of flight training devices will not be included in the study. An additional study is now examining the training value of PCATDs for maintaining pilots' skills and for substituting these devices for aircraft experience in maintaining instrument skills. According to a University of Illinois researcher, the airplane performance of 100 current instrument pilots will be tested and compared across four groups: A control group will receive no practice in instrument flight during a 6-month period; an airplane group will perform the six required approaches and holding patterns in the aircraft; a PCATD group will perform the same maneuvers in a PCATD; and a flight training device group will perform the same maneuvers in a flight training device. The performance of pilots on the final instrument proficiency check ride at the end of the 6-month experimental period will be compared with the performance on the initial instrument proficiencies check ride. Pilots may not receive recency credit for their practice on PCATDs. To meet FAA recency regulations toward maintaining current instrument skills, pilots must make at least six instrument approaches, holding procedures, and intercepting and tracking courses every 6 months. These maneuvers must be performed under either simulated or actual flight conditions; currently, flight training devices may be used to perform all six instrument currency approaches, but PCATDs may not. Page 9 GAO/RCED-99-143 Personal Computer- Based Aviation Training Device B-280735 Research and Experts The two FAA-commissioned studies by the University of Illinois and Support Training Embry-Riddle Aeronautical University are the most complete controlled studies to date on the training effectiveness of computer-based devices. Value of PCATDs, but Despite methodological limitations, the studies show that (1) the use of Devices' Impact on computer-based devices may modestly reduce the training time spent in the airplane (Illinois) and (2) the training effects of computer-based Long-Term Safety Is devices and a previously approved flight training device may not differ Unknown greatly (Embry-Riddle). The additional studies we reviewed also supported the use of computer- based devices. None of these studies examined the issue of how many hours of training on a computer-based device should be allowed in an instrument training curriculum. Academic and other flight training experts generally believed that computer-based devices offer training value to pilots obtaining an instrument rating. Since PCATDs have only recently been approved and few pilots have used them in training, their long-term risks and benefits are not yet known. While several studies examined the training effectiveness of computer-based devices, none directly addressed their effects on general aviation safety. Safety experts we interviewed did not believe that the use of computer-based devices and the granting of credit hours pose a threat to general aviation safety by adversely affecting the skills and abilities of pilots who receive instrument training using the devices. However, several other issues that could affect the long-term ability of PCATD-trained pilots to fly safely have been raised. Research Generally The Illinois study showed that the use of computer-based devices in an Supports Use of PCATDs in instrument course may modestly reduce the training time spent in the Training airplane-26.5 hours on the device during an instrument course reduced the amount of airplane training time needed by 3.9 hours. (See. fig. 2.) Page 10 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 Figure 2: University of Illinois Study: Hours to Proficiency in an Airplane Hours Hours using PC Hours in airplane Notes: N = Number of students trained. The airplane used was a Beechcraft Sports/Sundowner. The computer-based device was an MDM FS-100, not FAA-approved as a PCATD. Source: GAO's analysis of University of Illinois data. The Embry-Riddle study provided a preliminary indication that there may not be very large differences between the training effects of the previously approved flight training devices and the recently approved computer-based devices. The training trials and time needed to reach proficiency for students using two different types of computer-based devices and for students using a previously approved flight training device are shown in figures 3 and 4. (See app. II for a more detailed summary of the two FAA-sponsored studies.) Page 11 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 Figure 3: Embry- Riddle Study: Trials to Proficiency in an Airplane Trials Training device used Notes: N = Number of students trained. The airplane used was the Mooney 20J. The Elite PC was a computer- based device, an FAA-approved PCATD. The IFT PC was a computer- based device not approved by FAA as a PCATD. The Frasca FTD is a flight-training device approved by FAA. Source: GAO's analysis of Embry-Riddle Aeronautical University data. Page 12 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 Figure 4: Embry-Riddle Study: Hours to Proficiency in an Airplane Hours Training device used Notes: N = Number of students trained. The airplane used was the Mooney 20J. The Elite PC was a computer-based device, an FAA-approved PCATD. The IFT PC was a computer-based device not approved by FAA as a PCATD. The Frasca FTD is a flight-training device approved by FAA. Source: GAO's analysis of Embry-Riddle Aeronautical University data. The additional studies we reviewed also found some value in using computer-based devices. Three studies found computer-based devices helped train beginning pilots.7 An additional study found that performance in an airplane test was better for a group of students trained on 7Gustavo Ortiz, "Effectiveness of PC-Based Flight Simulation," The International Journal of Aviation Psychology 4 (3), p. 285, 1994; Kerry A Dennis and Don Harris, "Computer-Based Simulation as an Adjunct to Ab Initio Flight Training," The International Journal of Aviation Psychology 8 (3), pp. 261-276, 1998; Jefferson M. Koonce, Steven L. Moore, and Charles J. Benton, "Initial Validation of a Basic Flight Instruction Tutoring System (BFITS)," Proceedings of the Eighth International Symposium on Aviation Psychology, Columbus, Ohio, 1995. Page 13 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 computer-based devices than for a group trained on flight training devices.8 Experts See Training Value The academic and flight training experts as well as the flight instructors in Computer-Based we interviewed generally believed that PCATDs offer training value to Devices instrument students. Experts told us that training devices, if used properly, have the potential to be more effective than airplane training for certain uses; furthermore, training experts generally agree that a low-cost simulation device may be justified economically even if the training it provides is less effective than the training provided in an airplane. In addition, commercial airline training and military research support the value of computer-based training for certain training purposes. Several experts we interviewed believed that computer-based devices would become more effective as their technology continues to improve and as features are added to enhance their training effectiveness. Some experts believe that a variety of training equipment, including computer-based devices, can offer effective training if used properly. PCATDs are usually seen by experts as most appropriate for introducing equipment and procedures and for practicing certain maneuvers, flight scenarios, and emergency procedures. Flight instructors in one survey believed that computer-based devices were effective for teaching most tasks in an instrument course.9 Flight training experts at a major airline's training center told us that they and other airlines use a variety of training equipment in their operations, 8Sybil I. Phillips, Charles L. Hulin, and Paul J. Lamermayer, "Uses of Part- Task Trainers in Instrument Flight Training," University of Illinois, Proceedings of the Seventh International Symposium on Aviation Psychology, Columbus, Ohio, 1993. 9William F. Moroney, Steven Hampton, and David W. Biers, "Considerations in the Design and Use of Personal Computer-Based Aircraft Training Devices (PCATDs) for Instrument Flight Training: A Survey of Instructors," University of Dayton, Ohio, and Embry-Riddle Aeronautical University, Daytona Beach, Florida, Proceedings of the Ninth International Symposium on Aviation Psychology, Columbus, Ohio, 1997. According to another study (Brian K. Rogers, Capt., U.S. Air Force. "Microcomputer-Based Instrument Flight Simulation: Undergraduate Pilot Training Student Attitude Assessment," December 1991, Human Resources Directorate, Aircrew Training Research Division, Williams Air Force Base, Arizona, AL-TR-1991- 0039), surveyed students believe that computer-based devices have some training value. Almost all of the of U.S. Air Force flight students surveyed planned to use computer-based instrument simulators to replace at least some of their "chair-flying," or rehearsing procedures from a chair placed in front of a life-size photo of the airplane's cockpit. Over half said they would replace at least 75 percent of their chair-flying with computer-based simulation. The same study found increased student interest in using computer-based simulation as instrument training was expanded to include multiple procedures in real-time practice. Another study (D. Gopher, M. Weil, and T. Bareket, "Transfer of skill from a computer game trainer to flight," Human Factors, 36 (3) (1994), pp. 387-405) found that military flight students who played a complicated computer game performed better in later airplane training. Page 14 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 ranging from $20 million full-motion simulators to simple personal computer-based training, depending on whether their goal is to introduce a pilot to a new procedure or piece of equipment, to practice complete simulated flights, or to serve an intermediate purpose. For example, these airline training officials believed that simple personal computer equipment can introduce a pilot to a new navigation system or component more effectively than a full- motion simulator with its many competing demands for a pilot's attention. Safety and Academic According to safety experts from the National Transportation Safety Board Experts Generally Do Not and from an industry safety association, the use of computer-based Believe PCATDs Pose a devices and the granting of credit hours do not pose a threat to general Safety Risk aviation safety by adversely affecting the skills and abilities of pilots who receive instrument training using the devices. In general, academic and other flight training experts believe that the devices offer training value to pilots who want to obtain an instrument rating. According to these experts, good instrument and navigational skills are vital in avoiding bad weather and surviving it when necessary; two experts we interviewed believe that the low cost and wide availability of PCATDs could help maintain or increase the general level of pilots' instrument skills, which are critical to safe flying. Long-Term Safety Impact We found no empirical evidence on the long-term safety impact of of PCATDs Is Unknown computer- based devices, their potential safety benefits, or the long-term safety impact or benefits of the currently approved flight training devices. Many of the experts we interviewed believed either that computer-based devices were not likely to reduce aviation safety or that they offer long-term safety benefits. However, several experts noted that the potential for long-term safety risks from computer-based devices used in training remains a possibility. Objections to their use for credit towards the training hours needed for an instrument rating generally revolve around whether using a computer-based device in instrument training is more likely to reduce or increase an instrument pilot's margin of safety. Several issues were raised in our interviews with experts concerning PCATD safety. Among these were potential risks from the lack of physical similarity between the device and the airplane and associated issues, possible benefits to training from their use, and the lack of research showing that one hour on a training device equals one hour of training in an airplane. Some of these issues appear to support the value of PCATDs; others raise questions about their value; and the effects of other issues on Page 15 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 the value of PCATDs are unclear. Although not fully resolved, these issues may be relevant to future research. (See app. III for a summary of these issues.) FAA does not currently track how many hours an instrument pilot trains on a PCATD. However, FAA is now computerizing this information and, for unrelated reasons, plans to revise the form on which pilots apply for the instrument and other ratings. FAA's application for the instrument rating could easily and inexpensively be changed to include a box on the types of training devices a pilot has used, particularly since the form (FAA Form 8710-1) will soon be revised and its data computerized. With information on the training devices used by a particular pilot, along with other data that FAA now maintains or will soon maintain electronically, researchers will be better able to determine, within a few years, the safety of pilots trained using computer-based devices. These data will enable researchers to link accidents or incidents with the type of instrument training the pilots had received. While this type of examination would present some statistical challenges, experts we interviewed said that it would be possible to identify any large safety risks posed by computer-based devices. Conclusions Commercial aviation safety depends on many factors, including the training of general aviation pilots, who sometimes share common airspace and airports with the flying public. Because most accidents in general aviation are caused by pilot error, and many accidents occur when pilots without an instrument rating encounter bad weather or poor visibility and unexpectedly have to rely on their instruments, effective training that teaches, reinforces, and maintains safe instrument flying skills is vital. To the extent that the use of computer-based devices may be of training value to instrument pilots-which research supports and experts believe-the devices do not pose identifiable safety risks. The low cost and wide availability of computer-based devices could improve the general level of instrument training among pilots, according to some experts, and help to maintain or improve instrument pilots' skills, especially for students and pilots who would not otherwise have access to a training device. Several experts told us that FAA's check rides, which are required for a pilot to obtain an instrument rating, may not be sufficient indicators of a student's training, capabilities, and ability to fly safely. However, we found no studies on the link between a pilot's training equipment or flying proficiency at the time of training and that pilot's Page 16 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device B-280735 long-term ability to fly safely. Collecting information on pilots' training equipment and safety records would permit research in the future and could help to uncover any safety problems related to the use of computer-based and other training devices. Recommendations As a first step toward determining whether computer-based devices pose safety risks to general aviation in relation to other training methods, the Secretary of Transportation should direct the FAA Administrator to collect information from pilot applications for instrument ratings on how many hours students trained on PCATDs and flight training devices. This information can be used by FAA and others to study the relationship between the use of training devices and safety. FAA's revision to its rating application form allows the additional information to be obtained at minimal cost. Agency Comments We provided a draft copy of this report to the Office of the Secretary of Transportation and the Federal Aviation Administration (FAA) for review and comment. We discussed the report with FAA officials, including the Deputy Associate Administrator for Regulation and Certification. They generally agreed with the draft report. FAA officials also indicated that they concurred with and would implement our recommendation as soon as possible. They also provided technical corrections, which we incorporated into the report as appropriate. Scope and We reviewed FAA's decision to allow the use of PCATDs by reviewing the Methodology data the agency used to support its decision and interviewing a variety of FAA and aviation industry officials. We also reviewed research on the use of computer-based devices in training and interviewed a variety of experts on the potential safety impacts of their use. Appendix I provides our detailed scope and methodology. We conducted our review from June 1998 through June 1999 in accordance with generally accepted government auditing standards. As arranged with your offices, unless you announce its contents earlier, we plan no further distribution of this report until 7 days after the date of this report. At that time, we will make copies available to the Secretary of Page 17 GAO/RCED-99-143 Personal Computer- Based Aviation Training Device B-280735 Transportation and the Administrator, FAA. We will also make copies available to others on request. If you have any questions about this report, please contact me at (202) 512-3650. Major contributors to this report were Karen Bracey, David Ehrlich, and John Rose. Gerald L. Dillingham Associate Director, Transportation Issues Page 18 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Page 19 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Contents Letter 1 Appendix I 22 Scope and Methodology Appendix II 25 Summaries of Two University of Illinois Study 25 Embry-Riddle Aeronautical University Study 26 Major FAA-Sponsored Studies Appendix III 28 Other Issues Related Views on the Lack of Physical Similarity Between PCATDs and 28 Aircraft Instrument Panels and Controls to PCATD Safety Selected Bibliography 41 Figures Figure 1: Personal Computer-Based Aviation Training Device 5 Figure 2: University of Illinois Study: Hours to Proficiency in an 11 Airplane Figure 3: Embry-Riddle Study: Trials to Proficiency in an Airplane 12 Figure 4: Embry-Riddle Study: Hours to Proficiency in an 13 Airplane Figure III.1: Approved Flight Training Device 29 Figure III.2: Instrument Panel Photo of Cessna 172 32 Figure III.3: PCATD Depiction of Cessna 172 Instruments 33 Figure III.4: Flight Path Screen 35 Figure III.5: Screen to Set Failures 36 Figure III.6: Screen to Set Weather Conditions 37 Figure III.7: Example of Grandfathered Flight Training Device 39 Abbreviations CAMI Civil Aeromedical Institute FAA Federal Aviation Administration FTD flight training device PCATD personal computer-based aviation training device Page 20 GAO/RCED-99-143 Personal Computer- Based Aviation Training Device Page 21 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix I Scope and Methodology To examine the Federal Aviation Administration's (FAA) decision allowing the use of personal computer-based aviation training devices (PCATD), we reviewed the data the agency used to support its decision to issue Advisory Circular 61-126, reviewed other studies available on the topic, and interviewed a variety of experts within and outside government. In particular, we reviewed the supporting data FAA used in making its decision, including a 1994 study by Embry-Riddle Aeronautical University and a 1996 study by the University of Illinois on the effectiveness of PCATD training. We reviewed FAA's files, including archived e-mail messages, a chronology of events, and supporting information and documents. We met with FAA officials from Flight Standards, both in Washington, D.C., and at an Illinois Flight Standards District Office, as well as with FAA officials from Human Factors, the Office of Aviation Medicine, the Civil Aeromedical Institute (CAMI) in Oklahoma City, and the National Simulator Program in Atlanta. We asked several recognized academic and training experts to review our plans at several points during our research. Each of these experts has extensive academic, research, and training experience in aviation, including the fields of aviation psychology and human factors; their views are theirs alone rather than those of their institutions. These experts were Dr. Barry Strauch of the National Transportation Safety Board; Dr. Dee Andrews of the U.S. Air Force Research Laboratory, Mesa, Arizona; and Professor (Emeritus) Stanley Roscoe of the University of Illinois at Urbana-Champaign and New Mexico State University. In addition, we interviewed flight training, aviation psychology, human factors, military research, and aviation safety experts in academia and the military, as well as at the National Transportation Safety Board and the National Aeronautics and Space Administration. We interviewed flight instructors, association and airline officials, the manufacturers of each of the approved PCATDs, and the manufacturers of flight training devices and control equipment. In particular, we interviewed representatives of the following organizations: the National Association of Flight Instructors; the National Air Transport Association; the University Aviation Association; the Airplane Owners and Pilots Association and its affiliated Air Safety Foundation; two flight schools in the Chicago area; two flight schools in the Washington, D.C., area; and the chief flight instructors of the University of Illinois and Embry-Riddle Aeronautical University at Daytona Beach, and other selected flight instructors. Among the equipment Page 22 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix I Scope and Methodology manufacturers and distributors we interviewed were Aviation Supplies & Academics, Inc.; Aviation Teachware Technologies; Jeppesen Sanderson; and Frasca International, Inc. Among the academic experts we contacted, in addition to our advisory panel, were the authors of both the University of Illinois and Embry- Riddle Aeronautical University studies. At FAA, we spoke to officials in Flight Standards; the National Simulator Center (Atlanta); the Civil Aeromedical Institute (Oklahoma City); Human Factors technical scientists (Washington, D.C.); and FAA's Flight Standards District Office (West Chicago, Illinois). We also spoke to representatives of the U.S. Air Force, the U.S. Navy, the National Transportation Safety Board, and the National Aeronautics and Space Administration (Ames, California). To identify other studies relevant to FAA's decision, we obtained articles cited in recent studies. We also conducted additional literature searches of the collections at two major aviation libraries and a computerized literature search using several bibliographic databases and including the following keywords and phrases: personal computer aviation training device, PCATD, accident/incident, flight training device, ground training device, part task trainers, aviation safety, pilot training, air safety.10 Our computerized literature search identified about 700 studies and articles. Of these, we selected 68 dealing with personal computers, simulation issues, or aviation safety, including all the citations that appeared to involve, discuss, or present empirical research in these areas. Thirty-nine abstracts were further selected as relevant enough to obtain full text copies, including all those that appeared to involve empirical research on personal computer-based flight training. From all our sources, including electronic and nonelectronic literature searches, we found eight studies involving computer-based device effectiveness, and four of these met all of the following criteria: (1) examined computer-based flight simulation effectiveness; (2) compared computer-based device training to airplane-only training performance or to training with another approved training device; (3) had been conducted using an experimental design or survey, including random assignment; and (4) used data from an experiment not summarized elsewhere in our report (i.e., the two studies cited by FAA in its decision-one at Embry-Riddle Aeronautical University and one at the University of Illinois). In addition, we reviewed studies and 10The databases searched included the National Technical Information Service and Transportation Research Information Services; DIALOG, from which the following databases were searched: Dissertation Abstracts Online, GPO Monthly Catalog, PsychINFO, IAC Aerospace/Defense Markets, Aerospace Database, McGraw-Hill Publications, Periodical Abstracts Plus text, DIALOG Defense Newsletters, Business Dateline, IAC Newsletter Database; and the NEXIS database. Page 23 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix I Scope and Methodology articles on general aviation flight safety that did not directly mention either computer-based simulation or training. Many articles in this field are published in journals with limited circulation and in the proceedings of civilian and military conferences. Though these can be difficult to identify from conventional bibliographic databases, we attempted through these combined methods to obtain all those studies examining computer-based flight training that used empirical methods. We conducted our review from June 1998 through June 1999 in accordance with generally accepted government auditing standards. Page 24 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix II Summaries of Two Major FAA- Sponsored Studies This appendix summarizes the studies by the University of Illinois and the Embry-Riddle Aeronautical University. University of Illinois In the University of Illinois study,11 researchers compared the performance Study of students who received some training on a PCATD before their training in an airplane with the performance of a group of students trained entirely in an airplane. University of Illinois aviation students who were already private pilots and had no previous instrument coursework were randomly assigned to one of two groups for their instrument flight-training course. After this study's coursework, passing students took FAA check rides to attempt to obtain their instrument rating. Performance measures were available for 74 students; roughly half the students were in each group. Students were tested to demonstrate proficiency, and the time and number of trials were recorded for each student. The computer-based device used was Model FS-100 by MDM, a model not currently approved for use as a PCATD by FAA toward flight training credit hours; the airplane used was Beechcraft Sports/Sundowner, which has a single engine, a fixed-pitch propeller, and fixed landing gear. Students' performance in the airplane was not substantially different for those trained partly using a computer-based device and those trained solely in an airplane. However, the data showed small, but statistically significant, savings in the total airplane time needed to reach proficiency in the airplane by students trained on the PCATD device. Students using this device spent, on average, 3.9 fewer hours in the airplane than those who took their entire training in the airplane. Airplane-only students averaged 51.3 total hours to complete the course, while students using the PCATD averaged 47.4 hours in an airplane and 26.5 on the device. This study was not designed to determine the amount of loggable time FAA should grant for PCATD use, according to its authors. It did not include a student group trained on a currently approved flight training device. Such a comparison group is of interest because the 10 hours currently permitted for PCATD instrument training may be substituted for part of the 15 to 20 hours that can be earned using a flight training device. The study's authors believe that in an actual instrument course, where experimental controls would not require the use of the PCATD for all course lessons, the 3.9 hour savings of time in the airplane could be achieved with fewer PCATD hours 11Henry L Taylor, Gavan Lintern, Charles L. Hulin, Donald Talleau, Tom Emanuel, and Sybil Phillips; "Transfer of Training Effectiveness of Personal Computer-Based Aviation Training Devices," University of Illinois; November 1996, Final Technical Report ARL-96-3-/FAA-96-2, prepared for the Federal Aviation Administration, Oklahoma City, OK, Contract DTFA-94-G-044 Page 25 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix II Summaries of Two Major FAA- Sponsored Studies by targeting the use of the device to those early course lessons where training transfer is greatest. The study does not support a conclusion that a "negative transfer"-or learning interference caused by using PCATDs-occurred when the students trained on the personal computer device switched to the airplane. In the few instances in which this group of students needed more trials to demonstrate proficiency in the airplane than the group without PCATD training, the differences were so small that they were not statistically significant. Differences of the magnitude observed could occur even if the two groups of students had been given exactly the same training. While the study does not support a conclusion that negative transfer occurred on some tasks, it also does not rule out the possibility that it did occur. Small amounts of negative transfer in an individual task could have existed without being detected by statistical tests. Embry-Riddle The Embry-Riddle study measured the flight performance of students Aeronautical trained using PCATDs and compared their performance with that of students trained using a previously approved flight training device.12 All University Study the students had previously taken navigation and instrument coursework. Embry-Riddle students who volunteered for the study were randomly assigned to receive ground-based training using one of two PCATDs or using the flight training devices. Students used the assigned ground-training devices during the first part of their training and continued their instrument flight training in an airplane. Instructors were randomly assigned to students and taught students in each of the three groups. In the airplane, the performances of the three groups of students were measured by the total amount of time and the total number of trials students required to demonstrate proficiency in each of eight maneuvers. Performance measures were available for a total of 66 students. The PCATDs used were the Elite (20 students finished) and the IFT Pro 5.1 (26 students finished); both devices had 15-inch screens, 486/66MHz computers, and similar control features. The flight training device used was the Frasca 141 (20 students finished). The airplane used was a Mooney 20J single- engine plane. The study provided a preliminary indication that there may not be very large differences in the ground flight training provided by the approved 12Steven Hampton, William Moroney, Tom Kirton, and David W. Biers, "The Use of Personal Computer-Based Training Devices in Teaching Instrument Flying," Embry-Riddle Aeronautical University, June 15, 1994. Prepared for the Federal Aviation Administration, CAAR 15471931, Grant No. 92- G-0015. Page 26 GAO/RCED-99-143 Personal Computer- Based Aviation Training Device Appendix II Summaries of Two Major FAA-Sponsored Studies flight training devices and the two PCATDs studied. No statistically significant differences were found in the airplane performance of the groups trained on the three devices, as measured by trials and hours to meet FAA's established practical test standards. The study did not and was not intended to address the issue of how many hours of training on PCATDs FAA should grant for an instrument rating, according to one of the study's authors. The study also did not show how much, if any, training received on each of the ground-based devices was transferred to the airplane. The performance of a group of students who received all their training in an airplane would be required before an estimate of how much less time in an airplane students receiving training on a PCATD or flight training device might need. Because no empirical studies have been conducted on the effectiveness of training on a flight training device compared with airplane-only training for instrument students, this study's comparative data cannot be used to conclude that PCATDs provide effective instrument training. Page 27 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety We identified a number of other issues from interviews with safety, industry, and academic experts that could affect the long-term use of computer-based devices. These issues are the lack of the devices' physical similarity to actual airplanes and the potential for associated problems from negative learning from simulation-that is, a pilot's reverting to incorrectly learned behaviors in an emergency or under stress-and the possibility that the smaller-sized instruments on the devices could interfere with a pilot's instrument scan. Possible benefits from PCATDs include their availability and use in teaching concepts and procedures, such as the flexibility they provide by permitting a pilot to practice on a particular type of airplane, to see a graphic display of performance, and to fly under varying scenarios of instrument failure, system failure, and weather conditions. Several simulation experts believe that credit hours should not be granted for PCATD use, that the two major studies do not show the training effectiveness of either PCATDs or flight training devices, and that there is no evidence that one hour in any training device equals one hour in an airplane, particularly for the older, grandfathered flight training devices. Views on the Lack of One of the main concerns expressed by those opposed to granting credit Physical Similarity for instrument training time on PCATDs is that PCATDs do not have sufficient similarity, or physical fidelity, to an airplane's instrument panel and Between PCATDs and controls.13 For example, PCATDs have display screens showing instruments Aircraft Instrument that are smaller than they actually are in the cockpit, and the placement of some PCATD switches and dials represented on the display screen are in a Panels and Controls physically different position than in the aircraft-alongside or below a desktop computer monitor. Flight training devices, in contrast, can include more instruments-often actual aircraft instruments-that can be mounted on a dashboard panel larger than a computer screen, permitting greater physical similarity to an aircraft. On the other hand, flight training devices represent a standard aircraft configuration, while PCATD software can be easily changed to represent a specific aircraft. Figure III.1 shows an approved flight training device. 13The literature discusses at least three different kinds of fidelity, including physical fidelity-whether the simulation "looks right;" functional fidelity-whether the simulation "acts right;" and psychological fidelity-whether the simulation "feels right." Page 28 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety Figure III.1: Approved Flight Training Device Note: This is a previously approved flight training device with an optional visual system (Frasca 131). Source: Frasca International. Page 29 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety The potential impact of these differences between PCATDs and flight training devices on pilot training is not clear. Some experts and representatives of the flight training device industry believe that devices that are less similar to the airplane and have lower physical fidelity could interfere with learning. Some simulation experts believe that the more the simulator looks, feels, and "flies" like the airplane, the greater its training value will be, particularly for experienced pilots. However, other experts and many research studies suggest that high physical fidelity is not essential for a device to have training value; that students using low-fidelity devices often perform as well as those using more complex, higher-fidelity devices; and that the effectiveness of a training device depends on how the device is used in a training curriculum.14 Negative transfer- learning that can interfere with the performance of a task rather than improve it-can occur when a familiar instrument or knob is placed in a dramatically different position in a training device than in an aircraft. One expert on aviation psychology told us that negative transfer from PCATDs would be unlikely but possible. A pilot's switching from a training device to an airplane, or from one airplane to another, according to interviews with aviation psychologists, would rarely involve locating control yokes, knobs, or switches that are completely reversed. According to one expert, however, if a pilot trains in a plane with the retractable landing gear switch on the left side of the cockpit and the flap switch on the right and then flies another plane with the landing gear switch on the right and the flap switch on the left, negative transfer of learning could occur if a pilot were to confuse the two switches. In general, the academic, private, military, and airline flight training experts we interviewed did not believe that the issue of PCATDs' similarity to the airplane presented safety problems. Several safety experts noted that previously approved flight training devices do not typically represent one particular type of aircraft and often include instrument panels that differ from those on the plane in which the student will train and later fly; students will have to adjust from the training device to a somewhat different airplane. While familiarity with a particular type of plane is an important safety consideration, few students will fly only one type of plane and therefore will have to adjust, using training devices, to different 14Gavan Lintern, S.N. Roscoe, J.M. Koonce, and L.D. Segal, "Transfer of Landing Skill in Beginning Flight Training," Human Factors 32, (1990), pp. 319-327; Gavan Lintern, Henry L. Taylor, Jefferson M. Koonce, and Donald Talleur, "An Incremental Transfer Study of Scene Detail and Field of View Effects on Beginning Flight Training," Proceedings of the Eighth International Symposium on Aviation Psychology, Columbus, Ohio, 1995, p. 737. Page 30 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety instrument displays, a different feel to the controls, and different performance characteristics. Another potential danger could occur when a pilot, in an emergency, automatically reverts to incorrectly learned responses, skills, or procedures. This stressful reversion, according to some industry representatives, could be the result of the lower physical fidelity of PCATDs. Other experts dispute this, noting that instrument students are already pilots licensed to fly in good visibility conditions and already have some experience using airplane instrumentation; one expert noted that incorrect and dangerous initial instruction can take place in an airplane as well as while using a PCATD. Another expert said that quality instruction is more important than the type of training device, but this type of problem would be difficult to detect. Several experts were also concerned that the smaller size of PCATD instruments-usually about three-fourths actual size-coupled with the smaller size of the PCATD computer monitor screen, could affect the quality of a pilot's instrument scan-a critical skill to retain constant spatial orientation by using instruments alone.15 In poor visibility, a pilot must quickly scan the correct instruments that are most important to completing various maneuvers, as well as to maintain level flight. Instrument scan may also be impaired if PCATD-displayed instruments suffer from lower fidelity than flight training devices, some PCATD critics believe. Figures III.2 and III.3 show an actual instrument panel and the PCATD version of that panel. 15Spatial orientation is knowing the position of the plane in space and its speed, direction, and up, down, and sideways orientation. A blindfolded passenger in a plane-even an experienced pilotquickly loses the ability to accurately sense whether a plane is climbing, descending, or turning, in part because of the confusing multiple sensations of acceleration, deceleration, and turning from vestibular and other cues. G-forces and even a turning of the head during a turn or climb can cause equally misleading illusions. A pilot flying relying on visual cues can be tragically mistaken by subtle optical illusions under certain conditions that might be prevented by the better use of instruments to confirm the plane's situation. Page 31 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety Figure III.2: Instrument Panel Photo of Cessna 172 Source: The Cessna Aircraft Company. Page 32 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety Figure III.3: PCATD Depiction of Cessna 172 Instruments Source: Aviation Teachware Technologies While instrument and panel size could be potential problems for a pilot's instrument scan, experts responded to this potential problem in several ways. Several noted that a pilot's instrument scan must be altered to read different airplanes' instrument panels and so may be aided by practice on a PCATD that resembles the particular plane he or she flies because primary instruments would be correctly spaced although not full- sized. A flight instructor noted that larger computer monitors are becoming less expensive and that larger screens can increase the size of a PCATD's instrument display. A safety expert noted that whether an instrument is an actual airplane instrument or displayed on a video screen is not important. Other experts say that clarity on a video screen can be very good-most commercial airliners now display instruments on video screens in "glass cockpits." Another expert said that while very few general aviation Page 33 GAO/RCED-99-143 Personal Computer- Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety aircraft now have glass cockpits, they will become more common in the future. Views on the Advantages Experts we interviewed saw the main value of PCATDs in teaching of PCATDs procedures and concepts, rather than the complete set of skills needed to fly. Many experts and flight instructors believe that isolating a single procedure on a training device can help a student focus on that lesson and learn more effectively, particularly when introduced to a new topic, without the distractions of simultaneously flying the airplane. In general, the experts we interviewed believed that the potential safety advantages of PCATDs outweigh their potential risks. Among these possible benefits are the flexibility of PCATD software to mimic the instrument configuration and performance of a variety of airplanes; permitting the pilot to fly a half-dozen approaches in an hour compared with perhaps one in an airplane; allowing the pilot to see and save a chart of a flight as a learning tool; and allowing the simulation of systems failures, specific instrument failures, and a variety of weather conditions, such as wind speed and level of visibility. Figure III.4 shows a PCATD diagram of a flight path, figure III.5 shows a PCATD screen to set instrument failures and figure III.6 shows a PCATD screen to set weather conditions. Page 34 GAO/RCED-99-143 Personal Computer- Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety Figure III.4: Flight Path Screen Source: Aviation Teachware Technologies. Page 35 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety Figure III.5: Screen to Set Failures Source: Aviation Supplies & Academics, Inc. Page 36 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety Figure III.6: Screen to Set Weather Conditions Source: Aviation Teachware Technologies. Some safety experts also see the wider availability and use of relatively inexpensive PCATDs as a benefit, since most instrument students are not likely to have easy access to a flight training device.16 The wider use of PCATDs for continuing training could potentially improve the overall level of general aviation pilots' instrument skills and judgment, which in turn could improve pilot safety. Views on Granting Training According to several experts, PCATDs have a role in training but that Credit for PCATDs students using PCATDs should not be granted credit hours for instrument coursework. These experts believe that neither of the two major studies showed that PCATDs had achieved a level of training effectiveness so that one hour of device training equals one hour in the airplane; however, FAA 16According to PCATD manufacturers, many PCATDs are sold to individuals either for practice at home or possibily for individual Part 61 instruction. Page 37 GAO/RCED- 99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety assumes that one hour in the training device equals one hour in the airplane. These experts question whether the performances of the PCATD students in the Embry-Riddle Aeronautical University and University of Illinois studies were due to the PCATD, training received in the course curriculum or in the airplane itself during the check ride. They said that the previously approved flight training devices had also not demonstrated their effectiveness. These experts believed that both PCATDs and flight training devices should be included under one set of standards and that a device should not be approved for instrument training credit hours until it shows that (1) one hour in the device is as effective for training as one hour in the airplane and (2) the number of credit hours now permitted are the appropriate amounts that should be approved for instrument training. According to these and several academic flight training and safety experts, PCATDs are superior to the older, grandfathered flight training devices that are still approved for the full 15 to 20 hours of instrument flight training as well as to meet instrument recency (currency) requirements.17 Figure III.7 shows an older flight training device that was grandfathered in for use as a flight training device. 17These older model flight training devices may be mechanically driven desktop models that are not computer-driven. Page 38 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety Figure III.7: Example of Grandfathered Flight Training Device Note: This is an ATC 610 tabletop device. Source: Air Safety Foundation. Despite anecdotal evidence that there has been little difference in course completion hours between instrument students who are trained to proficiency in the airplane and those trained partly using flight training devices, we found no research showing that this level of instrument training effectiveness has been documented either for PCATDs or approved flight training devices. However, a 1971 study of beginning flight students found that an older training device, the Link GAT-1, transferred 100 percent of 11 hours of training value to the student's airplane performance, with a training effectiveness ratio of one-to-one; in other words, under certain conditions for beginning students, one hour in a flight training device can save one hour of time the student would Page 39 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Appendix III Other Issues Related to PCATD Safety otherwise need in the airplane to reach proficiency. In a 1973 study, also on beginning flight students, airplane training time savings appeared to be greater than one-to-one in the first few hours of training, where one hour in the device saved more than one hour in the airplane and appeared to decline in effectiveness to less than one-to-one as time in the flight training device rose to 11 hours.18 According to one expert, the assumption of one-to-one substitution of training device hours for airplane hours, though important, is secondary to other concerns because few students complete their instrument coursework in only the minimum required 35 to 40 hours but instead average about 70 total hours to reach proficiency. 18H.K. Povenmire and S.N. Roscoe, "An Evaluation of Ground-Based Flight Trainers in Routine Primary Flight Training." Human Factors (1971), 13 (2), pp. 109-116; _________," Incremental Transfer effectiveness of a ground-based general aviation trainer." Human Factors (1973). 15 (6), pp. 534-542. Page 40 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Selected Bibliography Amdor, S.L., F.W. Isley, and Byron J. Pierce. "Computer-Based Instruction/Simulator Program for Fighter Lead-In Training: Feasibility Research." July 1988. Air Force Human Resources Laboratory, Air Force Systems Command, Brooks Air Force Base, Texas, AFHRL-TP-87-64. Andrews, Dee H., Lynn A. Carroll, and Herbert H. Bell. "The Future of Selective Fidelity in Training Devices," March 1996, Human Resources Directorate, Aircrew Training Research Division, Mesa, Arizona, AL/HR-TR-1995- 0195. Barber, Sarah. "From PCs to Flight Simulators: Cost Effective Aircrew Training Devices for Aeronautical Decision Making and Crew Resource Management." The Ohio State University, Proceedings of the Ninth International Symposium on Aviation Psychology. Columbus, Ohio, 1997 Benton, Charles J., Paul Corriveau, Jefferson M. Koonce, and William C. Tirre. "Development of the Basic Flight Instruction Tutoring System (BFITS)," January 1992, Final Technical Paper for Period September 1989-December 1991, Air Forces Systems Command, Brooks Air Force Base, Texas, AL-TP-1991-0060. Beringer, Dennis B., and Howard C. Harris, Jr. "Assessment of Cockpit Navigation and Memory Aids Using PC-Based Flight Simulation," Civil Aeromedical Institute, Federal Aviation Administration, Oklahoma City, Oklahoma. Proceedings of the Eighth International Symposium on Aviation Psychology, Columbus, Ohio, 1995. Beringer, Dennis B. "Use of Off-The-Shelf PC-Based Flight Simulators for Aviation Human Factors Research," April 1996, Civil Aeromedical Institute, Federal Aviation Administration, U.S. Department of Transportation, Oklahoma City, Oklahoma, DOT/FAA/AM- 96-15. Burki-Cohen, Judith, Nancy N. Soja, and Thomas Longridge. "Simulator Platform Motion- The Need Revisited," The International Journal of Aviation Psychology 8 (3), 293-317. Caro, Paul W. "Some Current Problems in Simulator Design, Testing and Use," March 1977, Professional Paper 2-77, Air Force Office of Scientific Research, Air Force Systems Command, Bolling Air Force Base, HumRRO-PP-2-77. Page 41 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Selected Bibliography Collins, William E., and Carolyn S. Dollar. "Fatal General Aviation Accidents Involving Spatial Disorientation: 1976-1992," August 1996, Civil Aeromedical Institute, Federal Aviation Administration, U.S. Department of Transportation, Oklahoma City, Oklahoma, DOT/FAA/AM-96-21 Cordell, Captain Tom. "Computer-Based Procedural Training," United Airlines, Proceedings of the Sixth International Symposium on Aviation Psychology, Columbus, Ohio, 1991. Dennis, Kerry A., and Don Harris. "Computer-Based Simulation as an Adjunct to Ab Initio Flight Training," The International Journal of Aviation Psychology 8 (3) (1998) pp. 261-276, Emanuel, Tom W., Jr., Henry L. Taylor, Charles L. Hulin, Gavan Lintern, Sybil I. Phillips, and Donald A. Talleur. "Development of an Experimental Flight Syllabus for Testing PC Computer-Based Aviation Training Devices, "University of Illinois, Proceedings of the Eighth International Symposium on Aviation Psychology, Columbus, Ohio, 1995, p. 1198 Emanuel, Tom W., Jr. "PC Computer- Based Aviation Training Devices, Curriculum Development, and Student Affect," University of Illinois, Proceedings of the Ninth International Symposium on Aviation Psychology, Columbus, Ohio, 1997, p. 1198. Etem, Kamil, and Marcia Patten. "Communications- Related Incidents in General Aviation Dual Flight Training," Proceedings of the Ninth International Symposium on Aviation Psychology, Columbus, Ohio, 1997, p. 1204. Giffin, Walter C., and Thomas H. Rockwell. "A Methodology for Research on VFR Flight into IMC," Proceedings of the Fourth International Symposium on Aviation Psychology, Columbus, Ohio, 1987, p. 278. Gopher, D., M. Weil, & T. Bureket. "Transfer of skill from a computer game trainer to flight," Human Factors 36 (3) (1994), pp. 387-405. Hampton, Steven. "PC-Based Simulation: An Alternative for Teaching Instrument Flying Skills," Embry-Riddle Aeronautical University, Daytona Beach, Florida, in PC-Based Instrument Flight Simulation A First Collection of Papers, A. Robert Sadlowe, editor. Presented at the Winter Annual Meeting of the American Society of Mechanical Engineers, December 1-6, 1991. Page 42 GAO/RCED- 99-143 Personal Computer-Based Aviation Training Device Selected Bibliography William Moroney, Tom Kirton, and David W. Biers. "The Use of Personal Computer-Based Training Devices in Teaching Instrument Flying: A Comparative Study," June 15, 1994, Prepared for the Federal Aviation Administration, CAAR 15471931, Grant No. 92-G-0015 [Summarized in this report.]. Koenig, Robert L. "Standard Computer Hardware and Software Configured to Produce Useful Flight Simulator in Study." Human Factors & Aviation Medicine 43, No. 4 (July-August 1996), Flight Safety Foundation. Koonce, Jefferson M., "Predictive Validity of Flight Simulators as a Function of Simulator Motion." Human Factors 21 (2) (1979), pp. 215-223. Koonce, Jefferson M., and William J. Bramble, Jr. "Personal Computer-Based Flight Training Devices." The International Journal of Aviation Psychology 8 (3), (1998), pp. 277-292. _____, Steven L. Moore, and Charles J. Benton. "Initial Validation of a Basic Flight Instruction Tutoring System (BFITS)," Proceedings of the Eighth International Symposium on Aviation Psychology, Columbus, Ohio, 1995. Lintern, Gavan. "Transfer of Landing Skill after Training with Supplementary Visual Cues," Human Factors 22 (1), (1980), pp. 81-88. _____, Roscoe, S.N., Koonce, J.M., and Segal, L.D. "Transfer of Landing Skill in Beginning Flight Training." Human Factors 32, (1990), pp. 319-327. _____, Henry L. Taylor, Jefferson M. Koonce, and Donald Talleur. "An Incremental Transfer Study of Scene Detail and Field of View Effects on Beginning Flight Training," Proceedings of the Eighth International Symposium on Aviation Psychology, Columbus, Ohio, 1995, p. 737. Lombardo, D.A. "Effectiveness of Computer-Based Flight Simulation," in B.M. Barker (Ed.), Collegiate Aviation Review, University Aviation Fall Conference, Atlanta, 1993. Moroney, William F., Steven Hampton, and David W. Biers. "Considerations in the Design and Use of Personal Computer-Based Aircraft Training Devices (PCATDs) for Instrument Flight Training: A Survey of Instructors," University of Dayton, Ohio, and Embry- Riddle Page 43 GAO/RCED-99-143 Personal Computer- Based Aviation Training Device Selected Bibliography Aeronautical University, Daytona Beach, Florida, Proceedings of the Ninth International Symposium on Aviation Psychology, Columbus, Ohio, 1997. Moroney, William F. and Brian W. Moroney, "Utilizing a Microcomputer Based Flight Simulation in Teaching Human Factors in Aviation," Proceedings of the Human Factors Society 35th Annual Meeting, 1991, p. 523 "Nall Report: 1997." Airplane Owners and Pilots' Association Air Safety Foundation, Frederick, Maryland. O'Hare, David, and Stanley Roscoe. Flightdeck Performance: The Human Factor. Ames, Iowa: Iowa State University Press; 1990. O'Hare, David, and David Chalmers. "Risk Factors for Aviation Accidents and Incidents: A Nationwide Study," Proceedings of the Ninth International Symposium on Aviation Psychology, Columbus, Ohio, 1997, p. 1352. Ortiz, Gustavo. "Effectiveness of PC-Based Flight Simulation." The International Journal of Aviation Psychology 4 (3), p. 285, 1994. _____, Dominik Kopp, and Thomas Willenbucher. "Instrument Training Using a Computer-Based Simulation Device at Luftansa Airlines," Andrews University, Berrien Springs, Michigan, and Luftansa Airlines, Bremen, Germany, Proceedings of the Eighth International Symposium on Aviation Psychology, Columbus, Ohio, 1995. _____, Phillips, Sybil I., Charles L. Hulin, and Paul J. Lamermayer. "Uses of Part-Task Trainers in Instrument Flight Training." University of Illinois, Proceedings of the Seventh International Symposium on Aviation Psychology, Columbus, Ohio, 1993. _____, Henry L. Taylor, Gavan Lintern, Charles L. Hulin, Tom W. Emanuel, Jr.; Donald A. Talleur. "Developing Performance Measures for Evaluating Personal Computer- Based Aviation Training Devices Within an FAR Part 141 Pilot Training School," Proceedings of the Eighth International Symposium on Aviation Psychology, Columbus, Ohio, 1995, p. 1204- 1209. Pierce, Rodney, and Dr. Richard S. Jensen. "PC Based Assessment and Training of Aviation Decision Making Skills, The Ohio State University, Page 44 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Selected Bibliography Columbus, Ohio, Proceedings of the Ninth International Symposium on Aviation Psychology, Columbus, Ohio, 1997. Povenmire, H. K., & S. N. Roscoe. "An evaluation of ground based flight trainers in routine primary flight training." Human Factors 13 (1971),.pp. 109-116. _____. "Incremental transfer effectiveness of a ground-based general aviation trainer. "1973. Human Factors 15, pp. 534-542. Rogers, Brian K., Capt., U.S. Air Force. "Microcomputer-Based Instrument Flight Simulation: Undergraduate Pilot Training Student Attitude Assessment," December 1991, Human Resources Directorate, Aircrew Training Research Division, Williams Air Force Base, Arizona, AL-TR-1991- 0039. Rolfe, Dr. J.M., J.R. Cook, and Squadron Leader C.G. Durose. "Knowing What We Can Get from Training Devices: Substituting a Little Arithmetic for a Measure of Emotion," Proceedings of the Third International Symposium on Aviation Psychology, Columbus, Ohio, 1985, p. 617. Roscoe, Stanley N. "Simulator Qualification: Just as Phony as it Can Be," Illiana Aviation Sciences Limited, International Journal of Aviation Psychology 1, No. 4 (1991), p. 335-339. _____.( ed)., Aviation Psychology. Ames, Iowa: Iowa State University Press; 1980. Salas, Eduardo, Clint A. Bowers, and Lori Rhodenizer. "It is Not How Much You Have but How You Use It: Toward a Rational Use of Simulation to Support Aviation Training." International Journal of Aviation Psychology 8 (3) (1998), pp. 197-208 Svoboda, J.V., Ruth M. Heron, and H. Weinberg. "Emergency Manoevre Pilot Training in a Low-Cost Flight Simulator," Proceedings of the Seventh International Symposium on Aviation Psychology, Columbus, Ohio, 1993, p. 721. Talleur, Donald, Henry L. Taylor, Gavan Lintern, Charles L. Hulin, Tom W. Emanuel, Jr., and Sybil I. Phillips. "Effectiveness of an Integrated Instrument Training Syllabus Using a Personal Computer and a Certified Page 45 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Selected Bibliography Ground Trainer," University of Illinois, Proceedings of the Ninth International Symposium on Aviation Psychology, Columbus, Ohio, 1997. Taylor, Henry L., Gavan Lintern, Charles L. Hulin, Donald Talleur, Tom Emanuel, and Sybil Phillips; "Transfer of Training Effectiveness of Personal Computer-Based Aviation Training Devices," November 1996, Final Technical Report ARL-96-3-/FAA-96-2, prepared for the Federal Aviation Administration, Oklahoma City, OK, Contract DTFA- 94-G-044 [Summarized in this report.]. Wiener, Earl L. "Life in the Second Decade of the Glass Cockpit," Keynote Address, Seventh International Symposium on Aviation Psychology, Proceedings of the Seventh International Symposium on Aviation Psychology, Columbus, Ohio, 1993, p. 1. Williams, Kevin W., and Robert E. Blanchard. "Development of Qualification Guidelines for Personal Computer- Based Aviation Training Devices," February 1995, Final Report, Civil Aeromedical Institute, Federal Aviation Administration, U.S. Department of Transportation, Oklahoma City, Oklahoma, DOT/FAA/AM- 95-6. Williams, Kevin W.,"Qualification Guidelines for Personal Computer-Based Aviation Training Devices: Instrument Training," February 1996, Final Report, Civil Aeromedical Institute, Federal Aviation Administration, U.S. Department of Transportation, Oklahoma City, Oklahoma, DOT/FAA/AM-96-8. _____ (ed.) "DRAFT Proceedings of the Joint Industry-FAA Conference on Development and Use of PC-Based Aviation Training Devices: June 16-17, 1994, Oklahoma City, OK," July 1994, Civil Aeromedical Institute, Federal Aviation Administration, U.S. Department of Transportation, Oklahoma City, Oklahoma. Williams, Donna M., Robert W. Holt, and Deborah A. Boehm-Davis. "Training for Inter- Rater Reliability: Baselines and Benchmarks," Proceedings of the Ninth International Symposium on Aviation Psychology, Columbus, Ohio, 1997, pp. 514-520. Wilt, Donna Forsyth. "Clueless in the Cockpit: Situation Awareness of Instrument Flight Students," Proceedings of the Ninth International Symposium on Aviation Psychology, Columbus, Ohio, 1997, p. 1464. (348100) Page 46 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: info@www.gao.gov or visit GAO's World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States General Accounting Office Bulk Rate Washington, D.C. 20548-0001 Postage & Fees Paid GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested *** End of document. ***