Aerospace Medicine Technical Reports
FAA Office of Aerospace Medicine
Civil Aerospace Medical Institute
Report No: DOT/FAA/AM-10/17
Title and Subtitle: Effects of Video Weather Training Products, Web-Based Preflight Weather Briefing, and Local Vs. Non-Local Pilots on General Aviation Pilot Weather Knowledge and Flight Behavior, Phase 3
Report Date: November 2010
Authors: Knecht WR, Lenz M
Abstract: The primary purpose of Phases 1 and 2 of this research was to test the effects of video weather training products on weather-related risk-taking. During the investigation, two unexpected observations were made:
To assess if any of the IMC violations were willful (rather than inadvertent), we sent a brief questionnaire to the nine pilots of interest. Five responded. After analysis, the leading explanation seemed that their flight profiles were consistent with preflight terrain avoidance planning (TAP). These pilots seemed determined to fly straight and level above the highest known obstacle, even if that obstacle was distant and TAP altitude meant flying initial VFR-into-IMC.
The average group decline in certification exam scores was equally significant from a logical standpoint. Since knowledge retention tends to be a function of knowledge relevancy, if FAA test questions were uniformly relevant to real-world weather encounters, we would expect pilots' scores to increase with experience, not decrease. Since experience tends to increase with time, this should offset the normal decay process of forgetting.
However, this study shows that it did not. This was consistent with pilot anecdotes that FAA test questions often seemed, to them, "trick questions," or otherwise based on tasks that pilots rarely do and conditions rarely encountered. This suggests ways to improve FAA exams:
Critics may argue that the relatively small percentage of weather-related questions on any given exam could make the test hard to pass for some individuals due to sampling error. However, this could be addressed by computerized adaptive testing, which presents harder questions to a candidate after correct answers, and easier questions after incorrect answers. Computerized adaptive testing quickly homes in on a candidate's native ability level and self-terminates after reaching a preset reliability (e.g., 95%). Because adaptive tests tend to be more efficient (shorter and more reliable) than fixed-item tests, this would free up more testing time for weather-related items, making sampling error much less of a problem.
Key Words: Weather, Training, Pre-Flight Briefing, Weather Knowledge, Flight Behavior
No. of Pages: 16
Civil Aerospace Medical Institute
Report No: DOT/FAA/AM-10/17
Title and Subtitle: Effects of Video Weather Training Products, Web-Based Preflight Weather Briefing, and Local Vs. Non-Local Pilots on General Aviation Pilot Weather Knowledge and Flight Behavior, Phase 3
Report Date: November 2010
Authors: Knecht WR, Lenz M
Abstract: The primary purpose of Phases 1 and 2 of this research was to test the effects of video weather training products on weather-related risk-taking. During the investigation, two unexpected observations were made:
- Despite specific instructions to fly visual-flight-rules-only (VFR), nine of 50 Phase 1 pilots spent more than 10 min in simulated instrument meteorological conditions (IMC), plus three of those nine repeated that behavior in Phase 2;
- Whole-group (N=50) weather knowledge test scores were significantly lower (19%, p.001) than average FAA certification exam scores obtained by freshly licensed pilots, implying knowledge decay over time.
To assess if any of the IMC violations were willful (rather than inadvertent), we sent a brief questionnaire to the nine pilots of interest. Five responded. After analysis, the leading explanation seemed that their flight profiles were consistent with preflight terrain avoidance planning (TAP). These pilots seemed determined to fly straight and level above the highest known obstacle, even if that obstacle was distant and TAP altitude meant flying initial VFR-into-IMC.
The average group decline in certification exam scores was equally significant from a logical standpoint. Since knowledge retention tends to be a function of knowledge relevancy, if FAA test questions were uniformly relevant to real-world weather encounters, we would expect pilots' scores to increase with experience, not decrease. Since experience tends to increase with time, this should offset the normal decay process of forgetting.
However, this study shows that it did not. This was consistent with pilot anecdotes that FAA test questions often seemed, to them, "trick questions," or otherwise based on tasks that pilots rarely do and conditions rarely encountered. This suggests ways to improve FAA exams:
- Screen existing questions for real-world relevancy, eliminating those based solely on rote learning;
- Scale the relative number of weather-test items to the relative hazard and/or encounter frequency of real-world weather types (dangerous and common weather types deserve relatively more test questions);
- Computerize the testing procedure;
- Require pilots to pass a certain percentage of weather questions.
Critics may argue that the relatively small percentage of weather-related questions on any given exam could make the test hard to pass for some individuals due to sampling error. However, this could be addressed by computerized adaptive testing, which presents harder questions to a candidate after correct answers, and easier questions after incorrect answers. Computerized adaptive testing quickly homes in on a candidate's native ability level and self-terminates after reaching a preset reliability (e.g., 95%). Because adaptive tests tend to be more efficient (shorter and more reliable) than fixed-item tests, this would free up more testing time for weather-related items, making sampling error much less of a problem.
Key Words: Weather, Training, Pre-Flight Briefing, Weather Knowledge, Flight Behavior
No. of Pages: 16
Last updated: Sunday, February 14, 2016