The purpose of this evaluation methodology was to determine, by analyzing meteorological data, the responsiveness of the ASOS and whether the ASOS observations lead or lag the human observer in reporting changing weather conditions.
This analysis was performed on the same pre-commissioned ASOS systems used in the Validity—Blind Comparison of Automated and Manual Observations Methodology (see Table 3.7-1). Figure 3.7-1 is a map showing the ASOS sites included in this analysis.
An analysis was conducted on the ceiling and visibility elements of the ASOS observations during periods of changing weather. The objective was to determine whether the ASOS observations lead or lag the human observer during periods of changing weather conditions.
The observation from the human observer is considered the baseline for comparison purposes.
The observer was carrying out a Basic Weather Watch, therefore the human observer may not report a condition as soon as it occurs.
ASOS is designed to lag slightly behind the actual weather due to algorithms that are built into the system. The visibility measured by ASOS is the average one minute value from the past ten minutes. For ceiling, the ASOS algorithm evaluates 30 minutes of data. The last ten minutes of data is double-weighted to reflect more accurately the current conditions.
The Operator Interface Display, if available, was disabled so that the observer could not augment or backup the ASOS observation.
The following observational parameters were evaluated:
Ceiling height during periods of change
The primary data sources were:
ASOS data downloaded directly from the ASOS site.
Manual data downloaded from the University of Michigan weather page.
The tools used to analyze the data were the Parse Data software developed by NWS, Microsoft ACCESS Database, and Microsoft EXCEL Spreadsheet.
The analyst examined the data base compiled for the Validity—Blind Comparison of Automated and Manual Observations portion of this study (Section 3.4) and extracted those time periods where the human was reporting a changing condition in the Marginal Visual Flight Rules (MVFR) and Instrument Flight Rules (IFR) categories. The ASOS weather trend was evaluated near the same time period to determine whether the ASOS was leading or lagging the human in reporting the change.
When the ceiling was increasing, the ASOS reported a greater ceiling height than the human 74% of the time did on average for the six sites studied. The sample size for ceiling lead/lag consisted of 117.9 hours. Figure 3.7-2 is an overall comparison of ASOS lead/lag for ceiling. Although the algorithms in ASOS cause the reported ceiling to lag slightly behind the actual weather changes, there is also a natural tendency for the human observer to "wait and see." This introduces a lag from the time the observer notices and evaluates the improving change to the time that the observer changes the weather observation to reflect this improvement.
These findings are consistent with the 1995 ASOS Demo Report which found that: "analysis shows that the ASOS tends to lead the observer in reporting improving conditions despite algorithms which are designed to provide a slight delay in such reports." As described in Paragraph 3.4 of the Validity—Blind Comparison of Automated and Manual Observations Methodology, human estimation of the ceiling (in the absence of ceilometers to assist them), may actually result in the ASOS leading the observer in reporting improving conditions.
When the ceiling was decreasing, the ASOS led the human 40% of the time, on average, for the six sites studied (see Figure 3.7-3). The individual site percentages ranged from 25% at Wausau, WI (AUW) to 56% at Bluefield, West Virginia (BLF). These percentages were much more variable than in the case when the ceiling was increasing. This part of the analysis included 63.5 hours of decreasing ceiling data. The human observer leading the ASOS in almost two thirds of the decreasing ceiling cases also reflects the human tendency towards "bringing the ceiling down" more rapidly than ASOS.
Due to the relatively good weather, there were only 27.4 hours of either increasing or decreasing visibility data, which made it difficult to draw meaningful conclusions. Therefore, no analysis was done on visibility.