3.2 Validity—Analysis of Observations for Augmentation/backup
The purpose of this evaluation methodology was to determine the validity of the ASOS by analyzing the edited weather observations made by weather observers for augmentation and backup. Analyzing the weather observations made by the weather observers included ascertaining what changes or additions the observer made, i.e., weather elements not reported by the ASOS, weather elements missing, and/or weather elements that were considered non-representative.
The site selection criteria for sites utilized in this analysis methodology were:
Site was a commissioned ASOS site.
Site had an observer performing augmentation and back-up services. Observers could be FAA Contract Weather Observers (CWO), NWS observers (either NWS employees or CWOs), FAA Automated Flight Service Station (AFSS) specialists, or Air Traffic Control Specialists in a tower performing Limited Aviation Weather Reporting Station (LAWRS) services.
Site had an operational printer at the site that was accessible to the FAA for the collection of the printouts.
Five sites that met the selection criteria and had a history of user comments and/or complaints were analyzed. The site locations, airport location identifiers, type of weather observers, Service Levels (see Table 1.1-1), and ASOS commissioning dates are listed in Table 3.2-1. Figure 3.2-1 shows the locations of the ASOS sites included in the Analysis of Observations for Augmentation/Backup Methodology.
The process for this task was to analyze 30 days of weather observation data from selected ASOS systems. Printouts from each of the selected ASOS sites were examined. These printouts indicated the observations that were transmitted, each time the observer entered the "edit" mode in the system, any changes or additions the observer made to the observations, and the system/sensor availability status (when a sensor failed or was off-line for maintenance).
The edit entries were logged according to the weather element that was changed and the operational significance of the change was also noted. The total number of hourly and special observations created each day was noted. Procedural errors were logged for informational use outside of this assessment.
The following assumptions were made prior to the start of this analysis:
If an observation was changed or augmented, the observer was correct.
Certain changes/edits were considered operationally significant and others were not.
An operationally significant edit entry created a transmitted observation.
Augmentation requirements differ at the site depending upon the Service Level of the site, e.g., more augmentation entries are required at Boston (Service Level A) than at Ketchikan or Sioux City (both Service Level C). Therefore, the analytical value of the augmentation averages of all sites is questionable.
Operationally significant changes are defined as those that take an observation across one of the parameters established for the issuance of a special observation as listed in FAA Order 7900.5A. These parameters are:
Wind shift - wind direction changes of 45 degrees or more in less than 15 minutes with wind speed of 10 knots or more
Visibility changes to decrease below or increase to equal or above 3, 2, 1, or ½ mile
Tornado, funnel cloud, or waterspout observed to begin or end
Thunderstorm begins or ends
Hail begins or ends, freezing precipitation, or ice pellets begin, end or change intensity
Squall - wind increases 16 knots or more and is sustained at 22 knots for one minute
When broken or overcast clouds form or dissipate at or below 3,000 feet, 1,500 feet, 1,000 feet, 500 feet or 200 feet
These parameters are considered the changes that could have an impact upon the operation of aircraft at an airport and are the same parameters the human observer and the ASOS utilize for issuing special observations.
Observers who were physically located at the selected sites provided the printouts. The printouts provided the following data sets:
Number of observations, both hourly (Aviation Routine Weather Reports [METAR]) and "specials," (Aviation Selected Special Weather Report [SPECI]) transmitted longline through the ASOS system.
Number of times the observer entered the edit mode on the ASOS system:
Number of times each critical weather element was edited
Number of times these edits were operationally significant (based on definition established in Paragraph 3.2.3.3)
Number of times the edit reduced or increased operational criteria
Number of times edit mode was entered, but no changes made
Number of edits made to augment elements beyond the capability of the system in accordance with the service standards.
Number of outages:
Total ASOS system failures
Total sensor failures
Outages due to maintenance
Edits made to backup missing elements
While the intent of the methodology was not to identify system/sensor availability, it was documented for use in the availability portion of the assessment.
This assessment methodology utilized raw data without the manipulation of any variables. There were certain variables common to all sites and some unique to specific sites that must be considered in the analysis process.
Even though it is assumed that the observation of the human is considered correct when it differs from the ASOS mechanical observation, there is also another known potential variable at certain sites where air traffic controllers are providing augmentation and backup to the ASOS. When weather conditions are at or near certain break points, such as from visual flight rules to instrument flight rules or approach minima, there is the possibility of conflict between the accuracy of the observation and the air traffic control need for aircraft movement efficiency and airport capacity. This is an unknown variable and cannot be computed in a +/- percentage value.
It is acknowledged that there exists a wide range of observer technical ability. Therefore, it is reasonable to assume that the quality of observations will differ from site to site, and may even differ from observer to observer within a location. This is an unknown variable and cannot be computed in a +/- percentage value.
It is believed that there exists a wide range of ASOS acceptance levels among the observers and that the comfort level of an observer with the equipment may influence the confidence of the observer in the validity of the ASOS observed elements. This is an unknown variable and cannot be computed in a +/- percentage value.
Based upon analysis of the printouts, it is acknowledged that certain operator errors and ASOS interface procedural errors occur. These errors may have had an impact upon the creation of modified reports that do not meet the criteria established in FAA Order 7900.5A, Surface Weather Observing, METAR.
This analysis methodology produced data that indicated the frequency an observer at an ASOS site performed editing functions. These editing functions were performed to augment weather elements required at that site that were beyond the capability of the system, to backup failed sensors, or to change weather elements that the observer considered not representative of the airport environment. The data analyzed was the record of the actual editing activity. It was assumed that the most significant edits were those that changed a weather element to cross a threshold that would initiate a SPECI. Therefore, the data was presented as:
The percentage of times an edit entry was made but no elements were altered (no-change edits).
The percentage of times an edit was made to a value, but did not qualify as an operationally significant change.
The percentage of times an edit was made to enter augmentation data.
The percentage of times an edit was required to backup a missing element.
The percentage of times an operationally significant edit was required to change a non-representative value.
The average percentages of all sites are presented in the summary chart, Figure 3.2-2, and the individual site results are presented in Figures 3.2-3a-e. The summary indicates that, on an average, 72% of the edit entries the observers at these sites made were either no-change edits (entering the edit mode and exiting without changing any weather element), or non-operationally significant changes (those which would have no impact on aircraft operation). Missing element back-up entries accounted for an average 1% of all edits entries, and ASOS augmentation averaged 11% of all edit entries.
An average of 16% of all edits made were to change an ASOS reported weather value that was considered by the observer to be non-representative and that was operationally significant to aviation.
Figure 3.2-3 Percentage Results for Non-Representative Edits: All Sites