In addition to conducting site visits, several methodologies were employed to objectively assess how accurately the ASOS reflected actual weather conditions and transmitted the weather data to the user. The evaluation methodologies analyzed the ASOS in the areas of validity/representativeness, availability, reliability, and responsiveness. Pre-commissioning packages were also evaluated.
The areas of analysis included:
Validity/Representativeness
How often did the human edit the ASOS produced observation at a staffed site, what elements were edited, and were the edits operationally significant?
How did the ASOS produced observation correlate to a human observation in a blind comparison?
Availability
What was the availability of the ASOS observation to the end user?
What was the availability of the ASOS observation at the ASOS site?
What was the long line availability for the ASOS observation dissemination?
Reliability
How often was the ASOS system non-operational?
How often were individual sensors non-operational?
Responsiveness
a. How did the system respond to changing conditions in critical weather elements?
Pre-Commissioning Process
a. How thorough and complete is the ASOS pre-commissioning process?
More than one methodology may have been utilized to establish a measurement parameter. For example, the validity/representativeness of an ASOS observation was examined by analyzing the edits performed by an observer at a staffed ASOS site and the deviations between a fully manual observation and a non-commissioned ASOS (blind comparison). Each methodology alone will not produce an absolute conclusion. All of the analytical processes used in the methodologies will be utilized to support the findings and recommendations resulting from this study.
It is important to keep in mind the difference between reliability and validity (or representativeness). Comparing an automated observation to a manual observation produces a validity value. If the two observations agree, the validity is perfect. If the observations are unrelated, the validity is zero. However, a measurement is generally not wholly valid or invalid, but rather varies in its degree of validity. The applied analyses employ a correlation coefficient to measure the degree of validity.
Opposed to that line of thinking, reliability/availability is clearer cut. Either the observation is available to the user, or it is not. The question would be why is it not available? Is it due to ASOS failure, sensor failure or communications failure? It follows then, that availability/reliability can exist with or without validity; however, if the observation is not delivered to the user, then validity is irrelevant.
In order to conduct a valid assessment of the ASOS, it was essential to use objective measurement. Subjective measurement involves person-specific data, representing the judgments of one person; it is non-verifiable and potentially biased. This report contains some subjective information in the synopsis of user comments in the site visit reports. However, the requirement for objective analysis was paramount to this assessment and was utilized in the evaluation methodologies. The use of quantitative and qualitative analysis insured that objective processes supported the conclusions drawn.
The process for each methodology involved the following:
Statement of hypothesis, e.g. what do we intend to determine/discover with each individual assessment methodology
Analysis methodology/analytical formula
Assumptions utilized in the planning and execution of each analysis
Source(s) and method(s) of data compilation
Extraneous variables and the process of qualification to compensate for these variables