The forecasters understood and adjusted for model uncertainty. Models are not always accurate due to lack of observations and inadequacies of model physics. The forecasters estimated uncertainty at 2 stages: Background Weather forecasters examine current weather conditions upstream of the forecast area. They transform these conditions, according to the principles of atmospheric physics, into the future weather in the forecast area. Satellite Image MM5: Numerical Model Previous research suggests the steps in the forecasting process are: 1. Initialize understanding of synoptic (large) scale weather 2.Build and consolidate a qualitative mental model (QMM) of the local weather (Trafton, Kirschenbaum, Tsui, Miyamoto, Ballas & Raymond, 2000) –From quantitative information –Cause & effect relationships of elements in atmosphere 3. Verify and adjust QMM (Pilske, Klinger, Hutton, Crandall, Knight, Klein 1997, Hoffman, 1991) –Check against other information sources (e.g. numerical models) –Much of information gathering at this stage 4. Project forward to the time of forecast RESULTS There are special challenges involved in naval aviation forecasting. PROCEDURE: Cognitive task analysis of navy forecasters Collected think aloud verbal protocols. Subjects verbalized thoughts while completing the task. (Ericsson & Simon, 1984) Recorded 4 forecasters’ verbalizations and computer screen images as they each produced a forecast, in February, Transcribed audio recordings. Broke down into statements and coded transcripts for information source (e.g. satellite, numerical models, radar imagery). We suggest that these replace building and consolidating a qualitative mental model. Advantages: Reduces working memory load. It is a practical approach for individuals who are making rapid decisions amid constant interruptions. Disadvantages: When there is an insufficient match between present situation and “if” condition, adjustment is difficult without the benefit of an integrated mental model. Question: What value do the forecasters add to numerical model predictions? PROCEDURE: linear regression analysis - Dependent variable = observed wind - Predictors = Navy forecasters predictions; Numerical Model predictions RESULTS: Wind Speed Correlation between Navy Forecasters’ prediction and the observed wind speed was greater than the correlation between numerical models and observed wind speed. Conclusions Navy forecasters have a streamlined forecasting process. - Use few information sources - Rely on rules of thumb Navy forecasters understand model dynamics. - Evaluate model uncertainty - Use several standard evaluation techniques - Adjust the model prediction Navy forecasters add value to numerical model predictions. Question: What is the impact of time pressure and information overload on weather forecasting? RESULTS (cont.) The forecasters relied heavily on numerical models Question: What is the forecasters’ understanding of model dynamics? References Ericcson KA, and Simon HA Protocol Analysis: Verbal Reports as Data. Cambridge, MA: MIT Press. Hoffman RR Human factors psychology in the support of forecasting: The design of advanced meteorological workstations. Weather and Forecasting 6: Pilske R, Klinger D, Hutton R, Crandell B, Knight B, Klein G Understanding skilled weather forecasting: Implications for training and design of forecasting tools. Technical Report Al/HR-Cr Trafton GJ, Kirschenbaum SS, Tsui TL, Miyamoto RT, Ballas JA, Raymond PD Turning pictures into numbers: extracting and generating information from complex visualizations. International Journal of Human-Computer Studies 53: Wind Direction The human forecaster outperformed the numerical model. Naturalistic Decision-Making: A Cognitive Task Analysis of Naval Weather Forecasting Susan Joslyn, Karla Schweitzer & Earl Hunt University of Washington The variance in actual wind speed accounted for by the human forecast subsumed that accounted for by the numerical model. Multiple R= Human Forecast Actual Numerical Model Statements were coded qualitative or quantitative. The forecasters’ mental representation was predominantly qualitative, with the exception of pressure. Fact statements by parameter: percent qualitative versus percent quantitative Qualitative Quantitative Numerical Model alone=.15 Human Forecast=.46 Observed=1 Forecaster AForecaster BForecaster CForecaster D Synoptic Scale Analysis 40% Synoptic Scale Analysis 8% Synoptic Scale Analysis 50% Synoptic Scale Analysis 16% Estimate Specific Parameters 52% Estimate Specific Parameters 65% Estimate Specific Parameters 40% Estimate Specific Parameters 6% Write Forecast 9% Write Forecast 3% Write Forecast 11% Write Forecast 78% Consolidate Mental Model 0% Check Mental Model 38% Goal Structure for each forecaster: Percent of source statements under each goal Most of the forecasters used rule of thumb (if-then rule) reasoning. “If I see a system coming into the coast, I forecast strong southerly winds over Whidbey Island.” “These systems aren’t normal… it's going to act a little bit different. ” (Forecaster A) Forecasts must be posted every six hours while completing other tasks. Forecasters are interrupted as many as 12 times while writing a single forecast. Forecasters learn about current weather conditions from: - Surface Charts - Satellite Imagery - Radar Imagery Forecasters predict future weather using current observations and numerical models: - Numerical models are computer programs that make weather predictions which are fairly accurate for large-scale phenomena over a short period of time. Navy forecasting office Regional Forecasts Weather Discussion Surface Observations Satellite Radar Numerical Models Total number of statements by source Human Forecast versus Numerical Model: Forecast wind direction or variable winds Direction: % Variable: % closer correct Human Forecast Numerical Model Both/Tie Numerical Model: MM5 Satellite Evaluation involves comparing patterns (e.g. position of low) in numerical models and satellite image (actual weather). Is the model doing a good job? All of the forecasters started with the synoptic scale analysis Most of the forecasters omitted the step of consolidating and verifying the qualitative mental model - Only Forecaster B (the most experienced forecaster) conformed to the predicted pattern Most of the forecasters gathered information in the middle stages of the process - Surprisingly, Forecaster D gathered most of his information after he began writing his forecast - A strategy to avoid maintaining large quantities of information amid constant interruptions? This research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N Estimating specific parameter values: 41% of model uncertainty statements involved checking the specific values. Synoptic Scale Analysis: 59% of statements regarding model uncertainty were made as forecasters examined the evolution of large-scale weather patterns over time. Example: Forecaster D compared the observed pressure to what the model had predicted. Access current pressure Calculate difference =.05 between current and forecast (error) Access predicted pressure for forecast time Adjust predicted =29.52 pressure based on current error Adjust predicted =29.54 pressure based on model bias