MURI Quarterly Meeting 1/31/02 APL Presentations 12/3/2018 Applied Physics Laboratory

Applied Physics Laboratory Overview Three presentations David-Thoughts on the METOC Task Scott- Ensemble Verification Keith- Visualization Framework APL’s POC David-MURI proj. mgnt, CTA, & Navy METOC ops Scott-Mesoscale modeling, ensem., & verification Keith-METOC visualization & workflow Jim-Statistical issues w/ uncertainty 12/3/2018 Applied Physics Laboratory

Applied Physics Laboratory Updates Recent Visits Susan & David-NPMOF Whidbey: Nov Scott & David-FNMOC & Pt Magu: Dec David: NPMOC San Diego, USS Constellation, & NAVSPECWAR Mission Support Center (SEALS): Jan General impression- all forecasters deal w/ uncertainty but that uncertainty is not conveyed to user 12/3/2018 Applied Physics Laboratory

Applied Physics Laboratory Updates (cont.) What are we doing? Conducting CTA & other investigations to understand how uncertainty affects the domain Evaluating verification strategies for ensemble systems Exploring alternative design strategies & a visualization framework 12/3/2018 Applied Physics Laboratory

METOC Task Analysis: Literature Review “Task analysis, as opposed to task description, should be a way of producing answers to questions (i.e., identifying potential performance failures or training needs and indicating how these problems might be solved.)” Annett (2000) 12/3/2018 Applied Physics Laboratory

METOC Task Analysis: Literature Review Hoffman (1991) provides good review of task analysis for forecaster domain and human factor design considerations for Advance Meteorological Workstation, but: Task analysis of researchers, not forecasters There has been a change in the chartroom paradigm 12/3/2018 Applied Physics Laboratory

METOC Task Analysis: A General Model July 2000- Human Systems Checklist for METOC Forecasting (Appendix A) Work in Progress (Appendix B) Information Networks- “here be uncertainty” 12/3/2018 Applied Physics Laboratory

Applied Physics Laboratory A Specific METOC Task The Terminal Aerodrome Forecast (TAF) KNUW 200909 15025G35KT 9999 FEW018 SCT045 QNH2967INS TEMPO 1018 -RA SCT015 BKN040 BKN100 BECMG 1820 14015G25KT 9999 SCT020 BKN060 BKN100 BKN200 QNH2973INS TEMPO 2209 SHRA BKN020 BKN060 OVC100 12/3/2018 Applied Physics Laboratory

Applied Physics Laboratory A Specific METOC Task Flight Weather Briefing Form: (DD 175-1) “Where the rubber meets the road!” 12/3/2018 Applied Physics Laboratory

Visualizing Uncertainty in Mesoscale Meteorology Thoughts On Verifying Ensemble Forecasts 31 Jan 2002 Scott Sandgathe

36km Ensemble Mean and Selected Members SLP, 1000-500mb Thickness 2002 Jan 2200Z

12km Ensemble Mean and Selected Members SLP, Temperature, Wind 2002 Jan 2200Z

Verification of Mesoscale Features in NWP Models Baldwin, Lakshmivarahan, and Klein 9th Conf. On Mesoscale Processes, 2001

Tracking of global ridge-trough patterns from Tribbia, Gilmour and Baumhaufner

Current global forecast and climate models produce ridge-trough transitions; however, the frequency of predicted occurrence is much less than the frequency of actual occurrence

Creating Concensus From Selected Ensemble Members - Carr and Elsberry

Necessary Actions for Improved Dynamical Track Prediction (48 h) Small Spread (229 n mi) Large Error Large Spread (806 n mi) Error No forecaster reasoning possible. Help needed from modelers and data sources to improve prediction accuracy Recognize erroneous guidance group or outlier, and formulate SCON that improves on NCON Large Spread (406 n mi) Small Error Small Spread (59 n mi) Error No forecaster reasoning required -- use the non-selective consensus (NCON) Recognize situation as having inherently low predictability; must detect error mechanisms in both outliers to avoid making SCON>>NCON

Applied Physics Laboratory References Cannon, A. J., P.H. Whitfield, and E.R. Lord, 2002: Automated, supervised synoptic map-pattern classification using recursive partitioning trees. AMS Symposium on Observations, Data Assimilation, and Probabilistic Prediction, pJ103-J109. Carr. L.E. III, R.L. Elsberry, and M.A. Boothe, 1997: Condensed and updated version of the systematic approach meteorological knowledge base – Western North Pacific. NPS-MR-98-002, pp169. Ebert, E.E., 2001: Ability of a poor man’s ensemble to predict the probability and distribution of precipitation. Mon. Wea. Rev., 129, 2461-2480. Gilmour, I., L.A. Smith, R. Buizza, 2001: Is 24 hours a long time in synoptic weather forecasting. J. Atmos. Sci., 58, -. Grumm, R. and R. Hart, 2002: Effective use of regional ensemble data. AMS Symposium on Observations, Data Assimilation, and Probabilistic Prediction, pJ155-J159. Marzban, C., 1998: Scalar measures of performance in rare-event situations. Wea. and Forecasting, 13, 753-763. 12/3/2018 Applied Physics Laboratory

Visualization in the METOC Environment R. Keith Kerr 12/3/2018 Applied Physics Laboratory

Applied Physics Laboratory “Visualization” A broad definition in the context of our work The mental representation of concepts (spatially, temporally, and operationally) that serve to refine and enhance the efficiency and accuracy of a defined suite of tasks executed within a particular workflow context. 12/3/2018 Applied Physics Laboratory

The Visualization Framework Must blend Capturing of the “workflow” process for a suite of individual tasks that constitute a “product” Integrate a varied range of workflows within common user-interface paradigms 12/3/2018 Applied Physics Laboratory

Applied Physics Laboratory Framework (cont.) Might involve Automated “reasoning” and process control Analytic plots Geospatial (map-based) plots Data resource management Presentation “tools” Specialized “viewing” environments 12/3/2018 Applied Physics Laboratory

Visualization Flow Mental task model Ontological representation Reasoning (inference) engine Process controller METOC interface components Products

Software Engineering Goals Design “component” architecture for visualization of METOC information Help implement useful research results within software prototype Integrate prototype within METOC Information Management Framework Install, maintain and support prototype in chosen test environment(s) Research implementations of “cognitive paradigms” within workflow software 12/3/2018 Applied Physics Laboratory

Current Development Efforts Developing design requirements based on task analysis Refining design of previously developed components (data retrieval, inference, etc.) Working with METOC community to define a modern information framework Investigating relationship between ontology and the reasoning engine Working with preliminary results from cognitive analysis group to specify what portion of overall study might be implemented in software – and what those requirements should be. e.g., should/could software elements that monitor forecasting procedures and products be a component in measuring overall effectiveness of the methodology. APL is migrating (and in many cases re-designing) useful software components from previous METOC information programs (E.g., MIAWS – METOC Intelligent Agent Workflow System) The development of an information framework that can be manipulated in part or in whole by other software is essential for future development. The METOC community is diverse and somewhat disorganized in their approach to information management. This is costly, confusing, and in many cases, antiquated. It would be desirable to leverage new representational developments in markup languages and peer-to-peer object communication to generate rules for inferencing and control. Can we use XML, SOAP, and similar schemas? 12/3/2018 Applied Physics Laboratory

Current Development Efforts Developing design requirements based on task analysis Refining design of previously developed components (data retrieval, inference, etc.) Working with METOC community to define a modern information framework Investigating relationship between ontology and the reasoning engine Working with preliminary results from cognitive analysis group to specify what portion of overall study might be implemented in software – and what those requirements should be. e.g., should/could software elements that monitor forecasting procedures and products be a component in measuring overall effectiveness of the methodology. APL is migrating (and in many cases re-designing) useful software components from previous METOC information programs (E.g., MIAWS – METOC Intelligent Agent Workflow System) The development of an information framework that can be manipulated in part or in whole by other software is essential for future development. The METOC community is diverse and somewhat disorganized in their approach to information management. This is costly, confusing, and in many cases, antiquated. It would be desirable to leverage new representational developments in markup languages and peer-to-peer object communication to generate rules for inferencing and control. Can we use XML, SOAP, and similar schemas? 12/3/2018 Applied Physics Laboratory

Platform-independent, Three-tiered Services Arbitrary Data Sources – Web, METOC data bases, models, Local archives, etc. METOC Center(s) Server(s) – Java Enterprise Environment (Servlets, Server Pages, WebStart, Applets), Process Management, “Reasoning” engine Client Displays – Both static and dynamic interaction, Local process management and “reasoning”, METOC product creation tools, workflow monitoring

XIS – “one size fits all?” Extremely sophisticated programming model Excellent information handling and abstraction facilities Already adopted by some Naval units and DII/COE certified But…..as of today, lacks METOC annotational tools and fine-grained user interactivity with very large data sets (models) 12/3/2018 Applied Physics Laboratory

XIS Viewpoint