1. MURI Quarterly Meeting 1/31/02
APL Presentations
12/3/2018
Applied Physics Laboratory

2. Applied Physics Laboratory
Overview
Three presentations
David-Thoughts on the METOC Task
Scott- Ensemble Verification
Keith- Visualization Framework
APL’s POC
David-MURI proj. mgnt, CTA, & Navy METOC ops
Scott-Mesoscale modeling, ensem., & verification
Keith-METOC visualization & workflow
Jim-Statistical issues w/ uncertainty
12/3/2018
Applied Physics Laboratory

3. Applied Physics Laboratory
Updates
Recent Visits
Susan & David-NPMOF Whidbey: Nov
Scott & David-FNMOC & Pt Magu: Dec
David: NPMOC San Diego, USS Constellation, & NAVSPECWAR Mission Support Center (SEALS): Jan
General impression- all forecasters deal w/ uncertainty but that uncertainty is not conveyed to user
12/3/2018
Applied Physics Laboratory

4. Applied Physics Laboratory
Updates (cont.)
What are we doing?
Conducting CTA & other investigations to understand how uncertainty affects the domain
Evaluating verification strategies for ensemble systems
Exploring alternative design strategies & a visualization framework
12/3/2018
Applied Physics Laboratory

5. METOC Task Analysis: Literature Review
“Task analysis, as opposed to task description, should be a way of producing answers to questions (i.e., identifying potential performance failures or training needs and indicating how these problems might be solved.)”
Annett (2000)
12/3/2018
Applied Physics Laboratory

6. METOC Task Analysis: Literature Review
Hoffman (1991) provides good review of task analysis for forecaster domain and human factor design considerations for Advance Meteorological Workstation, but:
Task analysis of researchers, not forecasters
There has been a change in the chartroom paradigm
12/3/2018
Applied Physics Laboratory

7. METOC Task Analysis: A General Model
July Human Systems Checklist for METOC Forecasting (Appendix A)
Work in Progress (Appendix B)
Information Networks- “here be uncertainty”
12/3/2018
Applied Physics Laboratory

8. Applied Physics Laboratory
A Specific METOC Task
The Terminal Aerodrome Forecast (TAF)
KNUW G35KT 9999 FEW018 SCT045 QNH2967INS
TEMPO RA SCT015 BKN040 BKN100
BECMG G25KT 9999 SCT020 BKN060 BKN100 BKN200 QNH2973INS
TEMPO 2209 SHRA BKN020 BKN060 OVC100
12/3/2018
Applied Physics Laboratory

9. Applied Physics Laboratory
A Specific METOC Task
Flight Weather Briefing Form: (DD 175-1)
“Where the rubber meets the road!”
12/3/2018
Applied Physics Laboratory

10. Visualizing Uncertainty in Mesoscale Meteorology
Thoughts On
Verifying Ensemble Forecasts
31 Jan 2002
Scott Sandgathe

14. 36km Ensemble Mean and Selected Members
SLP, mb Thickness 2002 Jan 2200Z

15. 12km Ensemble Mean and Selected Members
SLP, Temperature, Wind 2002 Jan 2200Z

16. Verification of Mesoscale Features in NWP Models
Baldwin, Lakshmivarahan, and Klein
9th Conf. On Mesoscale Processes, 2001

17. Tracking of global ridge-trough patterns from Tribbia, Gilmour and Baumhaufner

18. Current global forecast and climate models produce ridge-trough transitions; however, the frequency of predicted occurrence is much less than the frequency of actual occurrence

19. Creating Concensus From Selected Ensemble Members
- Carr and Elsberry

20. Necessary Actions for Improved Dynamical Track Prediction
(48 h)
Small
Spread
(229 n mi)
Large
Error
Large
Spread
(806 n mi)
Error
No forecaster reasoning possible.
Help needed from modelers and data sources
to improve prediction accuracy
Recognize erroneous guidance group or outlier,
and formulate SCON that improves on NCON
Large
Spread
(406 n mi)
Small
Error
Small
Spread
(59 n mi)
Error
No forecaster reasoning required -- use the
non-selective consensus (NCON)
Recognize situation as having inherently low
predictability; must detect error mechanisms in
both outliers to avoid making SCON>>NCON

21. Applied Physics Laboratory
References
Cannon, A. J., P.H. Whitfield, and E.R. Lord, 2002: Automated, supervised synoptic map-pattern classification using recursive partitioning trees. AMS Symposium on Observations, Data Assimilation, and Probabilistic Prediction, pJ103-J109.
Carr. L.E. III, R.L. Elsberry, and M.A. Boothe, 1997: Condensed and updated version of the systematic approach meteorological knowledge base – Western North Pacific. NPS-MR , pp169.
Ebert, E.E., 2001: Ability of a poor man’s ensemble to predict the probability and distribution of precipitation. Mon. Wea. Rev., 129,
Gilmour, I., L.A. Smith, R. Buizza, 2001: Is 24 hours a long time in synoptic weather forecasting. J. Atmos. Sci., 58, -.
Grumm, R. and R. Hart, 2002: Effective use of regional ensemble data. AMS Symposium on Observations, Data Assimilation, and Probabilistic Prediction, pJ155-J159.
Marzban, C., 1998: Scalar measures of performance in rare-event situations. Wea. and Forecasting, 13,
12/3/2018
Applied Physics Laboratory

22. Visualization in the METOC Environment
R. Keith Kerr
12/3/2018
Applied Physics Laboratory

23. Applied Physics Laboratory
“Visualization”
A broad definition in the context of our work
The mental representation of concepts (spatially, temporally, and operationally) that serve to refine and enhance the efficiency and accuracy of a defined suite of tasks executed within a particular workflow context.
12/3/2018
Applied Physics Laboratory

24. The Visualization Framework
Must blend
Capturing of the “workflow” process for a suite of individual tasks that constitute a “product”
Integrate a varied range of workflows within common user-interface paradigms
12/3/2018
Applied Physics Laboratory

25. Applied Physics Laboratory
Framework (cont.)
Might involve
Automated “reasoning” and process control
Analytic plots
Geospatial (map-based) plots
Data resource management
Presentation “tools”
Specialized “viewing” environments
12/3/2018
Applied Physics Laboratory

26. Visualization Flow Mental task model Ontological representation
Reasoning (inference) engine
Process controller
METOC interface components
Products

27. Software Engineering Goals
Design “component” architecture for visualization of METOC information
Help implement useful research results within software prototype
Integrate prototype within METOC Information Management Framework
Install, maintain and support prototype in chosen test environment(s)
Research implementations of “cognitive paradigms” within workflow software
12/3/2018
Applied Physics Laboratory

28. Current Development Efforts
Developing design requirements based on task analysis
Refining design of previously developed components (data retrieval, inference, etc.)
Working with METOC community to define a modern information framework
Investigating relationship between ontology and the reasoning engine
Working with preliminary results from cognitive analysis group to specify what portion of overall study might be implemented in software – and what those requirements should be. e.g., should/could software elements that monitor forecasting procedures and products be a component in measuring overall effectiveness of the methodology.
APL is migrating (and in many cases re-designing) useful software components from previous METOC information programs (E.g., MIAWS – METOC Intelligent Agent Workflow System)
The development of an information framework that can be manipulated in part or in whole by other software is essential for future development. The METOC community is diverse and somewhat disorganized in their approach to information management. This is costly, confusing, and in many cases, antiquated.
It would be desirable to leverage new representational developments in markup languages and peer-to-peer object communication to generate rules for inferencing and control. Can we use XML, SOAP, and similar schemas?
12/3/2018
Applied Physics Laboratory

29. Current Development Efforts
Developing design requirements based on task analysis
Refining design of previously developed components (data retrieval, inference, etc.)
Working with METOC community to define a modern information framework
Investigating relationship between ontology and the reasoning engine
Working with preliminary results from cognitive analysis group to specify what portion of overall study might be implemented in software – and what those requirements should be. e.g., should/could software elements that monitor forecasting procedures and products be a component in measuring overall effectiveness of the methodology.
APL is migrating (and in many cases re-designing) useful software components from previous METOC information programs (E.g., MIAWS – METOC Intelligent Agent Workflow System)
The development of an information framework that can be manipulated in part or in whole by other software is essential for future development. The METOC community is diverse and somewhat disorganized in their approach to information management. This is costly, confusing, and in many cases, antiquated.
It would be desirable to leverage new representational developments in markup languages and peer-to-peer object communication to generate rules for inferencing and control. Can we use XML, SOAP, and similar schemas?
12/3/2018
Applied Physics Laboratory

30. Platform-independent, Three-tiered Services
Arbitrary Data Sources – Web, METOC data bases, models,
Local archives, etc.
METOC Center(s) Server(s) – Java Enterprise Environment
(Servlets, Server Pages, WebStart, Applets),
Process Management, “Reasoning” engine
Client Displays – Both static and dynamic interaction,
Local process management and “reasoning”,
METOC product creation tools, workflow monitoring

31. XIS – “one size fits all?”
Extremely sophisticated programming model
Excellent information handling and abstraction facilities
Already adopted by some Naval units and DII/COE certified
But…..as of today, lacks METOC annotational tools and fine-grained user interactivity with very large data sets (models)
12/3/2018
Applied Physics Laboratory

32. XIS Viewpoint