1 Using the Weather to Teach Computing Topics B. Plale, Sangmi Lee, AJ Ragusa Indiana University

2 Outline Forecasting Severe Storms –Why a better computing infrastructure is needed –Grid computing addresses the problem –Work being done in context of LEAD project Computing architecture to enable better weather forecasting Demo

3 Motivation for LEAD Each year, mesoscale weather – floods, tornadoes, hail, strong winds, lightning, and winter storms – causes hundreds of deaths, routinely disrupts transportation and commerce, and results in annual economic losses > $13B.

4 Conventional Numerical Weather Prediction OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites

5 OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Conventional Numerical Weather Prediction

6 Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Prediction PCs to Teraflop Systems Conventional Numerical Weather Prediction OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites

7 Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Prediction PCs to Teraflop Systems Product Generation, Display, Dissemination Conventional Numerical Weather Prediction OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites

8 Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Prediction PCs to Teraflop Systems Product Generation, Display, Dissemination End Users NWS Private Companies Students Conventional Numerical Weather Prediction OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites

9 Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Prediction PCs to Teraflop Systems Product Generation, Display, Dissemination End Users NWS Private Companies Students Conventional Numerical Weather Prediction OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites The Process is Entirely Serial and Pre-Scheduled: No Response to Weather! The Process is Entirely Serial and Pre-Scheduled: No Response to Weather!

10 Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Prediction PCs to Teraflop Systems Product Generation, Display, Dissemination End Users NWS Private Companies Students The LEAD Vision: No Longer Serial or Static OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites

11 Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Prediction PCs to Teraflop Systems Product Generation, Display, Dissemination End Users NWS Private Companies Students The LEAD Vision: No Longer Serial or Static OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites

12 The Value of Being Able to Respond to the Weather: Dynamic Adaptivity

13 Radar Observations of a Storm System In Kansas on 20 June 2001

14 11-hr Forecast

15 9-hr Forecast

16 5-hr Forecast

17 3-hr Forecast Moral: Need to do more short forecasts, because they are more accurate

18 The Value of Local Observations

19 What Do Operational Forecast Models Currently Predict? Bands of rain, and high and low pressure, but that’s about it.

20 What Causes the Problems? Do we really understand the conditions that result in a funnel cloud?

21 Why the Lack of Detail in Current Forecasts? This Thunderstorm Falls Through the Cracks

22 Why the Lack of Detail in Current Forecasts?

23 The Solution…. Fine-Scale Local Observations Fine Grid Spacing in Forecast Models +

24 Example: The March 28, 2000 Fort Worth Tornado

25 TV Radar Image of the Hook Echo

26 NWS 12-hr Forecast Valid Near Tornado Time (shading indicates precipitation)

27 6 pm 7 pm8 pm Radar Hourly Radar Observations (Fort Worth Shown by the Pink Star)

28 6 pm Radar Computer Forecast 2 hr

29 6 pm 7 pm Radar Computer Forecast 2 hr 3 hr

30 6 pm 7 pm8 pm Radar Computer Forecast 2 hr 3 hr 4 hr

31 Fcst w/o Radar 2 hr 3 hr 4 hr Radar 6 pm 7 pm8 pm

32 Outline The Weather –Why cyberinfrastructure is needed –LEAD project – addressing the problem Cyberinfrastructure based on a web service architecture for the Grid Prototype demo

33 What is the Grid? A collection of resources (computers, databases, telescopes, etc.) that can be used by a wide range of users with a wide range of skills. More than the Internet –Built on top of the Internet The “Grid” is a collection of web services layered on top of the Internet. Security Data Management Service Data Management Service Accounting Service Accounting Service Logging Event Service Policy Administration & Monitoring Administration & Monitoring Grid Orchestration Registries Reservations And Scheduling Reservations And Scheduling Web Services layer Internet Physical Resource Layer

34 Predicting Severe Storms Lightning Data Server NEXRAD Radar Data Server Satellite Data Server Surface and Upper-Air Data Server SUNY Albany Wisconsin/SSEC NASA, NOAAPort EROS Data Center I D D Historical Observations and Model Output Operational Model Grids and Server ProjectCONDUIT I D D NOMADSNCDC Hydrologic Data Server I D D NWS River Forecast Centers Air Quality Data Server EPA I D D GPS Meteorological Data Server SuomiNet I D D Oceanograp hic Data DODS Digital Library Holdings DLESE Demographic Data Server Field Program & User Generated Data UCAR/JOSS Individual Investigators Abilene/NGI I D D Large scale, real-time Simulation Grid The LEAD project Univ. of Oklahoma

35 Very Simple Scenario to Run Forecast Search for data set, run simulation, and catalog results. –Query metadata catalog for dataset –Use result of query a large WRF simulation –Allocate storage on remote resource –Move WRF output to that allocated space –Record output location and computation history in a metadata catalog. How does a user describe such a scenario as a workflow or distributed application? How do we free the user from details of distributed computing in a service oriented architecture? What does a service architecture mean in this context? Can it be done by a component composition approach?

36 Web Services Why does the web work? –A language with few verbs (get, put, post) and many nouns (documents). Corba & Java RMI are object models which present a problem. –Object identity and lifetime is bound to its container, –Whereas a web address is persistent. RPC/RMI requires too much synchronization –For reliability make “connections” implicit. –Communicate with simple “standard” message exchanges.

37 So what is a web service? A network “endpoint”, i.e. server, that implements one or more “ports” –Each port is defined by the message types that accepts and the messages it returns. A Web Service is specified by a “Web Service Definition Language” xml document. –Given the WSDL for a web service you know all you need to interact with it. Web Service Standards exist for security, policy, reliability, addressing, notification, choreography and workflow. –It is the basis for MS.NET, IBM Websphere, SUN, Oracle, BEA, HP, … –It is the basis for the new Grid standards like WSRF and OGSA.

38 Web Site vs Web Service The Web Site –Designed to pass http get/post/put request to between a browser and a web server. –Google has a web site. The Web Service –Designed for services to talk to other services by exchanging xml messages –Google also provides a web service so Google may be used in distributed apps Client’s Browser Web Server Web Server Web Service Web Service Web Service Web Service Web Service Web Service

39 An Example The program: –Run a query against a metadata catalog and extract simulation boundary conditions –Allocate storage for simulation output –Run the simulation –Save result metadata reference for output to the metadata catalog. –Record event log of execution to the catalog. Services/components in our example are –Metadata catalog –Storage Allocator –WRF Simulation Engine –Execution history recorder Metadata Catalog query input output Query results Metadata Catalog reference input output notification mdata

40 The Workflow – as specified by the scientist WRF Factory Storage requirements Space Allocator Space Allocator File Mover “done” Metadata Catalog “done” Resource info Experiment Name (Notification Topic) Output URL Notification Broker Final URL Parameter file Event Listener Event Listener “done” Metadata Catalog query

41 The Portal User’s View of the Grid A very sophisticated web browser. Lets a classroom teacher create an experiment (to run a forecast model for Hurricane Ivan), then submit the experiment to the “Grid” for computing. The results can be viewed graphically within the portal.

42 Portal as Point of Access to Grid Security Data Management Service Data Management Service Accounting Service Accounting Service Logging Event Service Policy Administration & Monitoring Administration & Monitoring Grid Orchestration Registries and Name binding Registries and Name binding Reservations And Scheduling Reservations And Scheduling Open Grid Service Architecture Layer Web Services Resource Framework – Web Services Notification Grid Portal Server Grid Portal Server https Physical Resource Layer SOAP & WS-Security Keeps information about all the different users

43 Portal Architecture (OGCE) Building on Standard Technologies –Portlet Design (JSR-168) IBM, Oracle, Sun, BEA, Apache –Grid standards: Java CoG, Web/Grid Services User configurable, Service Oriented Based on Portlet Design –A portlet is a component within the portal that provides the interface between the user and some service –Portlets can be exchanged, interoperate Portal contaner Local Portlets Grid Service Portlets Java COG API Java CoG Kit Grid Services Grid Protocols GRAM, MDS-LDAD MyProxy SOAP ws call Grid Services Web Services Client’s Browser

44 Factory myLEAD agent myLEAD agent WRF model Data mining task Data mining task workflow myLEAD service myLEAD service LEAD Portal service LEAD Portal service Storage Repository service Storage Repository service myLEAD portlet /var/tmp/wrf_tmp IU NCSA Putting it together

45 Managing Workflow 1.Portlets exist to submit jobs to a condor web-service and monitor results 2.BPEL4WS is web-service workflow standard. Interface is under development. 3.CCA components can also be managed from the portal.

46 Science Portal Deployments in Collaboration with OGCE, DOE Fusion Portal, NCSA, NPACI/SDSC and others

47

48

49

50

51 Thank You