1. The LEAD Effort at Unidata The Unidata Seminar will start at 1:30 PM MST

2. The LEAD Effort at Unidata Tom Baltzer, Brian Kelly, Doug Lindholm, Anne Wilson December 14, 2005

3. LEAD is funded by the National Science Foundation under the following Cooperative Agreements: ATM-0331594 ATM-0331591 ATM-0331574 ATM-0331480 ATM-0331579 ATM-0331586 ATM-0331587 ATM-0331578

4. Outline 1.Setting the Stage: Introduction to LEAD and Unidata’s LEAD Efforts: Anne 2.Application of current technology on the LEAD testbeds: Tom 3.The LEAD Hardware at Unidata: Brian 4.The THREDDS Data Repository: Doug

5. Setting the Stage: Introduction to LEAD and Unidata’s LEAD Efforts Anne Wilson

6. Current IT Barriers to Mesoscale Weather Research and Education Data and tools useable mainly by experts Researchers and educators constrained by hardware limitations Rigid, brittle technology can’t accommodate mesoscale weather research requirements: –real time, on demand, dynamic data processing and sensor steering

7. A Solution: Linked Environments for Atmospheric Discovery (LEAD) Funded by NSF Large Information Technology Research (ITR) award Produce a web service based, scalable framework for handling meteorological data and model output: –Identifying, accessing, preparing, assimilating, predicting, managing, analyzing, mining, visualizing –Independent of data format and physical location Dynamically adaptive workflows and steering of sensors

8. The LEAD Vision Data access via querying, and browsing Analysis and forecast tools that can be composed into workflows Workflows and sensors that respond to the weather Support users ranging from grade 6 to experienced researchers

9. LEAD Objectives Lower the barrier for entry and increase the sophistication of problems that can be addressed by complex end-to-end weather analysis and forecasting/simulation tools Improve our understanding of and ability to detect, analyze and predict mesoscale atmospheric phenomena by interacting with weather in a dynamically adaptive manner Result: Paradigm change in how experiments are conceived and performed

10. LEAD Challenges ChallengeRequirements Disparate, high volume data setsEfficient transmission, remote subsetting and aggregration, reliable, robust storage, format independence Huge computational demands, e.g. ensemble forecasting Distributed, load balanced computations Use of existing complex numerical models and data assimilation systems Make existing tools work in web service environment Lack of controlled vocabularyOntology, dictionary Support for 6 – 12, college, graduate, and advanced research Robust security, user aids, education modules, meaningful responses

11. Multidisciplinary Effort Meteorology Computer Science and Information Technology Education and Outreach

12. LEAD Institutions > 100 scientists, students, technical staff

13. LEAD Thrust Groups Data* Orchestration Portal Meteorology Grid and Web Services Test Bed* Education and Outreach Test Bed *Major Unidata areas

14. LEAD Data Subsystem Query Service Dictionary Ontology Service Resource Catalog myLEAD Catalog LEAD Data Repository (LDR) Public Data (e.g. IDD data) LEAD Portal

15. Unidata Technology Used in LEAD LDM/IDD Data Delivery: near real time data delivery THREDDS: catalogs of data and their associated metadata Common Data Model (CDM): single interface to multiple data formats THREDDS Data Server (TDS): integrated OPeNDAP and http data access Integrated Data Viewer (IDV): visualization THREDDS Data Repository (TDR): data storage framework Decoders

16. Unidata and LEAD Unidata also brings: –Experience with atmospheric data –Community of users –Robust, fielded software

17. Recent LEAD-Related Efforts 2. Application of current technology on our LEAD testbed: Tom 3. Structure of the LEAD testbed: Brian 4. THREDDS Data Repository: Doug Goal: Support both LEAD and our community

18. Application of Current Technologies on the LEAD Testbed Systems Tom Baltzer

19. Acronyms for LEAD Tools ADAS - ARPS Data Assimilation System (Center for Advanced Prediction of Storms at OU) ADaM - Algorithm Development and Mining (University of Alabama at Huntsville) IDV – Integrated Data Viewer (Unidata) LDM/IDD – Local Data Manager/Internet Data Distribution (Unidata) OPeNDAP – Open-source Project for a Network Data Access Protocol (OPeNDAP.org) THREDDS – Thematic Real-time Environmental Distributed Data Services TDS - THREDDS Data Server TDR – THREDDS Data Repository (Unidata) WRF – The Weather and Research Forecasting Model (ARW Core - NCAR) Also: WS-Eta – Workstation Eta Model

20. LEAD Testbed Systems Testbed systems at several LEAD locations to provide: – Data Near Real-Time data ingest, storage and access LEAD Data Product storage and access – Data Processing High Performance Computing Grid and Web Services Allow each institution to develop methods by which their capabilities fit into LEAD effort Single Web Portal system at Indiana Univ. to bring it all together and provide User Interface

21. Core Academic Partner + Grid Test Bed Core Academic Partner + Education Test Bed Core Academic Partner + Grid Test Bed + Education Test Bed Core Academic Partner CSU Unidata OU UI IU UAH UNC MU HU LEAD Grid

22. Data Aspects of LEAD Testbeds

23. LEAD Testbed Systems UPC Technologies being leveraged to facilitate LEAD needs – LDM/IDD – THREDDS – IDV – NetCDF Decoders – OPeNDAP (Unidata supported)

24. IDD Testbed System Forecast Model Output Weather station observations Aircraft data Radar data Typical LEAD Testbed (Current Source Data Configuration) Decoders THREDDS Catalog GridFTP OPeNDAP LEAD Grid System

25. IDD Testbed System Forecast Model Output Weather station observations Typical LEAD “Data” Testbed (Future Source Data Configuration) Decoders THREDDS Catalog GridFTP LEAD Grid System TDS & TDR Radar data Aircraft data Note: UPC plans ~ 6 month store OPeNDAP

26. LEAD Processing on the Unidata Testbed System

27. UPC Processing Testbed (Current Configuration) NCEP NAM (Eta) Forecast Precipitation Locator Center Lat/Lon OPeNDAP Access THREDDS Catalog Unidata LEAD Test Bed Regional Forecasts WS-Eta WRF Initial and Boundary Conditions - WRF being Steered by Chiz’s GEMPAK precipitation locator

28. Next Steps NCEP NAM (Eta) Forecast Precipitation Locator Center Lat/Lon OPeNDAP Access THREDDS Catalog Unidata LEAD Test Bed Regional Forecasts WS-Eta WRF Boundary Conditions CAPS ADAS Assimilation Initial Conditions Millersville ADaM Precip Locator

29. Longer Term NCEP NAM (Eta) Forecast Precipitation Locator Center Lat/Lon OPeNDAP Access THREDDS Catalog Unidata LEAD Test Bed Regional Forecasts WS-Eta WRF Boundary Conditions ADAS IDD Datasets Radar Surface & Upper air Satellite NCEP NAM ADaM

30. Ultimately NCEP NAM (Eta) Forecast Precipitation Locator Center Lat/Lon OPeNDAP Access THREDDS Catalog Unidata LEAD Test Bed Regional Forecasts WS-Eta Web Service WRF Boundary Conditions Web Service ADAS IDD Datasets Radar Surface & Upper air Satellite NCEP NAM Web Service ADaM LEAD Grid System

31. Objectives for UPC Testbed Testing ground for integration new UPC and LEAD technologies Determining ways to bring LEAD Technologies to the Unidata Community “Operational” environment for LEAD Processing cluster Data Storage –~6 months of IDD data –LEAD product data

32. The LEAD Hardware at Unidata Brian Kelly

33. Existing LEAD Infrastructure Lead1 GRID Server Development Tools NFS Server Cluster Node Lead3 HTTP Server THREDDS Server OpenDAP Server LDM Node NFS Server Cluster Node Lead2 GRID Server NFS Server Cluster Node Cluster Monitoring Lead4 TDS LDM Node NFS Server Cluster Node LeadStor 8 TB of Disk NFS Server

34. 40 TB Storage Cluster ~30 GFLOP Processing Cluster Portal Servers for Web, TDS, Grid and LDM Services UCAR/Unidata LEAD Infrastructure

35. LEAD Portal Systems Processing Cluster Head Node HTTP, TDS and Grid Server LDM Server Test Server Gigabit Network for NFS Storage Access Storage Cluster Gateway

36. Beowulf Cluster Connected by a Gigabit Fibre Network LEAD Processing Cluster Each Node contains Two Athlon 2400+ CPUs Cluster Uses OSCAR with the MPICH MPD Eight Nodes is ~30 GFLOPs

37. LEAD Storage Cluster LEAD Storage Gigabit Network LEAD Storage Nodes LEAD Storage Head Node

38. One (1) Guanghsing GHI-583 5U Case 24 hot swapable SATA trays 1000W 2+2 power supply ● One (1) Tyan Thunder K8SD Pro Motherboard Dual Opteron CPUs Four 64-bit 133/100 Mhz PCI-X Slots Two Gigabit Ethernet ports ● One (1) AMD Opteron 242 Processor 1.6 Ghz CPU ● Three (3) Broadcom RAIDCore BC4853 Eight SATA ports Controller spanning Advanced raid ● Twenty-Four (24) Seagate Barracuda ST3400832AS 7200 RPM 400GB SATA Drives LEAD Storage Node

39. Twenty-Four (24) 400 GB Drives Divided into Two (2) Eleven Column RAID 5 Arrays and Two Hot Spares Form Two (2) 4 TB LUNs Using bcraid Each Node Publishes the Two LUNS over iSCSI LEAD Storage Node

40. ● Mounts Each Node's Two (2) 4 TB LUNs Published via iSCSI ● Builds Two (2) 20 TB 6 column RAID 5 Meta-devices using mdadm ● Divides Each Meta-device into Volume using LVM ● Each Volume is Formatted with an XFS Filesystem ● Each Filesystem is Published with NFS LEAD Storage Gateway Result: 40 TB of mid-performance double-redundant storage

41. THREDDS Data Repository (TDR) Doug Lindholm

42. LEAD Architecture Data Storage Perspective LEAD Data Grid Unidata NCSA IU OU UAH

43. LEAD Architecture Data Storage Perspective LEAD Data Grid Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities

44. LEAD Architecture Data Storage Perspective LEAD Data Grid Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) Data Assimilation (ADAS) Forecast Model (WRF)

45. LEAD Architecture Data Storage Perspective LEAD Data Grid Portal Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) User Data Assimilation (ADAS) Forecast Model (WRF)

46. LEAD Architecture Data Storage Perspective LEAD Data Grid Portal Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) User Data Assimilation (ADAS) Forecast Model (WRF)

47. LEAD Architecture Data Storage Perspective LEAD Data Grid Portal Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) User Data Assimilation (ADAS) Forecast Model (WRF)

48. LEAD Architecture Data Storage Perspective LEAD Data Grid Portal Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) User Data Assimilation (ADAS) Forecast Model (WRF)

49. LEAD Architecture Data Storage Perspective LEAD Data Grid Portal Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) User Data Assimilation (ADAS) Forecast Model (WRF) Data Repository THREDDS Data Repository

50. Storage Locator locate- Storage() Data Mover move- Data() Unique ID Generator generate- UniqueID() Name Resolver mapID- ToURL() Metadata Generator generate- Metadata() Metadata Crosswalk translate- Metadata() Cataloger catalog- Metadata() THREDDS Data Repository Component Architecture putData()getData()discoverData() Data Storage THREDDS Data Repository

51. Storage Locator locate- Storage() Data Mover move- Data() Unique ID Generator generate- UniqueID() Name Resolver mapID- ToURL() Metadata Generator generate- Metadata() Metadata Crosswalk translate- Metadata() Cataloger catalog- Metadata() THREDDS Data Repository Component Architecture THREDDS Data Repository putData()getData()discoverData() Data Storage

52. Resource Broker locate- Storage() trebuchet move- Data() Unique ID Generator generate- UniqueID() RLS mapID- ToURL() THREDDS Metadata Generator generate- Metadata() THREDDS to LEAD Crosswalk translate- Metadata() myLEAD catalog- Metadata() THREDDS Data Repository Component Architecture THREDDS Data Repository putData()getData()discoverData() Data Storage LEAD Configuration

53. Storage Locator locate- Storage() Data Mover move- Data() generate- UniqueID() mapID- ToURL() generate- Metadata() translate- Metadata() THREDDS Catalog catalog- Metadata() THREDDS Data Repository Component Architecture THREDDS Data Repository putData()getData()discoverData() Data Storage Alternate Configuration THREDDS Metadata Generator

54. Unidata Architecture Internet Data Distribution (IDD) Data Storage Local Data Manager (LDM)

55. Unidata Architecture Internet Data Distribution (IDD) Data Storage Local Data Manager (LDM) access

56. THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog Data Storage Local Data Manager (LDM) discover access

57. THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog Data Storage Local Data Manager (LDM) Common Data Model (CDM) discover access

58. THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog THREDDS Data Server (TDS) Data Storage Local Data Manager (LDM) Common Data Model (CDM) discover access

59. THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog THREDDS Data Server (TDS) THREDDS Data Repository (TDR) Data Storage Local Data Manager (LDM) Common Data Model (CDM) discover access store

60. THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog THREDDS Data Server (TDS) THREDDS Data Repository (TDR) Data Storage Locally Generated Data Locally Generated Data Local Data Manager (LDM) Common Data Model (CDM) discover access store

61. THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog THREDDS Data Server (TDS) THREDDS Data Repository (TDR) E-mail Application (e.g. IDV) Service Data Storage Locally Generated Data Locally Generated Data Local Data Manager (LDM) Common Data Model (CDM) discover access store notify

62. Questions?