The LEAD Effort at Unidata The Unidata Seminar will start at 1:30 PM MST

The LEAD Effort at Unidata Tom Baltzer, Brian Kelly, Doug Lindholm, Anne Wilson December 14, 2005

LEAD is funded by the National Science Foundation under the following Cooperative Agreements: ATM ATM ATM ATM ATM ATM ATM ATM

Outline 1.Setting the Stage: Introduction to LEAD and Unidata’s LEAD Efforts: Anne 2.Application of current technology on the LEAD testbeds: Tom 3.The LEAD Hardware at Unidata: Brian 4.The THREDDS Data Repository: Doug

Setting the Stage: Introduction to LEAD and Unidata’s LEAD Efforts Anne Wilson

Current IT Barriers to Mesoscale Weather Research and Education Data and tools useable mainly by experts Researchers and educators constrained by hardware limitations Rigid, brittle technology can’t accommodate mesoscale weather research requirements: –real time, on demand, dynamic data processing and sensor steering

A Solution: Linked Environments for Atmospheric Discovery (LEAD) Funded by NSF Large Information Technology Research (ITR) award Produce a web service based, scalable framework for handling meteorological data and model output: –Identifying, accessing, preparing, assimilating, predicting, managing, analyzing, mining, visualizing –Independent of data format and physical location Dynamically adaptive workflows and steering of sensors

The LEAD Vision Data access via querying, and browsing Analysis and forecast tools that can be composed into workflows Workflows and sensors that respond to the weather Support users ranging from grade 6 to experienced researchers

LEAD Objectives Lower the barrier for entry and increase the sophistication of problems that can be addressed by complex end-to-end weather analysis and forecasting/simulation tools Improve our understanding of and ability to detect, analyze and predict mesoscale atmospheric phenomena by interacting with weather in a dynamically adaptive manner Result: Paradigm change in how experiments are conceived and performed

LEAD Challenges ChallengeRequirements Disparate, high volume data setsEfficient transmission, remote subsetting and aggregration, reliable, robust storage, format independence Huge computational demands, e.g. ensemble forecasting Distributed, load balanced computations Use of existing complex numerical models and data assimilation systems Make existing tools work in web service environment Lack of controlled vocabularyOntology, dictionary Support for 6 – 12, college, graduate, and advanced research Robust security, user aids, education modules, meaningful responses

Multidisciplinary Effort Meteorology Computer Science and Information Technology Education and Outreach

LEAD Institutions > 100 scientists, students, technical staff

LEAD Thrust Groups Data* Orchestration Portal Meteorology Grid and Web Services Test Bed* Education and Outreach Test Bed *Major Unidata areas

LEAD Data Subsystem Query Service Dictionary Ontology Service Resource Catalog myLEAD Catalog LEAD Data Repository (LDR) Public Data (e.g. IDD data) LEAD Portal

Unidata Technology Used in LEAD LDM/IDD Data Delivery: near real time data delivery THREDDS: catalogs of data and their associated metadata Common Data Model (CDM): single interface to multiple data formats THREDDS Data Server (TDS): integrated OPeNDAP and http data access Integrated Data Viewer (IDV): visualization THREDDS Data Repository (TDR): data storage framework Decoders

Unidata and LEAD Unidata also brings: –Experience with atmospheric data –Community of users –Robust, fielded software

Recent LEAD-Related Efforts 2. Application of current technology on our LEAD testbed: Tom 3. Structure of the LEAD testbed: Brian 4. THREDDS Data Repository: Doug Goal: Support both LEAD and our community

Application of Current Technologies on the LEAD Testbed Systems Tom Baltzer

Acronyms for LEAD Tools ADAS - ARPS Data Assimilation System (Center for Advanced Prediction of Storms at OU) ADaM - Algorithm Development and Mining (University of Alabama at Huntsville) IDV – Integrated Data Viewer (Unidata) LDM/IDD – Local Data Manager/Internet Data Distribution (Unidata) OPeNDAP – Open-source Project for a Network Data Access Protocol (OPeNDAP.org) THREDDS – Thematic Real-time Environmental Distributed Data Services TDS - THREDDS Data Server TDR – THREDDS Data Repository (Unidata) WRF – The Weather and Research Forecasting Model (ARW Core - NCAR) Also: WS-Eta – Workstation Eta Model

LEAD Testbed Systems Testbed systems at several LEAD locations to provide: – Data Near Real-Time data ingest, storage and access LEAD Data Product storage and access – Data Processing High Performance Computing Grid and Web Services Allow each institution to develop methods by which their capabilities fit into LEAD effort Single Web Portal system at Indiana Univ. to bring it all together and provide User Interface

Core Academic Partner + Grid Test Bed Core Academic Partner + Education Test Bed Core Academic Partner + Grid Test Bed + Education Test Bed Core Academic Partner CSU Unidata OU UI IU UAH UNC MU HU LEAD Grid

Data Aspects of LEAD Testbeds

LEAD Testbed Systems UPC Technologies being leveraged to facilitate LEAD needs – LDM/IDD – THREDDS – IDV – NetCDF Decoders – OPeNDAP (Unidata supported)

IDD Testbed System Forecast Model Output Weather station observations Aircraft data Radar data Typical LEAD Testbed (Current Source Data Configuration) Decoders THREDDS Catalog GridFTP OPeNDAP LEAD Grid System

IDD Testbed System Forecast Model Output Weather station observations Typical LEAD “Data” Testbed (Future Source Data Configuration) Decoders THREDDS Catalog GridFTP LEAD Grid System TDS & TDR Radar data Aircraft data Note: UPC plans ~ 6 month store OPeNDAP

LEAD Processing on the Unidata Testbed System

UPC Processing Testbed (Current Configuration) NCEP NAM (Eta) Forecast Precipitation Locator Center Lat/Lon OPeNDAP Access THREDDS Catalog Unidata LEAD Test Bed Regional Forecasts WS-Eta WRF Initial and Boundary Conditions - WRF being Steered by Chiz’s GEMPAK precipitation locator

Next Steps NCEP NAM (Eta) Forecast Precipitation Locator Center Lat/Lon OPeNDAP Access THREDDS Catalog Unidata LEAD Test Bed Regional Forecasts WS-Eta WRF Boundary Conditions CAPS ADAS Assimilation Initial Conditions Millersville ADaM Precip Locator

Longer Term NCEP NAM (Eta) Forecast Precipitation Locator Center Lat/Lon OPeNDAP Access THREDDS Catalog Unidata LEAD Test Bed Regional Forecasts WS-Eta WRF Boundary Conditions ADAS IDD Datasets Radar Surface & Upper air Satellite NCEP NAM ADaM

Ultimately NCEP NAM (Eta) Forecast Precipitation Locator Center Lat/Lon OPeNDAP Access THREDDS Catalog Unidata LEAD Test Bed Regional Forecasts WS-Eta Web Service WRF Boundary Conditions Web Service ADAS IDD Datasets Radar Surface & Upper air Satellite NCEP NAM Web Service ADaM LEAD Grid System

Objectives for UPC Testbed Testing ground for integration new UPC and LEAD technologies Determining ways to bring LEAD Technologies to the Unidata Community “Operational” environment for LEAD Processing cluster Data Storage –~6 months of IDD data –LEAD product data

The LEAD Hardware at Unidata Brian Kelly

Existing LEAD Infrastructure Lead1 GRID Server Development Tools NFS Server Cluster Node Lead3 HTTP Server THREDDS Server OpenDAP Server LDM Node NFS Server Cluster Node Lead2 GRID Server NFS Server Cluster Node Cluster Monitoring Lead4 TDS LDM Node NFS Server Cluster Node LeadStor 8 TB of Disk NFS Server

40 TB Storage Cluster ~30 GFLOP Processing Cluster Portal Servers for Web, TDS, Grid and LDM Services UCAR/Unidata LEAD Infrastructure

LEAD Portal Systems Processing Cluster Head Node HTTP, TDS and Grid Server LDM Server Test Server Gigabit Network for NFS Storage Access Storage Cluster Gateway

Beowulf Cluster Connected by a Gigabit Fibre Network LEAD Processing Cluster Each Node contains Two Athlon CPUs Cluster Uses OSCAR with the MPICH MPD Eight Nodes is ~30 GFLOPs

LEAD Storage Cluster LEAD Storage Gigabit Network LEAD Storage Nodes LEAD Storage Head Node

One (1) Guanghsing GHI-583 5U Case 24 hot swapable SATA trays 1000W 2+2 power supply ● One (1) Tyan Thunder K8SD Pro Motherboard Dual Opteron CPUs Four 64-bit 133/100 Mhz PCI-X Slots Two Gigabit Ethernet ports ● One (1) AMD Opteron 242 Processor 1.6 Ghz CPU ● Three (3) Broadcom RAIDCore BC4853 Eight SATA ports Controller spanning Advanced raid ● Twenty-Four (24) Seagate Barracuda ST AS 7200 RPM 400GB SATA Drives LEAD Storage Node

Twenty-Four (24) 400 GB Drives Divided into Two (2) Eleven Column RAID 5 Arrays and Two Hot Spares Form Two (2) 4 TB LUNs Using bcraid Each Node Publishes the Two LUNS over iSCSI LEAD Storage Node

● Mounts Each Node's Two (2) 4 TB LUNs Published via iSCSI ● Builds Two (2) 20 TB 6 column RAID 5 Meta-devices using mdadm ● Divides Each Meta-device into Volume using LVM ● Each Volume is Formatted with an XFS Filesystem ● Each Filesystem is Published with NFS LEAD Storage Gateway Result: 40 TB of mid-performance double-redundant storage

THREDDS Data Repository (TDR) Doug Lindholm

LEAD Architecture Data Storage Perspective LEAD Data Grid Unidata NCSA IU OU UAH

LEAD Architecture Data Storage Perspective LEAD Data Grid Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities

LEAD Architecture Data Storage Perspective LEAD Data Grid Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) Data Assimilation (ADAS) Forecast Model (WRF)

LEAD Architecture Data Storage Perspective LEAD Data Grid Portal Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) User Data Assimilation (ADAS) Forecast Model (WRF)

LEAD Architecture Data Storage Perspective LEAD Data Grid Portal Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) User Data Assimilation (ADAS) Forecast Model (WRF)

LEAD Architecture Data Storage Perspective LEAD Data Grid Portal Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) User Data Assimilation (ADAS) Forecast Model (WRF)

LEAD Architecture Data Storage Perspective LEAD Data Grid Portal Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) User Data Assimilation (ADAS) Forecast Model (WRF)

LEAD Architecture Data Storage Perspective LEAD Data Grid Portal Cataloger (myLEAD) Storage Locator Data Mover ID Generator Name Resolver Metadata Generator Metadata Crosswalk Unidata NCSA IU OU UAH “Atomic” Capabilities Application Services Data Mining (ADAM) Visualization (IDV) User Data Assimilation (ADAS) Forecast Model (WRF) Data Repository THREDDS Data Repository

Storage Locator locate- Storage() Data Mover move- Data() Unique ID Generator generate- UniqueID() Name Resolver mapID- ToURL() Metadata Generator generate- Metadata() Metadata Crosswalk translate- Metadata() Cataloger catalog- Metadata() THREDDS Data Repository Component Architecture putData()getData()discoverData() Data Storage THREDDS Data Repository

Storage Locator locate- Storage() Data Mover move- Data() Unique ID Generator generate- UniqueID() Name Resolver mapID- ToURL() Metadata Generator generate- Metadata() Metadata Crosswalk translate- Metadata() Cataloger catalog- Metadata() THREDDS Data Repository Component Architecture THREDDS Data Repository putData()getData()discoverData() Data Storage

Resource Broker locate- Storage() trebuchet move- Data() Unique ID Generator generate- UniqueID() RLS mapID- ToURL() THREDDS Metadata Generator generate- Metadata() THREDDS to LEAD Crosswalk translate- Metadata() myLEAD catalog- Metadata() THREDDS Data Repository Component Architecture THREDDS Data Repository putData()getData()discoverData() Data Storage LEAD Configuration

Storage Locator locate- Storage() Data Mover move- Data() generate- UniqueID() mapID- ToURL() generate- Metadata() translate- Metadata() THREDDS Catalog catalog- Metadata() THREDDS Data Repository Component Architecture THREDDS Data Repository putData()getData()discoverData() Data Storage Alternate Configuration THREDDS Metadata Generator

Unidata Architecture Internet Data Distribution (IDD) Data Storage Local Data Manager (LDM)

Unidata Architecture Internet Data Distribution (IDD) Data Storage Local Data Manager (LDM) access

THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog Data Storage Local Data Manager (LDM) discover access

THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog Data Storage Local Data Manager (LDM) Common Data Model (CDM) discover access

THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog THREDDS Data Server (TDS) Data Storage Local Data Manager (LDM) Common Data Model (CDM) discover access

THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog THREDDS Data Server (TDS) THREDDS Data Repository (TDR) Data Storage Local Data Manager (LDM) Common Data Model (CDM) discover access store

THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog THREDDS Data Server (TDS) THREDDS Data Repository (TDR) Data Storage Locally Generated Data Locally Generated Data Local Data Manager (LDM) Common Data Model (CDM) discover access store

THREDDS Client API Unidata Architecture Internet Data Distribution (IDD) THREDDS Catalog THREDDS Data Server (TDS) THREDDS Data Repository (TDR) Application (e.g. IDV) Service Data Storage Locally Generated Data Locally Generated Data Local Data Manager (LDM) Common Data Model (CDM) discover access store notify

Questions?