O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 1 Enabling Supernova Computations by Integrated Transport and Provisioning Methods Optimized for Dedicated Channels Nagi Rao, Bill Wing, Tony Mezzacappa Oak Ridge National Laboratory Malathi Veeraraghavan University of Virginia DOE MICS PI Meeting: High-Performance Networking Program September 14-16, 2004 Fermi National Laboratory

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Outline  Background  ORNL Tasks  Preliminary Results  UVA Tasks  Preliminary Results

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Terascale Supernova Initiative - TSI  Science Objective: Understand supernova evolutions  DOE SciDAC Project: ORNL and 8 universities  Teams of field experts across the country collaborate on computations  Experts in hydrodynamics, fusion energy, high energy physics  Massive computational code  Terabyte/day generated currently  Archived at nearby HPSS  Visualized locally on clusters – only archival data  Current Networking Challenges  Limited transfer throughput  Hydro code – 8 hours to generate and 14 hours to transfer out  Runaway computations  Find out after the fact that parameters needed adjustment

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Data and File Transfers (terabyte – petabyte)  Move data from computations on supercomputers  Supply data to visualizations on clusters and supercomputers Interactive Computations and Visualization  Monitor, collaborate and steer computations  Collaborative and comparative visualizations Visualization channel Visualization control channel Steering channel TSI Desired Capabilities Computation or visualization

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Background on NSF CHEETAH Project  Circuit-switched High-speed End-to-End Transport arcHitecture (CHEETAH)  Team: UVA, ORNL, NCSU, CUNY  Concept:  Share bandwidth on a dynamic call-by-call basis  End-to-end circuit:  Ethernet - Ethernet over SONET - Ethernet  Network  Second NICs at hosts in a compute cluster/viz cluster  Connected to MSPPs that perform Ethernet-SONET mapping  GMPLS-enabled SONET crossconnects  Transport protocols and middleware  To support file transfers on dedicated circuits  To support remote visualization and computational steering  Applications to support TSI scientists  SFTP  Ensight + new visualization programs

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Current DOE ORNL-UVA Project: Complementary Roles Project Components: Provisioning for UltraScience Net - GMPLS File transfers for dedicated channels Peering – DOE UltraScience Net and NSF CHEETAH Network optimized visualizations for TSI TSI application support over UltraScience Net + CHEETAH Peering ORNL UVA Visualization TSI Application Provisioning File Transfers This project leverages two projects DOE UltraScience Net NSF CHEETAH

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Peered UltraScienceNet-CHEETAH Enables coast-to-coast dedicated channels Phase I: TL1-GMPLS cross conversion Phase II: GMPLS-based

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY ORNL: Year 1 Activities Peering CHEETAH - UltraScienceNet Visualization Decomposable visualization pipeline Analytical formulation First implementation TSI support Monitoring Visualizations

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY ORNL Personnel Conference Papers M. Zhu, Q. Wu, N. S. V. Rao, S. S.Iyengar, “Adaptive Visualization Pipeline Partition and Mapping on Computer Network”, International Conference on Image Processing and Graphics, ICIG2004. On Optimal Mapping of Visualization Pipeline onto Linear Arrangement of Network Nodes”, International Conference on Visualization and Data Analysis, 2005M. Zhu, Q. Wu, N. S. V. Rao, S. S.Iyengar, “On Optimal Mapping of Visualization Pipeline onto Linear Arrangement of Network Nodes”, International Conference on Visualization and Data Analysis, 2005 Publications Nagi Rao, Bill Wing, Tony Mezzacappa (PIs) Qishi Wu (Post-Doctoral Fellow) Mengxia Zhu (Phd Student – Louisiana State Uni.)

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Modules of Visualization Pipeline  Visualization Modules  Pipeline consists of several modules  Some modules are better suited to certain network nodes  Visualization clusters  Computation clusters  Power walls  Data transfers between modules are of varied sizes and rates Note: Commercial tools do not support efficient decomposition

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Grouping Visualization Modules  Grouping  Decompose the pipeline into modules  Combine the modules into groups  Transfers on single node are generally faster  Between node transfers take place over the network  Align bottleneck network links between modules with least data requirements

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Optimal Mapping of Visualization Pipeline: Minimization of Total Delay Dynamic Programming Solution  Combine modules into groups  Align bottleneck network links between modules with least data requirements  Polynomial-time solvable – not NP-complete Note: 1.Commercial tools (Ensight) are not readily amenable to optimal network deployment 2.This method can be implemented into tools that provide appropriate hooks

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Optimal Mapping of Visualization Pipeline: Maximization of Frame Rate Dynamics Programming Solution  Align bottleneck network links between modules with least data requirements  Polynomial-time solvable – not NP-complete

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY First Implementation  Client/Server OpenGL implementation (leveraged from CHEETAH)  Case 1: small cube geometry or frame-buffer  Case 2: small geometry  Case 3: small geometry  CT scan: raw image or frame-buffer DimensionEstimated bandwidth Minimum delay Raw data size/delay Geometry size/delay FB size/delay Cube 1 10x6x80.284Mbps0.032sec8 K / 0.257sec1K / 0.032sec1.8M/50.73sec Cube 2 50x20x Mbps0.034sec610K / 16.3sec16K / 0.46sec1.8M/48.03sec Cube 3 150x210x Mbps0.033sec71.6M / 34.4min2.4M / 69.34sec1.8M/52.01sec Hand256x256x Mbps0.033sec81.9M / 45.69min NA1.8M/60.28sec

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY  Requirements  Light-weight server located at the computation site  Remote client provides constant monitoring of variables  Our first implementation  OpenGL server and client  Client  Geometric operations  Point, iso-surface, vector view  Commercial Visualization tools  Not light weight – server on supercomputers  Expensive – collaborative visualization by team  Not optimized for network deployment Monitoring Visualization

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY ORNL: Year 2 Activities MPLS Peering CHEETAH Visualizations Computational Monitoring Collaborative Visualization TSI support Collaborative Steering Integrated Data Transfers