1. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Project 3.3 Optimizing Shared Access to Tertiary Storage March, 2002 Presenter - Randy Burris Other Participants Associated with Project 1, task 1.3 - Agents Tom Potok, Joel Reed, Mark Elmore Associated with Project 2, task 2.1 - Cluster analysis Nagiza Samatova, George Ostrouchov Associated with Project 3, task 3.3 - HRM Arie Shoshani, Alex Sim Probe - Dan Million, Mike Gleicher Grid - Kasidit Chanchio

2. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Overview of this presentation  Actually going to cover two topics  Discuss Task 3.3, Optimizing shared access to tertiary storage  In the context of environments available or in use in Probe

3. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Stingray RS/6000 S80 Marlin RS/6000 H70 RS/6000 B80 External Esnet Router To NERSC Probe IBM and Compaq Supercomputers RS/6000 44P-170 Probe Production Sun E450 Intel Dual P-III Linux Agent Research Environment (Associated with Project 1 task 3) RS/6000 F50 SGI Origin 200 Sun E250 Compaq DS20 Available (Grid nodes)

4. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Cluster Analysis Research Environment (Associated with Project 2 task 1) R data analysis GGobi data visualization Java gnu Fortran, CAIX Fortran, Cgnu Fortran, C Stingray RS6000 S80, 6 processors, 2 GB memory 1 TB IDE FibreChannel RAID, 360 GB Sun RAID (A fully-configured S80 was world’s fastest commercial server in 2001)

5. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Task 3 of Project 3; specifically, HRM  HRM at ORNL and NERSC will provide long-awaited capability for Earth System Grid.  Alex Sim now has accounts for ORNL/Probe and for ORNL’s production HPSS  He’s installed HRM on one of Probe’s Sun machines.  He has extracted files from production HPSS via HRM, which:  Uses hsi to find files in HPSS  Extracts them from HPSS via parallel ftp  Sends them to NERSC via GridFTP  Alex’s testing uncovered a bug in GridFTP.  With his help, fix discovered  Due for release in next GT2 beta 2.0 version.

6. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Testing so far:  Used the new HRM command-line program  One set of tests: of 9 runs, 1 succeeded  Bug in GridFTP globus-gass-copy library killed the rest  In another test, successfully replicated a set of 20 files.  350-390 megabyte files  5 concurrent pftps for HPSS operations  3 concurrent GridFTPs (5 concurrent hit the bug)  2 parallel streams per GridFTP  Rates up to 11 megabytes/sec for single file; average for test was 3.88 megabytes/sec  Next step, try two-stripe files and partial file transfers.

7. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY HRM Research Environment (Associated with Project 3) Probe HPSS 180 GB Sun FibreChannel RAID STK Library ORNL Production HPSS Disk Cache Probe Production STK Tape Drive 180 GB Sun FibreChannel RAID STK Tape Drive Sleepy Sun E250 pftp, hsi, GridFTP Globus 2.0 beta Verified Two-stripe NERSC Production HPSS

8. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY HRM and HSI plans  HRM  Verify correct operation in retrieving a file from two-stripe HPSS class-of-service  Add Grid authentication service  Use hsi instead of parallel ftp when retrieving file from HPSS  Pass individual user proxy to HSI when getting files from HPSS  HSI  Add Grid authentication service (already has Kerberos and DCE)  Implement partial file transfers

9. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Other Environments  Some other environments lead to top-to-bottom integration  I believe these allow scientists access to raw data  NetCDF (and soon HDF5) directly using HPSS  HDF5 via MPI-IO over HPSS or ROMIO  I’m showing them to:  Indicate what is available now  Identify some of the work to be done  And raise some questions

10. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Target Software Environments NetCDF HPSS Client API Local HPSSRemote HPSS HDF5(active) Viz (ncview, GrADS) Added “-hpss” option to ncview and an option to netcdf “open” and “create” calls to specify HPSS files. Demonstrated its use with HPSS-resident Climate datasets in ORNL’s and NERSC’s HPSS. Similar plan for HDF5 but it’ll take a little longer – its library is more complex.

11. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Target Software Environments HRM GridFTP WAN transfers Local HPSSRemote HPSS Bitmap index Earth System Grid source filtering HDF5 or NetCDF Partial-file reads Striped files We have thisThink about integrating with: pftp HPSS extraction hsi file location

12. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Target Software Environments MPICH, IBM MPI, Sun HPC MPI HPSS HPSS Client API MPI-IO ADIO HDF5 MPICH, SGI MPI, NEC MPI, … HPSS MPI-IO Unix, PFS, PIOFS, XFS, NFS … HPSS (ORNL?)ROMIO (Argonne?) Disk/Tape HierarchyDisk “real” NetCDF HDF5 NetCDF HDF5 Questions: Are both really available to SDM now? Should both be available widely? How can we make things easier? “real” NetCDF HDF5 NetCDF

13. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY DISCUSSION?

14. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Backup Slide ORNL/Probe Technology on hand  Software  HPSS (unlimited instantiations) and HPSS development license  Inter-HPSS hsi application  HDF5, NetCDF  R, GGobi, Rjava  gcc suite  MPI/IO testbed  C on Solaris, AIX, IRIX and Tru64  Fortran on AIX  Oracle 8i and DB2 7.2 on AIX  Globus 2.0/AIX and Solaris  HRM  OPNET modeling product  18 nodes – IBM/AIX, Sun/Solaris, SGI/IRIX, Compaq/Tru64, Intel/Linux  GRID nodes – Sun/Solaris, IBM/AIX (and Linux if someone’s interested)  ESnet III OC12 externally, GigE jumbo and Fast Ethernet internally  Web100 and NET100 participation

15. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Probe – “Place to be” Overview of ORNL Probe Cell, February 2002 Stingray RS/6000 S80 Marlin RS/6000 H70 STK Silo 200 GB SCSI RAID Disks Sun E250 Compaq DS20 360 GB Sun FibreChannel Disks 360 GB STK FibreChannel Disks FibreChannel Switch GSN Switch Origin 2000 Reality Monster RS/6000 B80 External Esnet Router To NERSC Probe Sun Ultra 10 STK Silo IBM and Compaq Supercomputers 3494 Library GSN Bridge RS/6000 44P-170 Probe Production Sun E450 IBM F50 SGI Origin 200 Gigabit Ethernet Intel Dual P-III Linux

16. OAK RIDGE NATIONAL LABORATORY U.S. DEPARTMENT OF ENERGY Backup slide 