High Performance Storage System Harry Hulen

HSM: Hierarchical storage management Purposes of HSM: –Extend disk space –Back up disk files to tape –Managed permanent archive User sees virtually unlimited file system –Data migrates “down” the hierarchy –Migrated files may be asychronously purged from higher level (e.g. disk) to free up space Multiple classes of service in a single name space –Disk to tape –Tape only (SLAC approach) –Complex, e.g. Striped disk to mirrored tape File System disk robotic tape shelf tape data

Big storage, like big computing, is fundamentally an aggregation problem SAN or LAN A typical commercial SAN allocates a few high-function disk arrays among many non-shared file systems and data bases on many computers Our large shared-data SANs must aggregate many disk arrays among a few very large file systems and data bases shared by many computers C Reserve B A ABC SAN Administrator manages spare capacity SAN File System manages spare capacity

HPSS architecture Shared, secure global file system Aggregate disks, tapes, and bandwidth SAN and/or LAN connected Metadata-mediated via database based on IBM DB2 Highly distributed with multiple data movers and subsystems for scalability API for maximum control and performance (e.g. “hints”) Parallel FTP (PFTP) Multi-petabyte capability in a single name space (e.g. SLAC, LLNL, BNL, ECMWF, DOD) Robotic Tape Libraries Client Computers LAN SAN Disk Arrays Backup Core Server Metadata Disks Core Server Based on HPSS 6 Tape-Disk Movers

The HPSS Collaboration U.S Department of Energy Laboratories are Co-Developers –Lawrence Livermore National Lab. - Sandia National Laboratories –Los Alamos National Laboratory - Oak Ridge National Laboratory –Lawrence Berkeley National Lab. IBM Global Services in Houston, Texas –Access to IBM technology (DB2, for example) –Project management –Quality assurance and testing (SEI CMM Level 3) –Outreach: commercial sales and service Advantages of Collaborative Development –Developers are users: focus on what is needed and what works –Keeps focus on the high end: the largest data stores –A limited “open source” model for collaboration members and users “Since 1993”

HPSS performance trivia Capacity –Largest HPSS installation (BNL) has 2 petabytes in a single address space with no indications of an upper bound –Calculations show ability to handle 100s of millions of files in a name space File Access Rate (recent data with DB2, not tuned) –50 create-writes per second with 6-processor Power4 and AIX (ECMWF) –20 create-writes per sec with 4-processor XEON with Linux (Test lab) –Hope to achieve 100 c-w/sec with optimization and newer hw Data Bandwidth –Data rate benchmark 1 gigabyte per sec to 16 movers with 16 disks each (4 year old data) –2-way and 4-way striping of disk arrays and tapes for higher single- file transfers –Concurrent transfers among many clients, disk arrays, and tape libraries for very high aggregate transfer capability

Disaster Recovery: Difficulty grows with size For most cluster file systems, loss of disk corrupts entire file system –Entire file system must be rebuilt or restored from backup –Disk array availability about HPSS keeps metadata separate from data –Metadata kept in a DB2 database –HPSS disk files and tape files use the same metadata –Loss of an entire disk array causes only loss of data not migrated to tape (or to another disk), HPSS continues to run –Restoration of system = reloading metadata Recovery Performance –Capable of recovery in minutes from loss of any or all disk data (hours to days in other large systems) –Capable of recovery in hours from loss of all metadata (hours to days in other large systems)

HPSS Plans 2004 –New HPSS infrastructure based on DB2 and eliminating DCE (transparent to users) –HPSS for Linux and “HPSS Light” –LAN-less data transfers (SAN capability) –Include support for HTAR and HSI utility packages –Stand-alone PFTP offering and push protocol 2005 –ASCI Parallel Local File Movers for Lustre archive –Globus Grid Gridftp capability –True VFS interface (initially Linux) –Additional small file performance improvements –Exploit multilevel hierarchy (e.g. MAID) –Better integration with application agents (e.g. Objectivity) 2006 –Object-based disk technology –Exploit DB2 metadata engine for content management

HPSS for Linux will make HPSS more widely available HPSS serves 8 of the top 20 HPC sites HPSS for Linux will enable HPSS to extend down from XXL and XL to L later this year HPSS for Linux will be offered in lower- cost pre-configured packages 1000s 100s 10s HPSS Other HSMs HPSS for Linux D2D and D2D2T Backup ~1.5 PB ~.5 PB

ASCI Purple Parallel Local File Movers HPSS Lustre Disk Client Archive Agent Application Capability Or Capacity Platform  Simplicity (configuration, equipment expenditures, networking)  Performance potential  Minimize disk cache Lustre is a shared global file system in development by DOE, HP, and others. Site-provided agent controls migration based on file content and not on empirical data HPSS Parallel Local File Movers open, read, and write Lustre files using Unix semantics

Local Disks Global Disks 10 Records, 10 Files 10 Metadata Entries 30 Records, 3 Files 3 Metadata Entries Data is mirrored, format is not A file with multiple records is a container HTAR: Use Of Containers saves metadata overhead

A multi-level hierarchy Robotic Tape Libraries Client Computers LAN SAN Backup Core Server Metadata Disks Core Server Based on HPSS 6 Tape-Disk Movers Example 3 level –Disk Arrays –Massive Arrays of Idle Disks –Tape Libraries MAID will fill the “big middle” between disk and tape HPSS supports multilevel hierarchies today Disk Arrays Massive Array of Idle Disks (MAID)

HPSS Grid support Short-term plans include GSI FTP –LBL/NERSC has prototyped a GSI-enabled HPSS PFTP client and daemon –Tested by KEK lab (Japan) and U of Tokyo Long-term plans include HPSS-compatible GridFTP –Argonne National Lab is designing and implementing –Fully Globus compatible –Target is later this year (2004)

How to build a really large system to ingest and process data Institutional metadata Database engine Process Primary (e.g. GPFS) Process Primary (e.g. GPFS) Process Primary (e.g. SGFS) Writing concurrently does not interfere with processing access to primary disk Ingest Secondary (e.g. HPSS) Tertiary (e.g. HPSS) Batch into containers Institutional metadata can direct Process to Secondary disk in case of loss of Primary Single hierarchical archive file system Multiple primary file systems