Virtualization in the NCAR Mass Storage System Gene Harano National Center for Atmospheric Research Scientific Computing Division High Performance Systems Section

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 1 Outline Overview NCAR Mass Storage System (MSS) Architecture Device Virtualization in the NCAR MSS Benefits of Virtualization

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 2 National Center for Atmospheric Research

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 3

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 4 Archive Statistics MSS Statistics – End of Aug 2003 –1.457 PBs total stored in 19.9 M files –45 TB/month total net growth rate –1.2 M user MSS host reads and writes/month –80 TBs moved/month on MSS hosts –70 TBs moved/month internal migration, 2nd copy, data ooze - (total varies widely) –150 TBs moved/month

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 5

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 6 Computing Roadmap

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 7 Concept Unlimited number of files Unlimited file size Unlimited file retention Near zero access latency Hardware cost reduced 90% or more Personnel cost increased 90% or more Charge-back eliminated

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 8 Concept Data Virtualization Can only store “0” value bits!! 1

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 9 Architecture

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 10 The NCAR MSS Archive, not a file server Custom system operational since 1986 Based on the IEEE Mass Storage Model Tape based archive with a small amount of disk cache for small files. Major area of focus – multi- TeraByte internal disk cache Separation of control and data paths Implemented 3rd party transfers to optimize data movement and increase scalability

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 11 MSCP Traditional Server Control/Metadata/Data Control/ Data Host Server

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 12 MSCP 3 rd Party Transfer Control/Data Separation MFD HPDF Control/Metadata Data control Host MSCP

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 13 High Performance Data Fabric (SAN) NCAR MSS has used a SAN since Direct tape and disk access via the SAN 30+ heterogeneous hosts (including 2 nd campus 16km away) Separation of control and data paths

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 14 Transparency The Archive stores bitfiles String of bits without regard to content No need to maintain/understand record boundaries No need to maintain/understand record structure

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 15 Device Virtualization The NCAR MSS utilizes 2 types of device virtualization –Vendor supplied (in hardware) –Custom built (in software – application level)

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 16 Vendor Supplied Device emulation –RAID devices emulate SCSI disks –Tape devices emulate IBM 3490/3590 and SCSI Virtual tape –Available from vendors but not used by the NCAR MSS –How would you utilize a 1 TB capacity tape medium to it’s maximum capacity?

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 17 Custom Built Application level software – no OS mods String of bits stored without regard to device type Storage Manager component presents a socket interface to clients. In-band between client and storage device. Underlying storage device (disk, tape, ???) is not exposed to clients FC RAID and FC tape

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 18 Benefits

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 19 Benefits Quickly introduce new storage technologies –Coding modifications are localized –Vendor supplied virtualization may not require coding modifications (eg. Virtual tape, RAIT) Data can move easily move between storage media. In concert with bitfile transparency. Reduces system complexity

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 20 Benefits Enables new functionality –Split data streams Multiple simultaneous copies – Disaster Recovery, reliability RAIT – performance and reliability Simultaneous disk and tape migration/staging

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 21 Summary Virtualization is good! Vendor supplied virtualization is even better! Virtualization –Reduces complexity –Optimizes resource utilization –Lowers costs

CAS2K3 September 9, 2003 Copyright© 2003 University Corporation for Atmospheric Research 22