National Energy Research Scientific Computing Center (NERSC) The GUPFS Project at NERSC GUPFS Team NERSC Center Division, LBNL November 2003.

Slides:



Advertisements
Similar presentations
NAS vs. SAN 10/2010 Palestinian Land Authority IT Department By Nahreen Ameen 1.
Advertisements

CURRENT AND FUTURE HPC SOLUTIONS. T-PLATFORMS  Russia’s leading developer of turn-key solutions for supercomputing  Privately owned  140+ employees.
Performance, Reliability, and Operational Issues for High Performance NAS Matthew O’Keefe, Alvarri CMG Meeting February 2008.
© 2004 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice InfiniBand Storage: Luster™ Technology.
The Efficient Fabric Presenter Name Title. The march of ethernet is inevitable Gb 10Gb 8Gb 4Gb 2Gb 1Gb 100Mb +
Linux Clustering A way to supercomputing. What is Cluster? A group of individual computers bundled together using hardware and software in order to make.
Storage area Network(SANs) Topics of presentation
5/8/2006 Nicole SAN Protocols 1 Storage Networking Protocols Nicole Opferman CS 526.
Native InfiniBand Storage Deployment & Customer Experiences Dave Ellis Director, HPC Architecture LSI Logic IBTA & OFA DevCon San Francisco September 25,
Emerging Storage Options for Server Blade Architectures Server Blade Summit 2005.
Module – 7 network-attached storage (NAS)
Trends in Storage Subsystem Technologies Michael Joyce, Senior Director Mylex & OEM Storage Subsystems IBM.
CPP Staff - 30 CPP Staff - 30 FCIPT Staff - 35 IPR Staff IPR Staff ITER-India Staff ITER-India Staff Research Areas: 1.Studies.
Peter Stefan, NIIF 29 June, 2007, Amsterdam, The Netherlands NIIF Storage Services Collaboration on Storage Services.
Real Parallel Computers. Modular data centers Background Information Recent trends in the marketplace of high performance computing Strohmaier, Dongarra,
Storage Survey and Recent Acquisition at LAL Michel Jouvin LAL / IN2P3
PRESENTATION TITLE GOES HERE DAS to SAN -- iSCSI Offers a Compelling Solution Presented by: Jason Blosil, NetApp and Gary Gumanow, Dell.
© Copyright 2010 Hewlett-Packard Development Company, L.P. 1 HP + DDN = A WINNING PARTNERSHIP Systems architected by HP and DDN Full storage hardware and.
The NERSC Global File System (NGF) Jason Hick Storage Systems Group Lead CAS2K11 September 11-14,
Introducing Snap Server™ 700i Series. 2 Introducing the Snap Server 700i series Hardware −iSCSI storage appliances with mid-market features −1U 19” rack-mount.
Roland Dreier Technical Lead – Cisco Systems, Inc. OpenIB Maintainer Sean Hefty Software Engineer – Intel Corporation OpenIB Maintainer Yaron Haviv CTO.
Copyright DataDirect Networks - All Rights Reserved - Not reproducible without express written permission Adventures Installing Infiniband Storage Randy.
1 - Q Copyright © 2006, Cluster File Systems, Inc. Lustre Networking with OFED Andreas Dilger Principal System Software Engineer
Small File File Systems USC Jim Pepin. Level Setting  Small files are ‘normal’ for lots of people Metadata substitute (lots of image data are done this.
9/16/2000Ian Bird/JLAB1 Planning for JLAB Computational Resources Ian Bird.
Trends In Network Industry - Exploring Possibilities for IPAC Network Steven Lo.
InfiniSwitch Company Confidential. 2 InfiniSwitch Agenda InfiniBand Overview Company Overview Product Strategy Q&A.
 International Computers Limited, 2001 AXiS 20/3/2001 AXiS Trimetra User Group Glenn Fitzgerald Manager, Storage Solutions ICL High Performance Systems.
November 2, 2000HEPiX/HEPNT FERMI SAN Effort Lisa Giacchetti Ray Pasetes GFS information contributed by Jim Annis.
Mike Truitt Sr. Product Planner Bryon Surace Sr. Program Manager
Remote Direct Memory Access (RDMA) over IP PFLDNet 2003, Geneva Stephen Bailey, Sandburst Corp., Allyn Romanow, Cisco Systems,
Storage Networking Evolution Jim Morin VP Strategic Planning June 2001.
Copyright © 2014 EMC Corporation. All Rights Reserved. Windows Host Installation and Integration for Block Upon completion of this module, you should be.
CASPUR Storage Lab Andrei Maslennikov CASPUR Consortium Catania, April 2002.
OFED Usage in VMware Virtual Infrastructure Anne Marie Merritt, VMware Tziporet Koren, Mellanox May 1, 2007 Sonoma Workshop Presentation.
10/22/2002Bernd Panzer-Steindel, CERN/IT1 Data Challenges and Fabric Architecture.
Infiniband in EDA (Chip Design) Glenn Newell Sr. Staff IT Architect Synopsys.
SAN DIEGO SUPERCOMPUTER CENTER SDSC's Data Oasis Balanced performance and cost-effective Lustre file systems. Lustre User Group 2013 (LUG13) Rick Wagner.
O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Facilities and How They Are Used ORNL/Probe Randy Burris Dan Million – facility administrator.
O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Probe Plans and Status SciDAC Kickoff July, 2001 Dan Million Randy Burris ORNL, Center for.
Storage and Storage Access 1 Rainer Többicke CERN/IT.
Ultimate Integration Joseph Lappa Pittsburgh Supercomputing Center ESCC/Internet2 Joint Techs Workshop.
CASPUR / GARR / CERN / CNAF / CSP New results from CASPUR Storage Lab Andrei Maslennikov CASPUR Consortium May 2003.
Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft Implementation of a reliable and expandable on-line storage for compute clusters Jos van Wezel.
Mellanox Connectivity Solutions for Scalable HPC Highest Performing, Most Efficient End-to-End Connectivity for Servers and Storage April 2010.
December 26, 2015 RHIC/USATLAS Grid Computing Facility Overview Dantong Yu Brookhaven National Lab.
CERN Computer Centre Tier SC4 Planning FZK October 20 th 2005 CERN.ch.
Barriers to IB adoption (Storage Perspective) Ashish Batwara Software Solution Architect May 01, 2007.
RHIC/US ATLAS Tier 1 Computing Facility Site Report Christopher Hollowell Physics Department Brookhaven National Laboratory HEPiX Upton,
Office of Science U.S. Department of Energy NERSC Site Report HEPiX October 20, 2003 TRIUMF.
KIT – University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association STEINBUCH CENTRE FOR COMPUTING - SCC
SA1 operational policy training, Athens 20-21/01/05 Presentation of the HG Node “Isabella” and operational experience Antonis Zissimos Member of ICCS administration.
National Energy Research Scientific Computing Center (NERSC) NERSC Site Report Shane Canon NERSC Center Division, LBNL 10/15/2004.
LBNL/NERSC Site Report Cary Whitney NERSC
Pathway to Petaflops A vendor contribution Philippe Trautmann Business Development Manager HPC & Grid Global Education, Government & Healthcare.
LBNL/NERSC/PDSF Site Report for HEPiX Catania, Italy April 17, 2002 by Cary Whitney
20071 Native Infiniband Storage John Josephakis, VP, Data Direct Networks St. Louis – November 2007.
Introduction to Data Analysis with R on HPC Texas Advanced Computing Center Feb
GPFS Parallel File System
The Efficient Fabric Presenter Name Title.
Ryan Leonard Storage and Solutions Architect
Cisco Storage Networking iSCSI
Storage Area Network
Introduction to Networks
GGF15 – Grids and Network Virtualization
Joint Techs Workshop InfiniBand Now and Tomorrow
Storage Networking Protocols
Scalable Database Services for Physics: Oracle 10g RAC on Linux
Presentation transcript:

National Energy Research Scientific Computing Center (NERSC) The GUPFS Project at NERSC GUPFS Team NERSC Center Division, LBNL November 2003

2 Global Unified Parallel File System (GUPFS) Global/Unified –A file system shared by major NERSC production systems –Using consolidated storage and providing unified name space –Automatically sharing user files between systems without replication –Integration with HPSS and Grid is highly desired Parallel –File system providing performance that is scalable as the number of clients and storage devices increase

3 GUPFS Project Overview Five year project introduced in NERSC Strategic Proposal Purpose to make advanced scientific research using NERSC systems more efficient and productive Simplify end user data management by providing a shared disk file system in NERSC production environment An evaluation, selection, and deployment project –May conduct or support development activities to accelerate functionality or supply missing functionality

4 Current NERSC Storage Configuration Each system has its own separate direct-attached storage Each system has its own separate user file system and name space Data transfer between systems is over the LAN Include Large computational systems, small systems, and support systems

5 NERSC Storage Vision Single storage pool, decoupled from NERSC computational systems – Diverse file access - supporting both home file systems and large scratch file system – Flexible management of storage resource – All systems have access to all storage – require different fabric – Buy new storage (faster and cheaper) only as we need it High performance large capacity storage – Users see same file from all systems – No need for replication – Visualization server has access to data as soon as it is created Integration with mass storage – Provide direct HSM and backups through HPSS without impacting computational systems Potential Geographical distribution

6 Envisioned NERSC Storage Configuration (GUPFS)

7 Where We Are Now At the beginning of the 3 rd year of the three-year technology evaluation phase Evaluation of technology components needed for GUPFS: –Shared File System, –SAN Fabric, and –Storage Constructed complex testbed environment simulating envisioned NERSC environment Developed testing methodologies for evaluation Identifying and testing appropriate technologies in all three areas Collaborating with vendors to fix problems and influence their development in directions beneficial to HPC

8 GUPFS Tesbed (FY2004)

9 Technologies Evaluated File Systems – Sistina GFS 5.1, 5.2 Beta – ADIC StorNext (CVFS) File System 2.0 – Lustre 0.6 (1.0 Beta 1) – GPFS 1.3 for Linux – Panasas Fabric – FC (1Gb/s and 2Gb/s): Brocade SilkWorm and Qlogic SANbox2 – Ethernet (iSCSI): Cisco SN 5428, Intel iSCSI HBA, Adaptec TOE – Infiniband (1x and 4x): InfiniCon and Topspin IB to GE/FC bridges (SRP over IB, iSCSI over IB), – Inter-connect: Myrinnet 2000 (Rev D) Storage – Traditional Storage: Dot Hill, Silicon Gear, Chaparral – New Storage: Yotta Yotta NetStorager, EMC CX 600, 3PAR

10 Parallel I/O Tests I/O Tests used for file system and storage evaluation: –Cache Write –Cache Read –Cache Rotate Read –Disk Write –Disk Read –Disk Rotate Read

11 Storage Performance

12 Storage Scalability Traditional Storage: DotHill, 1 1Gb/s port, 6-way R0 New Storage: YottaYotta. 4 blades, 2 FC ports per blade, 32-way R0 Client: Dual P4 Xeon, 1GB Mem, Linux smp HBA: Qlogic QLA2340 2Gb HBA

13 Comparison of Fabric Technologies Storage: YottaYotta (1 2Gb/s FC Raid-0 LUN) GE: Intel E1000 FC: Qlogic QLA2340 2Gb/s HBA IB: InfiniCon Infin7000 Gen 1 switch & 4x HCA

14 Software iSCSI vs. iSCSI HBA Storage: Chaparral (1 2Gb/s FC Raid-0 LUN) iSCSI: Intel PRO/1000 IP Storage Adapter Switch: Cisco SN5428 iSCSI Switch (1 1Gb/s iSCSI port) Normalized CPU Utilization=%SYS/Throughput (sec per KB/s) Date: 07/2003

15 Performance Impact of Kernel Changes Storage: Chaparral (1 2Gb/s FC Raid-0 LUN) iSCSI HBA: Intel PRO/1000 IP Storage Adapter Switch: Cisco SN5428 iSCSI Switch (1 1Gb/s iSCSI port) Date: 07/2003

16 Infiniband SRP Performance Storage: 3PARdata & EMC 600 (2Gb/s FC Raid-5 LUNs) FC: Qlogic QLA2340 2Gb/s HBA IB: Topspin 90 switch & 4x HCA Date: 08/2003

17 When Storage is a Bottleneck Storage: DotHill (2 1Gb/s FC Raid-5 LUNs) FC: Qlogic QLA2340 2Gb/s HBA Date: 03/2003 – 04/2003

18 ADIC StorNext File System Storage: YottaYotta, Single 28-way RAID-0 LUN (exported on 8 2Gb/s ports) Clients: Dual P4 Xeon, 1GB Mem, Linux smp FC: Qlogic QLA 2340

19 GPFS/Linux 1.3 Performance SAN: 3PARdata (8 2Gb/s FC ports over 1 Raid-5 LUN) Clients: Dual P4 Xeon, 1GB Mem, Linux smp FC: Qlogic QLA2340 Interconnect: Myrinet (Rev D) LAPI NSD: 8 YottaYotta 4-way RAID-0 LUNs (8 2Gb/s ports) Clients: Dual P4 Xeon, 1GB Mem, Linux smp FC: Qlogic QLA 2340 Interconnect: GigE

20 GPFS/Linux I/O Scalability Linux Cluster: NERSC Alvarez (87 nodes) Storage (ALV): 2 storage nodes (IBM Raid-5 LUNs, 100MB/s max) Storage (GUS): YottaYotta 8 Raid-0 LUNs, 8 x 2Gb FC ports Interconnect: Myrinet 2000 (Rev C) Date: 08/2003 – 10/2003

21 Panasas File System Host: Linux Panasas: 1 Shelf, 4 x 1 GigE with trunking, 1 DirectorBlade, 10 StorageBlade, with Direct Flow Interconnect: GigE File Sizes per I/O process/client: 128 MB, 1GB, 4 GB

22 File System - Metadata Performance Storage: YottaYotta (Raid-0 LUNs, 8 2Gb/s FC ports) FC: Qlogic QLA2340 2Gb/s HBA Date: 03/2003

23 Near Term Evaluation Activities ● Continue GFS, ADIC StorNext, GPFS File, and Lustre evaluations ● Conduct test of additional file system, storage and fabric technologies as possible. Potential targets: ● Panasas object based file system (with Direct Flow) ● IBM SAN File System (StorageTank) ● IBRIX File System ● DataDirect Networks storage ● Cisco MD950x FC and iSCSI director class switch ● iSCSI HBAs ● Quadrics interconnect (with Lustre evaluation, etc). ● Myrinet Gigabit Ethernet blade (fabric bridge) ● 10Gb Ethernet ● 4Gb and 10Gb Fibre Channel ● 12x Infiniband

24 Thank You GUPFS Project Web Site – GUPFS FY02 Technical Report – Contact Info: – Project Lead: Greg Butler

25 GUPFS Team Greg Butler Project Lead) Will Baird Rei Lee Craig Tull Michael Welcome Cary Whitney