Large Scale Test of a storage solution based on an Industry Standard Michael Ernst Brookhaven National Laboratory ADC Retreat Naples, Italy February 2, 2011

Motivation Though dCache at BNL supports distributed analysis well (up to 5000 concurrent analysis jobs) we are looking into ways to improve the usability of our environment, e.g. include large-scale interactive analysis. We want to explore to what extent we can build on industrial products as far as the facility part is concerned w/o having to rely on community extensions and interfaces.  NFS 4.1 (pNFS) appealing because of performance, simplicity, level of integration w/ OS  BlueArc successfully used by PHENIX and STAR

Areas of Interest

Whys is NFS4.1(pNFS) attractive?

BlueArc System Performance

Transparent Data Mobility Tiered Storage for Data SAN Network Storage Cluster  Automatic and transparent data migration between tiers  Rules-based policy engine reduces manual intervention  Third-party or external storage devices as an integrated tier  Reduced dependence on high performance tier for peak demands The seamless migration of data across storage tiers within a single namespace Ease data management and reduce costs Back-end Storage SAN Tier 1 High Performance Tier 2 High Capacity Tier 0 Solid State Cache Tier 3 De-duplication Encryption Compression Existing NAS Users

BlueArc Platform Comparison Mercury 50Mercury 100Titan 3100 Product ClassLower Mid-rangeMid-range Cluster Nodes2 Up to 4Up to 8 Max Storage Capacity4 PB8 PB NFS Throughput700 MB/s1100 MB/s1200 MB/s Performance (specSFS IOPS)60,000100,000 Software / File ServicesAll software and filesystem options (NFS, CIFS, iSCSI) available Titan 3200 High End Up to 8 16 PB 200, MB/s Storage OptionsAll BlueArc storage array options are available with each platform © 2010 BlueArc Corporation

Traditional Network File System Proven architecture Enterprise features Open, standard protocols Open storage philosophy No Throughput Aggregation Metadata and data together BUT

How Does pNFS Change This? pNFS adds parallel I/O to the NFS protocol Eliminates the file server bottleneck Provides parallel data paths, even for a single file pNFS is part of the NFSv4.1 standard Approved by IETF Dec, 2008 RFCs completed editorial review Oct, 2009 RFC numbers issued Jan, 2010 Multiple implementations are in development Client software is expected to be embedded in leading OS distributions The Only Industry Standard Parallel File System

Key Elements of a pNFS System Separation of metadata and data –Metadata server with metadata file systems –Multiple data movers with data file systems –Different requirements for these Clients query metadata server for data location, then perform I/O directly and in parallel to/from data movers Configure for aggregated performance and policy driven value-based data placement 10

pNFS Architecture

BlueArc pNFS Based NAS Platform Highly scalable single metadata server –Clustered for HA –Architecture supports multiple clustered metadata servers Support for heterogeneous data servers Built on Enterprise Class Platform –Reliability –Full Featured NAS – Quota, Snapshots etc. 12

pNFS for HPC Performance Scaling Get Data Location Meta-Data File System Data files automatically spread across multiple data FS for load balancing and performance Individual data files optionally striped across multiple data FS for performance Extreme Flexibility: A single server can run the entire cluster (acting as both metadata server and data mover) Can scale performance by adding data movers (and relocate the FS) Can scale capacity by adding data file systems 13 Separate Data File Systems File A File B Direct IO Clients

Connectivity Initial release will use 10GbE connection to compute clusters BA exploring both 40GbE and 100GbE for next generation of servers, in

Product Availability and Test Bed at BNL BA’s pNFS – Mid 2011 –Up to 40 data movers with up to 8PB capacity each – growing to 128 data movers –Mercury - High performance metadata server –BlueArc Mercury or BlueArc Linux-based data mover appliances Performance Goals for Test Bed at BNL –Will become DDM endpoint w/ SRM, GridFTP, Xrootd –Up to 2k concurrent Analysis Jobs –Up to 4 GB/s throughput between storage system and clients –Aiming at having ~500 TB of disk space 15

Test Bed at BNL WA Network SRM GridFTP Xrootd