1. Large Scale Test of a storage solution based on an Industry Standard Michael Ernst Brookhaven National Laboratory ADC Retreat Naples, Italy February 2, 2011

2. Motivation Though dCache at BNL supports distributed analysis well (up to 5000 concurrent analysis jobs) we are looking into ways to improve the usability of our environment, e.g. include large-scale interactive analysis. We want to explore to what extent we can build on industrial products as far as the facility part is concerned w/o having to rely on community extensions and interfaces.  NFS 4.1 (pNFS) appealing because of performance, simplicity, level of integration w/ OS  BlueArc successfully used by PHENIX and STAR

3. Areas of Interest

4. Whys is NFS4.1(pNFS) attractive?

5. BlueArc System Performance

6. Transparent Data Mobility Tiered Storage for Data SAN Network Storage Cluster  Automatic and transparent data migration between tiers  Rules-based policy engine reduces manual intervention  Third-party or external storage devices as an integrated tier  Reduced dependence on high performance tier for peak demands The seamless migration of data across storage tiers within a single namespace Ease data management and reduce costs Back-end Storage SAN Tier 1 High Performance Tier 2 High Capacity Tier 0 Solid State Cache Tier 3 De-duplication Encryption Compression Existing NAS Users

7. BlueArc Platform Comparison Mercury 50Mercury 100Titan 3100 Product ClassLower Mid-rangeMid-range Cluster Nodes2 Up to 4Up to 8 Max Storage Capacity4 PB8 PB NFS Throughput700 MB/s1100 MB/s1200 MB/s Performance (specSFS IOPS)60,000100,000 Software / File ServicesAll software and filesystem options (NFS, CIFS, iSCSI) available Titan 3200 High End Up to 8 16 PB 200,000 1900 MB/s Storage OptionsAll BlueArc storage array options are available with each platform © 2010 BlueArc Corporation

8. Traditional Network File System Proven architecture Enterprise features Open, standard protocols Open storage philosophy No Throughput Aggregation Metadata and data together BUT

9. How Does pNFS Change This? pNFS adds parallel I/O to the NFS protocol Eliminates the file server bottleneck Provides parallel data paths, even for a single file pNFS is part of the NFSv4.1 standard Approved by IETF Dec, 2008 RFCs completed editorial review Oct, 2009 RFC numbers issued Jan, 2010 Multiple implementations are in development Client software is expected to be embedded in leading OS distributions The Only Industry Standard Parallel File System

10. Key Elements of a pNFS System Separation of metadata and data –Metadata server with metadata file systems –Multiple data movers with data file systems –Different requirements for these Clients query metadata server for data location, then perform I/O directly and in parallel to/from data movers Configure for aggregated performance and policy driven value-based data placement 10

11. pNFS Architecture

12. BlueArc pNFS Based NAS Platform Highly scalable single metadata server –Clustered for HA –Architecture supports multiple clustered metadata servers Support for heterogeneous data servers Built on Enterprise Class Platform –Reliability –Full Featured NAS – Quota, Snapshots etc. 12

13. pNFS for HPC Performance Scaling Get Data Location Meta-Data File System Data files automatically spread across multiple data FS for load balancing and performance Individual data files optionally striped across multiple data FS for performance Extreme Flexibility: A single server can run the entire cluster (acting as both metadata server and data mover) Can scale performance by adding data movers (and relocate the FS) Can scale capacity by adding data file systems 13 Separate Data File Systems File A File B Direct IO Clients

14. Connectivity Initial release will use 10GbE connection to compute clusters BA exploring both 40GbE and 100GbE for next generation of servers, in 2012 14

15. Product Availability and Test Bed at BNL BA’s pNFS – Mid 2011 –Up to 40 data movers with up to 8PB capacity each – growing to 128 data movers –Mercury - High performance metadata server –BlueArc Mercury or BlueArc Linux-based data mover appliances Performance Goals for Test Bed at BNL –Will become DDM endpoint w/ SRM, GridFTP, Xrootd –Up to 2k concurrent Analysis Jobs –Up to 4 GB/s throughput between storage system and clients –Aiming at having ~500 TB of disk space 15

16. Test Bed at BNL WA Network SRM GridFTP Xrootd