Large Scale Test of a storage solution based on an Industry Standard Michael Ernst Brookhaven National Laboratory ADC Retreat Naples, Italy February 2, 2011

Motivation Though dCache at BNL supports distributed analysis well (up to 5000 concurrent analysis jobs) we are looking into ways to improve the usability of our environment, e.g. include large-scale interactive analysis. We want to explore to what extent we can build on industrial products as far as the facility part is concerned w/o having to rely on community extensions and interfaces. NFS 4.1 (pNFS) appealing because of performance, simplicity, level of integration w/ OS BlueArc successfully used by PHENIX and STAR

Areas of Interest

Whys is NFS4.1(pNFS) attractive?

BlueArc System Performance

Traditional Network File System No Throughput Aggregation Metadata and data together BUT Proven architecture Enterprise features Open, standard protocols Open storage philosophy BlueArc is about high performance network storage through standards. We deliver our file services via NFS and CIFS for example. So just so we’re on the same page, this is a view of a typical, traditional NFS system. The benefits are… At BlueArc we deliver the fastest systems for mixed workloads in this architecture BUT there what you don’t get here is throughput aggregation to go beyond the fastest nodes we have, and metadata and data are typically residing together on the disks – essentially getting in the way of each other as you scale up in bandwidth performance 8

pNFS Architecture With that said, let’s take a look at the pNFS architecture and I’ll also point out some of the areas where BlueArc’s implementation go beyond the minimum that is defined in the standard. In this picture you have … walk through the general elements Essentially pNFS is a protocol standard, it defines the communication between the clients and the metadata server and belween the clients and the data servers (movers). The communication between the metadata server and the data movers is an implementation detail. Some of the items we add to the solution is building on our current technology portfolio. With our hardware accelerated IOPS performance we have a screaming metadata server that will not melt under heavy workloads. We also have the ability through out to cluster nodes wherever needed. This is a cluster of clusters. The performance profile for the data servers is different so we will optimize those for maximum bandwidth performance. The architecture is essentially limitless, you scale performance (bandwidth) by adding more data server nodes. Any questions? 11

BlueArc pNFS Based NAS Platform Highly scalable single metadata server Clustered for HA Architecture supports multiple clustered metadata servers Support for heterogeneous data servers Built on Enterprise Class Platform Reliability Full Featured NAS – Quota, Snapshots etc. 12

pNFS for HPC Performance Scaling Data files automatically spread across multiple data FS for load balancing and performance Individual data files optionally striped across multiple data FS for performance Extreme Flexibility: A single server can run the entire cluster (acting as both metadata server and data mover) Can scale performance by adding data movers (and relocate the FS) Can scale capacity by adding data file systems Clients Direct IO Get Data Location File A File B Meta-Data File System Separate Data File Systems 13 13

Product Availability and Test Bed at BNL BA’s pNFS – Mid 2011 Up to 40 data movers with up to 8PB capacity each – growing to 128 data movers Mercury - High performance metadata server BlueArc Mercury or BlueArc Linux-based data mover appliances Performance Goals for Test Bed at BNL Will become DDM endpoint w/ SRM, GridFTP, Xrootd Up to 2k concurrent Analysis Jobs Up to 4 GB/s throughput between storage system and clients Aiming at having ~500 TB of disk space 15 15

Test Bed at BNL SRM Xrootd WA Network GridFTP With that said, let’s take a look at the pNFS architecture and I’ll also point out some of the areas where BlueArc’s implementation go beyond the minimum that is defined in the standard. In this picture you have … walk through the general elements Essentially pNFS is a protocol standard, it defines the communication between the clients and the metadata server and belween the clients and the data servers (movers). The communication between the metadata server and the data movers is an implementation detail. Some of the items we add to the solution is building on our current technology portfolio. With our hardware accelerated IOPS performance we have a screaming metadata server that will not melt under heavy workloads. We also have the ability through out to cluster nodes wherever needed. This is a cluster of clusters. The performance profile for the data servers is different so we will optimize those for maximum bandwidth performance. The architecture is essentially limitless, you scale performance (bandwidth) by adding more data server nodes. Any questions? WA Network GridFTP 16