Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. An Optimized Entry-Level Lustre* Solution in a Small Form Factor Chakravarthy Nagarajan Solutions Architect, APJ

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. 2 Legal Information All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at Intel technologies may require enabled hardware, specific software, or services activation. Check with your system manufacturer or retailer. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. Other names and brands may be claimed as the property of others. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit © 2016 Intel Corporation

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. 2*OSS/2*MDS – Cost Overheads Harder to Absorb 3 IML Server Ethernet Switch (Grey) InfiniBand* Switch (Black) Object Storage Servers Metadata Servers IML Server Lustre Clients Metadata Servers Object Storage Servers Metadata servers add to perceived overheads when designing an entry-level Lustre solution Minimum server count for an HA Lustre file system is 4 servers, 2 storage arrays Cost of entry is high compared to NFS Performance is good, however, and framework is scalable Management Network High Performance Data Network

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. What if … ? Ethernet Switch (Grey) InfiniBand* Switch (Black) Lustre Clients MDS/ MGS OSS MGT MDT OST(s) MDS/MGS OSS IML Server Management Network High Performance Data Network

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. 2 Node Intel EE ® For Lustre* with shared storage array 5 Server-1 Net HBA Server-2 Net HBA Management Lustre / Data OST0 OST1 OST2 OST3 OSTn Heartbeat MDT0/MGT

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Scalability & Performance – Small Form Factor Lustre Clients … Building block for Scalability. Add storage servers to increase capacity and performance Basic Block MDS/ MGS OSS1 Additional Block – 1U JBOD(NVMe) OSS2 OSS3 … OSS4 OSS5 … Additional Block – Legacy JBOD(SAS/NL-SAS) MDS2 MDS3 … Additional Block – Flash MDTs - DNE

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Small Form Factor Lustre* Solution – An Overview 7 Provide an Entry Level Lustre storage solution in a high density form factor  Low cost, small footprint integrated Intel ® EE for Lustre* software storage platform  No single point of failure  Failover - HA resource agents to handle SAS link failure, Infiniband* link failure and ZFS* export/import  Scale out with performance within the same namespace  Monitoring - Intel ® Manager for Lustre*

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Test Bed - Lustre* 8  MDS and OSS Servers –2x E v2 CPU, 128 GB Memory, 4x 500GB SATA HDD, 1x FDR,4x 6Gb SAS –Centos* 7.2 –pcs el7.x86_64 –Intel ® EE for Lustre* software – v3.0 –zfs* and spl*  4 x Intel ® Storage System JBOD2224S2DP with 20x 146GB 15k rpm SAS drives each –8x OST(s) – RAIDZ2(10x 146GB HDDs) each –1x MGT/MDT – Mirror(4x 400GB SSDs)

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Test Bed – NFS* 9  NFS* Servers – 2x E v2 CPU, 128 GB Memory, 4x 500GB SATA HDD, 1x FDR,4x 6Gb SAS – Centos* 7.2 – pcs el7.x86_64 –zfs* and spl* –NFS*V4  4x Intel ® Storage System JBOD2224S2DP with 20x 146GB 15k rpm SAS drives each –1x ZFS* pool – RAIDZ2 with 80x 146GB HDDs

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Obdfilter-Survey 10 Test Environment: Please refer to slide-8 and slide-9. Command Used: thrhi=1 thrlo=1 nobjhi=1 nobjlo=128 size=$size case="disk" targets=$TARGETS rslt_loc=$OUTPUT /usr/bin/obdfilter-survey

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. IOR 11 Test Environment: Please refer to slide-8 and slide-9. Command Used: mpirun -np -f $IOR -F -e -w -r -b -t 1m -o

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. MDTEST(with sync) – 1/2 12 Test Environment: Please refer to slide-8 and slide-9. Command Used – File Ops: mpirun -np -f hostfile -rr $mdtest -d -i 3 -b 1 -z 1 -L -I -y -u -t –F Command Used – Dir Ops: mpirun -np -f hostfile -rr $mdtest -d -i 3 -b 1 -z 1 -L -I -y -u -t –D

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. MDTEST(with sync) - 2/2 13 NFS Stats operations are better than Lustre due to the following facts  Lustre to block waiting on a TXG to sync.  4x SSDs for MDT zpool on mirror and stripe Vs 80x SAS drives on NFS Zpool Test Environment: Please refer to slide-8 and slide-9. Command Used – File Ops: mpirun -np -f hostfile -rr $mdtest -d -i 3 -b 1 -z 1 -L -I -y -u -t –F Command Used – Dir Ops: mpirun -np -f hostfile -rr $mdtest -d -i 3 -b 1 -z 1 -L -I -y -u -t –D

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Key Benefits over NFS* 14  57 percent better performance for write operations and 25 percent better performance for read operations  94-percent storage efficiency for read and write operations  Up to 2.5x and 8.8x faster file creation and file removal, respectively  Up to 8.45x and 3.68x faster directory creation and removal, respectively  Support for high availability and manageability **The above results are the summary of IOR and mdtest observations as described in the slides 10,11,12 and 13. Test Environment: Please refer to slide-8 and slide-9.

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Design Considerations 15  Use SSDs for MDTs to achieve the best metadata performance on file and directory creation.  Use RAID-Z2 for OSTs and striped mirror for MDTs.  Sufficient memory on MDSs and OSSs.  Disable auto-failback to avoid conflicts on failures in pcs configuration—this configuration is designed to optimize the recovery time in case of failover.

Copyright ® Intel Corporation All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Future Work 16  Stats performance improvements  Optimize single thread write performance  Optimize small files performance