1. Parallel File System Benchmarking Project Tracey Wilson DICE, Program Manager CSC twilso23@csc.com

2. DICE Overview What is the DICE TAP? The File System Comparison Issue and Why? DICE Plan for Framework Normalization Summary Outline

3. Nationwide Test Environment for Technology Innovations

4. Testing & Validation Services Independent 3 rd Party Evaluations Evaluating HPC, Storage, Networking and Data Management Technologies Product & Technology Evaluation & Validations Proof-of-Concept & Performance Validations Technology Research & Development Projects Benchmark & Standards Development & Validation

5. DICE utilizes a geographically distributed test bed Real world research networks DICE sites indicative of HPC Data Centers DICE Architecture

6. Quite Simply… A Test Bed for Advancing Innovation Independent, Unbiased 3 rd Party Evaluations Genesis of a Vision by Government, Industry & Academia to Address Critical HPC Technology Challenges

7. The DICE Technical Advisory Panel Advises DICE on: Technology directions Project concepts Testing methodologies Utilizes working groups to focus on more specific issues Current members include: Community Input is Key Kevin Regimbal, PNNLChris Jordan, TACC Lee Ward, SandiaSteve Conway, IDC Kevin Wilson, P&GDan Duffy, NASA Ron Bewtra, NOAALloyd Slonaker, DoD Matt Leininger, LLNL

8. Lack of standardized metrics for performance Need exists for file system performance comparison Need exists for defined benchmarks for local and remote file systems Different HPC systems and their file system architectures have no direct correlations Scaling: Scaling of current file systems has unpredictable performance impacts Load: Performance varies with file systems with loads of 50-70% Fragmented data is not accounted for What is the File System Issue?

9. Centers need input for: New procurements User requirements Application performance File systems are a critical piece of an HPC solution How do you as a Center Director make an informed decision? Why do HPC Centers Care? “Can you prove or rather disprove the performance or value of one file system compared to another?”

10. In 2007, DICE received 3 proposals for evaluation projects: Panasas DoD Microsoft DICE Advisory Panel offered new solution “Develop a standard benchmark framework for file system evaluation” How did the Effort Begin? 1.Make the framework expandable to use 2.Make it fully available for the HPC community to use 3.Effort was to include normalization of results

11. Benefits of Standardization Standardized Metric One set of benchmarks for all parallel file systems Results will be normalized for comparison HPC Community needs non-biased benchmark(s) Increased understanding of the impacts on current and proposed upgrades to a center’s file system: The storage subsystem performance Scaling Load performance

12. Current Ideas Parameterized Benchmark Suite Simulate many different workloads Expose more tuning/configuration options in the benchmarks Suite Extensible so users can add new benchmarks Have unique benchmarks for synthetic workloads Transactions Streaming Random I/O Various Read/Write Ratios

13. What is the plan? Currently surveying the HPC benchmark and file system community including: Carnegie Melon University of California/Santa Cruz ORNL Argonne National Labs PNNL TACC LLNL And many more

14. Application I/O Patterns Many different I/O intensive applications Investigate the development of an I/O analyzer Recognize I/O utilization of applications Similar to trace and trace replay Add this functionality to the framework to simulate application I/O patterns Verification of performance Can be applied on different architectures

15. Normalization Normalization is needed between differing architectures Need a formula Performance Cost Additional Criteria will be evaluated Aging and fragmentation of data should be considered Difficult to develop Need to develop on a small scale first Will the normalization hold as you scale systems? Plan is to create the framework first then normalize

16. Summary Lot of collaborative effort from the Team and the Community is required Full project plan is in draft Survey underway to identify current efforts Plan to have initial framework for file system testing by early 2010 Plan to use this on other file system types

17. How To Get Involved Collaboration is key Developing a DICE Forum for this project Will be collecting traces DICE TAP will be forming specific working group for this effort Interested parties should send email to twilson@avetec.org DICE Website www.diceprogram.org