Storage Performance Benchmarking: Part 4 – File Components Ken Cantrell, NetApp Mark Rogov, Dell EMC October 20, 2016

SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.

About SNIA SNIA is made up of 160 unique member companies. We have 3,500 active contributing members that reach a community of over 50,000 IT end users and storage professionals worldwide.

About The Speakers Alex McDonald Ken Cantrell Mark Rogov NetApp Manager Perf Engineering @kencantrelljr Mark Rogov Dell EMC Advisory Systems Engineer @rogovmark Alex McDonald NetApp Vice Chair of the SNIA ESF @alextangent

Storage Performance Benchmarking July 30, 2015 OCT 20, 2015 March 8, 2016 Today ONE LEFT! Workload definitions Metrics and terminology Solution Under Test block Components File Components intro fs fs ops perf fun end

Session 1 – Terminology and Context OPS Count every protocol operation per second MB/S PAYLOAD sum Of every IOPS IO operation Response time Time target Takes to reply To an io CONTEXT MAKES METRICS MATTER Terminology And Basic Concepts Graph Fun intro fs fs ops perf fun end

Session 2 – The Slowest Component Matters Most DISK BOUND CLIENT BOUND Do Less Work Do Work Faster Increase Parallelism Slow Component Matters Most Bottlenecks Always Exist 3 Performance Principles intro fs fs ops perf fun end

Session 3 – Block Is The Foundation Random vs. Sequential Workloads Flash vs Disk Drive RAID Performance Implications intro fs fs ops perf fun end

Today: File & Filesystem Performance storage controller front-end connect clients / hosts Filesystem Filesystem counts 245 Filesystems NAS LOCAL intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Agenda Introduction Filesystem Layout Fundamentals INTRO What are Filesystem operations? Performance in a File Oriented World INTRO Graph fun with performance Filesystem Summary FS Ops File Perf FUN END

Term “Filesystem” was coined in 1964 What Is A Filesystem? Term “Filesystem” was coined in 1964 / etc dev home usr var ken mark log dir1 fileA fileB intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Filesystem Metadata Metadata / etc dev home usr var ken mark log dir1 name etc dev home usr var permissions ken mark log access times dir1 fileA … and more … data block pointers fileB data block pointers data block pointers intro fs fs ops perf fun end

Filesystem Basic Metadata Structure /home/ken/fileA Name: / Type: dir Size: 2KiB Owner: root Mode: 0700 …a few more… file & dir records Physical Area intro fs fs ops perf fun end

Filesystem Basic Metadata Structure /home/ken/fileA Name: / Type: dir Size: 2KiB Owner: root Mode: 0700 …a few more… file & dir records Name: home Type: dir Size: 2KiB Owner: root Mode: 0755 …a few more… file & dir records Physical Area intro fs fs ops perf fun end

Filesystem Basic Metadata Structure /home/ken/fileA Name: / Type: dir Size: 2KiB Owner: root Mode: 0700 …a few more… file & dir records Name: home Type: dir Size: 2KiB Owner: root Mode: 0755 …a few more… file & dir records Name: ken Type: dir Size: 2KiB Owner: ken Mode: 0755 …a few more… file & dir records Name: fileA Type: file Size: 20KiB Owner: ken Mode: 0755 …a few more… data block pointers Physical Area 1 3 2 intro fs fs ops perf fun end

Filesystem Metadata Metadata / etc dev home usr var ken mark log dir1 name etc dev home usr var permissions ken mark log access times dir1 fileA … and more … data block pointers fileB data block pointers data block pointers intro fs fs ops perf fun end

FILESYSTEMS provide STRUCTURE Filesystem Metadata FILESYSTEMS provide STRUCTURE / Metadata name etc dev home usr var permissions ken mark log access times dir1 fileA … and more … data block pointers fileB data block pointers data block pointers intro fs fs ops perf fun end

Agenda Introduction Filesystem Layout Fundamentals INTRO What are Filesystem operations? Performance in a File Oriented World INTRO Graph fun with performance Filesystem Summary FS Ops File Perf FUN END

User Action Converted Into Filesystem Operations Local File Operations User Action Converted Into Filesystem Operations $ echo “test” > data.txt STAT OPEN WRITE CLOSE intro fs fs ops perf fun end

User Action Converted Into Filesystem Operations Local File Operations User Action Converted Into Filesystem Operations $ echo “test” > data.txt STAT OPEN WRITE CLOSE $ cat data.txt STAT OPEN READ CLOSE intro fs fs ops perf fun end

User Action Converted Into Filesystem Operations Local File Operations User Action Converted Into Filesystem Operations $ echo “test” > data.txt STAT OPEN WRITE CLOSE $ cat data.txt STAT OPEN READ CLOSE $ rm data.txt STAT UNLINK intro fs fs ops perf fun end

Are There Only Six File Operations? Local File Operations Are There Only Six File Operations? $ echo “test” > data.txt READ STAT OPEN WRITE CLOSE WRITE OPEN counts 108 Filesystem Calls $ cat data.txt STAT STAT OPEN READ CLOSE UNLINK CLOSE $ rm data.txt …more… STAT UNLINK intro fs fs ops perf fun end

Relationship Between Filesystem Operations And Block IO Operations Local File Operations Relationship Between Filesystem Operations And Block IO Operations $ echo “test” > data.txt READ STAT OPEN WRITE CLOSE WRITE Block IO Read OPEN $ cat data.txt STAT Block IO Write STAT OPEN READ CLOSE UNLINK CLOSE $ rm data.txt …more… STAT UNLINK intro fs fs ops perf fun end

Create A New File Containing Just Three Spaces NAS File Operations Create A New File Containing Just Three Spaces intro fs fs ops perf fun end

Create A New File Containing Just Three Spaces NAS File Operations Create A New File Containing Just Three Spaces intro fs fs ops perf fun end

THERE ARE MORE OPS THAN JUST In The File World… THERE ARE MORE OPS THAN JUST READS and writes intro fs fs ops perf fun end

Agenda Introduction Filesystem Layout Fundamentals Intro What are Filesystem operations? Performance in a File Oriented World Intro Graph fun with performance Filesystem Summary FS Ops File Perf FUN END

Performance In a File Oriented World is Affected By … Caching Performance Differences Between Operations Data vs Metadata Ratios intro fs fs ops perf fun end

Performance Characteristics Of File Operations create / write read unlink / delete ls intro fs fs ops perf fun end

Performance Characteristics Of File Operations create / write read unlink / delete ls intro fs fs ops perf fun end

Performance Characteristics Of File Operations create / write read unlink / delete ls intro fs fs ops perf fun end

Performance Characteristics Of File Operations create / write read unlink / delete ls intro fs fs ops perf fun end

Performance In a File Oriented World Be Careful … Operation Performance Will Almost Always Be Affected By the SUT intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Performance In a File Oriented World is Affected By … Caching Performance Differences Between Operations Data vs Metadata Ratios intro fs fs ops perf fun end

Performance In a File Oriented World Does The Number Of Files Make A Difference? create 81x 16,005 sec 197 sec 8kib 4.1gib MD MD read 17x 8kib MD 1,707 sec 100 sec 8kib MD 540,000 total unlink 4,426x 729 sec 0.1647 sec 8kib MD 8kib MD ls 17,142x 12 sec 0.0007 sec 8kib MD stat 5,000x 1 sec 0.0002 sec 4.1 GiB intro fs fs ops perf fun end

Performance In a File Oriented World is Affected By … Caching Performance Differences Between Operations Data vs Metadata Ratios intro fs fs ops perf fun end

Performance in a File Oriented World uncached ‘ls’ cached ‘ls’ “Uncached ‘ls’” had a USB unmount just prior to the ‘ls’ command execution intro fs fs ops perf fun end

Performance In a File Oriented World is Affected By … Caching Performance Differences Between Operations Data vs Metadata Ratios intro fs fs ops perf fun end

Agenda Introduction Filesystem Layout Fundamentals Intro What are Filesystem operations? Performance in a File Oriented World Intro Graph Fun with Performance Filesystem Summary FS Ops File Perf FUN END

Which Is Better? intro fs fs ops perf fun end I have two results here, which is better? Uh…what the heck ARE these? What are we charting? What are my axes? (These are the WRITE op/s for SWBUILD and DATABASE intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Which Is Better? intro fs fs ops perf fun end Ok … this is better, but still doesn’t answer all the questions. We learned from our first webcast that “context makes metrics matter.” And, we put a pretty good focus on latency. So … what do these look like if you look at BOTH operations / second and data transfer rates, AND you chart using a y-axis of latency? These are the total KBps for SWBUILD (result1) and DB (result2) intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Which Is Better? intro fs fs ops perf fun end Ok … this is better, but still doesn’t answer all the questions. We learned from our first webcast that “context makes metrics matter.” And, we put a pretty good focus on latency. So … what do these look like if you look at BOTH operations / second and data transfer rates, AND you chart using a y-axis of latency? These are the total KBps for SWBUILD (result1) and DB (result2) intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Which Is Better? intro fs fs ops perf fun end We’re getting some where … but is this really just a re-cast of the first webcast? Actually … I notice you keep saying “Total KiB/s” … why is that? Could you break it out perhaps? These are the total KBps for SWBUILD (result1) and DB (result2) intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Which Is Better? intro fs fs ops perf fun end We’re getting some where … but is this really just a re-cast of the first webcast? Actually … I notice you keep saying “Total KiB/s” … why is that? Could you break it out perhaps? These are the total KBps for SWBUILD (result1) and DB (result2) intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Which Is Better? intro fs fs ops perf fun end We’re getting some where … but is this really just a re-cast of the first webcast? Actually … I notice you keep saying “Total KiB/s” … why is that? Could you break it out perhaps? These are the total KBps for SWBUILD (result1) and DB (result2) intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Which Is Better? intro fs fs ops perf fun end We’re getting some where … but is this really just a re-cast of the first webcast? Actually … I notice you keep saying “Total KiB/s” … why is that? Could you break it out perhaps? These are the total KBps for SWBUILD (result1) and DB (result2) intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Which Is Better? intro fs fs ops perf fun end We’re getting some where … but is this really just a re-cast of the first webcast? Actually … I notice you keep saying “Total KiB/s” … why is that? Could you break it out perhaps? These are the total KBps for SWBUILD (result1) and DB (result2) intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Which Is Better? Database Workload SOFTWARE BUILD Workload intro fs We’re getting some where … but is this really just a re-cast of the first webcast? Actually … I notice you keep saying “Total KiB/s” … why is that? Could you break it out perhaps? These are the total KBps for SWBUILD (result1) and DB (result2) intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Workload is yours The only that matters Which Is Better? Database SOFTWARE BUILD Workload We’re getting some where … but is this really just a re-cast of the first webcast? Actually … I notice you keep saying “Total KiB/s” … why is that? Could you break it out perhaps? These are the total KBps for SWBUILD (result1) and DB (result2) intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

Agenda Introduction Filesystem Layout Fundamentals Intro What are Filesystem operations? Performance in a File Oriented World Intro Graph fun with performance Filesystem Summary FS Ops File Perf FUN END

Storage Performance Benchmarking Today July 30, 2015 OCT 20, 2015 March 8, 2016 ONE LEFT! Workload definitions Metrics and terminology Solution Under Test block Components File Components intro fs fs ops perf fun end

After This Webcast A PDF and a PPT of the slides for this and all previous parts of this Webcast series will be posted to the SNIA Ethernet Storage Forum (ESF) website and available on-demand PPT and PDF: http://www.snia.org/forums/esf/knowledge/webcasts Presentation Recording: https://www.brighttalk.com/webcast/663/219127 A full Q&A from this webcast, including answers to questions we couldn't get to today, will be posted to the SNIA-ESF blog http://sniaesfblog.org/ Follow us on Twitter @SNIAESF, @RogovMark, @KenCantrellJr, @alextangent, @DrJMetz Final Webcast – First Half of 2017 “Storage Performance Benchmarking: Part 5”

THANK YOU!

Client Caching Affects Operation Mixes Lookup calls increase from 4% to 42% Getattr calls decrease from 65% to 14% Increasing # of Files, Active Dataset and Workload Intensity intro fs fs ops perf fun end

Operation Mixes For Different Applications Metadata IO Data IO PROPOSED intro fs fs ops perf fun end © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only