1. Storage Performance Benchmarking:
Part 4 – File Components
Ken Cantrell, NetApp
Mark Rogov, Dell EMC
October 20, 2016

2. SNIA Legal Notice
The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted.
Member companies and individual members may use this material in presentations and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without modification
The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee.
Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.
The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.

3. About SNIA
SNIA is made up of 160 unique member companies. We have 3,500 active contributing members that reach a community of over 50,000 IT end users and storage professionals worldwide.

4. About The Speakers Alex McDonald Ken Cantrell Mark Rogov NetApp
Manager Perf Engineering
@kencantrelljr
Mark Rogov
Dell EMC
Advisory Systems Engineer
@rogovmark
Alex McDonald
NetApp
Vice Chair of the SNIA ESF
@alextangent

5. Storage Performance Benchmarking
July 30,
2015
OCT 20,
2015
March 8,
2016
Today
ONE LEFT!
Workload
definitions
Metrics and
terminology
Solution
Under Test
block
Components
File
Components
intro fs
fs ops
perf
fun
end

6. Session 1 – Terminology and Context
OPS
Count every
protocol
operation
per second
MB/S
PAYLOAD sum
Of every
IOPS
IO operation
Response
time
Time target
Takes to reply
To an io
CONTEXT
MAKES
METRICS
MATTER
Terminology And Basic
Concepts
Graph Fun
intro fs
fs ops
perf
fun
end

7. Session 2 – The Slowest Component Matters Most
DISK BOUND
CLIENT
BOUND
Do Less
Work
Do Work
Faster
Increase
Parallelism
Slow Component
Matters Most
Bottlenecks
Always Exist
3 Performance
Principles
intro fs
fs ops
perf
fun
end

8. Session 3 – Block Is The Foundation
Random vs. Sequential
Workloads
Flash vs Disk Drive
RAID Performance
Implications
intro fs
fs ops
perf
fun
end

9. Today: File & Filesystem Performance
storage
controller
front-end
connect
clients /
hosts
Filesystem
Filesystem
counts
245
Filesystems
NAS
LOCAL
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

10. Agenda Introduction Filesystem Layout Fundamentals INTRO
What are Filesystem operations?
Performance in a File Oriented World
INTRO
Graph fun with performance
Filesystem
Summary
FS Ops
File Perf
FUN
END

11. Term “Filesystem” was coined in 1964
What Is A Filesystem?
Term “Filesystem” was coined in 1964 /
etc
dev
home
usr
var
ken
mark
log
dir1
fileA
fileB
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

12. Filesystem Metadata Metadata / etc dev home usr var ken mark log dir1
name
etc
dev
home
usr
var
permissions
ken
mark
log
access times
dir1
fileA
… and more …
data block pointers
fileB
data block pointers
data block pointers
intro fs
fs ops
perf
fun
end

13. Filesystem Basic Metadata Structure
/home/ken/fileA
Name: /
Type: dir
Size: 2KiB
Owner: root
Mode: 0700
…a few more…
file & dir
records
Physical Area
intro fs
fs ops
perf
fun
end

14. Filesystem Basic Metadata Structure
/home/ken/fileA
Name: /
Type: dir
Size: 2KiB
Owner: root
Mode: 0700
…a few more…
file & dir
records
Name: home
Type: dir
Size: 2KiB
Owner: root
Mode: 0755
…a few more…
file & dir
records
Physical Area
intro fs
fs ops
perf
fun
end

15. Filesystem Basic Metadata Structure
/home/ken/fileA
Name: /
Type: dir
Size: 2KiB
Owner: root
Mode: 0700
…a few more…
file & dir
records
Name: home
Type: dir
Size: 2KiB
Owner: root
Mode: 0755
…a few more…
file & dir
records
Name: ken
Type: dir
Size: 2KiB
Owner: ken
Mode: 0755
…a few more…
file & dir
records
Name: fileA
Type: file
Size: 20KiB
Owner: ken
Mode: 0755
…a few more…
data block pointers
Physical Area 1 3 2
intro fs
fs ops
perf
fun
end

16. Filesystem Metadata Metadata / etc dev home usr var ken mark log dir1
name
etc
dev
home
usr
var
permissions
ken
mark
log
access times
dir1
fileA
… and more …
data block pointers
fileB
data block pointers
data block pointers
intro fs
fs ops
perf
fun
end

17. FILESYSTEMS provide STRUCTURE
Filesystem Metadata
FILESYSTEMS
provide
STRUCTURE /
Metadata
name
etc
dev
home
usr
var
permissions
ken
mark
log
access times
dir1
fileA
… and more …
data block pointers
fileB
data block pointers
data block pointers
intro fs
fs ops
perf
fun
end

18. Agenda Introduction Filesystem Layout Fundamentals INTRO
What are Filesystem operations?
Performance in a File Oriented World
INTRO
Graph fun with performance
Filesystem
Summary
FS Ops
File Perf
FUN
END

19. User Action Converted Into Filesystem Operations
Local File Operations
User Action Converted Into Filesystem Operations
$ echo “test” > data.txt
STAT
OPEN
WRITE
CLOSE
intro fs
fs ops
perf
fun
end

20. User Action Converted Into Filesystem Operations
Local File Operations
User Action Converted Into Filesystem Operations
$ echo “test” > data.txt
STAT
OPEN
WRITE
CLOSE
$ cat data.txt
STAT
OPEN
READ
CLOSE
intro fs
fs ops
perf
fun
end

21. User Action Converted Into Filesystem Operations
Local File Operations
User Action Converted Into Filesystem Operations
$ echo “test” > data.txt
STAT
OPEN
WRITE
CLOSE
$ cat data.txt
STAT
OPEN
READ
CLOSE
$ rm data.txt
STAT
UNLINK
intro fs
fs ops
perf
fun
end

22. Are There Only Six File Operations?
Local File Operations
Are There Only Six File Operations?
$ echo “test” > data.txt
READ
STAT
OPEN
WRITE
CLOSE
WRITE
OPEN
counts
108
Filesystem Calls
$ cat data.txt
STAT
STAT
OPEN
READ
CLOSE
UNLINK
CLOSE
$ rm data.txt
…more…
STAT
UNLINK
intro fs
fs ops
perf
fun
end

23. Relationship Between Filesystem Operations And Block IO Operations
Local File Operations
Relationship Between Filesystem Operations And Block IO Operations
$ echo “test” > data.txt
READ
STAT
OPEN
WRITE
CLOSE
WRITE
Block IO
Read
OPEN
$ cat data.txt
STAT
Block IO
Write
STAT
OPEN
READ
CLOSE
UNLINK
CLOSE
$ rm data.txt
…more…
STAT
UNLINK
intro fs
fs ops
perf
fun
end

24. Create A New File Containing Just Three Spaces
NAS File Operations
Create A New File Containing Just Three Spaces
intro fs
fs ops
perf
fun
end

25. Create A New File Containing Just Three Spaces
NAS File Operations
Create A New File Containing Just Three Spaces
intro fs
fs ops
perf
fun
end

26. THERE ARE MORE OPS THAN JUST
In The File World…
THERE ARE MORE OPS THAN JUST
READS and writes
intro fs
fs ops
perf
fun
end

27. Agenda Introduction Filesystem Layout Fundamentals Intro
What are Filesystem operations?
Performance in a File Oriented World
Intro
Graph fun with performance
Filesystem
Summary
FS Ops
File Perf
FUN
END

28. Performance In a File Oriented World is Affected By …
Caching
Performance Differences Between Operations
Data vs Metadata Ratios
intro fs
fs ops
perf
fun
end

29. Performance Characteristics Of File Operations
create / write
read
unlink / delete ls
intro fs
fs ops
perf
fun
end

30. Performance Characteristics Of File Operations
create / write
read
unlink / delete ls
intro fs
fs ops
perf
fun
end

31. Performance Characteristics Of File Operations
create / write
read
unlink / delete ls
intro fs
fs ops
perf
fun
end

32. Performance Characteristics Of File Operations
create / write
read
unlink / delete ls
intro fs
fs ops
perf
fun
end

33. Performance In a File Oriented World
Be Careful … Operation Performance Will Almost Always Be Affected By the SUT
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

34. Performance In a File Oriented World is Affected By …
Caching
Performance Differences Between Operations
Data vs Metadata Ratios
intro fs
fs ops
perf
fun
end

35. Performance In a File Oriented World
Does The Number Of Files Make A Difference?
create
81x
16,005 sec
197 sec
8kib
4.1gib MD MD
read
17x
8kib MD
1,707 sec
100 sec
8kib MD
540,000 total
unlink
4,426x
729 sec
sec
8kib MD
8kib MD ls
17,142x
12 sec
sec
8kib MD
stat
5,000x
1 sec
sec
4.1 GiB
intro fs
fs ops
perf
fun
end

36. Performance In a File Oriented World is Affected By …
Caching
Performance Differences Between Operations
Data vs Metadata Ratios
intro fs
fs ops
perf
fun
end

37. Performance in a File Oriented World
uncached ‘ls’
cached ‘ls’
“Uncached ‘ls’” had a USB unmount just prior to the ‘ls’ command execution
intro fs
fs ops
perf
fun
end

38. Performance In a File Oriented World is Affected By …
Caching
Performance Differences Between Operations
Data vs Metadata Ratios
intro fs
fs ops
perf
fun
end

39. Agenda Introduction Filesystem Layout Fundamentals Intro
What are Filesystem operations?
Performance in a File Oriented World
Intro
Graph Fun with Performance
Filesystem
Summary
FS Ops
File Perf
FUN
END

40. Which Is Better? intro fs fs ops perf fun end
I have two results here, which is better?
Uh…what the heck ARE these?
What are we charting?
What are my axes?
(These are the WRITE op/s for SWBUILD and DATABASE
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

41. Which Is Better? intro fs fs ops perf fun end
Ok … this is better, but still doesn’t answer all the questions. We learned from our first webcast that “context makes metrics matter.” And, we put a pretty good focus on latency.
So … what do these look like if you look at BOTH operations / second and data transfer rates, AND you chart using a y-axis of latency?
These are the total KBps for SWBUILD (result1) and DB (result2)
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

42. Which Is Better? intro fs fs ops perf fun end
Ok … this is better, but still doesn’t answer all the questions. We learned from our first webcast that “context makes metrics matter.” And, we put a pretty good focus on latency.
So … what do these look like if you look at BOTH operations / second and data transfer rates, AND you chart using a y-axis of latency?
These are the total KBps for SWBUILD (result1) and DB (result2)
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

43. Which Is Better? intro fs fs ops perf fun end
We’re getting some where … but is this really just a re-cast of the first webcast?
Actually … I notice you keep saying “Total KiB/s” … why is that?
Could you break it out perhaps?
These are the total KBps for SWBUILD (result1) and DB (result2)
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

44. Which Is Better? intro fs fs ops perf fun end
We’re getting some where … but is this really just a re-cast of the first webcast?
Actually … I notice you keep saying “Total KiB/s” … why is that?
Could you break it out perhaps?
These are the total KBps for SWBUILD (result1) and DB (result2)
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

45. Which Is Better? intro fs fs ops perf fun end
We’re getting some where … but is this really just a re-cast of the first webcast?
Actually … I notice you keep saying “Total KiB/s” … why is that?
Could you break it out perhaps?
These are the total KBps for SWBUILD (result1) and DB (result2)
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

46. Which Is Better? intro fs fs ops perf fun end
We’re getting some where … but is this really just a re-cast of the first webcast?
Actually … I notice you keep saying “Total KiB/s” … why is that?
Could you break it out perhaps?
These are the total KBps for SWBUILD (result1) and DB (result2)
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

47. Which Is Better? intro fs fs ops perf fun end
We’re getting some where … but is this really just a re-cast of the first webcast?
Actually … I notice you keep saying “Total KiB/s” … why is that?
Could you break it out perhaps?
These are the total KBps for SWBUILD (result1) and DB (result2)
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

48. Which Is Better? Database Workload SOFTWARE BUILD Workload intro fs
We’re getting some where … but is this really just a re-cast of the first webcast?
Actually … I notice you keep saying “Total KiB/s” … why is that?
Could you break it out perhaps?
These are the total KBps for SWBUILD (result1) and DB (result2)
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

49. Workload is yours The only that matters Which Is Better? Database
SOFTWARE BUILD
Workload
We’re getting some where … but is this really just a re-cast of the first webcast?
Actually … I notice you keep saying “Total KiB/s” … why is that?
Could you break it out perhaps?
These are the total KBps for SWBUILD (result1) and DB (result2)
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

50. Agenda Introduction Filesystem Layout Fundamentals Intro
What are Filesystem operations?
Performance in a File Oriented World
Intro
Graph fun with performance
Filesystem
Summary
FS Ops
File Perf
FUN
END

51. Storage Performance Benchmarking
Today
July 30,
2015
OCT 20,
2015
March 8,
2016
ONE LEFT!
Workload
definitions
Metrics and
terminology
Solution
Under Test
block
Components
File
Components
intro fs
fs ops
perf
fun
end

52. After This Webcast
A PDF and a PPT of the slides for this and all previous parts of this Webcast series will be posted to the SNIA Ethernet Storage Forum (ESF) website and available on-demand
PPT and PDF:
Presentation Recording:
A full Q&A from this webcast, including answers to questions we couldn't get to today, will be posted to the SNIA-ESF blog
Follow us @DrJMetz
Final Webcast – First Half of 2017
“Storage Performance Benchmarking: Part 5”

53. THANK YOU!

54. Client Caching Affects Operation Mixes
Lookup calls increase from 4% to 42%
Getattr calls decrease from 65% to 14%
Increasing # of Files, Active Dataset and Workload Intensity
intro fs
fs ops
perf
fun
end

55. Operation Mixes For Different Applications
Metadata IO
Data IO
PROPOSED
intro fs
fs ops
perf
fun
end
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only