1. 1 Thesis Proposal Zachary Kurmas (v4.0– 24 April 03)

2. 2 Outline Motivation and discussion of problem Overview of of solution Contributions Proposal Future Work Timeline Details of solution (time permitting)

3. 3 Typical disk array Controller A Cache Controller B Cache SCSI Buses Fibre Channel Hosts

4. 4 Motivation Potential storage system designs and automated configuration algorithms must be evaluated with respect to some set of workloads. Ideally, these workloads are actual production workloads. This is usually impossible Two alternatives Replay traces of production workloads Construct and use synthetic workload

5. 5 Problem The currently available set of workload traces and synthetic workloads are not sufficient Can’t get enough of right traces Companies don’t like to give them out No traces of future workloads Quality of synthetic workloads too low High-quality synthetic workload must share certain key properties with production workload These properties are currently found by trial-and-error and domain expertise

6. 6 Solution Improve quality of synthetic I/O workloads Automatically determine what properties a synthetic workload must share with the production workload on which it is based (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Production Workload List of Properties SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... CDF of Response Time

7. 7 Contributions Prototype system to automatically determine what properties a synthetic workload must share with the production workload on which it is based Library of possible properties and corresponding generation techniques Algorithm for searching through library Examination of tradeoffs between size and complexity of properties and quality of synthetic workloads Evaluation of whether improved synthetic workloads enable us to make better design decisions Exploration of workload scaling using identified properties

8. 8 Workload Traces Advantage: Perfectly representative of production workload Can be used in place of production workload Disadvantages Difficult to collect and maintain Severe lack of block-level traces for researchers Large (10s MB/hour/LU) Production workload must exist Inflexible --- no “what-if” scenarios

9. 9 Workload Trace Workload is a series of I/O requests I/O request has four parameters Read/Write type Request Size Location Arrival Time Used to “replay” workload Workload Trace (R,1024,42912,10) (W,8192,12493,12) (W,2048,20938,15) (R,2048,43943,2) (W8192,98238,11) (W8192,76232,23) Read Write Request Size Location Arrival Timeq

10. 10 Representative Results of Evaluation (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Workload A Workload B (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Workload B represents Workload A if the results of the evaluation produces the same results for both workloads q

11. 11 Synthetic Workload List of requests is generated (more or less) randomly to have certain properties. For example 67% of I/Os reads Mean request size 8KB 23% I/Os sequential Just like “x” but twice as fast. (R,1024,42912,10) (W,8192,12493,12) (W,2048,20938,15) (R,2048,43943,2) (W8192,98238,11) (W8192,76232,23) Read Write Request Size Location Arrival Time Synthetic Workload

12. 12 Synthetic Workloads Advantages List of properties often compact Properties do not compromise privacy Should encourage companies to make more traces available Properties provide “knobs” for what-if scenarios Disadvantage Properties must be chosen very carefully Using too few properties (or the wrong properties) makes synthetic workload not representative Using too many properties makes representation not compact

13. 13 $64,000 Question What properties must a synthetic workload share with the production workload in order to be representative? In other words, what is the “essence” of the workload? (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Production Workload List of Properties SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... CDF of Response Time

14. 14 $64,000 Question What attribute-values must a synthetic workload share with the production workload in order to be representative? In other words, what is the “essence” of the workload? (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Production Workload List of Properties SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... CDF of Response Time

15. 15 $64,000 Question What attribute-values must a synthetic workload share with the production workload in order to be representative? Currently answered on a case-by-case basis using trial-and-error and domain expertise (if answered at all) My goal: To automate this process and remove need for human expertise

16. 16 Review of problem Not enough input for evaluations of storage systems Too few workload traces Traces not always right answer Synthetic workloads are not practical Don’t know precisely what makes synthetic workloads representative Trial-and-error too cumbersome Can’t maintain every conceivable attribute-value

17. 17 Outline Discussion of problem Overview of of solution Contributions Proposal Future Work Timeline Details of solution (time permitting)

18. 18 Attributes / Attribute-Values An attribute is the name or description of a property Read percentage Mean interarrival time An attribute-value is the actual value of the measurement (i.e., the actual property.) Read percentage of 67 Mean interarrival time of.8ms

19. 19 Requirements of attributes Attribute-values are properties of only the workload. Response time not an attribute because attribute-value depends on both workload and disk array Attributes must be quantifiable “Locality” and “burstiness” are qualitative concepts. “runCount” and “Hurst parameter” are attributes

20. 20 The Distiller Automate process of choosing necessary attribute-values Input: workload trace and large set of attributes Output: set of attributes that identifies those attribute-values that synthetic workload must share with target Helps identify type of any necessary attribute missing from library (if no known set of attribute-values leads to a representative synthetic workload)

21. 21 Basic Idea Begin with simple attribute-values (distributions of I/O request parameters) Iteratively add attribute-values until evaluation of original and synthetic workloads is similar (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Production Workload Attribute-value List SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... CDF of Response Time

22. 22 Why empirical distributions? The Distiller’s initial set of attributes is always the empirically observed set of I/o parameter Distributions. Random numbers always drawn from some distribution Drawing from an implicit distribution would product a synthetic workload with incorrect as opposed to missing attribute-values q

23. 23 Challenges: Which attribute to add Each iteration takes many minutes; therefore, we must limit the number of iterations Addition of necessary attribute-values does not always result immediately in improvement Fewer attributes better Smaller compact representation Less complex generation techniques Generation techniques for some attributes can interfere with each other E.g., distribution of location and jump distance

24. 24 Improvement not immediate Current synthetic workload: Mean 10ms Target workload: Mean 7ms Improvement #1: Increase mean by 5ms Improvement #2: Decrease mean by 8ms Neither #1 nor #2 will result in a “better” synthetic workload CDF of Response Time Faster Wkld Slower Wkld q

25. 25 Generation conflicts Easy to reproduce distribution of location values Not as easy to reproduce of jump distances What happens if loc = 1 and jump distance = -10 Don’t know how to maintain both distributions What happens if loc = 100 and jump distance = 6? Excluding values from jump distance distribution skews final observed distribution Location Jump distance 10011092931001041029091 9-210 0063 -3 q

26. 26 Outline Discussion of problem Overview of of solution Contributions The Distiller itself An analysis of the key attributes for many different workloads A an analysis of the potential uses of synthetic workloads Proposal Future Work Timeline Details of solution (time permitting)

27. 27 The Distiller itself Distiller makes generating representative synthetic workloads practical Encourage companies to make evaluation workloads available More accurate / relevant research results Basis for “what-if” evaluations Future estimations Stability estimates Improved relevance of old evaluation workloads (possibly) Distiller provides library of attributes and corresponding generation techniques

28. 28 Analysis of key attributes for many workloads Attribute-values that lead to representative synthetic workloads describe what makes the workload behave like or unlike other workloads The “essence” of the workload Possible to study essence of many different (workload, storage system) pairs and look for interesting trends or patterns

29. 29 Potential benefits of analyses Help us learn about how workloads and storage systems interact Attribute-values contain all info necessary to predict behavior Focus researchers’ attention on concentrated information Help development of analytical models Identify potential areas of improvement q

30. 30 Outline Discussion of problem Overview of of solution Contributions Proposal Evaluate the correctness of the Distiller Examine the attributes chosen for different workload/storage system pairs Show that the resulting synthetic workloads are useful Future Work Timeline Details of solution (time permitting)

31. 31 Evaluate correctness of Distiller Show that the Distiller works for: One definition of “representative”: response time distribution Up to three storage systems: FC-60, FC-30, and JBOD (Just a Bunch Of Disks) Several artificial workloads Five production workloads: Open Mail, TPC-C, TPC-H, file system trace Stopping Condition: Distiller can correctly identify key attributes for artificial workloads

32. 32 Definition of “representative” Design decisions almost always based on performance. Thus, matching response time distributions should be a stronger condition than most design decisions Distribution of response time stronger condition than mean response time Many decisions decide between competing configurations. Showing applicability across storage system configurations is my next evaluation Workloads are considered representative when RMS difference between distributions of response time is sufficiently small Seconds Number of IOs representative Not representative

33. 33 Disk array properties FC-60 and FC-30 10s of disks arranged on several SCSI busses Fairly large cache Write back cache – writes are “free” Can do prefetching and other optimizations JBOD (Just a Bunch of Disks) Set of disks on single SCSI bus No fancy stuff (e.g., cache, “free” writes, prefetching, RAID) q

34. 34 Disk array availability Can’t guarantee results on all three disk arrays: Access to FC-60 may go away FC-30 may break. No repair budget Access to JBOD at HP may go away May or may not be able to build one here q

35. 35 Stopping conditions (1) Much randomness in system OS scheduling of trace replayer Rotational position of disk Network traffic / stray I/Os Difficult to put FC-60 in known state Multiple replays of same workload have RMS differences of 5% or more This is a lower bound for the quality of the Distiller’s answers q

36. 36 Stopping conditions (2) Distiller limited by set of attributes in library. Library may not be complete enough to handle all real workloads Instead we will show how Distiller’s hints can be used to improve library We limit ourselves to one improvement per real workload because developing new attributes is not main focus of this research. This is not an issue for artificial workloads generated based on attributes in library Using artificial workloads tests Distiller independently from its library q

37. 37 Wild success vs. minimal success Wild success Distiller easily handles all workloads, even real workloads Learn that synthetic workloads need not be as complicated as previously though Minimal success Distiller can’t even handle artificial workloads We learn that the disk array interactions that produce performance are much more complicated that previously believed My thesis becomes a study of surprising disk array behaviors that affect predictability q

38. 38 Outline Discussion of problem Overview of of solution Contributions Proposal Demonstrate that the Distiller works Examine the attributes chosen for different workload/storage system pairs Show that the resulting synthetic workloads are useful Future Work Timeline Details of solution (time permitting)

39. 39 Learn about attributes (1) Determine if attributes depend on the workload. My Guess: Yes. Locality attributes are probably different for write-only workload on FC-60 Determine if attributes depend on the storage system? Answer: They must. Storage system with constant 2min response time has no important attributes Better objective: Compare attributes chosen for similar storage systems / system configurations. How much overlap?

40. 40 Learn about attributes (2) Determine which set of attributes does best overall (for a given storage system configuration) average over all workloads best worst-case Can either of these be used in practice for all wklds? Attempt to find a single set of attributes that works for almost all workloads (e.g. take union of all chosen attributes) Examine complexity (e.g., number of attributes) of such a set

41. 41 Learn about attributes (3) Examine changes in attributes and attribute-values over time. Compare traces of a file system taken in 1992, 1996, 1999, and 2002. Attempt to develop scaling rules. Examine tradeoffs between accuracy and complexity. Attempt assign a “percent contribution” to each attribute and/or attribute group?

42. 42 Outline Discussion of problem Overview of of solution Contributions Proposal Demonstrate that the Distiller works Examine the attributes chosen for different workload/storage system pairs Show that the resulting synthetic workloads are useful Future Work Timeline Details of solution (time permitting)

43. 43 Apply to real life Show that synthetic workloads can be used to make design decisions Show that currently available traces not adequate Show usefulness of “knobs”

44. 44 Synthetic workloads useful Show that synthetic workloads can be used in place of real workloads to make simple design decision Cache size Prefetch length High-water mark of write-back cache Complex design decisions basis for entire Ph.D. theses. Can’t practically reproduce at Tech. Use Pantheon disk simulator to simulate effects of changing above parameters

45. 45 Synthetic workloads useful (2) Take production workload trace Simulate performance given different prefetch lengths. Choose best Take synthetic workload based on production workload Simulate performance given different prefetch lengths Compare best to best for production workload For cache size, find best performance/$ mark.

46. 46

47. 47 Show available traces inadequate Use Pantheon disk simulator to show that using the cello92 and cello02 traces to evaluate simple design decisions results in different answers. From this we infer that using cello92 traces to justify more complex design decisions also produces incorrect answer

48. 48 Turning “knobs” useful Show that turning “knobs” of compact representation better than ad-hoc modifications to workload traces Show that turning arrival time knob better than contracting interarrival times Show that turning request size knob better than ad-hoc doubling of request size and location values. Evaluate turning of knobs versus removing ½ of cello I/Os based on process ID

49. 49 Future Work Optimality Find “smallest” set of attributes per workload (e.g. set of attributes with smallest compact representation) Find smallest set of attributes per storage system (if possible) Use chosen attributes to develop analytical model of performance Formula for performance, not simulation

50. 50 Timeline April  June: Run Distiller on many workloads Submit results to MASCOTS June  July: Analyze changes over different workloads / storage systems. Submit results to CMG conference July  August Find best overall set of attributes. Find best worst-case September  October: Attempt to develop set of attributes that works for all workloads on a given storage system Submit results to SIGMETRICS and/or FAST November  December: Evaluate different what-if scenarios. February 2004: Defense January 2004  February 2004: write March 2004  April 2004: interview May 2004  July 2004: write August 2004: graduate

51. 51 Outline Discussion of problem Overview of of solution Contributions Proposal Future Work Timeline Details of solution (time permitting)

52. 52 Generating Synthetic Workload To generate synthetic workload, randomly choose value for each element in table Attribute-values put restrictions on values chosen Adding attribute-values reduces the difference between synthetic and production workloads (R,1024,42912,10) (W,8192,12493,12) (W,2048,20938,15) (R,2048,43943,2) (W8192,98238,11) (W8192,76232,23) Read Write Request Size Location Arrival Time

53. 53 Mean Arrival Time Arrival Time Dist. Hurst Parameter Mean Request Size Request Size Dist. Request Size Attrib 3 Request Size Attrib 4COV of Arrival Time Dist. of LocationsRead/Write ratio Mean run length Markov Read/Write Jump DistanceR/W Attrib. #3 Proximity MungeR/W Attrib #4 Mean Read Size D. of (R,W) Locations Read Rqst. Size Dist.Mean R,W run length Mean (R, W) SizesR/W Jump Distance (R, W) Size Dists.R/WProximity Munge Mean Arrival Time Arrival Time Dist. Hurst Parameter Mean Request Size Request Size Dist. Request Size Attrib 3 Request Size Attrib 4COV of Arrival Time Dist. of LocationsRead/Write ratio Mean run length Markov Read/Write Jump DistanceR/W Attrib. #3 Proximity MungeR/W Attrib #4 Mean Read Size D. of (R,W) Locations Read Rqst. Size Dist.Mean R,W run length Mean (R, W) SizesR/W Jump Distance (R, W) Size Dists.R/WProximity Munge Choosing Attribute Wisely Challenge Not all attributes useful Some attributes partially redundant Can’t test all attributes My Solution Group attributes Evaluate whole groups at once Attributes

54. 54 Attribute Groups Attributes measure one or more parameters Mean Request Size Request Size Distribution of LocationLocation BurstinessInterarrival Time Request Size Read/Write Attributes grouped by parameter(s) measured Location = {mean location, distribution of location, locality, mean jump distance, mean run length,...} Arrival Time = {mean interarrival time, Markov model of interarrival time, Hurst parameter, etc. } Distribution of Read Size

55. 55 Attribute Groups Each group corresponds to each column or set of columns Operation Type Request Size {Arrival Time, Location} Measures patterns within column(s) (R,(R,1024,42912,10) (W,(W,8192,12493,12) (W,(W,2048,20938,15) (R,(R,2048,43943,2)2) (W,(W,8192,98238,11) (W,(W,8192,76232,23) Read Write Request Size Location Arrival Time Workload

56. 56 12 13 15 Do I need (more) attributes from the {Arrival Time} group? Idea #1: Add “best” attribute from {Arrival Time} and measure improvement Amount of improvement implies potential benefit R/W RSLocAT R/W RSLocAT Current Attributes Attributes for Test R, 1024, 10242 W, 2048, 11224 R, 1024, 10252 Current Current R, 1024, 10242 W, 2048, 11224 R, 1024, 10252 Current 12 14 15 Perfect

57. 57 Problem with idea #1 Errors involving other parameters can interfere Very random reads can overshadow moderate queuing effects R/W RSLocAT R/W RSLocAT Perfect Current Current Attributes Attributes for Test Current

58. 58 Idea #2 --- Idea #1 “backwards” Look at a synthetic workload in which everything except Arrival Time is “perfect”. Change in performance implies importance of group. R/WRSLoc AT Current Current Arrival Time Attributes Perfect Everything Perfect R/W RSLocAT Production Workload Perfect Workload Trace R/W RSLocAT Workload Trace

59. 59 Problem with idea #2 Workload on left missing not only {Arrival Time} Also missing {Arrival Time, Request Size}, {Arrival Time, Location} and {Arrival Time, Operation Type} Cause of any difference not clear R/W RSLoc AT R/W RSLocAT Current Operation Type Attributes Workload Trace Production Workload Production Workload Current

60. 60 Solution Remove {Arrival Time, Request Size}, {Arrival Time, Location} and {Arrival Time, Operation Type} from workload trace by “rotating” arrival times. Only difference between workloads is {Arrival Time} R/W RSLocAT Current R/W RSLocAT Current Operation Type Attributes “Rotated” Arrival Time Production Workload Production Workload Production Workload

61. 61 Process Add {Operation Type} attributes until two workloads below are representative Repeat for other attribute groups R/W RSLocAT Current R/W RSLocAT Current Operation Type Attributes “Rotated” Operation Types Production Workload Production Workload Production Workload

62. 62 Hints If Distiller is unable to find attributes for a particular group, it identifies the deficiency Helps people develop new attributes Attributes for multi-parameter groups must be compatible with single parameter groups {Operation Type, Location} attribute must maintain same properties as chosen {Operation Type} and {Location} parameters

63. 63 End Of Talk

64. 64 Problem: Lack of traces for researchers …. Papers use same … traces Traces used may or may not be representative of actual production workloads When traces not sufficient, really bad synthetic workloads used instead We don’t know how to easily produce representative synthetic workload Lack of synthetic workload generation ability suggests lack of understanding of disk array and storage system interactions

65. 65 Proposed Solution Improve our ability to generate synthetic workloads (Discuss previous work)

66. 66 Workload Characteristic Characteristic: A property of a workload (or workload trace) that can be measured. 27% reads Mean request size of 8KB Must be property of workload alone Response time not workload characteristics, but characteristics of both workload and storage system Must be concrete measurable property. “burstiness” and “locality” too vague. Also called “attribute-values”

67. 67 Attributes Attribute: The “name” of a characteristic AttributeCharacteristic eye colorblue eyes Read percentage27% reads mean request size mean size: 8KB Hence, characteristics also called “attribute-values”

68. 68 How the Distiller works Partition known attributes into groups All characteristics in each group contain similar information Choose a “complete” set of characteristics from each group. i.e. choose a set of characteristics that contains all the necessary information from the group

69. 69 $21,000 question What attribute-values must a synthetic workload should share with the production workload in order to be representative? Do the attributes depend on the workload? Do the attributes depend on the storage system? If so, how can we find them easily? If not, what are they?

70. 70 Trivial “solution” doesn’t work Trivial solution: Use many attribute-values Problems with trivial solution Many attribute-values contain irrelevant info. Many attribute-values contain duplicate info. High-level description too large and complex Negates advantages of synthetic workload Generating synthetic workload too difficult Obvious algorithms for generating attribute-values often interfere with each other.

71. 71 Challenges of Useful Solution Solution: Choose small set of “important” attribute-values That is, attribute-values that have the most impact on evaluation Challenges Estimating impact of single attribute-value on evaluation Finding small set of attribute-value with “disjoint” information

72. 72 Goal of Distiller CDF of Response Time will have evaluation similar to original. (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload Given a workload and storage system, automatically find a set of attributes, so Attribute List SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... synthetic workloads with the same values

73. 73 High-level approach Divide and Conquer Partition attributes into groups according to “type of information” Recall some attributes describe similar info. Find a set of attribute-values that contains all the information for a particular group (No, its not that simple …)

74. 74 Workload I/O request has four parameters Read/Write type Request Size Location Arrival Time shown in ms Workload series of I/O requests Trace can be viewed as a table with four columns (R,1024,42912,10) (W,8192,12493,12) (W,2048,20938,15) (R,2048,43943,2) (W8192,98238,11) (W8192,76232,23) Read Write Request Size Location Arrival Time

75. 75 “Engineering” Contributions Finding representative synthetic workloads becomes practical Basis for evaluations when traces are unavailable Basis for “what-if” evaluations Provides basis for workload similarity metric “Table-based” models Highlight what workload features a storage system handles best Help configure storage system for new workload

76. 76

77. 77 Apply to “real life” Attempt to generate a representative synthetic workload when no trace exists Choose workload trace and “hide” it Use lessons from previous slides to choose attributes based on similar workloads Compare synthetic workload to trace Compare “what-if” workload based on chosen attributes to ad-hoc “what-if” workload Play workload twice as fast “bootstrapping” Attempt to find attributes that determine whether to use Raid 1/0 or Raid 5

78. 78 Proposal: Apply to Real Life (2) Attempt to build “table-based” model of performance n-dimensional table Each axis represents one attribute fill element (w, x, y, …) with performance of workload with attribute-values w, x, y, … Given new workload compute attribute-values w, x, y, … Value in corresponding table element estimate of performance

79. 79 Problem Both alternatives have problems: Workload traces Companies don’t like to give them out Don’t always meet the researcher’s needs Synthetic workloads Difficult and tedious to generate correctly

80. 80 Overview Motivation: Storage system design studies and automated management systems require workloads to drive evaluations Problem: Neither traces of production workloads nor simple synthetic workloads are sufficient to drive experimental evaluation My solution: Improve the quality of synthetic storage workloads by automatically determining what properties synthetic workloads must share with the production workloads they model