1. Distillation of Performance- Related Characteristics kurmasz@cc.gatech.edu

2. 29 January 20022 Introduction zWant synthetic workload to maintain certain realistic properties or attributes yWant representative behavior (performance) zResearch Question: yHow do we identify needed attributes? zWe have a method...

3. 29 January 20023 Goal CDF of Response Time will have performance similar to original. (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload Given a workload and storage system, automatically find a list of attributes , so Attribute List  SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... synthetic workloads based on values for 

4. 29 January 20024 Why? zPredicting performance of complex disk arrays is extremely difficult. yMany unknown interactions to account for. zList of attributes much easier to analyze than large, bulky workload trace. zList of attributes tells us: yWhich patterns in a workload affect performance yHow those patterns affect performance zPossible uses of attribute lists: yOne possible basis of “similarity” for workloads yStarting point for performance prediction model

5. 5 Challenge zAttribute List may be different for every workload/storage system pair zRequire general method of finding list of attributes ynot just one workload’s attribute-values zUseful method will require little human attention CDF of Response Time (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload Attribute List  SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131...

6. 29 January 20026 Road Map zIntroduction yGoal yChallenges zDescription of Method zImplementation of Method zCase Analysis zLocality Attributes zConclusion

7. 29 January 20027 Basic Idea zBasic Idea: Add attributes until performance of original and synthetic workloads is similar. (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload Attribute List  SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131...

8. 29 January 20028 Mean Arrival Time Arrival Time Dist. Hurst Parameter Mean Request Size Request Size Dist. Request Size Attrib 3 Request Size Attrib 4COV of Arrival Time Dist. of LocationsRead/Write ratio Mean run length Markov Read/Write Jump DistanceR/W Attrib. #3 Proximity MungeR/W Attrib #4 Mean Read Size D. of (R,W) Locations Read Rqst. Size Dist.Mean R,W run length Mean (R, W) SizesR/W Jump Distance (R, W) Size Dists.R/WProximity Munge Mean Arrival Time Arrival Time Dist. Hurst Parameter Mean Request Size Request Size Dist. Request Size Attrib 3 Request Size Attrib 4COV of Arrival Time Dist. of LocationsRead/Write ratio Mean run length Markov Read/Write Jump DistanceR/W Attrib. #3 Proximity MungeR/W Attrib #4 Mean Read Size D. of (R,W) Locations Read Rqst. Size Dist.Mean R,W run length Mean (R, W) SizesR/W Jump Distance (R, W) Size Dists.R/WProximity Munge Choosing Attribute Wisely zProblem: yNot all attributes useful yCan’t test all attributes zSolution: yGroup attributes yEvaluate entire groups at once Attributes zHow are they grouped? zHow are they evaluated?

9. 29 January 20029 Iterative Process Choose Attribute Chose Attribute Group (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload  List of Attributes Evaluate  Are we done yet? Does  contain enough info? Which attribute group contains the attribute we should add? Which attribute should we add? Evaluate  Are we done yet? Does  contain enough info?

10. 10 Evaluation CDF of Response Time Quantify difference between response time distributions (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload Take original workload Obtain attribute-values specified by  Attribute List  SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Generate Synthetic workload using 

11. 29 January 200211 Quantify Difference zWe use root-mean-square metric ySquare root of sum of squares of horizontal differences. yUsed by Ruemmler and Wilkes, Ganger, etc. zAny reasonable metric will work. zIs RMS the best metric? yDifferences at all scales weighted equally yTherefore, differences in cache behavior are emphasized less than differences in seek time yCan be good or bad zIs there a non-arbitrary stopping point?

12. 29 January 200212 Iterative Process Choose Attribute Evaluate  Chose Attribute Group (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload  List of Attributes Are we done yet? Does  contain enough info? Which attribute group contains the attribute we should add? Which attribute should we add?

13. 29 January 200213 zWorkload is series of requests y(Read/Write Type, Size, Location, Interarrival Time) zAttributes measure one or more parameters yMean Request Size Request Size yDistribution of LocationLocation yBurstinessInterarrival Time yRequest Size yRead/Write zAttributes grouped by parameter(s) measured yLocation = {mean location, distribution of location, locality, mean jump distance, mean run length,...} How Attributes Grouped Distribution of Read Size

14. 29 January 200214 “Additive” Method zAdd “every” attribute in group at once and observe change in performance. zAmount of change in performance estimator of most effective attribute “All” (Size, R/W) “All” Request Size “All” Location

15. 29 January 200215 “All” Location attribute “All” (Location, Request Size) attribute The “All” Attribute zThe list of values for some parameter contains every attribute in that group zAttributes in that group will have same value for both original and synthetic workload z List represents “perfect knowledge” of group

16. 29 January 200216 “Subtractive” Method zRemove “All” attribute from original workload. yReplace an observed list of request parameters with randomly generated values. Remove “all” location attributes by making all locations random

17. 29 January 200217 Challenges zSubtleties to applying methods yMust avoid mutually exclusive attributes zChange in performance can be ambiguous yResulting workload can be faster or slower yEffects of attributes in same group can cancel each other. (Have net effect of zero.) zNot looking for strictly decreasing RMS “Faster” Workload “Slower” Workload

18. 29 January 200218 Current Focus zThus far, solutions have been apparent zHowever, we want to reduce human involvement zCurrent Focus: yFind general description of ambiguities yDevelop algorithms to recognize and address them

19. 29 January 200219 Iterative Process Choose Attribute Evaluate  Chose Attribute Group (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload  List of Attributes Are we done yet? Does  contain enough info? Which attribute group contains the attribute we should add? Which attribute should we add?

20. 29 January 200220 Add from chosen group zThis is most difficult part. yOnly a few useful attributes known, so we must develop most attributes from scratch. ySuggestions for attributes? ySuggestions for related work? zThis should get easier as technique used and “attribute library” grows. yFuture Work: We will eventually need an intelligent method of searching library.

21. 29 January 200221 Road Map zIntroduction yGoal yChallenges zMethod zCurrent Status / Implementation zCase Analysis zLocality zConclusion

22. 29 January 200222 New Programs z“Distiller” yPerl program that prepares Rome input and parses resulting output of SSP tools. xRubicon, Buttress, Rome, srtLite, etc. z“GenerateSRT” yC++ program that generates SRT trace given set of attribute-values in Rome format. yVaguely like Pylon without capacity to run workload. zBoth programs need better names!

23. 29 January 200223 Functionality zInput: yWorkload description (SRT file, list of LUNS) yAttribute list (in Rome format) zMake Rubicon configuration zRun Rubicon zMake attribute-value list (Rome format) zRun Generate SRT zRun Buttress zProcess miio file (mi2srt) zCollect Performance (CDF in Rome format)

24. 29 January 200224 Future Functionality zCurrently the user must: yApply additive or subtractive method and generate a new attribute list. yLook at resulting CDFs and choose attribute group by hand. yChoose specific attribute. xThis could easily done by brute force. yDevelop new attributes when needed. zAt these points Distiller application asks user for answer. (Actually, sends me e-mail.)

25. 29 January 200225 Incremental Implementation zDistiller application designed to allow incremental implementation. yWe can implement code to recognize handle specific cases, and leave the rest to the user. zPlan: Implement code to recognize additional cases until distiller application runs with almost no human intervention. yWill probably always require a occasional help.

26. 29 January 200226 Road Map zIntroduction yGoal yChallenges zMethod zCurrent Status / Implementation zCase Analysis zLocality zConclusion

27. 29 January 200227 Experimental Environment zWorkload: Trace of Open Mail ye-mail application for 4,500 users y1400 active users during trace collection yMean request rate: 75.52 I/Os per second yMean request size: 7115 bytes yMean throughput: 524.5KB per second zStorage System: Optimus disk array yThirty 18GB disks (.5 TB total) yTwo disk controllers each with Fibre Channel port yMax I/Os per second: about 100 y256 MB Write-back cache backed by non-volatile RAM xThus, writes are “free”

28. 29 January 200228 Starting Point zOriginal attribute list (  ) is simply four distributions, one for each parameter. zSimplest possible explicit attribute list yAnything simpler must make implicit assumptions about distribution for random numbers. yThis would cause mis-information, not missing information.

29. 29 January 200229 RMS:.9477 RMS/Mean:.1877 Performance of real workload Synthetic workload Values chosen independently at random

30. 29 January 200230 First Iteration zApply additive method to 4 single- parameter attribute groups. zThis allows us to examine intra-parameter relationships. ye.g. “What would performance be if locality was correctly generated?” zNo inter-parameter relationships present in synthetic workload. ye.g. only relationships within location created.

31. 29 January 200231 * Big change for Location * Small changes for everything else * Goal: Choose locations so they exhibit locality like this

32. 29 January 200232 RMS:.7739 RMS/Mean:.3143

33. 29 January 200233 Second Iteration zUse subtractive method to check for inter- parameter relationships. zNow that all intra-parameter relationships are included in attribute-list, replacing perfect attribute with current attribute should only result in a change in performance if there are inter-parameter relationships

34. 29 January 200234 Large change for Location and R/W Type Small change for Request Size and Arrival Time

35. 29 January 200235 RMS/Mean : Original:.1877 Current:.0918

36. 29 January 200236 Road Map zIntroduction yGoal yChallenges zMethod zCurrent Status / Implementation zResults zLocality zConclusion

37. 29 January 200237 Failed Attempts zJump Distance yChoose jump distance instead of location. yDifficult to choose intelligently. zJump Distance Markov yMarkov model for zero, and non-zero jumps yAlso tried zero, forward, and backward jumps zProximity yConsider distance between 25 most recent requests, not just most recent request.

38. 29 January 200238 Common Problem: Footprint zTwo parts to choosing locations: yFootprint--- Which sectors of disk accessed yLocality --- In which order sectors accessed zLocality schemes will fail on FC-60 unless footprint is correct yFC-60 has a huge cache. yCache hits and cache misses have order of magnitude difference in response time ySmall footprint results in faster workload yLarge footprint results in slower workload

39. 29 January 200239 “Loose Histogram” Problem zHistogram of location had fewer bins than number of unique locations zRNG chooses bin according to distribution. Specific value chosen uniformly from bin zIn reality, request locations are not uniformly distributed through bins Observed Location Distribution Synthetic Location Distribution

40. 29 January 200240 “Tight” Histogram Problem zGive each unique location value its own bin zMakes footprint too small yNot every bin gets chosen x(when number of bins  number of requests) yYahtzee “Large Straight” problem zAlso too large to be useful attribute-value Observed Location Distribution Synthetic Location Distribution

41. 29 January 200241 Choose Footprint Explicitly zExplicitly choose set of unique location values for correct footprint zHistogram must be tight zHowever; distribution is “bursty” zThis makes pre-determined bin-widths inefficient. yMany empty fixed-width bins y“log” histogram makes bins too large at end

42. 29 January 200242 Location Percentile zWe fix the number of unique location values in each bin. ySpecify starting and ending sector for each bin. zAlternatively, we fix the number of requests in each bin. yWhen we do this, heavily used unique location values are placed in smaller (more precise) bins

43. 29 January 200243 Evaluate Footprint Generator zIf we simply permute locations, footprint is exactly correct, but no locality yToo large to be a practical attribute-value yUseful to evaluate other attributes describing the footprint

44. 29 January 200244 Main Ideas zNew method of automatically finding performance-related attributes: yMeasure completeness of list by comparing performance of synthetic workloads yUseful method of grouping attributes yEffective method of evaluating entire groups of attributes yAvoid evaluation of useless attributes zkurmasz@cc.gatech.edu zwww.cc.gatech.edu/~kurmasz

45. END OF LONG TALK Remaining Slides are extra

46. 29 January 200246 Goal (v1) zGiven a workload,  automatically find the set of characteristics , such that  any synthetic workload based only on  zwill have performance similar to original. (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload Response Time  (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Synthetic Workload

47. 29 January 200247 zAdd attributes until performance is similar (unused) (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload Response Time  (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Synthetic Workload   

48. 29 January 200248 New Application Choose Attribute Evaluate  Chose Attribute Group (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload  List of Attributes Which attribute is best? Collect performance info Compute RMS of CDFs Prepare synthetic workloads Run synthetic workloads Compute

49. 29 January 200249 Goal (used for parts) CDF of Response Time will have performance similar to original. (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload Given a workload and storage system, automatically find a list of attributes , so Attribute List  SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... synthetic workloads based on values for 

50. 29 January 200250 CDF of Response Time (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131... Original Workload Attribute List  SyntheticWorkload (R,1024,120932,124) (W,8192,120834,126) (W,8192,120844,127) (R,2048,334321,131...