1 Ashish Gupta Bin Lin Peter Dinda Department of Computer Science Northwestern University

2 Overview Our Goal –Introduce resource borrowing and its motivation –Current model and further possibilities –What we want to study ? Methodology –Actually try it out ! –Design of a system which emulates resource borrowing and captures feedback Design of our application –What it does ? –Extensive System Monitoring –Resource Exercisers we have For technical details refer to paper and tech report System Design – the client server system, creating and analyzing testcases The controlled study –What the user was asked to do, user details etc., set of testcases ( ramp and step) Results –Aggregated –Context –Person –Frog in the pot ? –Hardware ? Internet Wide Study Conclusions and Results

3 Outline Motivation behind Resource Borrowing Measuring user comfort System Design for Measuring User Comfort Controlled User Study Its Results Conclusions

4 New Models of Distributed Computing Break Screen Saver time Working 9 am 11 am 1 pm3 pm4 pm5 pm 9 am

5 Observation and ideas Idea: Why not borrow the unused resources ? Unused Resources Problem: Performance Slowdown

6 Effect of performance slowdown

7

8 The question we want to answer Depends on Other Factors as well… What level of resource borrowing leads to user discomfort for significant fraction of users ?

9 CPURAM Disk

10 ContextPersonHardware Others ?

11 Power UsersTypical UsersBeginners

12

13 Outline Motivation behind Resource Borrowing Measuring user comfort System Design for Measuring User Comfort Controlled Study Its Results Conclusions

14 How can we find this out ? Actually test it with user feedback A system for emulating resource borrowing and measuring user discomfort using direct feedback

15 How can we find this out ? 1 Measure existing contention

16 How can we find this out ? Emulate Resource Borrowing Real Contention Borrowed Contention Final Contention 2 % CPU Usage vs. Time

17 How can we find this out ? Allow User Feedback 3 F11

18 How can we find this out ? Find MAX Level of tolerable contention Borrowed Contention Final Contention 4 MAX % CPU Usage vs. Time

19 How is it useful ? Comfort = f (Contention) To find a quantitative relationship between resource borrowing and user comfort

20 How is it useful ? Enhances the degree of workstation sharing

21 Interactivity Aware Scheduling Interactive More intelligent scheduling of interactive tasks possible via user feedback and our results

22 Outline Motivation behind Resource Borrowing Measuring user comfort System Design for Measuring User Comfort Controlled Study Its Results Conclusions

23 Testcase Transfer Results Transfer Server Client Resource borrowing using testcase profile Can do large scale user studies and maintain testcase control over wide-area

24 The client design

25 Testcases Each testcase stores details of resource borrowing Testcase consists of exercise functions for each resource Exercise function is a vector of contention values Example: CPU contention: [0,0.5,1.0,1.5,2.0] at 1 Hz

26 Exercise Functions Time Contention Exercise Function 1 Exercise Function 2 Exercise Function 3 Exercise Function 4 Has all sort of strange exercise functions Power-law, exp, pareto Plot our real testcases

27 Flowchart of Testcase Execution No Testcase running Resource Borrowing Phase Begins (Testcase Runs) No user feedback User expressed discomfort Testcase Exhausts after some time Testcase Ends immediately User feedback

28 Resource Exercisers CPU Exerciser –Contention describes the expected extra number of threads in ready queue –Fractional resource borrowing using stochastic scheduling methods –Validated to contention level of 10 Disk Exerciser –Random seek and read/write in a large file (twice the memory) –Validated to contention level of 7 Memory Exerciser –Borrows a fraction of physical memory: f rom 0 to 1

29 Outline Motivation behind Resource Borrowing Measuring user comfort System Design for Measuring User Comfort Controlled Study Its Results Conclusions

30 The control study description 33 undergrad and grad students 1.5 hrs each Each user was assigned 4 popular tasks to do –MS Word –MS Powerpoint –MS Explorer  searching and saving information –Quake III

31 Machine Configuration

32 Testcases for the Control Study Three types of testcases: step, ramp and blank Ramp allows us to know the point of discomfort for each resource Step and Ramps: comparing the manner of borrowing resources Blank: To test the background level of discomfort

33 Frog in the Pot Hypothesis % CPU Contention vs. Time Step/Ramp Dynamics

34 Testcases for the Control Study Testcases different for each application –Some apps more intensive than others Example ： Word and Quake Requires Calibration

35 Outline Motivation behind Resource Borrowing Measuring user comfort System Design for Measuring User Comfort Controlled Study Its Results Conclusions

36 Resource borrowing vs User Discomfort CDFs for discomfort contention level Not all contentions cause discomfort: exhausted region

37 C 0.05 = 0.35 ( aggregated over all applications) Exhausted Region Discomfort Region Aggregated Results

38 Resource borrowing vs User Discomfort CDFs for discomfort contention level Not all contentions cause discomfort: exhausted region CDFs allow us to read %age of people discomforted for a given contention –Metric c 0.05 : At what contention do we discomfort only 5% of the people ?

39 Results overview Aggregated Results Dependence on Resource Dependence on Context Dependence on Users Dependence on Step/Ramp

40 Aggregated Results

41 C 0.05 = 0.35 ( aggregated over all applications) Exhausted Region Discomfort Region Aggregated Results

42 Can borrow a lot, discomforting only a small fraction\ C0.05=0.33 Aggregated Results Exhausted Region Discomfort Region

43 Can borrow a lot, discomforting only a small fraction C 0.05 = 1.11 Aggregated Results Exhausted Region Discomfort Region

44 Dependence on Resource

45 An overview of factors Dependence on Resource User sensitivity by task and resource (Low, Medium, High) Users more tolerant for disk and memory borrowing, than CPU Results Qualitative, but the testcase range is wide

46 CPU Dependence on Resource

47 Memory Dependence on Resource

48 Disk Dependence on Resource

49 Dependence on Context

50 Comparison of CPU discomfort Word IE Powerpoint Quake Dependence on Context

51 Dependence On Context - CPU Word Powerpoint IE Quake Dependence on Context

52 Dependence on Context - Memory Word Powerpoint IE Quake Dependence on Context

53 Dependence on Context - Disk Word Powerpoint IE Quake Dependence on Context

54 Fraction of users discomforted in Disk Word Powerpoint IE Quake Dependence on Context

55 Internet Wide-Area Study No controls here, hope to get more general and complete data Users can download a client from Clean installation/un-installation and privacy options Join us !

56 Outline Motivation behind Resource Borrowing Measuring user comfort System Design for Measuring User Comfort Controlled Study Its Results Conclusions

57 Conclusion Comfort = f(contention) Disk and memory can be borrowed more aggressively, CPU less so A throttle for resource borrowing  very useful CDFs can be used to set the throttle

58 Conclusion Comfort = f(contention) Knowing the user context helps Using Direct user feedback can be useful

59 Current and Future Work User feedback for interactive VM Scheduling Real time Scheduling to give exact resources to VMs Minimal resources required to keep the user happy

60 Acknowledgements Special Thanks to: Don Norman Andrew Ortony Ben Watson Jack Tumblin Leslie Lamport and All our beta testers Study participants

61 Conclusion Comfort = f(contention) Disk and Memory surprisingly easy to borrow, CPU borrowing also liberal Set of empirical CDFs provided Depends of other factors: Resource, Context and Users

62 Dependence on User

63 Dependence on User Power Typical Beginner Dependence on User

64 Step/Ramp Dynamics

65 Frog in the Pot Hypothesis % CPU Contention vs. Time Step/Ramp Dynamics

66 An example with PPT % CPU Contention vs. Time 0.98 Step/Ramp Dynamics

67 The placebo effect Users express discomfort even when no testcase is running Can never guarantee that the user will never express feedback

68 Memory Exerciser Contention indicates fraction of total memory allocated in addition to occupied memory Pages are touched frequently to keep them in physical memory 0MB 1024 MB Occupied Mem. 0.5 Memory Contention Clipping Point

69

70 Related Work User Frustration in HCI Computer Response to User Frustration [K99] Sensing and measurement of frustration with computers [R01] Impact of latency of user utility Behavioral aspects of text editors [EN81] Psychological Upper and lower limits of system response time and user’s preference on skill level [K97]

71 Virtual Machine Scheduling Virtuoso: Distributed Computing with Virtual Machines User pays for the resources he uses Need to know minimal resources needed to keep an interactive VM user happy

72 Questions and Answers Other types of irritation Bursty natures of Desktop Computing, bursts indicate sudden action-response tasks Large number of factors for irritation Since there is usually a power law for process run times, why do we need this model ? (CPU is idle most of the time) Why can’t we just run the bg app at a lower priority ? It may benefit much more than just running at a fixed contention. Memory exerciser: What does contention mean ? How do you find out free physical memory ? How do you avoid thrashing ? Why do you avoid thrashing ? How does the CPU exerciser work ? Why do you disable buffering in the Disk exerciser ? The aggregated graphs are not useful. Memory usage may be much more dynamic then in your tests due to user multitasking etc. So this may not represent reality ? –Our study is a start in this direction and we realize that there are limitations to the data we have collected. But we expect to collect and analyze more results from the Internet Wide area study and generalize our results to more real environments. There are no controls in the internet wide area study. The control study was meant to specifically answer the questions we have posed, like the affect of other factors in user discomfort and resource borrowing.

73 Lower priority ? –Disk and memory ? –There is a limit (max) possible amount of cycles stolen. Our goal is to s=estimate that. Seperately, there are many possible sched mech that would get closer to that max. Lower priority is an example. Linger longer is an example. Screen saver is an example. –Quake example –Don’t argue for/against a particular scheduling scheme –It is a step in the right direction… (for results)

74 Testcase Transfer Results Transfer Server Client Resource borrowing using testcase profile

75 The client-server system Data Exchange Hotsync –Results transfer –Testcase update Testcase Tools

76

77

78 An irritating Interface

79 Interactive Computation Bursty Nature of Interactive Computation

80

81

82 Understanding Testcases CPU ramp 1 ramp 2 ramp 3 step 1 step 2 ….. MEM ramp 1 ramp 2 ramp 3 step 1 step 2 ….. Ramp Step

83 Combination of Resources CPU Usage Memory Usage Hard Disk activity Network Traffic Comfort Zone surface

84 % CPU Contention vs. Time

85