1. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation Bin Lin Department of Electrical Engineering & Computer Science, Northwestern University binlin@cs.northwestern.edu Aaron Brown IBM T.J. Watson Research Center abbrown@us.ibm.com Towards an understanding of Decision Complexity in IT Configuration

2. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 2 Towards an understanding of Decision Complexity in IT Configuration Context: Quantifying IT Process Complexity  Technical problem –Identify metrics and develop methodology for quantifying the exposed operational complexity of IT processes  Importance –Complexity of systems management processes drives labor cost –Labor cost reductions are extremely important to services/outsourcing organizations and customers –A quantitative framework for complexity can guide process improvements to reduce labor cost –Opportunities for deploying autonomic computing in IT environment

3. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 3 Towards an understanding of Decision Complexity in IT Configuration Previous work  Initialize a model of configuration complexity and demonstrates its value for a change management system.  Metrics that indicate some configuration complexity, including execution complexity, parameter complexity, and memory complexity.  Example: complexity of J2EE provisioning  Process complexity: manual Execution 59 steps, 27 context switches Parameter 32 parameters used 61 times, 18 outside of source context Source score: 125 Memory (LIFO stack model) Size: max 8, avg 4.4 5 1 17 0 94 0 0 automated See: Brown, A.B., A. Keller, and J.L. Hellerstein. A Model of Configuration Complexity and Its Application to a Change Management System. Proceedings of the Ninth IFIP/IEEE International Symposium on Integrated Network Management (IM 2005), Nice, France, May 2005.

4. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 4 Towards an understanding of Decision Complexity in IT Configuration s Next Step: Decision Complexity  Previous metrics assume expert skill –Do not consider complexity arising from decision-making  Capturing complexity impact of decisions along a specific procedure’s path –Parameterized by skill level  Understanding the overall complexity across all possible procedures  Quantifying the tradeoff between flexibility and simplicity goal s vs. Procedure Design Space

5. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 5 Towards an understanding of Decision Complexity in IT Configuration Install/Config Procedure for J2EE App Install DB2 UDB + WAS ND Install Cloudscape + WAS Express Need Enterprise Clustering ? Y N... Decision Complexity (An initial model & methodology)  Factors that affect complexity –constraints e.g. compatibility between software products, capabilities of a machine consequences e.g. functionality, performance levels of guidance e.g. documentation, previous configuration experience  Manifestation –task time, user-perceived difficulty, error probability  A starting point to drive data collection (user study)  After we have the real-world data, refine the model

6. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 6 Towards an understanding of Decision Complexity in IT Configuration Model details: levels of guidance  Global information –E.g. documentation, design guide, deployment patterns  Short-term goal-oriented information –E.g. wizard-based prompts indicating the appropriate next step  Confounding information –E.g. alternate configuration instructions for a different platform than the target  Position information –E.g. feedback on the current state of the system and the effect of the previous action

7. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 7 Towards an understanding of Decision Complexity in IT Configuration Decision Complexity (challenge & solution)  Hard to conduct a full user study to validate the model (constraints, consequences, levels of guidance) using real IT processes  Sol: measuring decision complexity in a simplified domain: Route-planning –navigating a car from one point to another

8. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 8 Towards an understanding of Decision Complexity in IT Configuration Decision Complexity (user study design)  Web-based study –larger subject pool –accurate timing data –standardized information  Questionnaire to collect user background  Recording user interaction –time spent, each decision point –comparison b/w user path & optimal path –user ranking of the complexity for testcases

9. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 9 Towards an understanding of Decision Complexity in IT Configuration Testcase selection  Testcases –Different combinations of factors Static traffic Dynamic traffic Expert path GPS Difference in travel times Position information –Selected 10 most relevant testcases –Example: dynamic traffic (speed updates) + expert path Dynamic traffic Expert path

10. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 10 Towards an understanding of Decision Complexity in IT Configuration User Study: Overview & Analysis Approach  Overview –3 experiments, 10 testcases with 1 warm-up –1 st stage, 35 users –2 nd stage, 23 users, with refined experiment  Metrics –Average time spent per step (e.g. time / no. of steps) –User rating (in the end of each experiment) –Error rate (user picked non-optimal path)  Analysis approach –Step I: general statistical analysis of all data Each testcase measured as an independent data point Goal: identify factors that explain the most variance –Step II: pair-wise testcase comparisons Get more insight into specific effects of factor value Goal: remove inter-user variance

11. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 11 Towards an understanding of Decision Complexity in IT Configuration Summary of results  Significantly different impacts on user-perceived difficulty than on objective measures (e.g. time and error rate)  Time is influenced by: –Constraints static constraints > dynamic; static constraints > without constraints –Guidance (goal) without short-term goal oriented guidance > with such guidance  Rating is influenced by: –Guidance (goal) –Guidance (position) without position guidance > with such guidance –Constraints static constraints > dynamic  Error rate: hard to say statistically, except –error rate is reduced when g uidance (goal) is present –error rate is reduced when guidance (position) is not present

12. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 12 Towards an understanding of Decision Complexity in IT Configuration Summary of results (cont) –Depending on its goal (user, time or error rate), optimization for less complexity will have different focus, examples: An installation procedure with easily-located clear info (e.g. wizard- based prompts) for the next step will reduce both task time and user-perceived complexity A procedure with feedback on the current state of the system and the effect of the previous action (e.g. message windows following a button press) will only reduce user-perceived complexity, but unlikely to improve task time or error rate Omitting positional feedback (i.e., not showing users effects of their actions) may, counterintuitively, increase user accuracy, but at cost of significantly higher perceived complexity and task time

13. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 13 Towards an understanding of Decision Complexity in IT Configuration Proposal for a new user study Validate the model in the IT configuration domain

14. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 14 Towards an understanding of Decision Complexity in IT Configuration Analogy between two studies Driving time per segment Global map Traffic Goal (reach the destination) Number of features achieved per step Flowchart of the overall process (text) Soft compatibility / machine capacity limit Achieve the max number of features

15. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 15 Towards an understanding of Decision Complexity in IT Configuration Further step AvgTimePerStep Rating (User perceived complexity) Error Rate Operation time Skill levels Probability (downtime) Cost ($) Complexity (Constraints, Guidance, Consequence, …) Apply the model to assess IT decision complexity

16. IBM T.J. Watson Research Center & Northwestern University © 2006 IBM Corporation 16 Towards an understanding of Decision Complexity in IT Configuration Conclusions  We investigated decision complexity in IT configuration procedures –Used an carefully-mapped analogous domain to explore complexity space –Conduct an extensive user study –Quantitative results showing the key factors –Some guidance for system designers seeking to reduce complexity –Next steps are to explore further in simulated IT environment