1. Can Autonomic Management Systems be Trusted? A Perspective from Business-Driven IT Management Jacques Sauvé - UFCG LANOMS 2007 Panel

2. Trust in IT Management Why can I talk about this? What’s my special perspective? I do research in Business- Driven IT Management (BDIM) Currently looking at ITIL processes and including a "business perspective" 2

3. An experiment closer to home... How many of you mistrust some result presented at LANOMS 2007 or other conferences? – Why? How many of you mistrust results presented in your own papers? – Why? 3

4. Where does mistrust come from? Methodology! – We are deficient in validation Questions: – Did you do any kind of validation to see if the problem is solved? – Do you know (versus hope) that your solution is useful? – Did you cover the problem space well? – Do you know where your solution is good and where it is not? We plant the seeds of mistrust … 4

5. Why are we deficient? Validation easy to do (in theory) – Formulate a hypothesis: "My management software improves metric X by Y%” – Get a lot of data (from a large user base) – See if it supports the hypothesis – If it does, accept the hypothesis Your claims will now be valid when selling the solution Sweep the problem space to see what you can claim 5

6. But the real world creeps in... Marketing people sell their own “hypothesis” How do you get a large user base to get data? Pressure to push products to market before they are validated How do you test your own baby? “Of course, it’s good... I built it!” 6

7. “Good enough” solutions It takes time to validate and by then you have new modules, new problems,... Given the lifecycle of management solutions, how much time can we afford to spend validating? – Our solutions last only a (very few) years Validating systems is not like validating general theories – Stop when the validation is “good enough” 7

8. Conclusion 1: Full scientific validation is usually not an option But the following remains true... – Trust is gained through user experience – Level of trust changes over time – Building trust is a process – Overselling may be good for the bottom line but may hurt trust 8

9. How to Lose Trust? Is it helpful to get it right 99% of the time? How about 95%? Getting it wrong once on a while makes managers fidgety – “When can I trust it?" Losing trust is (almost) immediate and tools get boycotted – Some processes are more critical – Ex. Getting diagnostics wrong 9

10. Conclusion 2: Sell the right message Sell the message that the systems are “useful”, not “always right” or “best” Use metrics and baselines to enable comparison – Use aggregate metrics But... 10

11. Conclusion 3: Automatic Solutions Should not Cause Grand Disasters! Example: automatic scheduler loops and can’t choose any job to run Example: Automatic capacity allocation should not flip-flop and thrash Grand disasters reduce trust to zero 11

12. What Kind of Metrics Should we Use to Build Trust? Are business metrics are better than technical metrics? – Technical: BD throughput, resource utilization, response time, etc. – Business: business transaction throughput, financial gain or loss, risk, business impact, etc. 12

13. Non-Conclusion 1: Are Business Metrics Better for Trust? I think business metrics are more useful but they are not directly measurable They may or may not help build trust more than easy-to-measure technical metrics – It’s hard to trust models, easier to trust measurements Use of business metrics helps the dialog with business people and should help validation and thus gain trust – This is an untested hypothesis 13

14. The Effects of Scale Humans can’t handle: – Number of elements and – Number of dimensions present in today’s scenarios 14

15. Conclusion 4: Autonomic will Rule Because of Scale Managers have no choice but to rely to automatic solutions because of scale – Even if trust is not perfect 15

16. Other Possible Solutions Put user in the feedback loop! – To calibrate tools – To make tool learn – To let user decide how much the tool should do Automate versus suggest actions 16

17. Thank you. 17