1. 1 Jack J. Phillips, Ph.D. Patti P. Phillips, Ph.D. Show Me The Money!

2. 2 Learning Objectives Identify the major reasons for the growth and use of ROI as a measurement tool Identify the key steps and principles in the ROI Methodology Identify the issues in a successful implementation and examine the trends for use of ROI in the future

3. 3 The Value Evolution

4. The Many Applications of the ROI Methodology CommunicationsPolicies/Procedures Consulting/ODProcurement/Supply Chain Educational SystemsPublic Policy Programs Human Resources/Human CapitalPublic Relations / Public Affairs Leadership and CoachingQuality / Six Sigma Learning and DevelopmentResearch and Development Marketing/AdvertisingSocial Programs Meetings and EventsTechnology / Systems 4

5. The “New” Definition of Value Value Must: Be balanced, with qualitative and quantitative data Contain financial and non-financial perspectives Reflect strategic and tactical issues Represent different time frames Satisfy all key stakeholders Be consistent in collection and analysis Be grounded in conservative standards Come from credible sources Reflect efficiency in its development Create a call for action 5

6. The Changing Face of Value OldNew Activity cost centerResults profit center Expense controlMaximize Value Human Resources as expensesHuman capital as investment Rule centeredClient centered We value what IBM and GE valueOur value systems are unique Tolerate overheadOutsource or automate overhead Add value with small piecesAdd value with integrated applications Most of us are in a support roleAll of us are in a sales role Our measures are based on benchmarking Our measures are based on what we need We view value from one perspective We view value from everyone’s perspective Just another days work in the cubicleWe do something of value A jobA performance Treat old ideas as new ideasTreat old ideas as old ideas 6

7. The “Show Me” Evolution TermIssue Show Me!Collect Impact Data Show Me the Money!And Convert Data to Money Show Me the Real Money! And Isolate the Effects of the Project Show Me the Real Money, And Make Me Believe it! And Compare the Money to the Cost of the Project 7

8. How “Show Me the Money” Connects 8 SHOW ME THE MONEY!! Organizational Measures (Sponsor) Personal Measures (Stakeholders) Financial Measures (CFO)

9. Varieties of Terms and Applications NameExample Program A leadership development process for senior executives Project A re-engineering project for the plastics division System A fully networked system for all branches Initiative A faith-based initiative to reduce recidivism Policy A new pre-school policy for the disadvantaged Procedure A new scheduling procedure for truck drivers Event A golf outing for customers Meeting US Coast Guard innovations conference Process Quality sampling process People Staff additions in the customer care center ToolA new selection tool for the hotel staff 9

10. Results reacted very positively to the program and found it to be very relevant to their work; learned new skills and gained new insights about themselves; utilized the skills and insights routinely with their teams, although they had some difficulty in a few areas; improved several important work unit measures, with some measures improving as much as 28%; achieved an impressive 105% return on investment; and reported an increase in job satisfaction in the work unit. 10

11. Key Issues with this Level of Analysis Objectives? Credibility of data? Source of data? Consistent methodology? Scope? Standards? Use of data? Cost of process? Fear of data? 11

12. Project or Program Value Chain LevelMeasurement Focus 0.InputMeasures input such as volume and efficiencies 1. Reaction & Planned Action Measures participant reaction to the program and captures planned actions 2.LearningMeasures changes in knowledge, skills, and attitudes 3.ApplicationMeasures changes in on-the-job behavior or actions and progress with actions 4.Business ImpactCaptures changes in business impact measures 5.ROICompares program benefits to the costs

13. Global Trends in Measurement and Evaluation Organizations are moving up the value chain in their evaluation strategies Investment is increasing to 3-5% of the budget Increase focus is driven by clients and sponsors ROI is the fastest growing metric 13... and

14. Global Trends in Measurement and Evaluation Evaluation data is used to drive improvement and secure funding Evaluation is addressed early and often in the implementation cycle Processes are systematic and methodical, often designed into the delivery and implementation processes Technology is significantly enhancing processing 14

15. Profitability PROGRAM IMPACT Strategic Accountability Evaluation BOTTOM LINE CONTRIBUTION Effectiveness Vital Signs Benefits vs. Costs ECONOMIC VALUE ADDED PERFORMANCE STANDARDS Shareholder Value Balanced Scorecard Value Based ROI

16. ROI Dilemma 70-80% of organizations want to use ROI 15-20% of organizations are currently using ROI Wish List ROI ----- Use List ----- ROI Why the gap? HIGH LOW

17. 17 Activity BasedResults Based No business need for the program Program linked to specific business No assessment of performance issues Assessment of performance effectiveness No specific measurable objectives Specific objectives for behavior & business impact No effort to prepare program participants to achieve results Results expectations communicated to participants Shifting Paradigms... and

18. 18 Activity BasedResults Based No effort to prepare the work environment to support application Environment prepared to support application No efforts to build partnerships with key managers Partnerships established with key managers and clients No measurement of results or ROI analysis Measurement of results and ROI analysis Planning and reporting is input focused Planning and reporting is outcome focused Shifting Paradigms

19. 19 The ROI Process Collects Six Types of Data: Reaction and Planned Actions Learning and Confidence Application and Implementation Business Impact Return on Investment Intangible Measures ….and includes a technique to isolate the effects of the program or solution.

20. 20 The use of the Impact / ROI Process is Impressive Process refined over a 25-year period Thousands of impact studies conducted each year More than 100 case studies published More than 15,000 have attended a two-day ROI workshop More than 3,000 individuals certified More than 16 books developed to support the process ROI Process adopted by hundreds of organizations in 44 countries - See ROI Fact Sheet-

21. Why Use Impact and ROI Analysis? 21 Show contributions of selected programs Justify/defend budgets Identify inefficient programs that need to be redesigned or eliminated Reactive

22. Why Use Impact and ROI Analysis? 22 Aligns learning to business needs Earn respect of senior management / administrators Improve support for projects Enhance design and implementation processes Identify successful programs that can be implemented in other areas Earn a “seat at the table” Proactive

23. 23 Applications Learning and Development Career Development Competency Systems Diversity Programs E-Learning Executive Coaching Gainsharing Meetings and Events Leadership Development Organization Development Orientation Systems Recruiting Strategies Safety & Health Programs Self-Directed Teams Skill-Based/Knowledge-Based Compensation Technology Implementation Quality Management Wellness/Fitness Initiatives

24. Private Sector Organizations 24 Over 2,000 Private Sector Organizations

25. Government Organizations 25 US Department of Defense US Department of Labor US Department of Veteran Affairs US Government Accountability Office US Office of Personnel Management NASA EPA European Patent Office British Government New Zealand Singapore Italy Australia Canada Poland Over 200 Government Units

26. ROI Quiz 26

27. An Evaluation Framework Case Applications and Practice A Process Model Operating Standards and Philosophy Implementation Basic Elements

28. Reaction & Planned Action Learning & Confidence Application & Implementation Impact ROI Isolate the Effects of the Program Chain of Impact Intangible Benefits

29. Needs Program AssessmentObjectivesEvaluation Business ImpactBusiness NeedsObjectivesImpact Job PerformanceApplicationApplication NeedsObjectives Skills/Knowledge LearningLearning NeedsObjectives PreferencesSatisfactionReaction Objectives 44 3 3 2 2 11 Potential ROIROI PayoffsObjectives 5 5

30. 30 Collect Data During SolutionImplementationCollect SolutionImplementation Develop Objectives of Solution (s) Develop Objectives of Solution (s) DevelopEvaluation Plans and Baseline Data DevelopEvaluation Plans and Baseline Data Collect Data After SolutionImplementationCollect SolutionImplementation Level 2: Learning and Confidence Level 4: Business Impact Level 1: Reaction and Planned Actions Level 3: Application and Implementation Evaluation Planning Data Collection THE ROI PROCESS Calculating the Return on Investment of a Business Performance Solution

31. 31 Level 5: ROI Intangible Measures Data AnalysisReporting IdentifyIntangibleMeasuresIdentifyIntangibleMeasures Calculate the Return on Investment Calculate the Return on Investment Tabulate Costs of Solution Tabulate Costs of Solution GenerateImpactStudyGenerateImpactStudy Convert Data to Monetary Value Convert Data to Monetary Value Isolate the Effects Effects

32. 32 Collecting Follow-up Data Surveys/QuestionnairesSurveys/Questionnaires   Observation On the JobObservation On the Job  InterviewsInterviews  Follow-Up Focus GroupsFollow-Up Focus Groups  Action PlanningAction Planning   Performance ContractingPerformance Contracting   Performance MonitoringPerformance Monitoring  Level 3Level 4Method Type of Data

33. 33 Method 1 1. Comparison Group Analysis35% 2. Trend/Forecasting Analysis20% 3. Expert Estimation50% 4. Other20% 1 Listed by credibility 2 Percentages exceed 100% Best Practice Use 2 Isolating the Effects of the Program

34. 34 Converting Data to Money Credibility Resources Needed Standard valuesHighLow Records/Reports analysisHigh DatabasesModerate Expert EstimationLow

35. Direct Program Materials Coaching Fees Facilities Travel Indirect Needs Assessment Program Development Participant Time Administrative Overhead Evaluation 35 Tabulating Program Costs

36. 36 Intangible Benefits Complaints Conflicts Stress Job Satisfaction Commitment Teamwork Customer Service Engagement

37. 37 Defining the Return on Investment Benefits/Cost Ratio ROI Monetary Benefits Program Costs Net Monetary Benefits Program Costs = = X 100

38. ROI Example 38 $240,000 $80,000 $240,000 - $80,000 $80,000 BCR = ROI = X 100 = 3.00 = 200%

39. 39 ROI Target Options Set the value as with other investments, e.g.15% Set slightly above other investments, e.g. 25% Set at break even - 0% Set at client expectations Private sector organizations usually go with option #2; public sector organizations prefer #3.

40. ROI Best Practices 1. Implemented as a process improvement tool not a performance evaluation tool 2. Impact studies are conducted involving 5-10% of the projects and programs 3. A variety of data collection methods is used 4. The effects of the program are isolated 5. Business impact data are converted 40

41. ROI Best Practices 6. ROI evaluation targets are developed 7. The ROI methodology generates a micro level scorecard 8. Data are being integrated to create a macro scorecard 9. Implemented for about 3-5% of the budget 10. Routine ROI forecasting 11. Used as a tool to strengthen/improve 41

42. 42 Plan for evaluation early in the process Build evaluation into the process Share the responsibilities for evaluation Require participants to conduct major steps Use short-cut methods for major steps Use sampling to select the most appropriate programs for ROI analysis Cost-Saving Approaches and...

43. 43 Use estimates in the collection and analysis of data Develop internal capability to implement the ROI process Streamline the reporting process Utilize web-based software to reduce time Cost-Saving Approaches

44. Level 5: 5-10% Level 1: 90-100% Level 2: 40-60% Level 3: 30% Level 4: 10-20% 44 Evaluation Targets

45. 45 Criteria for Selecting Programs for Levels 4 & 5 Expected life cycle of the program The importance of the program in meeting the organization’s goals Cost of the program Visibility of the program The size of the target audience Extent of management interest

46. 46 What Happens If We Maintain a Status Quo? Budget?Budget? Influence?Influence? Support?Support? Other Issues?Other Issues?

47. 47 ROI Reality Information is desired by clients/executives The process provides a balanced, credible approach with six types of data All types of organizations are routinely using Impact/ROI The process can be implemented without draining resources The process is a long-term goal for many organizations

48. Impact / ROI Standards 48

49. Southeast Corridor Bank Program Title: Managing Retention Target Group: Bank Tellers Solution: Skills Based Pay System to Reduce Employee Turnover 49

50. Southeast Corridor Bank Level 1: Reaction Composite Rating: 4.2 out of 5 Level 2: Learning Positive self assessment on program understanding Positive self assessment on each course with few exceptions (only two failed to be promoted because of performance in training) 50

51. Level 3: Application 95% participation rate 86 requests for training compared to 46 the year before 138 review situations 257 promotions compared to 139 the year before Level 4: Impact Turnover reduced from 71% to 35% Staffing level reduced by 4% 51 Southeast Corridor Bank

52. Level 5: ROI BCR: 3.58 ROI: 258% Intangible Benefits Customer Satisfaction Job Satisfaction Product Sales Cross Selling 52 Southeast Corridor Bank

53. Technique to Isolate Effects of Program: Estimates from branch managers and branch staff, adjusted for error Technique to Convert Data to Monetary Value: External studies at similar institutions and standard values (for staffing) Fully-loaded Program Costs: $857,196 First year; $433,200 Second year 53 Southeast Corridor Bank

54. Sprint/Nextel Program Title: Diversity Target Group: Managers and Employees Solution: All-Inclusive Workforce Program (AIW) 54

55. Sprint/Nextel Level 1: Reaction Composite Rating: 4.39 out of 5 (for six items) Level 2: Learning Averaged 4.28 out of 5 (for learning on six objectives) 55

56. Level 3: Application Managers: Supports AIW (87%) Addresses Problems (81%) Encourages Staff (78%) Employees: Supports AIW (65%) Identifies Differences (63%) Encourages Staff (60%) 91% of Managers successful completed action plans 56 Sprint/Nextel

57. Level 4: Impact Attrition Rate Improvement = 9.77% Level 5: ROI BCR: 2.6 ROI: 163% Intangible Benefits Employee Satisfaction Communication Cooperation Diversity Mix Teamwork 57 Sprint/Nextel

58. Technique to Isolate Effects of Program: Manager’s estimate, adjusted for error Technique to Convert Data to Monetary Value: Standard cost item ($89,000 per Turnover) Fully-loaded Program Costs: $1,216,836 58

59. Healthcare, Inc. Program Title: Preventing Sexual Harassment Target Group: Primary First and Second Level Managers (655) Secondary: All employees through group meetings (6,844) Solution: Workshops and meetings on company policy and sexual harassment regulations 59

60. Healthcare, Inc. Level 1: Reaction Overall rating 4.11 out of 5 93% provided action items Level 2: Learning 65% increase post-test versus pre-test Skill practice demonstration 60

61. Level 3: Application 96% conducted meetings and completed meeting record 4.1 out of 5 on behavior change survey 68% report all action items complete 92% report some action items complete Level 4: Impact Turnover reduction: $2,840,632 Complaint reduction: $360,276 Total improvement: $3,200,908 61 Healthcare, Inc.

62. Level 5: ROI 1,051% Intangible Benefits Job Satisfaction Absenteeism Stress Reduction Better Recruiting 62 Healthcare, Inc.

63. Technique to Isolate Effects of Program: Trendline analysis; participant estimation Technique to Convert Data to Monetary Value: Historical costs; internal experts Fully-loaded Program Costs: $277,987 63 Healthcare, Inc.

64. Sample of Published ROI Studies 64