Measuring ROI in the Public Sector Jack J. Phillips, Ph.D. March 9, 2004

Agenda Review current status of ROI implementation Explore the myths and mysteries of ROI Identify the key steps involved in the ROI methodology Examine the key issues involved in ROI implementation in the public sector

Paradigm Shift for a Results-Based Approach Activity Based Results-Based Approach No business need for the program Program linked to specific business needs

Results-Based Approach No assessment of performance issues Assessment of performance effectiveness Paradigm Shift for a Results-Based Approach Activity Based

Results-Based Approach No specific measurable objectives Specific objectives for behavior and business impact Paradigm Shift for a Results-Based Approach Activity Based

Results-Based Approach No effort to prepare program participants to achieve results Results expectation communicated to participants Paradigm Shift for a Results-Based Approach Activity Based

Results-Based Approach No effort to prepare the work environment to support transfer Environment prepared to support transfer Paradigm Shift for a Results-Based Approach Activity Based

Results-Based Approach No efforts to build partnerships with key managers Partnerships established with key managers and clients Paradigm Shift for a Results-Based Approach Activity Based

Results-Based Approach No measurement of results or benefit-cost analysis Measurement of results and benefit-cost analysis Paradigm Shift for a Results-Based Approach Activity Based

Results-Based Approach Planning and reporting is input focused Planning and reporting is output focused Paradigm Shift for a Results-Based Approach Activity Based

Key Measurement Issues Time? Costs? Data Collection? Management Support? Skills

Why ROI? ROI is the ultimate level of evaluationROI is the ultimate level of evaluation ROI is a concept familiar to most managersROI is a concept familiar to most managers ROI has a rich history of applicationROI has a rich history of application Many top executives are requiring ROI informationMany top executives are requiring ROI information The ROI issue cannot be ignored!

The ROI Process Reaction, Satisfaction, and Planned Action Learning Application and Implementation Business Impact Return on Investment Intangible Measures ….and includes a technique to isolate the effect of the program or solution. Generates six types of data

ROI by the Numbers Process refined over a 20-year period Thousands of impact studies conducted each year 100 case studies published 2,000 individuals certified 15 books developed to support the process 500 member professional network ROI Process adopted by hundreds of organizations in 38 countries

Payoff Aligns programs to business needsAligns programs to business needs Show contributions of selected programsShow contributions of selected programs Earn respect of senior management/ administratorsEarn respect of senior management/ administrators Build staff moraleBuild staff morale Justify/defend budgetsJustify/defend budgets

Payoff Improve support for human resources/ learning/developmentImprove support for human resources/ learning/development Enhance design and implementation processesEnhance design and implementation processes Identify inefficient programs that need to be redesigned or eliminatedIdentify inefficient programs that need to be redesigned or eliminated Identify successful programs that can be expandedIdentify successful programs that can be expanded

ROI Applications Associate Relations ProgramsAssociate Relations Programs Career Development ProgramsCareer Development Programs Competency SystemsCompetency Systems Diversity ProgramsDiversity Programs E-LearningE-Learning Executive CoachingExecutive Coaching Executive EducationExecutive Education Gain Sharing ProgramsGain Sharing Programs Global LeadershipGlobal Leadership Organization DevelopmentOrganization Development Orientation SystemsOrientation Systems Recruiting StrategiesRecruiting Strategies Safety & Health ProgramsSafety & Health Programs Self-Directed TeamsSelf-Directed Teams Skill-Based/Knowledge- Based CompensationSkill-Based/Knowledge- Based Compensation Technology ImplementationTechnology Implementation Total Quality ManagementTotal Quality Management Wellness/Fitness InitiativesWellness/Fitness Initiatives

Who is Using the ROI Process? Private Sector Organizations Allstate Insurance Amazon.com Apple Computer AT&T Bristol-Myers Squibb Coca Cola Dell Computers Deloitte & Touche Delta Airlines DHL Worldwide Express Federal Express General Motors Georgia Pacific Hewlett Packard Hilton Hotels Home Depot Intel Lockheed Martin Motorola NCR Nextel Olive Garden Restaurant PricewaterhouseCoopers Shell Oil Singapore Airlines TD Bank Verizon Communications Wachovia Bank

Who is Using the ROI Process? Government Organizations US Department of Defense US Internal Revenue Service US National Security Agency US Department of Veteran Affairs NASA State of New York State of Mississippi State of Texas City of Richmond, Virginia Government of New Zealand Government of Singapore Government of Italy Government of Australia Provincial Government of Ontario, Canada

ROI Quiz

Appropriate Interventions Performance Improvement Training and Learning Solutions Organization Development HR Programs Change Initiatives Technology Implementation

Public Sector Issues Absence of revenues and profitsAbsence of revenues and profits Absence of hard dataAbsence of hard data Whose ROI?Whose ROI? Government services are essential and, therefore, may not need this level of evaluationGovernment services are essential and, therefore, may not need this level of evaluation Restricted range of options to correct problemsRestricted range of options to correct problems

An Evaluation Framework Case Applications and Practice A Process Model Operating Standards and Philosophy Implementation ROI: The Pieces of the Puzzle

Evaluation Levels Level Measurement Focus 1.Reaction & Planned Action Measures participant satisfaction with the program and captures planned actions 2.Learning Measures changes in knowledge, skills, and attitudes 3.Application Measures changes in on-the-job behavior 4.Business Impact Measures changes in business impact variables 5.Return on Investment Compares program benefits to the costs

Characteristics of Evaluation Levels Chain of Value ofCustomer FrequencyDifficulty of ImpactInformationFocusof UseAssessment SatisfactionLowestConsumerFrequent Easy LearningApplicationImpact ROIHighestClientInfrequentDifficult Customers Consumers: The customers who are actively involved in the process. Client: The customers who fund, support, and approve the project

Linking Assessment with Evaluation Needs Program AssessmentObjectivesEvaluation Business ImpactBusiness NeedsObjectivesImpact Job PerformanceApplicationApplication NeedsObjectives Skills/Knowledge LearningLearning NeedsObjectives PreferencesSatisfactionReaction Objectives

Matching Evaluation Levels with Objectives

Develop Objectives of Solution (s) DevelopEvaluation Plans and Baseline Data Collect Data During SolutionImplementationCollect Data After SolutionImplementation Level 2: Learning Level 4: Business Impact Level 1:Reaction, Satisfaction, and Planned Actions Level 3: Application/ Implementation Evaluation Planning Data Collection THE ROI PROCESS Calculating the Return on Investment of a Business Performance Solution THE ROI PROCESS Calculating the Return on Investment of a Business Performance Solution

Isolate the Effects Convert Data to Monetary Value Calculate the Return on Investment Tabulate Costs of Solution IdentifyIntangibleMeasures GenerateImpactStudy Level 5: ROI Intangible Measures Data AnalysisReporting

Collecting Follow-Up Surveys Follow-Up Questionnaires Observation On the Job Interviews with Participants Follow-Up Focus Groups Program Assignments Action Planning Performance Contracting Program Follow-Up Session Performance Monitoring Level 3  Level 4 

Isolating Use of a control group arrangement Trend line analysis of performance data Use of forecasting methods of performance data Participant’s estimate of program impact (percent) Supervisor’s estimate of program impact (percent) Management’s estimate of program impact (percent) Use of previous studies Subordinate’s report of other factors Calculating/Estimating the impact of other factors Use of customer input

Converting output to contribution - standard value Converting the cost of quality - standard value Converting employee’s time Using historical costs Using internal and external experts Using data from external databases Linking with other measures Using participants’ estimates Using supervisors’ and managers’ estimates Using staff estimatesConverting

Needs Assessment (Prorated) Development Costs (Prorated) Program Materials Instructor/Facilitator Costs Facilities Costs Travel/Lodging/Meals Participant Salaries and Benefits Administrative/Overhead Costs Evaluation CostsTabulating

Intangible Benefits Increased Job Satisfaction Increased Organizational Commitment Improved Teamwork Improved Customer Service Reduced Complaints Reduced Conflicts Reduced Stress

Return on Investment Benefits /Program Benefits Costs = Ratio Program Costs Net Program Benefits ROI = Program Costs

Calculating ROI BCR= = 3.0 ROI = = ???% $240,000 $80,000 Total Benefits - Program Costs Program Costs

BCR= = 3.0 ROI = = 200% $240,000 $80,000 $240,000 - $80,000 $80,000 Calculating ROI

ROI Target Options Set the value as with other investments, e.g. 15% Set slightly above other investments, e.g. 25% Set at break even - 0% Set at client expectations

When properly implemented, high ROI values can be achieved with programs on: LeadershipLeadership Team BuildingTeam Building Management DevelopmentManagement Development Supervisor TrainingSupervisor Training Sales TrainingSales Training % to 700% ROI is not uncommon

ROI Best Practices 1.Implemented as a process improvement tool not a performance evaluation tool 2.Impact studies are conducted involving 5-10% of the programs 3.A variety of data collection methods are used 4.The effects of learning/development are isolated 5.Business impact data are converted

6.ROI evaluation targets are developed 7.The ROI methodology generates a micro level scorecard 8.Data are being integrated to create a macro scorecard 9.Implemented for about 3-5% of the budget 10.Routine ROI forecasting 11.Used as a tool to strengthen/improve ROI Best Practices

Building a Scorecard Provides macro-level perspective of successProvides macro-level perspective of success Serves as a brief report versus detailed studyServes as a brief report versus detailed study Shows connection of training’s contribution to business objectivesShows connection of training’s contribution to business objectives Integrates various types of dataIntegrates various types of data Demonstrates alignment between programs, strategic objectives, and operating goalsDemonstrates alignment between programs, strategic objectives, and operating goals

Micro Level Scorecard Macro Level Scorecard

Seven Categories of Data Indicators Reaction and Planned Action Learning Application Business Impact ROI Intangibles

Scorecard Reporting First University

Key Implementation Actions Determine/establish responsibilities Develop skills/knowledge with ROI Develop transition/implementation plan Conduct ROI studies Prepare/revise evaluation policy/procedures/ guidelines Train/brief managers on the ROI Process Communicate progress/results

Plan for evaluation early in the process Build evaluation into the process Share the responsibilities for evaluation Require participants to conduct major steps Use short-cut methods for major steps Use sampling to select the most appropriate programs for ROI analysis Cost-Saving Approaches

Use estimates in the collection and analysis of data Develop internal capability to implement the ROI process Streamline the reporting process Utilize web-based software to reduce time Cost-Saving Approaches

Recommended Targets Level 1 - Reaction Level 2 - Learning Level 3 - Application Level 4 – Business Impact Level 5 - ROI 100% 60% 30% 10% 5%

Criteria Expected life cycle of the program The importance of the program in meeting the organization’s goals Cost of the program Visibility of the program The size of the target audience Extent of management interest

What Happens If We Do Nothing? Budget? Influence? Support? Other Issues?

ROI Reality ROI information is desired by clients/executives The ROI process provides a balanced, credible approach with six types of data All types of organizations are routinely using the ROI process The ROI process can be implemented without draining resources The ROI process is a long-term goal for many organizations

Status of Major Learning and Development Metrics

In Training Magazine’s 2003 Top 100 survey, 92% measure through Business Impact and 67% measure through ROI

Program Analysis Dominates Satisfaction of Training Participants Impact of Training on Performance ROI of Training 80% 88% 27% 78% 11% 63% Organizations Utilizing Specific L&D Metrics Organizations Wanting Specific L&D Metrics

ROI in training is a fast growing metric % of companies have ROI on their wish list Corporate Leadership Council 2002

Sample of Published ROI Studies