1. Measuring ROI in the Public Sector
Jack J. Phillips, Ph.D March 9, 2004

2. Agenda Review current status of ROI implementation
Explore the myths and mysteries of ROI
Identify the key steps involved in the ROI methodology
Examine the key issues involved in ROI implementation in the public sector

3. Paradigm Shift for a Results-Based Approach
Activity Based
No business need for the program
Results-Based Approach
Program linked to specific business needs

4. Paradigm Shift for a Results-Based Approach
Activity Based
No assessment of performance issues
Results-Based Approach
Assessment of performance effectiveness

5. Paradigm Shift for a Results-Based Approach
Activity Based
No specific measurable objectives
Results-Based Approach
Specific objectives for behavior and business impact

6. Paradigm Shift for a Results-Based Approach
Activity Based
No effort to prepare program participants to achieve results
Results-Based Approach
Results expectation communicated to participants

7. Paradigm Shift for a Results-Based Approach
Activity Based
No effort to prepare the work environment to support transfer
Results-Based Approach
Environment prepared to support transfer

8. Paradigm Shift for a Results-Based Approach
Activity Based
No efforts to build partnerships with key managers
Results-Based Approach
Partnerships established with key managers and clients

9. Paradigm Shift for a Results-Based Approach
Activity Based
No measurement of results or benefit-cost analysis
Results-Based Approach
Measurement of results and benefit-cost analysis

10. Paradigm Shift for a Results-Based Approach
Activity Based
Planning and reporting is input focused
Results-Based Approach
Planning and reporting is output focused

11. Key Measurement Issues
Time?
Costs?
Data Collection?
Management Support?
Skills

12. Why ROI? ROI is the ultimate level of evaluation
ROI is a concept familiar to most managers
ROI has a rich history of application
Many top executives are requiring ROI information
The ROI issue cannot be ignored!

13. Generates six types of data
The ROI Process
Generates six types of data
Reaction, Satisfaction, and Planned Action
Learning
Application and Implementation
Business Impact
Return on Investment
Intangible Measures
….and includes a technique to isolate the effect of the program or solution.

14. ROI by the Numbers Process refined over a 20-year period
Thousands of impact studies conducted each year
100 case studies published
2,000 individuals certified
15 books developed to support the process
500 member professional network
ROI Process adopted by hundreds of organizations in 38 countries

15. Payoff Aligns programs to business needs
Show contributions of selected programs
Earn respect of senior management/ administrators
Build staff morale
Justify/defend budgets

16. Payoff Improve support for human resources/ learning/development
Enhance design and implementation processes
Identify inefficient programs that need to be redesigned or eliminated
Identify successful programs that can be expanded

17. ROI Applications Associate Relations Programs
Career Development Programs
Competency Systems
Diversity Programs
E-Learning
Executive Coaching
Executive Education
Gain Sharing Programs
Global Leadership
Organization Development
Orientation Systems
Recruiting Strategies
Safety & Health Programs
Self-Directed Teams
Skill-Based/Knowledge-Based Compensation
Technology Implementation
Total Quality Management
Wellness/Fitness Initiatives

18. Who is Using the ROI Process? Private Sector Organizations
Allstate Insurance
Amazon.com
Apple Computer
AT&T
Bristol-Myers Squibb
Coca Cola
Dell Computers
Deloitte & Touche
Delta Airlines
DHL Worldwide Express
Federal Express
General Motors
Georgia Pacific
Hewlett Packard
Hilton Hotels
Home Depot
Intel
Lockheed Martin
Motorola
NCR
Nextel
Olive Garden Restaurant
PricewaterhouseCoopers
Shell Oil
Singapore Airlines
TD Bank
Verizon Communications
Wachovia Bank

19. Who is Using the ROI Process? Government Organizations
US Department of Defense
US Internal Revenue Service
US National Security Agency
US Department of Veteran Affairs
NASA
State of New York
State of Mississippi
State of Texas
City of Richmond, Virginia
Government of New Zealand
Government of Singapore
Government of Italy
Government of Australia
Provincial Government of Ontario, Canada

20. ROI Quiz

21. Appropriate Interventions
Performance Improvement
Training and Learning Solutions
Organization Development
HR Programs
Change Initiatives
Technology Implementation

22. Public Sector Issues Absence of revenues and profits
Absence of hard data
Whose ROI?
Government services are essential and, therefore, may not need this level of evaluation
Restricted range of options to correct problems

23. ROI: The Pieces of the Puzzle
An Evaluation
Framework
Case Applications
and Practice
Implementation
Operating
Standards and
Philosophy
A Process
Model

24. Evaluation Levels Level Measurement Focus 1. Reaction & Planned Action
Measures participant satisfaction with the program and captures planned actions
2. Learning
Measures changes in knowledge, skills, and attitudes
3. Application
Measures changes in on-the-job behavior
4. Business Impact
Measures changes in business impact variables
5. Return on Investment
Compares program benefits to the costs

25. Characteristics of Evaluation Levels
Chain of Value of Customer Frequency Difficulty of
Impact Information Focus of Use Assessment
Satisfaction Lowest Consumer Frequent Easy
Learning
Application
Impact
ROI Highest Client Infrequent Difficult
Customers
Consumers: The customers who are actively involved in the process.
Client: The customers who fund, support, and approve the project

26. Linking Assessment with Evaluation
Needs Program
Assessment Objectives Evaluation
Business Impact Business
Needs Objectives Impact 4 4
Job Performance Application Application
Needs Objectives 3 3
Skills/Knowledge Learning Learning
Needs Objectives 2 2
Preferences Satisfaction Reaction
Objectives 1 1

27. Matching Evaluation Levels with Objectives

28. THE ROI PROCESS Evaluation Planning Data Collection Develop
Calculating the Return on Investment
of a Business Performance Solution
Evaluation
Planning
Data Collection
Level 1:Reaction, Satisfaction, and Planned Actions
Level 3:
Application/
Implementation
Develop
Objectives of
Solution (s)
Develop
Evaluation
Plans and
Baseline Data
Collect
Data During
Solution
Implementation
Collect
Data After
Solution
Implementation
Level 2:
Learning
Level 4:
Business Impact

29. Data Analysis Reporting Tabulate Costs of Solution Isolate the Effects
Convert Data
to Monetary
Value
Calculate the
Return on
Investment
Generate
Impact
Study
Level 5:
ROI
Identify
Intangible
Measures
Intangible Measures

30. Collecting Follow-Up Surveys Follow-Up Questionnaires
Observation On the Job
Interviews with Participants
Follow-Up Focus Groups
Program Assignments
Action Planning
Performance Contracting
Program Follow-Up Session
Performance Monitoring
Level 3 
Level 4 

31. Isolating Use of a control group arrangement
Trend line analysis of performance data
Use of forecasting methods of performance data
Participant’s estimate of program impact (percent)
Supervisor’s estimate of program impact (percent)
Management’s estimate of program impact (percent)
Use of previous studies
Subordinate’s report of other factors
Calculating/Estimating the impact of other factors
Use of customer input
NOTES: Which techniques are appropriate in your organization?

32. Converting Converting output to contribution - standard value
Converting the cost of quality - standard value
Converting employee’s time
Using historical costs
Using internal and external experts
Using data from external databases
Linking with other measures
Using participants’ estimates
Using supervisors’ and managers’ estimates
Using staff estimates

33. Tabulating Needs Assessment (Prorated) Development Costs (Prorated)
Program Materials
Instructor/Facilitator Costs
Facilities Costs
Travel/Lodging/Meals
Participant Salaries and Benefits
Administrative/Overhead Costs
Evaluation Costs

34. Intangible Benefits Increased Job Satisfaction
Increased Organizational Commitment
Improved Teamwork
Improved Customer Service
Reduced Complaints
Reduced Conflicts
Reduced Stress

35. Return on Investment Benefits / Program Benefits Costs =
Ratio Program Costs
Net Program Benefits
ROI =
Program Costs

36. Total Benefits - Program Costs
Calculating ROI
BCR = = 3.0
ROI = = ???%
$240,000
$80,000
Total Benefits - Program Costs
Program Costs

37. Calculating ROI BCR = = 3.0 $240,000 $80,000 ROI = = 200%
$240,000 - $80,000

38. ROI Target Options Set the value as with other investments, e.g. 15%
Set slightly above other investments, e.g. 25%
Set at break even - 0%
Set at client expectations

39. Management Development Supervisor Training Sales Training
When properly implemented, high ROI values can be achieved with programs on:
Leadership
Team Building
Management Development
Supervisor Training
Sales Training
% to 700% ROI is not uncommon

40. ROI Best Practices
Implemented as a process improvement tool not a performance evaluation tool
Impact studies are conducted involving 5-10% of the programs
A variety of data collection methods are used
The effects of learning/development are isolated
Business impact data are converted

41. ROI Best Practices ROI evaluation targets are developed
The ROI methodology generates a micro level scorecard
Data are being integrated to create a macro scorecard
Implemented for about 3-5% of the budget
Routine ROI forecasting
Used as a tool to strengthen/improve

42. Building a Scorecard Provides macro-level perspective of success
Serves as a brief report versus detailed study
Shows connection of training’s contribution to business objectives
Integrates various types of data
Demonstrates alignment between programs, strategic objectives, and operating goals

43. Micro Level Scorecard Macro Level Scorecard1 2 3 4 1 2 3 4 1 2 3 4 5 1 2 3

44. Seven Categories of Data
Indicators
Reaction and Planned Action
Learning
Application
Business Impact
ROI
Intangibles

45. Scorecard Reporting First University

46. Key Implementation Actions
Determine/establish responsibilities
Develop skills/knowledge with ROI
Develop transition/implementation plan
Conduct ROI studies
Prepare/revise evaluation policy/procedures/ guidelines
Train/brief managers on the ROI Process
Communicate progress/results

47. Cost-Saving Approaches
Plan for evaluation early in the process
Build evaluation into the process
Share the responsibilities
for evaluation
Require participants to
conduct major steps
Use short-cut methods for major steps
Use sampling to select the most appropriate programs for ROI analysis

48. Cost-Saving Approaches
Use estimates in the collection and analysis of data
Develop internal capability to implement the ROI process
Streamline the reporting process
Utilize web-based software to reduce time

49. Recommended Targets 100% 60% 30% 10% 5% Level 1 - Reaction
Level 2 - Learning
30%
Level 3 - Application
10%
Level 4 – Business Impact 5%
Level 5 - ROI

50. Criteria Expected life cycle of the program
The importance of the program in meeting the organization’s goals
Cost of the program
Visibility of the program
The size of the target audience
Extent of management interest

51. What Happens If We Do Nothing?
Budget?
Influence?
Support?
Other Issues?

52. ROI Reality ROI information is desired by clients/executives
The ROI process provides a balanced, credible approach with six types of data
All types of organizations are routinely using the ROI process
The ROI process can be implemented without draining resources
The ROI process is a long-term goal for many organizations

53. Status of Major Learning and Development Metrics

54. In Training Magazine’s 2003 Top 100 survey, 92% measure through Business Impact and 67% measure through ROI

55. Program Analysis Dominates
88%
78%
80%
63%
27%
11%
Satisfaction of Training Participants
Impact of Training on Performance
ROI of Training
Organizations Utilizing Specific L&D Metrics
Organizations Wanting Specific L&D Metrics

56. ROI in training is a fast growing metric
ROI in training is a fast growing metric % of companies have ROI on their wish list
Corporate Leadership Council 2002

57. Sample of Published ROI Studies