1. Show Me the Money: Moving From Impact to ROI
Patti Phillips, Ph.D.

2. Objectives Isolate the effects of a program
Convert benefits to monetary value
Identify the intangible benefits
Tabulate the fully-loaded costs
Calculate ROI .

3. The ROI calculation is simple.
Program Benefits
Program Costs
BCR =
Net Program Benefits
Program Costs
X 100
ROI =

4. Try it!
$750,000
$425,000
BCR =
$750,000 - $425,000
$425,000
ROI =
X 100

5. Try it!
$750,000
$425,000
= 1.76:1
BCR =
$750,000 - $425,000
$425,000
ROI =
X 100
= 76%

6. While we don’t strive for a negative ROI, negative is not always bad!
What Makes a Good ROI?
Set the value at the same level as other investments – 15%
Set slightly above other investments – 25%
Set at break even – 0%
Set at client expectations
While we don’t strive for a negative ROI, negative is not always bad!

7. xxx Training Presenter Guide
Levels of Evaluation
Measurement Focus 7
Typical Measures
0. Inputs and Indicators
The input into the project in terms of scope, volume, efficiencies, costs
Participants, Hours, Costs, Timing
1. Reaction & Perceived Value
Reaction to the project or program, including the perceived value
Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action
2. Learning
Learning to use the content and materials, including the confidence to use what was learned
Skills, Knowledge, Capacity, Competencies, Confidence, Contacts
3. Application & Implementation
Use of content and materials in the work environment, including progress with actual items and implementation
Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use
4. Business Impact
The consequences of the use of the content and materials expressed as business impact measures
Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement
5. ROI
Comparison of monetary benefits from program to program costs
Benefit-Cost Ratio (BCR), ROI%, Payback Period

8. ROI Methodology Stage 1 Stage 2 Data Collection Evaluation Planning
Collect Data
After Solution
Implementation
Level 3
Level 4
Develop
Objectives
Of Solution
Develop
Evaluation
Plans and
Baseline Data
Collect Data
During Solution
Implementation
Level 1
Level 2

9. Stage 4 Communicate Results
Capture
Costs
Of Solution
Generate
Impact Study
Report
Stage Communicate Results
Stage Data Analysis
Isolate the
Effects of
Solution
Convert Data
to Monetary
Value
Calculate
the Return On
Investment
Level 5
Identify
Intangibles
Measures
Intangible Benefits

10. Operating Standards Report the complete story Conserve resources
Use the most credible sources
Choose the most conservative alternatives
Isolate the effects of the program
No data no improvement
Adjust estimates for error
Throw out the extreme and unsupported claims
Use first year benefits for short-term programs
Include fully-loaded costs
Report intangible benefits
Communicate results to all stakeholders

11. Isolate the Effects of the Program
Reaction
Learning
Application
Impact
ROI
Isolate the Effects of the Program
Intangible Benefits
Inputs

12. Methods to Isolate Program Effects
Use of a control group arrangement
Trend line analysis of performance data
Use of forecasting methods of performance data
Participant’s estimate of program impact (percent)
Supervisor’s estimate of program impact (percent)
Manager’s estimate of program impact
Use of expert/previous studies
Calculate/estimate the impact of other factors
Customer input
NOTES: Which techniques are appropriate in your organization?

13. Isolating the Effects of the Program Frequency of Methods Used
* Survey of Users, N = 235

14. Classic Control Group Design What is the difference in improvement?
Experimental Group
Pre-Measure
Program
Post-Measure
Control Group
Post-Measure
Pre-Measure

15. Post-Program Only Design What is the difference?
Experimental Group
Program
Post-Measure
Control Group
Post-Measure

16. Example of Trend Line Analysis
1.85% Pre Program Average
CPI Program Conducted 2% 1%
1.45% Projected Average
ERROR
RATE
.7% Post Program Average
J F M A M J J A S O N D J
MONTHS

17. Average Confidence (%)
Example of Estimation
Monthly increase in credit card accounts: 175 (fact)
Contributing Factors
Consensus
Impact (%)
Average Confidence (%)
Sales Training
32%
83%
Incentives
41%
87%
Management Reinforcement
14%
62%
Market Fluctuations
11%
91%
Other _________ 2%
100%

18. Example of Estimations
Influence
Fact
% Contrib.
Est. Impact
Confidence
Adjusted Impact
Sales Training
175
32% 56
83%
46.48
Fact: 175 New Credit Card Accounts
% Contribution: 32%
Est. Impact: 56
Uncertainty 17%
Margin of Error: +/
65.52 56
46.48

19. Data are converted by:
Converting output to contribution – standard value
Converting the cost of quality – standard value
Converting employee’s time – standard value
Using historical costs
Using internal and external experts
Using data from external databases
Linking with other measures
Using participants’ estimates
Using supervisors’ and managers’ estimates
Using staff estimates

20. Cost of A Sexual Harassment Complaint
35 Complaints
Actual Costs from Records
Additional Estimated Costs from Staff
Legal Fees, Settlements, Losses, Material, Direct Expenses
EEO/AA Staff Time, Management Time
$852,000 Annually
$852,000 35
Cost per complaint = $24,343

21. 5 Steps to Convert Impact to Money
Step 1: Focus on a unit of measure
Step 2: Determine the value (V) of each unit
Step 3: Calculate the change in performance (∆P)
Step 4: Determine the annual amount of the change (A∆P)
Step 5: Calculate the total annual value of the improvement (A∆P x V)

22. Example Using Internal Experts
Step 1: One grievance
Step 2: V = $6,500 (from Director of nursing and HR Experts)
Step 3: ∆P = average of 7 out of 10 grievances prevented per month
Step 4: Annual ∆P =
Step 5: A∆P x V =

23. Example Using Internal Experts
Step 1: One grievance
Step 2: V = $6,500 (from Director of nursing and HR Experts)
Step 3: ∆P = average of 7 out of 10 grievances prevented per month
Step 4: Annual ∆P = 84
Step 5: A∆P x V = $546,000

24. Example Using Standard Values
Step 1: One sale
Step 2: V = 30% profit margin
Step 3: ∆P = $20,000 revenue per month
Step 4: Annual ∆P =
Step 5: A∆P x V =

25. Example Using Standard Values
Step 1: One sale
Step 2: V = 30% profit margin
Step 3: ∆P = $20,000 revenue per month
Step 4: Annual ∆P = $240,000
Step 5: A∆P x V = $240,000 x .30 = $72,000

26. Fully-Loaded Cost Profile
Assessment Costs (Prorated)
Development Costs (Prorated)
Program Materials
Instructor/Facilitator Costs
Facilities Costs
Travel/Lodging/Meals
Participant Salaries and Benefits
Administrative/Overhead Costs
Evaluation Costs
Notes: Which cost categories are included in your calculations?
____________________________________

27. Financial ROI = 258% Program Skills Based Pay Systems Business Impact
Employee Turnover
Staffing Level
Customer and Job Satisfaction
Product Sales
Cross Selling
ROI = 258%

28. Telecommunications ROI = 163% Program
All-Inclusive Workforce Program (AIW)
Business Impact
Attrition Rate
Employee Satisfaction
Communication
Cooperation & Teamwork
Diversity Mix
ROI = 163%

29. Health Systems ROI = 1,051% Program
Sexual Harassment Prevention Workshop
Business Impact
Turnover Reduction
Complaint Reduction
Job Satisfaction
Absenteeism
Stress Reduction
Recruiting
ROI = 1,051%

30. Technology ROI = -85% Program Customer Service Training
Business Impact (Target)
Customer Complaint Escalation
ROI = -85%

31. The people who like ROI do so because the ROI Methodology:
Why do people like ROI?
The people who like ROI do so because the ROI Methodology:
Generates a balanced set of measures
Employs a step-by-step process
Bridges evaluation disciplines
Balances research and reality
Offers a flexible evaluation solution
Provides a credible approach to measurement and evaluation

32. Next Steps Assess your readiness for ROI
Identify stakeholders and their data needs
Determine the purpose of your evaluation practice
Identify programs suitable for ROI
Develop capability in the ROI Methodology

33. Success with measurement begins with your worldview
There are no absolutes.
If you are measuring, you are estimating.
Be able to explain what you did, how you did it, and why you did it that way.
Sometimes the crowd knows best.
Process without standards is no process.
A statistic is an estimate of what probably is… maybe.

34. The rest is merely a balancing act.
Benefits
Costs